acceptodds
Under review as a conference paper at ICLR 2027

ANSER-Bench: A Comprehensive Benchmark for Analytical Search

Abstract

Search and retrieval systems are increasingly used to support complex professional analysis and decision-making, where users seek not merely isolated facts but evidence-grounded conclusions to analytical questions. Meeting these needs requires retrieving heterogeneous evidence, performing multi-step aggregation and reasoning, and producing conclusions that remain traceable to their supporting evidence. Yet existing benchmarks often evaluate retrieval, question answering, and reasoning in isolation, leaving end-to-end analytical search insufficiently tested. We introduce ANSER-Bench,Benchmark is available at https://anonymous.4open.science/r/ANSER-Bench-8947. a bilingual benchmark of 2,235 tasks across law, finance, and scientific research, spanning descriptive, predictive, and prescriptive analytical needs. Each task pairs a query with a domain-specific corpus, a reference conclusion, and critical-evidence annotations, enabling separate evaluation of answer quality, evidence quality, and efficiency. Our evaluation reveals substantial heterogeneity across analytical needs: across 14 task settings, no system leads in more than five, and methods that perform strongly in one setting often fail to transfer to others. This variability highlights task-dependent challenges in evidence acquisition and downstream analytical reasoning, positioning ANSER-Bench as a testbed for diagnosing such failures and advancing reliable, evidence-grounded analytical search.

Primary area:
datasets and benchmarks
Keywords:
Analytical Search, Benchmark, Information Retrieval, Retrieval-Augmented Generation, Search Agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DExiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ANSER-Bench, a benchmark for analytical search. ANSER-Bench evaluates end-to-end performance from evidence retrieval to analytical reasoning in professional settings, covering 2,235 Chinese and English tasks across law, finance, and scientific research. The authors systematically compare Vanilla LLMs, RAG, and various agent paradigms, and furthur analyze performance across domains, languages, evidence acquisition, etc.

Critical strengths
  1. The motivation is clear. Analytical search naturally emerges as search tasks become more complex and increasingly coupled with reasoning. For such a challenging problem, constructing sufficiently difficult end-to-end tasks and decomposing the underlying capability bottlenecks are equally important.

  2. The paper contributes a valuable dataset. The authors have invested substantial effort in building a highly specialized benchmark, which is solid.

Critical weaknesses
  1. The D/P/R partition seems not that suitable as a taxonomy. P and R appear to build upon D——A predictive task may first require summarizing observed evidence before making a prediction. However, the paper reports D as the most difficult category and attributes this to the difficulty of target-population aggregation, which seems somewhat unreasonable.

  2. The evaluation of answer quality, evidence quality, and efficiency is limited. The paper lacks a quantitative stage-wise error analysis showing how many end-to-end failures occur during evidence acquisition/reasoning, and how failures distribute across Agents and subsets.

  3. The paper lacks detailed discussion and analysis of the interaction between Agent architectures and specific tasks. The adesign of an agent may be closely related to task performance, and certain architectural choices may substantially benefit particular tasks. For example:

(1) With Qwen3.5-122B-A10B, FinSight and ReAct appear particularly unsuitable for Law tasks, even though the latter is a general-purpose paradigm. However, Tongyi DR does not seem to exhibit the same issue.

(2) With DeepSeek-V4.1-Flash, LawThinker often achieves leading performance on several Finance tasks. As a benchmark paper, it is of course not required to propose new methods, but providing guidance for future system design would clearly be more valuable than merely reporting method rankings.

  1. The conclusion of RQ3 is somewhat fragile. The negative correlations may be more plausibly attributed to agent architectures being unsuitable for particular tasks. By varying reasoning effort or increasing test-time compute through pass@K, etc. would make the conclusion more convincing.
Questions for authors

See above the weaknesses section.

If the authors can address my concerns, I will consider raising my score.

3 comments
AuthorsOpenReview ↗

Response to Reviewer DExi (Part 1)

Thank you for recognizing the motivation of ANSER-Bench and the effort invested in constructing the benchmark. We address each concern below, split into two separate comments due to space limits.

1. D/P/R taxonomy and the difficulty of descriptive tasks

D/P/R classifies tasks by their final analytical objective (Section 2.1 and Appendix A). We agree that a prediction may begin by summarizing observations and that a prescriptive judgment may begin by establishing facts. These are intermediate operations within tasks whose final objectives are to infer an outcome or relationship, or to determine what follows under a rule or constraint. Importantly, D/P/R represents categories of analytical objectives, rather than mutually exclusive sets of underlying operations.

The difficulty of a descriptive operation also depends on its scope. Summarizing a few historical filings for a forecast, for example, differs substantially from identifying every eligible case in a large corpus and computing statistics over the complete target population. Descriptive tasks themselves range from extracting a specific fact to filtering and aggregating an entire population. Thus, some P/R tasks involve descriptive operations, while some standalone D tasks can still be more demanding for current systems. The lower descriptive scores in RQ1 are an empirical finding for the evaluated settings, consistent with the substantial population-coverage and aggregation requirements of several descriptive tasks.

2. Evaluation and stage-wise error attribution

ANSER-Bench is designed to evaluate end-to-end analytical search systems, rather than to impose a unified process evaluation on their internal execution workflows. Different agents organize planning, retrieval, tool use, evidence management, and answer generation in fundamentally different ways. Consequently, there is no consistent stage boundary that can be applied across all evaluated systems.

More importantly, both failure and stage-wise failure are task-dependent. For example, retrieving some relevant documents may be sufficient for a precedent-application task but inadequate for a population-level statistical task requiring exhaustive coverage. Even when the final answer is incorrect, the underlying cause may involve incomplete evidence acquisition, incorrect evidence selection, aggregation errors, or reasoning over correctly retrieved evidence. These operations can also interact within a single execution. Without a common operational definition of stage-level success and failure, assigning every unsuccessful run to a specific stage and reporting corresponding error percentages would risk producing misleading comparisons.

Nevertheless, we provide two complementary forms of diagnostic analysis. RQ2 quantitatively examines the relationship between evidence acquisition and answer quality. Dense RAG improves evidence scores over Sparse RAG in all 28 setting–backbone pairs, but answer quality improves in only 20 and declines in eight. For example, on Qwen Law-EN-P, evidence Set F1 increases by 1.96 points while relation accuracy drops by 58.95 points. On English financial prediction, Recall@5 increases by 33.40/26.60 points with Qwen/DeepSeek, while answer accuracy decreases by 18.33/10.00 points. These results demonstrate that better evidence acquisition does not necessarily translate into better analytical conclusions.

Appendix I further examines representative execution traces and failure cases, illustrating distinct failure mechanisms: incomplete population coverage despite retrieving relevant documents (Table 24), incorrect statistical conclusions despite complete evidence coverage (Table 23), incorrect aggregation despite an exact evidence set (Table 25), and incorrect rule application despite retrieving the correct evidence and rule (Table 26). Together, RQ2 and Appendix I provide quantitative and case-based insights into the distinction between acquiring evidence and using it correctly, without imposing potentially misleading stage-level attribution across heterogeneous agents.

AuthorsOpenReview ↗

Response to Reviewer DExi (Part 2)

3. Agent workflows and task requirements

We expanded RQ1 in Section 3.2 to analyze how task-specific evidence requirements interact with agent workflows, using the implementations in Appendix G and the results in Table 20.

With Qwen on Law-EN-P, FinSight and ReAct rely on bounded document search and inspection, achieving only 0.30 and 2.13 Set Recall, respectively. Tongyi DR can identify eligible cases through SQL and reaches 47.31. However, on Law-EN-R, which requires retrieving and applying a specific precedent rather than identifying an entire target population, FinSight and ReAct slightly outperform Tongyi DR. This contrast highlights the importance of matching evidence-acquisition capabilities to task requirements.

With DeepSeek, LawThinker leads both financial corpus-statistics subsets. Compared with Tongyi DR, it achieves higher RA@10% on both Chinese and English tasks (45.92 vs. 31.00; 41.41 vs. 36.00) and higher Set Recall (89.64 vs. 58.45; 66.93 vs. 49.42). The two agents share SQL, deterministic-calculation, and population-selection tools, while LawThinker additionally employs task-local evidence memory and a separate DeepVerifier.

Although these comparisons do not isolate the causal contribution of individual components, they suggest a broader design implication: effective analytical agents should integrate task-appropriate structured operations with explicit evidence management and verification, rather than relying solely on generic search-and-inspect workflows.

4. Interpretation of RQ3 and the reasoning-effort experiment

RQ3 examines the relationship between token usage and answer quality across systems, rather than establishing a universal negative correlation. In the revised Figure 5a, 21 of 28 correlations are positive, with median correlations of 0.37 for Qwen and 0.27 for DeepSeek. We clarify that the observed relationship can vary with both task requirements and agent–task compatibility.

To further examine the effect of additional test-time computation, we added a low-to-high reasoning-effort comparison using DeepSeek-V4.1-Flash on Science-EN-P (Appendix H.5, Table 23), holding the workflow, tools, and tasks fixed within each method. Token usage increases in six of eight configurations. Among these six, both answer metrics improve in three, decline in two, and move in opposite directions in one. For the five configurations with evidence measurements, both evidence metrics improve in one, decline in two, and remain unchanged in two. Notably, LawThinker achieves higher evidence scores but lower answer scores at higher reasoning effort.

These results reinforce our interpretation that additional computation does not guarantee improvements in either evidence acquisition or final answer quality; its effectiveness depends on how the agent uses the additional reasoning budget to address the task's analytical requirements.

Thanks for the response. Some of my concerns have been addressed. However, I remain unconvinced by the task taxonomy. The categories overlap, which calls the classification into question. Moreover, their difficulty comparisons mainly reflect the specific task distribution rather than underlying capability bottlenecks, limiting the insights gained. I therefore maintain my score.

Reviewer_V9uXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces ANSER-Bench, a bilingual (Chinese/English) benchmark of 2,235 “analytical search” tasks in law, finance, and science, organized into 14 settings by domain, language, and analytical need (descriptive, predictive, prescriptive). Each task pairs a query with a domain corpus of documents plus extracted structured records, a reference conclusion, and gold evidence. About 76% of tasks are newly built; the rest are adapted from CLERC, RegBench, FinAuditing, MetaSyn, and official Chinese court cases. Systems are scored on answer quality, evidence quality, and token cost. Experiments with eight systems on two backbones suggest that no method leads broadly, that better retrieval doesn’t reliably improve answers, and that more compute rarely pays off.

Critical strengths
  1. Population-level analytical tasks with complete, deterministic gold evidence over large professional corpora. The law and finance descriptive and association tasks require identifying and aggregating every qualifying case or company among thousands of records, for example counting judgments that meet both structured and textual conditions. Answers are computed from adjudicated evidence sets rather than written freely.

  2. Joint answer, evidence, and efficiency evaluation that exposes dissociations. Scoring evidence coverage and cost alongside the answer surfaces failure modes that answer-only benchmarks hide. This makes the benchmark diagnostic rather than a single leaderboard, which matters for the stated goal of advancing evidence-grounded systems.

Critical weaknesses
  1. The baseline comparison is confounded, so the headline empirical findings are unreliable.
  • During adaptation, the authors gave Tongyi DR and LawThinker SQL and calculation tools over the structured tables, while ReAct, DeerFlow (whose native code execution was disabled), and FinSight received only document search. Comparing whole systems is reasonable, but these tool sets were assigned by the authors rather than inherent to the systems, and the difference is mentioned only in the appendix. The main text instead attributes performance gaps to paradigm and domain specialization. Without an ablation that varies tools while holding the agent fixed, the results can’t tell us whether tools, control loops, or other design choices drive performance, and the paper’s conclusions about domain transfer are not supported.
  • Apparent pipeline failures are presented as findings. In Table 3, Dense RAG with Qwen scores 0.00 on Law-EN-P, while Sparse RAG scores 58.95 and Dense RAG with DeepSeek scores 55.88. The task uses a three-way label (positive / negative / no clear relationship), so a score of zero more likely signals a parsing or formatting failure than a real result. Yet RQ2 uses this exact case as its first example (“relation accuracy falls by 58.95,” lines 365–366), counts it among the eight regressions behind its conclusion, and highlights it as the most extreme point in Figure 4a.
  • Results lack uncertainty estimates, so the rankings are fragile. Table 3 reports single runs with no confidence intervals, and some settings are small (20 tasks for Science-D, 57 for Law-EN-R, 63 for Science-P). With 20 tasks, one question can shift a score by up to 5 points. The repeated-run check covers only 2 of 14 settings and finds standard deviations of up to 6.98 points (App. H.3), yet several leaders win by far less (76.64 vs. 76.61 on Law-ZH-R; 50.00 vs. 48.33 on Fin-EN-P). The leader counts and the claim that “no system leads in more than five” settings, are therefore not statistically supported.
  1. Weak evidence that the benchmark measures search, and QC can be unreliable.
  • A model answering from memory alone, with no access to the corpus, comes very close to the best system on several settings: Law-ZH-R (76.61 vs. 76.64), Law-EN-R (89.68 vs. 96.61), and both science settings. It even beats every agent on English financial forecasting Contamination in “holdout” tasks.
  • The forecasting tasks ask about FY2024 results, which models released in 2026 have very likely already seen, so the “holdout” design doesn’t really hold. For a benchmark meant to test search, this is a serious gap that the paper doesn’t address.
  • Thin quality-control evidence. QC relies on annotators whose expertise doesn’t match parts of the task: Chinese-law PhDs reviewing US-law tasks, and finance PhDs reviewing Basel and XBRL rule application. Also no error rates or agreement statistics are reported.

Together these weaken both the validity of the benchmark as a test of analytical search and its claimed coverage of domains and analytical needs.

Questions for authors
  1. Tool parity. Tongyi DR and LawThinker had read-only SQL, calculation, and population-selection tools; ReAct, DeerFlow, and FinSight had only retrieval. Can you rerun with a matched tool suite (or ablate SQL from the stronger agents)? Do the heterogeneity, domain-transfer, and RQ2/RQ3 conclusions survive?
  2. The closed-book LLM nearly matches the best system on Law-ZH-R, Law-EN-R, and both science settings, and ranks second on Fin-EN-P. What fraction of tasks per setting can be solved without the corpus? How do you rule out pretraining exposure to FY2024 outcomes, CLERC, and the MetaSyn systematic reviews?
  3. Anomalous zero scores. What causes the 0.00 results (e.g., Qwen Dense RAG and DeerFlow on Law-EN-P, a roughly three-way task)? Do the RQ2 counts and lead example still hold after fixing them?
  4. Annotation quality. What error rates did the 10% evidence audit and the blind answer re-derivation find, and what was inter-annotator agreement? How were the 352 English (US) legal tasks and the Basel/XBRL tasks validated, given the annotators’ backgrounds?
  5. Statistical robustness. Can you give confidence intervals for Table 3, and recount how many settings each system leads (Sec. 3.2), treating results within the intervals as ties?
3 comments
AuthorsOpenReview ↗

Response to Reviewer V9uX (Part 1)

Thank you for recognizing the value of ANSER-Bench's population-level analytical tasks and its joint evaluation of answer quality, evidence quality, and efficiency. We appreciate the constructive feedback and address each concern below, split into three separate comments due to space limits.

1. Baseline Adaptation and Controlled Tool Ablation (Question1/Weakness1.1)

We agree that tool availability can substantially affect end-to-end agent performance and that our original discussion did not sufficiently distinguish tool effects from agent-design effects.

Baseline adaptation and tool selection. We follow an architecture-preserving, component-scoped adaptation principle: retaining the published workflow of each selected system or component while adapting its data-access interfaces to ANSER-Bench's closed-corpus setting, rather than enforcing identical tool inventories. Specifically, retrieval-oriented baselines use benchmark-local search and document-reading interfaces, while selected tool-oriented workflows are equipped with structured-data operations, deterministic calculation, and population selection. These adaptations preserve the respective system workflows.

Controlled tool ablation. To examine the contribution of structured tools, we conducted additional ablations on Tongyi DeepResearch, separately removing SQL and deterministic calculation while keeping all other configurations unchanged.

Removing SQL consistently degrades performance. On Law-EN-P, relation accuracy drops from 61.82 to 50.00 with Qwen and from 81.05 to 58.18 with DeepSeek. On Finance-ZH, non-statistical descriptive accuracy decreases from 74.07 to 66.67 and from 73.15 to 66.67, respectively, while statistical MAE increases from 3.87 to 7.23 and from 4.17 to 6.31. These results highlight SQL's role in systematically filtering structured records and constructing complete evidence populations, which is difficult to achieve through BM25 retrieval and case-by-case reading alone.

The effect of deterministic calculation is less consistent: removing it reduces Law-EN-P relation accuracy by 12.67 and 9.01 points, but produces mixed results across Finance-ZH tasks and backbones. Overall, tool benefits depend on task requirements and backbone capabilities rather than increasing uniformly with tool availability.

Clarification of evaluation scope and future directions. ANSER-Bench primarily evaluates complete end-to-end systems under their implemented configurations, rather than isolating intrinsic architectural capabilities. We have added a clarification in RQ1 that our comparisons focus on complete end-to-end systems under their respective configurations, rather than isolating the effects of individual tools or workflow designs. Accordingly, RQ1–RQ3 findings should be understood as empirical observations across complete system configurations, not causal evidence of architectural superiority.

For future work, we envision two complementary evaluation tracks: (1) an end-to-end system track, allowing native workflows and tools to assess practical system performance; and (2) a standardized tool track, providing common interfaces and controlled capability boundaries to isolate the effects of retrieval, structured operations, and agent control strategies.

AuthorsOpenReview ↗

Response to Reviewer V9uX (Part 2)

2. Closed-book Performance and Contamination Audits (Question2/Weakness2.1-2)

Evaluation principle. Analytical search does not require models to lack prior domain knowledge or familiarity with source materials. It requires them to retrieve relevant evidence from a designated corpus, integrate information, verify facts and rules, and produce traceable conclusions. We therefore evaluate answer quality and evidence quality separately: a correct closed-book answer does not establish that the system has retrieved and verified the evidence supporting it.

Closed-book performance. Using full credit on the existing judge rubric's conclusion criterion, Qwen and DeepSeek answer 63/100 and 64/100 Law-ZH-R tasks, 44/57 and 53/57 Law-EN-R tasks, 9/20 and 11/20 Science-EN-D tasks, and 42/63 and 54/63 Science-EN-P tasks correctly. However, when full credit on the citation or precedent criterion is additionally required, DeepSeek's successful counts drop from 64 to 31 on Law-ZH-R and from 53 to 33 on Law-EN-R. Similarly, in Science, recalling an effect's direction does not establish that it holds for the specific population and outcome. On Finance-EN-P, DeepSeek ranks second with 29/60 multiple-choice answers correct (48.33%), but exactly recovers only 2/60 numerical forecast targets. Thus, correct conclusions, particularly on multiple-choice tasks, do not necessarily demonstrate evidence-grounded analytical search.

Targeted contamination audits. We agree that directly recalling a substantial number of task-specific answers from pretraining would undermine benchmark validity. We therefore conduct targeted tests for the three potential sources raised by the reviewer:

  • FY2024: In a Closed-book Exact Fact Recovery test, models receive the company, fiscal period, financial field, and unit, but not the filings, and must reproduce the reported value exactly. Across 376 company–field pairs, Qwen recovers 0/376 values and DeepSeek 6/376 (1.60%), providing little evidence of systematic exact recall of FY2024 disclosures.
  • CLERC: In a Slot Guessing test, models must recover the original case name, reporter citation, and holding from rewritten questions. Among 56 tasks with verified references, Qwen recovers both the case name and citation on 4/56 tasks (7.14%), and DeepSeek on 6/56 (10.71%); neither recovers all three fields under normalized exact matching.
  • MetaSyn: MetaSyn was publicly released in June 2026, after the training data cutoffs of both Qwen3.5-122B-A10B and DeepSeek-V4.1-Flash. Neither model could therefore have encountered the publicly released benchmark and its reference answers during pretraining.

Overall, These tests provide little evidence of widespread or systematic answer leakage. While these tests cannot rule out all prior familiarity with individual facts or source documents, they suggest that pretraining contamination is unlikely to undermine the benchmark as a whole.

3. Correction of Anomalous Scores and Revised RQ2 Findings (Question3/Weakness1.2)

Thank you for flagging the 0.00 in Table 3. We checked the saved outputs and confirmed a scoring error. The evaluator discarded valid relation labels when accompanying numerical fields failed validation, for example when the reported odds ratio was zero. The evaluator now scores the relation label independently of those fields. This correction affects two underlying scores, both on Law-EN-P with Qwen. Dense RAG changes from 0.00 to 58.95 and Vanilla LLM changes from 15.79 to 52.96.

We updated Table 3, the full results and macro summaries in Appendix H (Tables 20 and 21), and the affected analyses in Figures 3–5. In the revised RQ2 and Figure 4(a), the previously reported 58.95-point drop becomes a tie in relation accuracy, alongside a 1.96-point gain in Set F1. Across the 28 comparisons, Dense RAG improves evidence scores in all 28, while answer scores improve in 20, decline in seven, and tie in one. The revised results therefore still support the RQ2 conclusion that better evidence retrieval does not always lead to better answers.

AuthorsOpenReview ↗

Response to Reviewer V9uX (Part 3)

4. Annotation Quality and Domain-specific Expertise (Question4/Weakness2.3)

We have supplemented the annotation quality-control results and clarified the annotators' backgrounds and preparation.

Annotation quality and agreement. We report three complementary measures in Appendices C.5, D.5, and E.3: evidence error rates from independent audits of a sample of finalized evidence sets; answer error rates based on errors identified through independent re-derivation; and query-review agreement measured by Fleiss' κ\kappa across three annotators' independent validity judgments before adjudication. Across the three domains, overall error rates remain below 5%, and query-review Fleiss' κ\kappa exceeds 0.75, supporting the reliability and consistency of our annotation process.

Annotator expertise and preparation. For the English (US) legal tasks, one annotator specialized in international law and had substantial experience working with European and U.S. legal materials. The other two, whose primary training was in Chinese law, undertook focused preparation on fundamental U.S. legal concepts, judicial opinion structures, and case holdings before annotation. For the Basel and XBRL tasks, our finance annotation team included students with backgrounds in international finance and accounting. Their relevant coursework covered banking regulation, capital adequacy, and financial reporting, providing foundations for understanding Basel requirements and XBRL-based disclosures. Other annotators had also completed relevant foundational courses, which primarily introduced these topics under U.S. regulatory and accounting frameworks.

Further quality assurance. Within the practical constraints of annotator availability and resources, we have made every effort to safeguard annotation quality through careful annotator selection, task-specific preparation, independent quality checks, and quantitative agreement assessments. Nevertheless, we recognize the value of broader specialist coverage. We are therefore expanding recruitment through international annotation platforms to involve more annotators with relevant English-language legal and financial expertise. We are also extending evidence annotation toward full coverage of the benchmark rather than a 10% sample (actually we have already completed for the scientific subtasks), further strengthening its traceability and quality assurance.

5. Statistical Robustness and Ranking Stability (Question5/Weakness1.3)

We acknowledge that some settings have relatively small test sets, warranting caution when interpreting close performance differences. However, 10 of the 14 settings contain at least 100 tasks, providing a substantial evaluation basis for most comparisons. Since task difficulty is intrinsic to each benchmark setting, we focus our additional analysis on execution variability under fixed evaluation sets.

Repeated-run stability. Appendix H.3 reports repeated-run experiments on two representative settings, covering five agent-based systems under both Qwen and DeepSeek. Our additional ranking analysis shows that the primary answer rankings are relatively consistent across the first two complete runs. The average Spearman's ρ and Kendall's τ are 0.80 and 0.70 for Qwen, and 0.81 and 0.74 for DeepSeek, respectively. Notably, the top-ranked method remains unchanged across all four setting–backbone comparisons. Evidence scores and token usage exhibit greater run-to-run variability in some configurations, which we also analyze in Appendix H.3. These results suggest that the observed answer-quality rankings are reasonably stable in the examined settings.

Further evaluation. We are extending repeated-run experiments to more settings and additional independent executions to provide a more comprehensive assessment of performance variability. We will report these results in future benchmark updates. Meanwhile, we clarify that the leader counts in Section 3.2 are intended to summarize the observed performance patterns under the evaluated settings, rather than to establish definitive or universal advantages of particular systems.

Reviewer_EcNaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces ANSER-Bench, a bilingual benchmark for analytical search.

Critical strengths
  • The paper addresses the analytical needs in 3 domains (law, finance, and scientific research)
  • The experiments provide limitations of RAG-based and agent-based systems
Critical weaknesses
  • The authors claim a contribution to developing an evidence-grounded and efficiency-aware framework, but it is not supported or described in the paper. Instead, they still use one scoring metric for reporting, raising a question about this framework. Thus, no insights are gained from this setup.
  • The paper introduces a bilingual benchmark. However, no insightful information is gained from the bilingual benchmark: no statistics, no key takeaways from the bilingual setting. The experimental results in RQ1 are natural to me; they are not surprising.
  • The experimental setup is limited to 2 models, without frontier systems (OpenAI or Google Gemini).
  • The novelty of this paper is limited; prior work has introduced benchmarks in an information-seeking setting, requiring the gathering of information across sources.
Questions for authors

See Weaknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer EcNa

Thank you for recognizing the benchmark’s coverage and findings. We address the concerns below:

1. Evaluation framework and findings. ANSER-Bench evaluates answer quality, evidence quality, and efficiency separately. Table 3 reports one primary answer metric per setting, rather than a composite score: the three dimensions capture different aspects of system performance and use different scales, so combining them would require arbitrary weighting. Section 2.3 defines the evaluation protocols, and Appendix H.1 reports all three dimensions.

These dimensions provide complementary insights. In RQ2, for DeepSeek on Finance-EN-P, Dense RAG improves Recall@5 by 26.60 points over Sparse RAG, while answer accuracy decreases by 10 points. An answer-only evaluation would miss the improvement in evidence retrieval. In RQ3, LawThinker achieves 53.27 RA@10% on DeepSeek Law-EN-D, compared with 0.50 for ReAct, while using approximately 275K rather than 640K tokens. These examples show why retrieval, answer quality, and computational cost should be examined separately.

2. Purpose and findings of the bilingual design. The benchmark evaluates analytical search across Chinese- and English-language professional settings; it does not aim to construct translation-equivalent tasks or isolate language as a controlled variable. Method advantages do not consistently transfer between corresponding language-specific settings: across six Chinese–English domain–need pairs (Law D/P/R and Finance D/P/R), the top-performing method differs between languages in five pairs with Qwen and four with DeepSeek. For example, with DeepSeek, FinSight outperforms Tongyi DR on Chinese filing analysis (84.62 vs. 73.15), but trails it on English filing analysis (65.63 vs. 76.56). These differences may also reflect corpus sources, data structures, and task construction, and should not be interpreted as isolated causal effects of language. For clarity, we have revised the Abstract and Introduction to describe the benchmark’s Chinese and English coverage explicitly.

3. Coverage of frontier models. We agree that the current evaluation does not include proprietary models from OpenAI or Google. However, ANSER-Bench primarily evaluates complete end-to-end analytical-search systems, rather than exhaustively comparing model backbones. Given its scale (2,235 tasks across 14 settings) and the inference costs of multi-step agents, evaluating many backbones across all systems and tasks would substantially increase the cost. We therefore prioritized eight representative systems and two shared, contemporary open-weight backbones, Qwen3.5-122B-A10B and DeepSeek-V4.1-Flash. This enables systematic comparisons across retrieval pipelines and agent workflows while keeping the backbone consistent within each comparison. Thus, we prioritize system- and task-level coverage over exhaustive backbone coverage. Evaluating proprietary frontier models remains an important direction for future work.

4. Novelty relative to prior information-seeking benchmarks. We agree that gathering information across multiple sources is not, by itself, new. ANSER-Bench is designed around the end-to-end Analytical Search paradigm: systems must retrieve heterogeneous evidence from a specified corpus, perform the analytical operations required by each query, and produce evidence-grounded conclusions. As compared in Table 1, its contribution is to bring together four dimensions rarely covered simultaneously by existing benchmarks: professional-domain analytical needs, heterogeneous evidence acquisition, evidence-quality evaluation, and efficiency evaluation. It covers 2,235 tasks across 14 settings, three domains, two languages, and three analytical needs (descriptive, predictive, and prescriptive). For example, a legal case-statistics task requires identifying all cases that satisfy specified conditions in a fixed corpus and computing the corresponding statistic; retrieving a few relevant documents is insufficient. By jointly evaluating conclusions, supporting evidence, and end-to-end cost, ANSER-Bench offers a comprehensive evaluation of analytical search beyond information gathering or open-ended synthesis alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.