- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ANSER-Bench, a benchmark for analytical search. ANSER-Bench evaluates end-to-end performance from evidence retrieval to analytical reasoning in professional settings, covering 2,235 Chinese and English tasks across law, finance, and scientific research. The authors systematically compare Vanilla LLMs, RAG, and various agent paradigms, and furthur analyze performance across domains, languages, evidence acquisition, etc.
Critical strengths
-
The motivation is clear. Analytical search naturally emerges as search tasks become more complex and increasingly coupled with reasoning. For such a challenging problem, constructing sufficiently difficult end-to-end tasks and decomposing the underlying capability bottlenecks are equally important.
-
The paper contributes a valuable dataset. The authors have invested substantial effort in building a highly specialized benchmark, which is solid.
Critical weaknesses
-
The D/P/R partition seems not that suitable as a taxonomy. P and R appear to build upon D——A predictive task may first require summarizing observed evidence before making a prediction. However, the paper reports D as the most difficult category and attributes this to the difficulty of target-population aggregation, which seems somewhat unreasonable.
-
The evaluation of answer quality, evidence quality, and efficiency is limited. The paper lacks a quantitative stage-wise error analysis showing how many end-to-end failures occur during evidence acquisition/reasoning, and how failures distribute across Agents and subsets.
-
The paper lacks detailed discussion and analysis of the interaction between Agent architectures and specific tasks. The adesign of an agent may be closely related to task performance, and certain architectural choices may substantially benefit particular tasks. For example:
(1) With Qwen3.5-122B-A10B, FinSight and ReAct appear particularly unsuitable for Law tasks, even though the latter is a general-purpose paradigm. However, Tongyi DR does not seem to exhibit the same issue.
(2) With DeepSeek-V4.1-Flash, LawThinker often achieves leading performance on several Finance tasks. As a benchmark paper, it is of course not required to propose new methods, but providing guidance for future system design would clearly be more valuable than merely reporting method rankings.
- The conclusion of RQ3 is somewhat fragile. The negative correlations may be more plausibly attributed to agent architectures being unsuitable for particular tasks. By varying reasoning effort or increasing test-time compute through pass@K, etc. would make the conclusion more convincing.
Questions for authors
See above the weaknesses section.
If the authors can address my concerns, I will consider raising my score.
3 comments
Response to Reviewer DExi (Part 2)
3. Agent workflows and task requirements
We expanded RQ1 in Section 3.2 to analyze how task-specific evidence requirements interact with agent workflows, using the implementations in Appendix G and the results in Table 20.
With Qwen on Law-EN-P, FinSight and ReAct rely on bounded document search and inspection, achieving only 0.30 and 2.13 Set Recall, respectively. Tongyi DR can identify eligible cases through SQL and reaches 47.31. However, on Law-EN-R, which requires retrieving and applying a specific precedent rather than identifying an entire target population, FinSight and ReAct slightly outperform Tongyi DR. This contrast highlights the importance of matching evidence-acquisition capabilities to task requirements.
With DeepSeek, LawThinker leads both financial corpus-statistics subsets. Compared with Tongyi DR, it achieves higher RA@10% on both Chinese and English tasks (45.92 vs. 31.00; 41.41 vs. 36.00) and higher Set Recall (89.64 vs. 58.45; 66.93 vs. 49.42). The two agents share SQL, deterministic-calculation, and population-selection tools, while LawThinker additionally employs task-local evidence memory and a separate DeepVerifier.
Although these comparisons do not isolate the causal contribution of individual components, they suggest a broader design implication: effective analytical agents should integrate task-appropriate structured operations with explicit evidence management and verification, rather than relying solely on generic search-and-inspect workflows.
4. Interpretation of RQ3 and the reasoning-effort experiment
RQ3 examines the relationship between token usage and answer quality across systems, rather than establishing a universal negative correlation. In the revised Figure 5a, 21 of 28 correlations are positive, with median correlations of 0.37 for Qwen and 0.27 for DeepSeek. We clarify that the observed relationship can vary with both task requirements and agent–task compatibility.
To further examine the effect of additional test-time computation, we added a low-to-high reasoning-effort comparison using DeepSeek-V4.1-Flash on Science-EN-P (Appendix H.5, Table 23), holding the workflow, tools, and tasks fixed within each method. Token usage increases in six of eight configurations. Among these six, both answer metrics improve in three, decline in two, and move in opposite directions in one. For the five configurations with evidence measurements, both evidence metrics improve in one, decline in two, and remain unchanged in two. Notably, LawThinker achieves higher evidence scores but lower answer scores at higher reasoning effort.
These results reinforce our interpretation that additional computation does not guarantee improvements in either evidence acquisition or final answer quality; its effectiveness depends on how the agent uses the additional reasoning budget to address the task's analytical requirements.
Thanks for the response. Some of my concerns have been addressed. However, I remain unconvinced by the task taxonomy. The categories overlap, which calls the classification into question. Moreover, their difficulty comparisons mainly reflect the specific task distribution rather than underlying capability bottlenecks, limiting the insights gained. I therefore maintain my score.
Response to Reviewer DExi (Part 1)
Thank you for recognizing the motivation of ANSER-Bench and the effort invested in constructing the benchmark. We address each concern below, split into two separate comments due to space limits.
1. D/P/R taxonomy and the difficulty of descriptive tasks
D/P/R classifies tasks by their final analytical objective (Section 2.1 and Appendix A). We agree that a prediction may begin by summarizing observations and that a prescriptive judgment may begin by establishing facts. These are intermediate operations within tasks whose final objectives are to infer an outcome or relationship, or to determine what follows under a rule or constraint. Importantly, D/P/R represents categories of analytical objectives, rather than mutually exclusive sets of underlying operations.
The difficulty of a descriptive operation also depends on its scope. Summarizing a few historical filings for a forecast, for example, differs substantially from identifying every eligible case in a large corpus and computing statistics over the complete target population. Descriptive tasks themselves range from extracting a specific fact to filtering and aggregating an entire population. Thus, some P/R tasks involve descriptive operations, while some standalone D tasks can still be more demanding for current systems. The lower descriptive scores in RQ1 are an empirical finding for the evaluated settings, consistent with the substantial population-coverage and aggregation requirements of several descriptive tasks.
2. Evaluation and stage-wise error attribution
ANSER-Bench is designed to evaluate end-to-end analytical search systems, rather than to impose a unified process evaluation on their internal execution workflows. Different agents organize planning, retrieval, tool use, evidence management, and answer generation in fundamentally different ways. Consequently, there is no consistent stage boundary that can be applied across all evaluated systems.
More importantly, both failure and stage-wise failure are task-dependent. For example, retrieving some relevant documents may be sufficient for a precedent-application task but inadequate for a population-level statistical task requiring exhaustive coverage. Even when the final answer is incorrect, the underlying cause may involve incomplete evidence acquisition, incorrect evidence selection, aggregation errors, or reasoning over correctly retrieved evidence. These operations can also interact within a single execution. Without a common operational definition of stage-level success and failure, assigning every unsuccessful run to a specific stage and reporting corresponding error percentages would risk producing misleading comparisons.
Nevertheless, we provide two complementary forms of diagnostic analysis. RQ2 quantitatively examines the relationship between evidence acquisition and answer quality. Dense RAG improves evidence scores over Sparse RAG in all 28 setting–backbone pairs, but answer quality improves in only 20 and declines in eight. For example, on Qwen Law-EN-P, evidence Set F1 increases by 1.96 points while relation accuracy drops by 58.95 points. On English financial prediction, Recall@5 increases by 33.40/26.60 points with Qwen/DeepSeek, while answer accuracy decreases by 18.33/10.00 points. These results demonstrate that better evidence acquisition does not necessarily translate into better analytical conclusions.
Appendix I further examines representative execution traces and failure cases, illustrating distinct failure mechanisms: incomplete population coverage despite retrieving relevant documents (Table 24), incorrect statistical conclusions despite complete evidence coverage (Table 23), incorrect aggregation despite an exact evidence set (Table 25), and incorrect rule application despite retrieving the correct evidence and rule (Table 26). Together, RQ2 and Appendix I provide quantitative and case-based insights into the distinction between acquiring evidence and using it correctly, without imposing potentially misleading stage-level attribution across heterogeneous agents.