acceptodds
Under review as a conference paper at ICLR 2027

FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceability

Abstract

Financial search poses a demanding challenge for LLM agents, requiring not only correct final answers but also responses grounded in temporally valid information, authoritative sources, and correctly aligned units and definitions. Yet existing benchmarks predominantly evaluate only final answer correctness, obscuring both the causes of error and the completeness of supporting evidence. Our analysis revealed that 17.48% of correctly answered tasks lack a complete and verifiable evidence chain. To address this gap, we introduce **FinFIRST** (**F**inancial **I**nformation **R**etrieval, **S**ourcing and **T**raceability), a financial-search benchmark that jointly evaluates answers and evidential support through atomic rubrics. FinFIRST comprises 123 expert-authored tasks spanning a graduated difficulty spectrum, derived from aggregate patterns of real-world financial scenarios and informed by an 18-field taxonomy, a 6-axis coverage blueprint, and a registry of 138 financial sources. Task construction involved contributions from over 50 finance experts and a 6-stage quality-control pipeline. Each task is paired with an evidence-grounded reference package decomposed into atomic criteria across three dimensions: raw-information acquisition, source verification, and computation and answer formation. We evaluate multiple model configurations under a unified tool setting: Claude-Opus-5 achieves the highest Loose Pass rate at 87.61%, while GPT-5.6-Sol attains the highest Strict Pass rate at 71.54%, with computation and answer formation consistently lagging behind raw-information acquisition across all systems. FinFIRST thus retains final-answer correctness as the primary objective while making the supporting research process measurable, verifiable, and diagnosable.

Primary area:
datasets and benchmarks
Keywords:
search agent, benchmark, finance

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZQwYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FinFIRST, a benchmark for evaluating financial search agents beyond final-answer correctness. It contains 123 expert-authored tasks with 701 atomic rubric criteria. The criteria cover raw-information acquisition, source verification, and computation and answer formation. The benchmark also records requirements on entity, period, unit, definition, source authority, and evidence traceability. The authors evaluate 15 model configurations under a shared tool environment. Results show that final task completion is substantially lower than partial rubric performance. Computation and answer formation also consistently lag behind information acquisition. The benchmark further reports unsupported-correct answers to demonstrate that final-answer accuracy alone can hide incomplete evidence.

Critical strengths
  1. Financial search requires more than producing the correct number. Entity alignment, reporting periods, definitions, source authority, and evidence support are all important. FinFIRST provides a structured way to evaluate these requirements.
  2. The decomposition into raw-information acquisition, source verification, and computation provides more diagnostic information than a binary final-answer score. The unsupported-correct analysis is particularly useful.
Critical weaknesses
  1. FinFIRST emphasizes research traceability and evidence-grounded search. However, the rubric judge receives the question, rubric, and final response. It does not evaluate the agent's actual search trajectory or verify how the retrieved evidence was used to derive the answer. Therefore, the benchmark primarily evaluates the final reported evidence rather than the underlying research process. A model could potentially produce a correct answer with apparently appropriate citations without demonstrating the claimed evidence-grounded reasoning process. This is particularly important because process-level evaluation is presented as the main distinction from existing financial benchmarks.
  2. The benchmark contains only 123 questions and each model configuration is evaluated using one designated run at temperature 1.0. There is therefore no estimate of run-to-run variance. This matters for the relatively fine-grained comparisons between models and for conclusions drawn from differences of only a few percentage points. The judge validation on 50 sampled instances is useful, but it does not address evaluation variance across model runs or benchmark items.
Questions for authors
  1. How can the authors verify that the reported evidence was actually used to obtain the answer? Since the judge only sees the final response, what prevents a model from arriving at the answer independently and then attaching plausible supporting sources?
  2. How stable are the reported model rankings under repeated runs? The current evaluation uses one run per model. Can the authors report variance across multiple runs or justify why a single run is sufficient?
2 comments
AuthorsOpenReview ↗

Response to Reviewer ZQwY (1/2)

Dear Reviewer ZQwY,

Thank you for your thoughtful questions about evidence traceability and evaluation stability. Below, we clarify what the rubric judge assesses and the conclusions supported by our evaluation protocol.

1. W1/Q1

The judge only sees the final response. How can the authors verify that the reported evidence was actually used to obtain the answer, rather than attached afterwards?

The benchmark assigns atomic scores to the facts, evidence, and derivations presented in the final response, extending evaluation beyond answer correctness. As specified in Appendix E, the judge uses the question, expert-authored rubric, and final response to assess information acquisition, source verification, and computation and answer formation.

This design distinguishes correct answers from answers that also satisfy the evidence requirements. In financial research, final reports are themselves subject to scrutiny: readers need clearly stated sources, reporting periods, metric definitions, and calculations to check its conclusions. Even a numerically correct response may cite the wrong period, omit a required source, or leave out a requested calculation. Atomic scoring identifies these failures that binary answer scoring overlooks. Our results show that 17.48% of answer-correct responses fail at least one evidence-layer criterion, demonstrating the practical value of this distinction.

Regarding post-hoc citations, we distinguish whether the reported evidence satisfies the rubric from whether that evidence actually contributed to generating the answer. The current evaluation assesses the former: it neither infers the model’s internal reasoning sequence nor independently accesses cited pages. Evidence and derivations added afterwards can therefore receive credit if they satisfy the rubric.

We will clarify this boundary and revise references to the research “process” to describe the evidence and derivations presented in the response. FinFIRST’s contribution is to make these supporting elements individually scorable and their deficiencies diagnosable. Extending this framework with tool-trajectory analysis and interventions on retrieved evidence would help examine how that evidence influences answer generation.

AuthorsOpenReview ↗

Response to Reviewer ZQwY (2/2)

2. W2/Q2

How stable are the model rankings under repeated runs? Can the authors report variance across multiple runs or justify why a single run is sufficient?

Thank you for your thoughtful suggestion. To assess the stability of model scores and rankings across runs, we conducted three independent runs on the same 123 questions and report results as mean ± sample standard deviation in the table below.

ModelLoose Pass (%)Strict Pass (%)
GPT-5.6-SOL85.88 ± 0.5571.27 ± 1.24
Claude-Opus-587.63 ± 0.5969.11 ± 1.63
Gemini-3.7-Flash76.83 ± 0.7544.99 ± 2.05
Qwen3.8-Max77.44 ± 0.7254.47 ± 1.63
Qwen3.8-Flash81.29 ± 0.6461.79 ± 1.63
DeepSeek-V4-Pro75.32 ± 0.7650.95 ± 2.05
DeepSeek-V4-Flash71.29 ± 0.8348.51 ± 2.05
GLM-5.380.63 ± 0.5660.98 ± 1.63
GLM-5.3-Flash76.60 ± 0.6856.91 ± 1.63
GLM-5.265.94 ± 0.7643.09 ± 1.63
Kimi-K380.84 ± 0.8059.62 ± 2.05
Qwen3.8-27B77.23 ± 0.7655.01 ± 2.05
MiniMax-M360.76 ± 0.8337.67 ± 2.05
Hunyuan365.25 ± 0.7940.92 ± 2.05
Ling-3.0-Flash-Fin75.01 ± 0.6852.85 ± 1.63

Overall rankings are consistent across runs, though closely performing models occasionally swap positions. Across the three pairwise comparisons, the mean Spearman rank correlation is 0.973 for Loose Pass (range 0.954–0.982) and 0.961 for Strict Pass (range 0.930–0.986). Claude ranks first on Loose Pass in all three runs; GPT achieves the highest mean Strict Pass rate, though Claude leads in the third run; GLM-5.3 and Kimi also exchange positions on Loose Pass. These shifts illustrate that high overall rank agreement can coexist with reordering among closely performing models.

The micro-averaged UCR values across the three runs are 17.48%, 17.32%, and 17.86%, confirming that incomplete evidence among answer-correct responses is a consistent finding rather than an artifact of a single run.

Thank you again for encouraging a more complete assessment of evaluation stability. Please let us know if you have any further questions or concerns.

Best regards,
The Authors of Paper 30129

Reviewer_3x8wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FinFIRST, a benchmark for financial information search agents that aims to jointly evaluate final answers and the evidence and derivations presented in support of them. The benchmark comprises 123 tasks authored and verified by finance experts, together with 701 atomic rubric items, and scores models’ final responses at a fine-grained level along three dimensions: raw-information acquisition, source verification, and computation and answer formation. The authors evaluate 15 model configurations under a unified tool environment consisting of Search, Visit, and Python. They analyze model performance using the Atomic score, the weighted Loose Pass score, the Strict Pass rate, and the Unsupported-Correct Rate. The results show that 17.48% of responses satisfying all answer-layer criteria still fail at least one evidence-layer criterion. Performance on computation and answer formation is also generally lower than on raw-information acquisition. The main contributions of the paper are the construction of an evaluation dataset for financial search and the use of fine-grained scoring to reveal response-completeness issues that aggregate scores tend to obscure.

Critical strengths
  1. The research motivation is well-founded, and the task is practically valuable and challenging. Financial information search places high demands on retrieval accuracy, source authority, temporal validity, and the correctness of calculations and derivations. Jointly evaluating answer correctness and the completeness of the presented evidence and derivations is therefore well motivated from a domain perspective.
  2. The paper organizes rubric items into three dimensions: raw-information acquisition, source verification, and computation and answer formation. It combines these with partial-credit scores, the Strict Pass rate, and UCR to distinguish different types of response deficiencies. The experiments also show that models with similar overall scores differ markedly across rubric dimensions. This design improves the interpretability of the evaluation results and helps reveal issues that a single answer-level score tends to obscure.
  3. The data construction and reference annotation are carefully designed. Each reference package includes precise evidence locations, financial definitions, time-specific data versions, calculation steps, acceptable alternative solutions, and numerical tolerances. Quality is controlled through independent re-solving, cross-validation, and rubric auditing. These design choices help reduce ambiguity in metric definitions and annotation errors in financial tasks, while avoiding treating a single reference solution as the only correct path.
Critical weaknesses
  1. Incremental novelty. FinFIRST mainly recombines existing elements: expert financial search tasks (FinSearchComp, ICLR 2026), atomic rubrics (Finance Agent Benchmark, arXiv 2025), and source-attributed criteria (FrontierFinance, arXiv 2026). Its claimed traceability still comes from judging only the final output (Appendix E). Mind2Web 2 (NeurIPS 2025 D&B) already verifies cited sources with an agent-as-a-judge, yet the paper never compares against it.

  2. Insufficient presentation of the actual benchmark tasks and their correspondence to the annotation scheme: Although Appendix A.1 introduces the 18 annotation fields and Figure 2 reports partial composition statistics, the paper lacks representative questions drawn from the actual evaluation set. It also does not show how the complete field values and annotation rules correspond to actual tasks. The only complete case, in Appendix B, is explicitly stated to be outside the evaluation set and relies on hypothetical evidence records. As a result, it remains difficult to assess which financial research questions the benchmark actually contains, and whether the tasks adequately reflect the domain expertise and real-world demand that the paper emphasizes. I recommend that the authors include examples of actual benchmark tasks: at least one real question with its complete rubric for each of the easy, medium, and hard levels, covering both Chinese and English. The authors should also provide the value definitions and distribution for each taxonomy field.

  3. Appendix E states that the judge observes only the final response, without accessing cited pages or using tool-call trajectories. Consequently, Source Verification measures whether a response satisfies the rubric’s textual requirements regarding sources and dates, and cannot verify whether the cited content actually supports the reported values. However, Section 3.4 describes the judge as “verifying numerical values, units, time ranges, financial definitions, sources, calculations, and output requirements,” and the paper interprets UCR as indicating a lack of “a complete and verifiable evidence chain,” which may be inconsistent with what the evaluation can actually assess. Moreover, under the partition defined in Section 4.2, the evidence layer for retrieval questions includes the formatting and instruction-following items of Computation & Answer Formation. An “unsupported” outcome in UCR may therefore stem solely from formatting non-compliance. The paper also does not distinguish whether failures in this dimension arise from calculation errors or from omissions in presentation.

  4. The “computation bottleneck” may be an artifact of rubric structure. Computation items depend on the retrieved inputs. In Appendix B, the 30-point subtraction item fails whenever either input value is wrong. As a result, retrieval errors also fail the downstream computation items, and computation scores will trail retrieval scores even for an agent that calculates perfectly. The paper should report computation accuracy restricted to cases where all inputs were retrieved correctly.

  5. Missing protocol details. Results come from “a single designated run” at T = 1.0, with no explanation of how that run was chosen and no variance across runs. Tool-call budgets and timeouts are “held fixed” (Appendix C.2), but their values are never reported. This matters because GPT-5.6-Sol makes up to 281 calls and has 6 missing outputs, which are scored as failures and affect the rankings. Agents also search the live web while reference values are frozen, so an answer that is correct at evaluation time but differs from the frozen value is marked wrong. The paper does not report when evaluation took place relative to each task’s reference cutoff.

Questions for authors

Q1. Does source verification detect fabricated evidence? Please construct responses with correct values but manipulated citations: (1) a fabricated document that bears the expected document’s name, and (2) a genuine source that does not contain the reported value. What share still receives full Source Verification credit and correct classification? If most pass, the limitation acknowledged in Appendix E means the abstract’s claim that UCR identifies a lack of “a complete and verifiable evidence chain” is not supported. How do the authors justify that claim and the “Traceability” in the title?

Q2. Does the computation bottleneck survive once input errors are removed? Please report computation-and-answer-formation accuracy only on responses where every required raw-information item passed. If the gap with raw-information acquisition largely disappears, is “computation lags retrieval” still a finding about the agents, rather than about how the rubric items depend on each other?

Q3. Are source-verification criteria justified by the actual prompts? Appendix A.3 prohibits rubric requirements absent from the prompt. Yet the only complete example (Table 4, ~800 characters) explicitly requests sources, dates, and calculations, while actual benchmark prompts average just 187 characters. What proportion of prompts explicitly requires source attribution? Please provide actual benchmark examples demonstrating that source-verification criteria are grounded in prompt requirements rather than unstated expectations. How many criteria were revised or removed for violating this rule during the rubric audit?

10 comments
AuthorsOpenReview ↗

Response to Reviewer 3x8w (1/9)

Dear Reviewer,

Thank you for your detailed questions about FinFIRST’s contribution, task design, and evaluation methodology. Below, we clarify the benchmark’s diagnostic value, address the correspondence between prompts and rubrics, and outline the additional analyses needed to assess evidence validity and computation errors.

1. W1

FinFIRST mainly combines existing elements, and the paper does not compare its evaluation approach with Mind2Web 2.

Thank you for pointing out this related work. As our benchmark focuses on financial search, the original discussion prioritized closely aligned financial benchmarks and did not separately compare against Mind2Web 2.

Mind2Web 2 [1] focuses general-purpose search evaluation, covering complex information-gathering tasks across domains. Its Agent-as-a-Judge framework checks both answers and cited webpages, offering a valuable approach to source attribution assessment. FinFIRST instead translates financial research requirements into independently scored criteria across information acquisition, source verification, and computation and answer formation. Compared with binary task-level scoring [2-3], this decomposition identifies specific unmet requirements beyond answer correctness and reveals distinct capability profiles among models with similar overall scores.

We will add this discussion to the paper to clarify the differences in task scope and evaluation methodology.

[1] MIND2WEB: Towards a Generalist Agent for the Web.

[2] FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning

[3] FinanceQA: A Benchmark for Evaluating Financial Analysis Capabilities of Large Language Models

AuthorsOpenReview ↗

Response to Reviewer 3x8w (2/9)

2. W2/Q3

Please provide actual benchmark questions with complete rubrics and taxonomy annotations. What proportion of prompts explicitly requires source attribution, and how many criteria were revised or removed during the audit?

Thank you for your careful questions. We need to clarify that source citations, calculation steps, and intermediate results are explicit requirements in the System Prompt:

You are a deep research assistant. Your core responsibility is to conduct thorough, multi-source investigations on any topic. You must handle both open-ended questions spanning broad domains and inquiries within specific academic fields. For every request, synthesize information from reliable and diverse sources to provide comprehensive, accurate, and objective responses. When you have gathered sufficient information and are ready to deliver your final response, enclose the complete answer within tags. In your response, show all raw inputs, calculation steps, and intermediate results, and cite sources for each claim._

Current date: <current_date>

Current working directory: <work_directory>

Scoring requirements are therefore aligned with the full instructions received by the model, comprising both the unified System Prompt and the specific question. The mean length of 187 characters excludes these unified System Prompt and the example in Table 4 is aimed to illustrate the task design more clearly.

We will supplement the full system prompt and clarify Appendix A.3: each rubric must be grounded in the complete instructions available to the model.

During rubric auditing, we removed 83 candidate tasks (~6.60%) with requirements that exceeded the scope of the task instructions. For example, a rubric might additionally require quarterly interest rates when the question asks only for an annual rate and those quarterly values are not needed to derive it. Such additional outputs differ from the inputs and intermediate steps required to explain the requested calculation.

Below, we provide three actual benchmark tasks illustrating easy, medium, and hard research demands, covering Chinese and English. Each includes its question, rubric, reference answer, supporting evidence, and 18 annotation fields.

AuthorsOpenReview ↗

Response to Reviewer 3x8w (3/9)

2.1 Example 1 (Easy)

Question

As of the prospectus signing date, what was the exploration area, in square kilometers, of China Uranium’s overseas exploration right EPL-3602? Report the disclosed value to four decimal places.

Reference answer: 164.7881 km²

Scoring rubric

#CriterionCategoryPoints
1Locates the China Uranium prospectus or a link to it.Source verification20
2Identifies the exploration area of EPL-3602 as 164.7881 km², matching the disclosed value exactly to four decimal places.Raw-information acquisition80
Total100

Supporting evidence

The prospectus disclosed on November 27, 2025, reports an exploration area of 164.7881 km² for EPL-3602 [1].

  • Location: Printed page 1-1-198, PDF page 199.
  • Section: “(II) Intangible Assets → 1. Exploration Rights,” table row 6.

Annotation fields

FieldValue
question_decompositionDetermine the exploration area of China Uranium’s overseas exploration right EPL-3602 as of the prospectus signing date.
relation_typeSingle question
conditional_filteringNo conditional filtering
subject_categoryCompany
metricExploration area
measurement_scopeOverseas exploration right EPL-3602; prospectus signing date
regionChina
time_constraint_typeEvent date
time_descriptionProspectus signing date
verification_methodOfficial data
calculation_typeNone
output_format_requirementUnit: square kilometers
languageChinese
source_specificationPublic web search; specified source
source_accessibilityPublicly accessible
source_mainstreamnessMainstream
source_countSingle
intent_categoryData retrieval
Reviewer_R2CKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FinFIRST is a 123-question expert-authored benchmark for financial search agents. Each question ships with a reference package (answer, sources, definitions, calculations, numerical tolerances, data cutoff). Answers are graded by an atomic rubric over three dimensions: raw-information acquisition, source verification, computation & answer formation. 15 models are tested under a unified ReAct harness with Search/Visit/Python tools. Key finding: 17.48% of "answer-correct" responses still fail at least one evidence-layer criterion—aggregate scores overstate traceability.

Critical strengths
  • The atomic rubric is the right move. Splitting grading into retrieval / verification / computation and reporting UCR (unsupported-correct rate) directly exposes the "confident wrong-source" failure mode that binary scoring hides.
  • Construction hygiene is good: 50+ finance experts, 6-stage quality control (value-scope, independent re-solving, cross-validation, rubric audit, stress testing, consistency check), 9.78% acceptance rate, 18-field taxonomy, 6-axis coverage blueprint, 138-source map. This is closer to a professional benchmark than most LLM eval papers.
  • The cross-country/cross-source slice analysis (Fig 3) is useful: Chinese models do better on China-market questions, English models on US, source-multiplicity hurts everyone—these are actionable observations.
Critical weaknesses
  • 123 questions is small. With 15 models × 3 rubric groups × sliced analysis, per-cell counts can be in the single digits. The 17.48% UCR headline is on ~1,770 decomposable attempts, which is fine, but slice-level claims ("Hong Kong questions 65% vs 54%") are noisy.
  • GLM-5.3 is the rubric judge. Cohen's κ = 0.816 against human labels is reported, but the judge is also a model in the comparison pool (GLM-5.3 appears in Table 2). That's a conflict of interest in grading—even if the judge isn't allowed to see its own outputs, the rubric itself was calibrated against GLM-5.3's readings.
  • The benchmark is heavy on lookup/retrieval (59.5% of rubric points in raw-info acquisition). There's very little multi-step financial modeling, no time-series reasoning beyond a single cutoff, no adversarial/trap questions. The "18-field taxonomy" sounds broad but the actual task distribution skews narrow.
  • Tool budget (12.62 calls/question for Claude, up to 313 for Kimi K3) is reported as an observation, not controlled. A model that gets more tool calls looks better partly because it gets more chances—that confound isn't resolved.
Questions for authors

Q1. 59.5% of rubric points sit in raw-information acquisition, 17% in source verification, 23.5% in computation. This is closer to a retrieval benchmark than a financial reasoning benchmark. Do you plan to add tasks requiring multi-step calculation, time-series reasoning, or cross-source reconciliation? Q2. GLM-5.3 is both the rubric judge and one of the 15 evaluated models. Even with a blind judge, the rubric thresholds were calibrated against GLM-5.3's readings. How do the rankings change if you swap the judge to an independent model (e.g., GPT-5.6 or Claude Opus) on a subset?

4 comments
AuthorsOpenReview ↗

Response to Reviewer R2CK (1/4)

Dear Reviewer R2CK,

Thank you for your thoughtful questions about task coverage, judge independence, statistical reliability, and tool-use budgets. Below, we clarify the evaluation design and distinguish the findings supported by our current results from questions requiring further validation.

1. W3/Q1

The benchmark is heavy on lookup/retrieval. Do you plan to add tasks requiring multi-step calculation, time-series reasoning, or cross-source reconciliation?

The benchmark is scoped as a financial search benchmark (retrieval, scourcing, tracebility) rather than a reasoning benchmark. It evaluates agents’ ability to retrieve financial information, verify sources, and support answers with evidence, with computation and synthesis assessed as integral parts of the search workflow.

The tasks extend beyond simple retrieval: the 59.5% figure describes rubric weight, not the proportion of retrieval-only tasks. A single task can require information acquisition, source verification, and computation together. As reported in §2.4 and Figure 2, 61.8% of tasks require explicit computation, over four-fifths contain multiple related sub-questions, 32.5% multi-score, multi-source, and 75.6% require open search without a specific feed source.

Regarding your suggestions on multi-step calculation, time-series reasoning, and cross-source reconciliation, we fully agree these deserve further coverage, and future extensions could build on FinFIRST's atomic evaluation framework to encompass longer calculation chains and broader evidence integration, enabling more comprehensive end-to-end assessment of agents' financial research abilities. Thank you for your valuable suggestions.

AuthorsOpenReview ↗

Response to Reviewer R2CK (2/4)

2. W2/Q2

GLM-5.3 is both the rubric judge and an evaluated model. How do the rankings change if the judge is replaced with an independent model?

Thank you for your thoughtful suggestions. We would like to clarify that the rubric judge is GLM-5.1, which is not among the evaluated models, as specified in§3.4 and the Reproducibility Statement. The comparison pool includes GLM-5.3, GLM-5.3-Flash, and GLM-5.2, but not GLM-5.1. Thus, there is no direct overlap between the judge and the evaluated models.

Expert annotation provides an independent check of judge reliability. As reported in §3.4, we randomly sampled 50 evaluation instances for expert annotation. Agreement between GLM-5.1 and the expert labels reached Cohen’s κ = 0.816 at the atomic rubric-item level, supporting its reliability in applying the rubric.

AuthorsOpenReview ↗

Response to Reviewer R2CK (3/4)

3. W1

123 questions is small, and slice-level comparisons may be noisy.

The final set of 123 questions reflects substantial construction effort and stringent quality control:each task carries a full reference package, ~5.7 atomic criteria, and undergoes expert authoring, independent re-solving, and cross-review by multiple finance experts, with candidates required to pass six stages of quality control at an acceptance rate of just 9.78% (§2.4), ensuring the accuracy, reproducibility, and scoring consistency of the retained 123 questions.

We agree that the benchmark size limits fine-grained comparisons; however, the analysis is descriptive observations intended to guide further investigation rather than to establish general claims about market-specific or regional capabilities.

AuthorsOpenReview ↗

Response to Reviewer R2CK (4/4)

4. W4

Differences in tool-call counts may confound the performance comparison.

The benchmark focuses on evaluating whether agents can produce correct, evidence-supported answers through autonomous search, source verification, and computation; reaching a correct answer through extensive searches, checks, and revisions is itself a capability we aim to assess and encourage, reflecting the gains from inference-time scaling. Within the shared execution constraints, models are therefore free to determine their own search strategies without artificial truncation to equalize tool-call counts.

Tool-call counts serve as an efficiency indicator rather than an additional scoring constraint, and we report both task performance and tool usage together to characterize what models achieve alongside the resources they consume.

Thank you again for your thoughtful feedback, which has helped us clarify the benchmark’s scope and evaluation design! Please let us know if you have any further questions or concerns.

Best,
The authors of Paper 30129

Comments on the paper

1 comment
Area_Chair_2s58OpenReview ↗

Author-Reviewer Discussion

Dear Reviewers,

The authors have provided the rebuttal. Please have a look and let the authors know if you need further clarification or still have follow-up questions.

Thanks, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.