- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces FinFIRST, a benchmark for evaluating financial search agents beyond final-answer correctness. It contains 123 expert-authored tasks with 701 atomic rubric criteria. The criteria cover raw-information acquisition, source verification, and computation and answer formation. The benchmark also records requirements on entity, period, unit, definition, source authority, and evidence traceability. The authors evaluate 15 model configurations under a shared tool environment. Results show that final task completion is substantially lower than partial rubric performance. Computation and answer formation also consistently lag behind information acquisition. The benchmark further reports unsupported-correct answers to demonstrate that final-answer accuracy alone can hide incomplete evidence.
Critical strengths
- Financial search requires more than producing the correct number. Entity alignment, reporting periods, definitions, source authority, and evidence support are all important. FinFIRST provides a structured way to evaluate these requirements.
- The decomposition into raw-information acquisition, source verification, and computation provides more diagnostic information than a binary final-answer score. The unsupported-correct analysis is particularly useful.
Critical weaknesses
- FinFIRST emphasizes research traceability and evidence-grounded search. However, the rubric judge receives the question, rubric, and final response. It does not evaluate the agent's actual search trajectory or verify how the retrieved evidence was used to derive the answer. Therefore, the benchmark primarily evaluates the final reported evidence rather than the underlying research process. A model could potentially produce a correct answer with apparently appropriate citations without demonstrating the claimed evidence-grounded reasoning process. This is particularly important because process-level evaluation is presented as the main distinction from existing financial benchmarks.
- The benchmark contains only 123 questions and each model configuration is evaluated using one designated run at temperature 1.0. There is therefore no estimate of run-to-run variance. This matters for the relatively fine-grained comparisons between models and for conclusions drawn from differences of only a few percentage points. The judge validation on 50 sampled instances is useful, but it does not address evaluation variance across model runs or benchmark items.
Questions for authors
- How can the authors verify that the reported evidence was actually used to obtain the answer? Since the judge only sees the final response, what prevents a model from arriving at the answer independently and then attaching plausible supporting sources?
- How stable are the reported model rankings under repeated runs? The current evaluation uses one run per model. Can the authors report variance across multiple runs or justify why a single run is sufficient?
2 comments
Response to Reviewer ZQwY (2/2)
2. W2/Q2
How stable are the model rankings under repeated runs? Can the authors report variance across multiple runs or justify why a single run is sufficient?
Thank you for your thoughtful suggestion. To assess the stability of model scores and rankings across runs, we conducted three independent runs on the same 123 questions and report results as mean ± sample standard deviation in the table below.
| Model | Loose Pass (%) | Strict Pass (%) |
|---|---|---|
| GPT-5.6-SOL | 85.88 ± 0.55 | 71.27 ± 1.24 |
| Claude-Opus-5 | 87.63 ± 0.59 | 69.11 ± 1.63 |
| Gemini-3.7-Flash | 76.83 ± 0.75 | 44.99 ± 2.05 |
| Qwen3.8-Max | 77.44 ± 0.72 | 54.47 ± 1.63 |
| Qwen3.8-Flash | 81.29 ± 0.64 | 61.79 ± 1.63 |
| DeepSeek-V4-Pro | 75.32 ± 0.76 | 50.95 ± 2.05 |
| DeepSeek-V4-Flash | 71.29 ± 0.83 | 48.51 ± 2.05 |
| GLM-5.3 | 80.63 ± 0.56 | 60.98 ± 1.63 |
| GLM-5.3-Flash | 76.60 ± 0.68 | 56.91 ± 1.63 |
| GLM-5.2 | 65.94 ± 0.76 | 43.09 ± 1.63 |
| Kimi-K3 | 80.84 ± 0.80 | 59.62 ± 2.05 |
| Qwen3.8-27B | 77.23 ± 0.76 | 55.01 ± 2.05 |
| MiniMax-M3 | 60.76 ± 0.83 | 37.67 ± 2.05 |
| Hunyuan3 | 65.25 ± 0.79 | 40.92 ± 2.05 |
| Ling-3.0-Flash-Fin | 75.01 ± 0.68 | 52.85 ± 1.63 |
Overall rankings are consistent across runs, though closely performing models occasionally swap positions. Across the three pairwise comparisons, the mean Spearman rank correlation is 0.973 for Loose Pass (range 0.954–0.982) and 0.961 for Strict Pass (range 0.930–0.986). Claude ranks first on Loose Pass in all three runs; GPT achieves the highest mean Strict Pass rate, though Claude leads in the third run; GLM-5.3 and Kimi also exchange positions on Loose Pass. These shifts illustrate that high overall rank agreement can coexist with reordering among closely performing models.
The micro-averaged UCR values across the three runs are 17.48%, 17.32%, and 17.86%, confirming that incomplete evidence among answer-correct responses is a consistent finding rather than an artifact of a single run.
Thank you again for encouraging a more complete assessment of evaluation stability. Please let us know if you have any further questions or concerns.
Best regards,
The Authors of Paper 30129
Response to Reviewer ZQwY (1/2)
Dear Reviewer ZQwY,
Thank you for your thoughtful questions about evidence traceability and evaluation stability. Below, we clarify what the rubric judge assesses and the conclusions supported by our evaluation protocol.
1. W1/Q1
The benchmark assigns atomic scores to the facts, evidence, and derivations presented in the final response, extending evaluation beyond answer correctness. As specified in Appendix E, the judge uses the question, expert-authored rubric, and final response to assess information acquisition, source verification, and computation and answer formation.
This design distinguishes correct answers from answers that also satisfy the evidence requirements. In financial research, final reports are themselves subject to scrutiny: readers need clearly stated sources, reporting periods, metric definitions, and calculations to check its conclusions. Even a numerically correct response may cite the wrong period, omit a required source, or leave out a requested calculation. Atomic scoring identifies these failures that binary answer scoring overlooks. Our results show that 17.48% of answer-correct responses fail at least one evidence-layer criterion, demonstrating the practical value of this distinction.
Regarding post-hoc citations, we distinguish whether the reported evidence satisfies the rubric from whether that evidence actually contributed to generating the answer. The current evaluation assesses the former: it neither infers the model’s internal reasoning sequence nor independently accesses cited pages. Evidence and derivations added afterwards can therefore receive credit if they satisfy the rubric.
We will clarify this boundary and revise references to the research “process” to describe the evidence and derivations presented in the response. FinFIRST’s contribution is to make these supporting elements individually scorable and their deficiencies diagnosable. Extending this framework with tool-trajectory analysis and interventions on retrieved evidence would help examine how that evidence influences answer generation.