- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents BigFinanceBench, a 928-item benchmark for evaluating financial-research agents through auditable derivations rather than final answers alone. Each item contains an expert reference answer and a point-weighted rubric, and the study evaluates ten headline systems in a 50-step open-book setting while adding broader leaderboard results, a third judge, expert audits, and additional diagnostic analyses.
Critical strengths
- The benchmark evaluates an important workflow.
- The dataset construction is well-written and the authors provided many details, e.g., expert-authored questions, validation from finance professionals, and so on.
- The paper provides a rich empirical analysis.
Critical weaknesses
-
the paper should clarify how the router and GPT-5.5-generated workflow/source labels are obtained, whether the gains are evaluated without question-level outcome leakage.
-
In the abstract, the authors claimed that "the best system reaches only 58.8% rubric score", which is different from results in Table 3.
-
Possible typos: Confludes (Lines 946)
-
the paper should separate tool reliability from performance. (16.5% are off-target, 10.3% are truncated, and 3.5% are empty or boilerplate in Lines 1398-1403)
Questions for authors
Please refer to suggestions in Critical Weaknesses.
2 comments
Thank you for your response. I decide to keep my positive rating.
We thank the reviewer for the positive assessment and for the specific, actionable suggestions. We address each below and have revised the paper accordingly.
W1: Router and workflow/source labels
We agree, and thank the reviewer for raising this; the paper did not make it clear. A deployed router would only need to label each question and look up its model. In our analysis, however, the labels are produced with access to the reference answer and rubric, and each group's model is selected and evaluated on the same questions, so the reported gain is an in-sample estimate. Our intent was to show how much a model choice conditioned on task type could recover, given that the leading models are strong on different workflows. We have revised Section 4.2 and Appendix L to describe how the labels are obtained and that the gains are measured in-sample.
W2: 58.8% in the abstract versus Table 3
Thank you for catching this. The 58.8% refers to the best of the ten headline systems evaluated in April 2026 (Claude Opus 4.7 and GPT-5.5). Table 3 extends the evaluation to 31 systems run through September 2026, where the best system reaches 67.1%. We have revised the abstract and the conclusion (Section 5) to state both numbers and the scope of each. The finding is unchanged: even the strongest system evaluated leaves a third of the rubric credit unearned.
W3: Typo
Thank you. The typo comes from the original rubric, written by the practitioner who authored the Dayforce item. We had deliberately reproduced the example rubrics verbatim, including such typos, to show the items exactly as written. On reflection, a corrected version serves readers better, and we have fixed it in the paper (Appendix D) and in the dataset.
W4: Separating tool reliability from model performance
We agree that the paper should make this distinction explicit, and we have added it to the fetch audit in Appendix M. In the finance-expert audit of 400 fetches, the 16.5% off-target fetches returned readable content from a source the agent chose, so they reflect retrieval decisions that the benchmark is designed to measure. The truncated (10.3%) and empty (3.5%) outputs do originate on the tool side; truncation can be recovered by refetching with a targeted query or a larger token budget, and only one fetch was garbled by the parser. Since all models use identical tools, these effects apply to every model equally.