acceptodds
Under review as a conference paper at ICLR 2027

BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents

20 views

Abstract

Financial-research answers are decision-relevant only when another analyst can audit how they were produced: which source was chosen, which period and accounting definition were used, which assumptions were made, and how the calculation was performed. Existing finance benchmarks largely evaluate isolated subskills or final answers, leaving the auditable derivation itself under-measured. We introduce BigFinanceBench, a 928-item expert-authored benchmark of open-ended financial-research tasks in which each item pairs a ground-truth reference answer with a point-weighted rubric that decomposes the derivation into independently checkable steps. BigFinanceBench is workflow-grounded in that it rates the whole derivation, rather than just the final output. Across 36,241 rubric points, the benchmark supports partial-credit evaluation and localization of failures across the analyst workflow. Evaluating ten current frontier and open-weight agents, we find substantial headroom: the best system reaches only 58.8% rubric score, final-answer accuracy is a useful but lossy proxy for derivation quality, and model capability varies non-uniformly across financial workflows.

Primary area:
datasets and benchmarks
Keywords:
financial question answering, financial research agents, agentic financial retrieval, workflow-grounded benchmarks, rubric-based evaluation, multi-source retrieval, SEC filings, numerical reasoning, tool-using agents, professional task evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XdbrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents BigFinanceBench, a 928-item benchmark for evaluating financial-research agents through auditable derivations rather than final answers alone. Each item contains an expert reference answer and a point-weighted rubric, and the study evaluates ten headline systems in a 50-step open-book setting while adding broader leaderboard results, a third judge, expert audits, and additional diagnostic analyses.

Critical strengths
  • The benchmark evaluates an important workflow.
  • The dataset construction is well-written and the authors provided many details, e.g., expert-authored questions, validation from finance professionals, and so on.
  • The paper provides a rich empirical analysis.
Critical weaknesses
  • the paper should clarify how the router and GPT-5.5-generated workflow/source labels are obtained, whether the gains are evaluated without question-level outcome leakage.

  • In the abstract, the authors claimed that "the best system reaches only 58.8% rubric score", which is different from results in Table 3.

  • Possible typos: Confludes (Lines 946)

  • the paper should separate tool reliability from performance. (16.5% are off-target, 10.3% are truncated, and 3.5% are empty or boilerplate in Lines 1398-1403)

Questions for authors

Please refer to suggestions in Critical Weaknesses.

2 comments

We thank the reviewer for the positive assessment and for the specific, actionable suggestions. We address each below and have revised the paper accordingly.

W1: Router and workflow/source labels

"the paper should clarify how the router and GPT-5.5-generated workflow/source labels are obtained, whether the gains are evaluated without question-level outcome leakage."

We agree, and thank the reviewer for raising this; the paper did not make it clear. A deployed router would only need to label each question and look up its model. In our analysis, however, the labels are produced with access to the reference answer and rubric, and each group's model is selected and evaluated on the same questions, so the reported gain is an in-sample estimate. Our intent was to show how much a model choice conditioned on task type could recover, given that the leading models are strong on different workflows. We have revised Section 4.2 and Appendix L to describe how the labels are obtained and that the gains are measured in-sample.

W2: 58.8% in the abstract versus Table 3

"In the abstract, the authors claimed that 'the best system reaches only 58.8% rubric score', which is different from results in Table 3."

Thank you for catching this. The 58.8% refers to the best of the ten headline systems evaluated in April 2026 (Claude Opus 4.7 and GPT-5.5). Table 3 extends the evaluation to 31 systems run through September 2026, where the best system reaches 67.1%. We have revised the abstract and the conclusion (Section 5) to state both numbers and the scope of each. The finding is unchanged: even the strongest system evaluated leaves a third of the rubric credit unearned.

W3: Typo

"Possible typos: Confludes (Lines 946)"

Thank you. The typo comes from the original rubric, written by the practitioner who authored the Dayforce item. We had deliberately reproduced the example rubrics verbatim, including such typos, to show the items exactly as written. On reflection, a corrected version serves readers better, and we have fixed it in the paper (Appendix D) and in the dataset.

W4: Separating tool reliability from model performance

"the paper should separate tool reliability from performance. (16.5% are off-target, 10.3% are truncated, and 3.5% are empty or boilerplate in Lines 1398-1403)"

We agree that the paper should make this distinction explicit, and we have added it to the fetch audit in Appendix M. In the finance-expert audit of 400 fetches, the 16.5% off-target fetches returned readable content from a source the agent chose, so they reflect retrieval decisions that the benchmark is designed to measure. The truncated (10.3%) and empty (3.5%) outputs do originate on the tool side; truncation can be recovered by refetching with a targeted query or a larger token budget, and only one fetch was garbled by the parser. Since all models use identical tools, these effects apply to every model equally.

Thank you for your response. I decide to keep my positive rating.

Reviewer_UzMzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BigFinanceBench, a benchmark of 928 practitioner-authored financial-research tasks with 15,656 weighted rubric criteria. Its central contribution is to evaluate auditable intermediate work—including source selection, financial definitions, accounting adjustments, and calculations—using the agent’s visible trajectory rather than only its final answer. The headline experiments evaluate ten models in a shared ReAct-style harness, while Appendix B extends the evaluation to 31 systems. The paper also studies judge agreement, human performance, workflow-specific model strengths, and calculation performance conditional on a correct retrieval-and-definition setup.

Critical strengths

Important problem. The paper addresses the need to evaluate auditable financial-research workflows beyond final-answer correctness.

Strong dataset construction. Practitioner-authored tasks and independent expert review provide a solid foundation for the benchmark.

Substantial validation. The evaluation includes multiple judges, repeated trials, human baselines, and expert regrading.

Critical weaknesses
  1. The methodological contribution needs stronger construct validation. I think this paper missed comparison between grading the final response alone versus grading the full visible trajectory using the same rubric, followed by expert adjudication of disagreements. This would help establish whether access to the trajectory actually improves the assessment of source support, methodological correctness, and auditability.

  2. the paper provides evidence of careful rubric construction, but not yet sufficient evidence that scoring is robust to alternative legitimate methodologies.

Questions for authors

Can you provide an expert-adjudicated comparison of final-response-only and full-trajectory grading, including alternative valid derivations and contradictory or unnecessarily verbose trajectories? How many of the potential reference/rubric errors identified in Table 10 are confirmed?

How are routing policies selected and evaluated? Are policy selection and evaluation performed on disjoint questions, and are all routing features derived without reference answers, rubrics, or post-solution information?

How do the main findings change when representative models receive oracle source documents or correct structured inputs, or when the step budget and termination policy are varied?

2 comments

We thank the reviewer for the positive assessment and for questions that go to the core of the methodology. We address each below and have revised the paper accordingly.

W1 / Q1: Final-response grading versus full-trajectory grading

"I think this paper missed comparison between grading the final response alone versus grading the full visible trajectory using the same rubric, followed by expert adjudication of disagreements. This would help establish whether access to the trajectory actually improves the assessment of source support, methodological correctness, and auditability."

Thank you for this suggestion. In our harness, agents submit their final answer through a terminal tool, and for most models this response is only the figure itself (median under 20 characters), so a grader that sees only the final response can check little beyond the answer. We therefore took the question to be whether access to the full work changes the assessment, and tested it against the form of evidence used in practice.

In finance, an analyst hands in a workbook, and a senior colleague checks it step by step: which sources were used, which values were taken, and how each figure was derived. Our rubrics were written against that kind of workpaper. An agent's trajectory records the same information, every retrieval, tool output, and calculation behind the answer, so we grade the trajectory as the agent's equivalent of the workbook. We assumed this to be the simpler of the two options: it needs no separate deliverable from the agent and no conversion of spreadsheets for an LLM judge.

To validate this assumption, we compared both forms of evidence on the same tasks. In a workbook variant of the harness, the agent is also asked to save an Excel workbook of its analysis that a supervisor could check, and the judges grade that workbook with the final answer instead of the trajectory, under the same rubrics. We ran GPT-5.5 on the 50 public questions with three attempts each; the agent produced a workbook in 146 of 150 runs.

Judge reads trajectoryJudge reads workbook
Rubric score59.6%59.8%
Final-answer accuracy45.7%47.7%

The rubric score is unchanged (difference +0.2 points, 95% interval -5.0 to +5.7), which confirms that the trajectory carries the same evidence as the workbook. The remaining per-question differences go both ways: work the agent did without showing it in the trajectory becomes visible in workbook formulas, while a few workbooks contain broken formulas or a value other than the one the agent computed. We have added this comparison as Appendix Q.

W2: Robustness to alternative legitimate methodologies

"the paper provides evidence of careful rubric construction, but not yet sufficient evidence that scoring is robust to alternative legitimate methodologies."

The practitioner solutions provide direct evidence. Nine finance professionals solved 90 tasks authored by other practitioners, given only the question and not the rubric, and therefore chose their own methods. Their workpapers nonetheless earn 91.7% of the rubric credit. A manual audit of 33 of these workbooks found the missing credit to consist mostly of steps the experts did not record, rather than valid steps the rubric failed to recognize. This is consistent with how the rubrics were written: authors and reviewers included a step only if nearly all experts would agree it belongs in any correct answer, regardless of method, which keeps method-specific steps out of the rubric.

Q1 (continued): Contradictory and verbose trajectories, and Table 10

"including alternative valid derivations and contradictory or unnecessarily verbose trajectories? How many of the potential reference/rubric errors identified in Table 10 are confirmed?"

The judges award a criterion only when the trajectory contains positive evidence for it, but we have not tested trajectories that list several candidate values or contradict themselves. We agree this is a relevant risk for trajectory grading and have added it to the limitations in Section 5.

On Table 10, the 9.4% category is an LLM classifier label, not a confirmed error, and these cases have not yet been fully reviewed. Many concern precision: LLMs, the classifier included, struggle to choose the right level of precision for a financial figure, a judgment practitioners make by convention, and so flag correctly rounded references as errors. The references themselves are well supported: every reference answer was recomputed by a second practitioner, and experts solving the tasks without the rubric earn 91.7% of its credit. We have stated in Appendix N that the category is unconfirmed.

Q2: Routing

"How are routing policies selected and evaluated? Are policy selection and evaluation performed on disjoint questions, and are all routing features derived without reference answers, rubrics, or post-solution information?"

Each routing policy groups the questions by task type (workflow, skill, source) and sends every group to the model that scores best on it. These policies are selected and evaluated on the same questions, not on disjoint ones. The workflow and skill labels come from a GPT-5.5 classifier that reads the reference answer and rubric in addition to the question.

The reported gains are therefore in-sample estimates of what routing by task type can recover. The policies themselves are simple: a deployed router would label each question and look up its model. We have revised Section 4.2 and Appendix L to describe the selection and the labels.

Q3: Oracle inputs and step budget

"How do the main findings change when representative models receive oracle source documents or correct structured inputs, or when the step budget and termination policy are varied?"

Our clean-setup analysis (Figure 7) is close to the correct-inputs condition. Conditioning on trajectories with full Retrieval and Definition credit, models earn 85% of Calculation credit on average, and the remaining differences are small for most models. Our conclusion that setup rather than arithmetic is the main bottleneck rests on this conditioning. Exhausting the 50-step budget is concentrated in three systems (Kimi K2.6, Gemini 3 Flash, GLM 5.1), which we report as a looping failure mode. Runs with oracle documents and a larger budget would further separate retrieval from analysis; we have added them as future work in Section 5.

Reviewer_jxzhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a benchmark for evaluating agents on finance related tasks.Each task includes a (expert manually written and reviewed) question and answer and rubric.Different models perform tasks in a unified ReAct-style harness,and the authors evaluate both final-answer accuracy and intermediate rubric attainment.

Critical strengths

The authors provide a relatively complete account of the annotation process,essentially describing their data collection-validation-test process in full,and compared with typical benchmarks,they provide relative detailed rubric annotations,which facilitate evaluation of runs on the benchmark at a finer level of detail.

Critical weaknesses
  1. I find the novelty of this work relatively limited.Based on the comparison with other benchmarks in finance provided by the authors,the main distinction from existing work is the provision of dervation-level evaluation,but benchmarks such as APEX have already provided relatively detailed criteria and standards for scoring intermediate steps,so the novelty in this respect is also not particularly limited

  2. I think the validity of their derivation-level evaluation needs further clarification,and I find their validation of these rubrics and the derivation-level evaluation relatively not so sufficient,for example, they only report the answer scores and rubric scores of expert answers(in this case the rubric scores are even lower than the answer scores),but they do not show that for responses with incorrect final answers,the rubrics can provide scores that distinguish the quality of intermediate steps,and I hope the authors can provide additional detailed experiments to demonstrate the value of their annotations.

Questions for authors

See weakness 2

2 comments

We thank the reviewer for the feedback. We address both concerns, novelty relative to APEX and validity of the derivation-level evaluation, and have revised the paper accordingly.

W1: Novelty relative to APEX

"I find the novelty of this work relatively limited [...] benchmarks such as APEX have already provided relatively detailed criteria and standards for scoring intermediate steps"

We agree that APEX also scores intermediate steps; the submission did not make the distinction explicit, and APEX was missing from Table 1. The distinction is who determines those steps and what evidence the grader observes.

  1. APEX specifies the procedure; BigFinanceBench requires the agent to determine it. In APEX, the task prompt provides both the inputs and the procedure: the source files are included in the prompt, and the intermediate quantities are listed as deliverables. The investment-banking example asks the model to "calculate ... PP&E, capex, other operating assets, net working capital ..." under stated assumptions; tool use is left to future work. A BigFinanceBench question states only the decision to be made, e.g. whether Dayforce's adjusted EBIT is overstated and by how much. The agent must locate the evidence itself through web and EDGAR search and determine the relevant filing, fiscal year, and adjustments, as a practitioner would.
  2. APEX verifies requested outputs; BigFinanceBench assesses necessary steps the question does not name. Its rubric therefore verifies that the requested values were produced. Our rubrics instead assess steps that the question does not name but that a correct derivation requires, such as each component of the Dayforce GAAP-to-non-GAAP bridge.
  3. APEX grades the final response; BigFinanceBench grades the full trajectory. The APEX judge receives only the criterion and the final response. Our judges observe the full trajectory, including tool calls, tool outputs, and calculations, so credit requires supporting evidence in the run.

We have added APEX to Table 1 and renamed its last column to "Trajectory grading".

The same gap separates BigFinanceBench from the other finance benchmarks we cite. A banker, buy-side investor or equity research analyst starts from a decision about one real company at one point in time: is this adjusted EBIT overstated, what is this deal worth, did the quarter beat. To answer it they decide which filings and periods matter, which line items and adjustments apply, build the figure, and show the work so a reviewer can check it. Each prior benchmark removes at least one of these steps, so a model can score well on it without doing the part of the job that is actually hard:

  • FinQA, TAT-QA, DocFinQA and FinBen hand over the evidence. Questions are written against a given report page or table, and most answers are a single ratio or a span copied from it. None asks the model to decide where the answer comes from.
  • FinanceBench, FinRetrieval and FinDeepResearch name the line item and the period. A typical FinanceBench question reads "What is Boeing's FY2022 cost of goods sold". The analyst's judgment, which number answers the business question, is already made in the prompt.
  • Finance Agent Benchmark is the closest prior work, but still spells out the method. Its agent also searches EDGAR without given documents, but its questions often list the recipe, e.g. every component of Amazon's "M&A firepower", and its rubrics check only the final answer.
  • FinanceMath, FAMMA and FinTextQA test exam knowledge. They pose textbook problems about hypothetical companies with every input given. A practitioner applies such knowledge to a real company whose disclosures are incomplete.
  • PRBench and ProfBench have practitioner authors and weighted rubrics, but no retrieval. PRBench tasks are "designed to be solvable through reasoning alone", and ProfBench supplies pre-selected documents. In ProfBench's own ablation, o3's finance score falls from 60.9 to 43.4 without the documents.
  • FinResearchBench, FinTradeBench, FinDER and FinAgentBench do not check a figure. Much of financial analysis has an objectively verifiable outcome: a computed figure is either correct or not. These benchmarks give up that property. FinResearchBench has no reference answers and scores the structure of a report, so a report with incorrect numbers can still score well. FinTradeBench compares responses against LLM-generated reference answers. FinDER is mostly qualitative, and FinAgentBench scores rankings of documents rather than an answer. Their scores therefore cannot show whether the analysis was correct.

BigFinanceBench keeps all of these steps in the task: practitioners wrote each question about a real company and date, the agent gets no documents, and the rubric grades the choices the question leaves open.

W2: Validity of the rubrics

"the validity of their derivation-level evaluation needs further clarification [...] they only report the answer scores and rubric scores of expert answers"

We address validity in three respects: (1) whether the rubrics contain the steps of correct solutions, (2) whether they can be applied reliably, and (3) whether rubric scores track the quality of the work (W3).

Expert rubric scores. We regard the expert results as the most direct validation of the rubrics. Nine finance professionals solved 90 tasks authored by other practitioners, given only the question and not the rubric. Their workpapers nonetheless earn 91.7% of the rubric credit: experts working independently perform nearly all steps the rubrics require. Rubrics encoding a single author's idiosyncratic method would not be reproduced this closely.

The gap to their 98.9% answer accuracy reflects shortcomings in the human work. A manual audit of 33 workbooks, checking every uncredited criterion against the cells, formulas, and comments, finds mostly steps the experts did not record, such as line items reported in aggregate where the rubric requires each component, and some genuine mistakes that did not affect the final figure, such as a mislabelled fiscal year. In a bank or investment firm, a reviewing colleague would raise these issues before the analysis is relied upon. The rubric registers them; final-answer accuracy does not.

Each rubric was also audited during construction by a second practitioner who independently recomputed the reference answer.

Reliability of grading. A rubric with vague or subjective criteria would not be applied consistently by independent graders. A finance expert, blind to the judges' grades, regraded 387 criteria across 30 trajectories covering all ten models. The expert's scores correlate with the two-judge scores at Pearson r = 0.946, the aggregate scores differ by 0.4 points, and where both judges agree, the expert agrees with them on 98.1% (kappa = 0.961). The criteria are specific enough that an independent expert reaches the judges' verdicts.

W3: Rubric scores on incorrect answers

"they do not show that for responses with incorrect final answers,the rubrics can provide scores that distinguish the quality of intermediate steps"

Final-answer grading assigns every incorrect response the same score. We analysed the 19,091 trajectories whose final answer both judges mark incorrect and find that the rubric separates them along two dimensions: how much of the analysis was done correctly, and where in the analyst workflow it failed. Most of the variation in their rubric scores lies between incorrect answers to the same question, so it reflects the work rather than only the question's difficulty.

1. Rubric credit measures how much of the analysis is correct. If the rubric captures intermediate progress, an incorrect answer with more credit should be closer to a correct solution. We test this against two outcomes derived from final answers alone, independent of the rubric verdicts: (a) the distance of the submitted figure from the reference, for the 7,664 answers with a single-number reference, and (b) how often the same model answers the same question correctly on its other two attempts.

Rubric score of the incorrect answer0-20%20-40%40-60%60-100%
(a) Median relative error to the reference27.4%11.6%5.8%3.0%
(b) Other attempts at the same question correct6.1%7.8%11.9%17.7%

Both outcomes improve steadily with rubric credit: incorrect answers with high credit are near misses, and the models that produced them are three times as likely to solve the question on another attempt. Since (b) also holds within questions, easier questions earning more credit do not explain it.

2. Rubric credit locates where the workflow broke. Our stage analysis (Figure 7) attributes rubric credit to Retrieval, Definition, and Calculation, but pools correct and incorrect answers. We repeated the analysis on incorrect answers only and came to similar conclusions: every model earns far less credit in Calculation than in Retrieval and Definition, and the model ranking is preserved. The difference is where failures occur. Over all answers, a quarter of trajectories reach a clean setup with full Retrieval and Definition credit, and these then earn 85% of Calculation credit on average. Among incorrect answers, only 7% reach a clean setup, so most failures originate in finding and framing the inputs. The few that do reach it earn about half of the Calculation credit, a level that no longer separates models; for these, the rubric locates the error in the calculation itself.

The converse also holds: on average 43% of correct final answers still miss rubric credit, mostly for omitted retrieval and calculation steps. Neither distinction is visible to final-answer grading. We have added both analyses as Appendix P.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.