- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper releases a fully public benchmark of 101 tasks whose input is only a workbook and instructions, and the output is a complete financial model built end to end. Grading uses one fixed rubric of 129 criteria: final calculation accuracy is checked, and the remaining quality criteria are graded by an agentic judge, since many of them need context. Across more than 20 agents evaluated, even the strongest fails most tasks, and current agents still cannot deliver professional-grade spreadsheets.
Critical strengths
The judge is validated well. If the judge were unreliable, a low pass rate could reflect judge errors rather than agent failures. The paper rules this out: 0.26 percent false positive rate on controlled examples and 1.61 percent on real attempts against human experts. Both are low enough that the lenient pass rule works.
By hiding the rubric, the evaluation ensures that the pass rate measures ability rather than whether an agent was missing instructions. The Score discount makes the fallback metric usable by removing the inflation from criteria that an unchanged input workbook can already satisfy, so low-end Score differences reflect something real.
Critical weaknesses
The pass rate is not shown to be tight. The rule already allows several violations at each importance level, and the paper reports false negatives but not how violation counts are distributed. Calling the pass rate an upper bound is therefore not enough, since readers cannot tell how loose the bound is.
Judge drift is not measured. The judge is a current LLM grader, and reproducibility covers code and data but not the judge. Agreement with a stronger judge supports the choice of judge, but it does not show that the same judge gives stable verdicts after model updates.
The technical contribution is thin. The criteria come from public modeling codes, and LLM grading is now standard. The main contributions are resources rather than methods: a task set, a taxonomy, a validated judge, and a large evaluation.
Questions for authors
How are violation counts distributed near the pass boundary? If failing workbooks miss several criteria, the reported false-negative rate matters little. If many cluster at the boundary, the headline pass rate is inflated and should be read as a lower bound on failure.
Has any attempt been graded under two backbone versions to test judge drift? Agreement with a stronger judge supports the judge choice, but it does not show that the same judge gives stable verdicts after an update. So how long the leaderboard stays comparable is unclear.
If the violation distribution is concentrated near the boundary and judge drift is large, what should a reader do differently?
3 comments
Individual Response to Reviewer ndEy
We thank the reviewer for the thorough and constructive review, and are especially grateful that the reviewer recognizes the judge validation experiment, and how the hidden rubric and score discount make the metrics meaningful.
"Since readers cannot tell how loose the bound is…How are violation counts distributed near the pass boundary?"
We thank the reviewer for inspiring us for the following additional experiments. We verify that most failing workbooks are far from the boundary. First, 53% of the attempts fail the deterministic final-answer check, which results in an automatic failure regardless of the pass rule. Of the remaining, 81% were 3 or more failed criteria over the limit (aggregate across importance tiers). Additionally, we find that moving all tier's allowance by ±1 only changes the overall pass rate across all workbooks from 2.5% to between 0.5% and 7.1%. The conclusion that agents rarely pass therefore holds across this range, even if all four tiers are relaxed at once.
"Has any attempt been graded under two backbone versions to test judge drift?"
All results in the paper use one OpenAI Sol 5.6 snapshot with pinned inference settings. As such, grades within the paper are unaffected by drift and we can always use the same judge version. Furthermore, since we release the judge code and prompts, any agent attempts can be re-graded under a fixed judge version if Sol 5.6 snapshot is retired.
"If the violation distribution is concentrated near the boundary and judge drift is large, what should a reader do differently?"
Our analyses show neither condition holds for the results in the paper. Most failures are far from the boundary and all of our grades come from one pinned version of the judge. We agree wholeheartedly with the reviewers that the results should only be compared under the same pinned judge, and will add a discussion on this and the pass boundary results.
Dear Reviewer ndEy, as the end of the discussion comes near, we would greatly appreciate the chance to discuss further and hopefully address your concerns.
We are grateful for your time reviewing the paper, and for the astute points on the finer details in pass rate definition. To address the latter, we have performed additional experiments to show that violations are not mostly on the boundary, and therefore the main conclusion that most agents rarely passes, holds.
We hope the above satisfactorily addresses your concerns, and are happy to discuss further should any concern remains.
General Response
We sincerely thank the reviewers for their thoughtful feedback. We are grateful that all three reviewers appreciate our extensive validation of the judge (uYW5, w88T, ndEy), value the scale and originality of our 101 financial modeling tasks (uYW5, w88T, ndEy), and appreciate the breadth of the evaluation across 20+ agents, including commercial Excel AI products (uYW5, w88T, ndEy). We are particularly glad that the reviewers recognize that the rubric goes beyond numerical accuracy to evaluate readability, auditability, and maintainability of workbooks (uYW5, w88T, ndEy), since this is exactly how a finance professional’s work would be evaluated in practice, and a key dimension in which our benchmark differentiates from the prevalent evaluation paradigm. The main shared concerns of the reviewers are on further clarification of our settings and novelty relative to existing spreadsheet benchmarks (uYW5, w88T, ndEy). We are thankful for this opportunity to provide further clarifications and experimental results, and are hopeful that we can satisfactorily address them and other concerns in this round.
On settings and contributions relative to existing benchmarks.
Our central contributions are (a) the frontier setting where agents are tasked to construct spreadsheets from scratch (b) one of the most comprehensive and difficult tasks set in financial modeling, (c) and a carefully designed set of rubrics that are generally applicable.
Multiples are formatted with suffix ‘x’, as in BTB). Our rubrics therefore provides a comprehensive taxonomy for good modeling practices, and its general applicability also allows us to develop and validate the judge harness for each criteria.The aspects above situate the work substantially different from existing benchmarks, and we appreciate the chance to clarify the uniqueness of our contribution. The reviewers have raised other points regarding pass rule and judge validation, and we have performed additional annotation and experiments (e.g. we verify that pass rule aligns with expert judgement, and provide statistics on inter-rater agreement. See response to reviewers uYW5 and ndEy).
We believe these clarifications and additional experiments have strengthened the paper, and will include them in the camera-ready version with the extra page provided. We thank the reviewers for the discussion that led to these improvements.
[1] Patwardhan et al., GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks. arXiv:2510.04374
[2] Yang et al., ProgramBench: Can Language Models Rebuild Programs From Scratch? arXiv: 2605.03546
[3] Terminal-Bench-Science: Evaluating AI agents on research workflows across scientific domains. https://www.tbench.ai/news/terminal-bench-science-0-1