acceptodds
Withdrawn from ICLR 2027

SpreadsheetSmith: Evaluating AI Agents on Constructing Financial Spreadsheets From Scratch

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spreadsheets are the lingua franca of modern finance, driving valuation, forecast, audit, and many other core workflows of the industry. As LLM agents advance to automate knowledge work, spreadsheet construction is a natural and pertinent testbed. However, most existing spreadsheet benchmarks evaluate agents on fill-in-the-blank style tasks, where an agent performs minor edits in otherwise complete spreadsheets. This protocol fails to capture the most central and time-consuming part of the spreadsheet workflow: the modeling decisions, format design, and sheet layout regular knowledge workers need to toil through to build a workbook from scratch. We introduce SpreadsheetSmith, a benchmark of 101 original spreadsheet construction tasks, the hardest of which is estimated to take 33 hours for a modeler with 3+ years of experience. To holistically evaluate spreadsheet construction, we introduce 129 criteria spanning 12 categories that operationalize quality beyond the accuracy of the spreadsheet. Evaluating over 20 agents across 8 models and 4 harnesses, we find that frontier agents rarely construct spreadsheets that are both accurate and meet our quality criteria: the top agent achieves a pass rate of only 13% on the benchmark. The benchmark demonstrates a substantial gap in spreadsheet building capability of current agents.

Primary area:
datasets and benchmarks
Keywords:
spreadsheet construction, finance, end-to-end financial tasks, LLM agents, benchmark, LLM-as-judge

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ndEyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper releases a fully public benchmark of 101 tasks whose input is only a workbook and instructions, and the output is a complete financial model built end to end. Grading uses one fixed rubric of 129 criteria: final calculation accuracy is checked, and the remaining quality criteria are graded by an agentic judge, since many of them need context. Across more than 20 agents evaluated, even the strongest fails most tasks, and current agents still cannot deliver professional-grade spreadsheets.

Critical strengths

The judge is validated well. If the judge were unreliable, a low pass rate could reflect judge errors rather than agent failures. The paper rules this out: 0.26 percent false positive rate on controlled examples and 1.61 percent on real attempts against human experts. Both are low enough that the lenient pass rule works.

By hiding the rubric, the evaluation ensures that the pass rate measures ability rather than whether an agent was missing instructions. The Score discount makes the fallback metric usable by removing the inflation from criteria that an unchanged input workbook can already satisfy, so low-end Score differences reflect something real.

Critical weaknesses

The pass rate is not shown to be tight. The rule already allows several violations at each importance level, and the paper reports false negatives but not how violation counts are distributed. Calling the pass rate an upper bound is therefore not enough, since readers cannot tell how loose the bound is.

Judge drift is not measured. The judge is a current LLM grader, and reproducibility covers code and data but not the judge. Agreement with a stronger judge supports the choice of judge, but it does not show that the same judge gives stable verdicts after model updates.

The technical contribution is thin. The criteria come from public modeling codes, and LLM grading is now standard. The main contributions are resources rather than methods: a task set, a taxonomy, a validated judge, and a large evaluation.

Questions for authors

How are violation counts distributed near the pass boundary? If failing workbooks miss several criteria, the reported false-negative rate matters little. If many cluster at the boundary, the headline pass rate is inflated and should be read as a lower bound on failure.

Has any attempt been graded under two backbone versions to test judge drift? Agreement with a stronger judge supports the judge choice, but it does not show that the same judge gives stable verdicts after an update. So how long the leaderboard stays comparable is unclear.

If the violation distribution is concentrated near the boundary and judge drift is large, what should a reader do differently?

3 comments

General Response

We sincerely thank the reviewers for their thoughtful feedback. We are grateful that all three reviewers appreciate our extensive validation of the judge (uYW5, w88T, ndEy), value the scale and originality of our 101 financial modeling tasks (uYW5, w88T, ndEy), and appreciate the breadth of the evaluation across 20+ agents, including commercial Excel AI products (uYW5, w88T, ndEy). We are particularly glad that the reviewers recognize that the rubric goes beyond numerical accuracy to evaluate readability, auditability, and maintainability of workbooks (uYW5, w88T, ndEy), since this is exactly how a finance professional’s work would be evaluated in practice, and a key dimension in which our benchmark differentiates from the prevalent evaluation paradigm. The main shared concerns of the reviewers are on further clarification of our settings and novelty relative to existing spreadsheet benchmarks (uYW5, w88T, ndEy). We are thankful for this opportunity to provide further clarifications and experimental results, and are hopeful that we can satisfactorily address them and other concerns in this round.

On settings and contributions relative to existing benchmarks.

Our central contributions are (a) the frontier setting where agents are tasked to construct spreadsheets from scratch (b) one of the most comprehensive and difficult tasks set in financial modeling, (c) and a carefully designed set of rubrics that are generally applicable.

  • Setting: The agent receives only an input workbook and instructions and must build the complete model from scratch. This is a common theme in frontier benchmarks where agents are increasingly tasked with long-horizon, difficult tasks only expected of professionals (e.g. GDPval [1], ProgramBench [2], Terminal-Bench-Science [3]). We agree with Reviewer w88T that these tasks are much more complex than common workflows, but we view this as an important evaluation choice to understand frontier agent capability. This differentiates us from typical simpler benchmarks on finance and/or spreadsheets. We thank Reviewer uYW5 for raising a few related work that serve as clarifying examples:
    • FrontierFinance (Zhang et al.) is a typical QA benchmark. The tasks therein involve finding and analyzing information from existing documents, but distinctly do not evaluate agent’s capability to construct spreadsheets.
    • Finch (Dong et al.) evaluates based on workflows consisting of retrieval and simple spreadsheet edits (e.g. Example 1, review several sheets, retrieve info, and edit one main sheet with appropriate facts). Only 15 of its tasks involve spreadsheets, and these tasks are prototypical cases where a near-complete spreadsheet is already provided, and the agent is only tasked with minor edits on top, but is not expected to build models from scratch.
  • Task Set Size, Difficulty and Breath: We have created 101 bespoke modeling tasks, which is uniquely large in scale for agentic evaluation (e.g. compared to 50 in BTB, 4 in GDPVal, 38 public in MBABench). Besides sheer scale, the benchmark also covers most financial modeling use cases (in stark contrast to BTB and GDPVal, see Figure 4), and focuses on much more difficult tasks than MBABench.
  • Evaluation: we have carefully designed the rubric to extensively cover aspects of good modeling practices (129 rubrics vs. 17 in MBABench) and to be generally applicable (e.g. we avoid overly task-specific rubrics such as Multiples are formatted with suffix ‘x’ , as in BTB). Our rubrics therefore provides a comprehensive taxonomy for good modeling practices, and its general applicability also allows us to develop and validate the judge harness for each criteria.

The aspects above situate the work substantially different from existing benchmarks, and we appreciate the chance to clarify the uniqueness of our contribution. The reviewers have raised other points regarding pass rule and judge validation, and we have performed additional annotation and experiments (e.g. we verify that pass rule aligns with expert judgement, and provide statistics on inter-rater agreement. See response to reviewers uYW5 and ndEy).

We believe these clarifications and additional experiments have strengthened the paper, and will include them in the camera-ready version with the extra page provided. We thank the reviewers for the discussion that led to these improvements.

[1] Patwardhan et al., GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks. arXiv:2510.04374

[2] Yang et al., ProgramBench: Can Language Models Rebuild Programs From Scratch? arXiv: 2605.03546

[3] Terminal-Bench-Science: Evaluating AI agents on research workflows across scientific domains. https://www.tbench.ai/news/terminal-bench-science-0-1

Individual Response to Reviewer ndEy

We thank the reviewer for the thorough and constructive review, and are especially grateful that the reviewer recognizes the judge validation experiment, and how the hidden rubric and score discount make the metrics meaningful.

"Since readers cannot tell how loose the bound is…How are violation counts distributed near the pass boundary?"

We thank the reviewer for inspiring us for the following additional experiments. We verify that most failing workbooks are far from the boundary. First, 53% of the attempts fail the deterministic final-answer check, which results in an automatic failure regardless of the pass rule. Of the remaining, 81% were 3 or more failed criteria over the limit (aggregate across importance tiers). Additionally, we find that moving all tier's allowance by ±1 only changes the overall pass rate across all workbooks from 2.5% to between 0.5% and 7.1%. The conclusion that agents rarely pass therefore holds across this range, even if all four tiers are relaxed at once.

"Has any attempt been graded under two backbone versions to test judge drift?"

All results in the paper use one OpenAI Sol 5.6 snapshot with pinned inference settings. As such, grades within the paper are unaffected by drift and we can always use the same judge version. Furthermore, since we release the judge code and prompts, any agent attempts can be re-graded under a fixed judge version if Sol 5.6 snapshot is retired.

"If the violation distribution is concentrated near the boundary and judge drift is large, what should a reader do differently?"

Our analyses show neither condition holds for the results in the paper. Most failures are far from the boundary and all of our grades come from one pinned version of the judge. We agree wholeheartedly with the reviewers that the results should only be compared under the same pinned judge, and will add a discussion on this and the pass boundary results.

Dear Reviewer ndEy, as the end of the discussion comes near, we would greatly appreciate the chance to discuss further and hopefully address your concerns.

We are grateful for your time reviewing the paper, and for the astute points on the finer details in pass rate definition. To address the latter, we have performed additional experiments to show that violations are not mostly on the boundary, and therefore the main conclusion that most agents rarely passes, holds.

We hope the above satisfactorily addresses your concerns, and are happy to discuss further should any concern remains.

Reviewer_w88TOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper deals with evaluating AI agents in the context of spreadsheet generation focusing on financial tasks, which is motivated by its practical importance. The paper is well-written and easy to follow. The evaluation is based on 101 tasks generated by financial modelers that most likely are not in already available benchmarks, which is an excellent idea. The tasks are sorted considering their difficulty level. The evaluation comprises a vast number of different criterions that are put together. The evaluation is based on the agentic judge principle, which is a least debatable. To make this reasonable, the authors also provided a judge-evaluation against experts on a limited number of attempts, which is again an excellent idea. The provided judge accuracy of more than 96% seems to be sufficient for the evaluation. The presented results are interesting showing that generating spreadsheets using AI agents is not that simple and cannot be sufficiently achieved, especially when the difficulty level is higher. This seems to be in line with similar work and also personal experience.

Critical strengths
  • The paper tackles a recent and import problem of using agentic coding for spreadsheet generation.
  • The paper comes with an experimental evaluation comprising more than 100 tasks.
  • All open issues like providing evidence for agentic judges are reasonably solved.
Critical weaknesses
  • The application domain is a very specific one. Results might not be the same when going to ordinary (and simple) spreadsheets most people use these days.
Questions for authors

There are no further questions.

3 comments

General Response

We sincerely thank the reviewers for their thoughtful feedback. We are grateful that all three reviewers appreciate our extensive validation of the judge (uYW5, w88T, ndEy), value the scale and originality of our 101 financial modeling tasks (uYW5, w88T, ndEy), and appreciate the breadth of the evaluation across 20+ agents, including commercial Excel AI products (uYW5, w88T, ndEy). We are particularly glad that the reviewers recognize that the rubric goes beyond numerical accuracy to evaluate readability, auditability, and maintainability of workbooks (uYW5, w88T, ndEy), since this is exactly how a finance professional’s work would be evaluated in practice, and a key dimension in which our benchmark differentiates from the prevalent evaluation paradigm. The main shared concerns of the reviewers are on further clarification of our settings and novelty relative to existing spreadsheet benchmarks (uYW5, w88T, ndEy). We are thankful for this opportunity to provide further clarifications and experimental results, and are hopeful that we can satisfactorily address them and other concerns in this round.

On settings and contributions relative to existing benchmarks.

Our central contributions are (a) the frontier setting where agents are tasked to construct spreadsheets from scratch (b) one of the most comprehensive and difficult tasks set in financial modeling, (c) and a carefully designed set of rubrics that are generally applicable.

  • Setting: The agent receives only an input workbook and instructions and must build the complete model from scratch. This is a common theme in frontier benchmarks where agents are increasingly tasked with long-horizon, difficult tasks only expected of professionals (e.g. GDPval [1], ProgramBench [2], Terminal-Bench-Science [3]). We agree with Reviewer w88T that these tasks are much more complex than common workflows, but we view this as an important evaluation choice to understand frontier agent capability. This differentiates us from typical simpler benchmarks on finance and/or spreadsheets. We thank Reviewer uYW5 for raising a few related work that serve as clarifying examples:
    • FrontierFinance (Zhang et al.) is a typical QA benchmark. The tasks therein involve finding and analyzing information from existing documents, but distinctly do not evaluate agent’s capability to construct spreadsheets.
    • Finch (Dong et al.) evaluates based on workflows consisting of retrieval and simple spreadsheet edits (e.g. Example 1, review several sheets, retrieve info, and edit one main sheet with appropriate facts). Only 15 of its tasks involve spreadsheets, and these tasks are prototypical cases where a near-complete spreadsheet is already provided, and the agent is only tasked with minor edits on top, but is not expected to build models from scratch.
  • Task Set Size, Difficulty and Breath: We have created 101 bespoke modeling tasks, which is uniquely large in scale for agentic evaluation (e.g. compared to 50 in BTB, 4 in GDPVal, 38 public in MBABench). Besides sheer scale, the benchmark also covers most financial modeling use cases (in stark contrast to BTB and GDPVal, see Figure 4), and focuses on much more difficult tasks than MBABench.
  • Evaluation: we have carefully designed the rubric to extensively cover aspects of good modeling practices (129 rubrics vs. 17 in MBABench) and to be generally applicable (e.g. we avoid overly task-specific rubrics such as Multiples are formatted with suffix ‘x’ , as in BTB). Our rubrics therefore provides a comprehensive taxonomy for good modeling practices, and its general applicability also allows us to develop and validate the judge harness for each criteria.

The aspects above situate the work substantially different from existing benchmarks, and we appreciate the chance to clarify the uniqueness of our contribution. The reviewers have raised other points regarding pass rule and judge validation, and we have performed additional annotation and experiments (e.g. we verify that pass rule aligns with expert judgement, and provide statistics on inter-rater agreement. See response to reviewers uYW5 and ndEy).

We believe these clarifications and additional experiments have strengthened the paper, and will include them in the camera-ready version with the extra page provided. We thank the reviewers for the discussion that led to these improvements.

[1] Patwardhan et al., GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks. arXiv:2510.04374

[2] Yang et al., ProgramBench: Can Language Models Rebuild Programs From Scratch? arXiv: 2605.03546

[3] Terminal-Bench-Science: Evaluating AI agents on research workflows across scientific domains. https://www.tbench.ai/news/terminal-bench-science-0-1

Individual Response to Reviewer w88T

We are deeply grateful for the reviewer's positive assessment, and are glad that the reviewer finds the expert-authored task set and the judge validation against experts convincing, and the results consistent with practical experience.

"The application domain is a very specific one. Results might not be the same when going to ordinary (and simple) spreadsheets most people use these days."

We agree with the reviewer that results on simple, everyday spreadsheets are important, and may differ from our conclusion. We will state this explicitly in the limitation section. We chose financial modeling deliberately for three reasons:

  • Importance of the domain We believe Finance is one of the crucial application domains for AI, as the flurry of recent AI products in Finance has shown.
  • Codifiable standards. Finance has published professional modeling standards (FMI, ICAEW), which allowed us to ground the rubric in practitioners' criteria rather than our own taste.
  • High stakes. Errors in these models carry real costs, so the gap between a plausible looking, but flawed workbook and an acceptable one matters most here.
  • Transferable criteria. Many of the 129 criteria are general spreadsheet good practice rather than finance-specific, such as a clear structure, no formula errors, reasonable file size, and consistent formatting.

That said, we recognize it’s also important to evaluate agents on simpler spreadsheet usage. We consider simpler spreadsheet manipulation to be already covered by SpreadsheetBench [1], and the easier spreadsheet construction task covered by the Easy and Very-Easy segments of MBABench [2]. There, agents already perform relatively well. Our benchmark therefore targets the harder end of the spectrum, where current agents still fall short.

[1] Ma et al., SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation. NeurIPS 2024 Datasets and Benchmarks.

[2] Yen et al., MBABench: Evaluating LLM Agents on End-to-End Spreadsheet Tasks in Finance. The Fortieth Annual Conference on Neural Information Processing Systems Evaluations and Datasets Track

Dear Reviewer w88T, we would like to thank you for your time reviewing the paper and are very encouraged by your positive outlook of the work.

If any concerns remain, we'd be happy to provide more clarifications or perform additional analysis within the discussion period.

Thank you for your time and effort reviewing the paper!

Reviewer_uYW5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SpreadsheetSmith is a benchmark of 101 expert-authored financial spreadsheet-from-scratch construction tasks, graded via expert-authored rubrics and a LLM judge. The authors write that their benchmark covers more task-types than existing spreadsheet benchmarks. They evaluate many model & harness configurations (including GUI-based usage and popular Excel AI products) and report agent failure modes that include: fixed horizon builds that must be manually extended, sheets not set up for reading, sense checks that say nothing about plausibility, unguarded inputs, inconsistent legends and finance display rules, and incorporating the logic into long single-cell formulas instead of helper rows.

Critical strengths

The benchmark is sizable at 101 tasks covering many financial modeling scenarios, and it targets a meaningful challenge for agents: constructing usable spreadsheets.

Rubrics evaluate many key factors of agent outputs such as accuracy, maintainability, documentation, adaptability, and error checking.

The authors study many models and commercially-relevant harnesses, and properly validate that their LLM-judge is a reasonable grader.

Critical weaknesses

I don’t see enough novelty in this work for a top venue like ICLR. The authors acknowledge there are several spreadsheet benchmarks with similar styles of rubrics and an agentic grader. What exactly is the main differentiated contribution of this work?

The authors write: “ most existing spreadsheet benchmarks evaluate agents on fill-in-the-blank style tasks, where an agent performs minor edits in otherwise complete spreadsheets”

But I have seen many spreadsheet benchmarks that go far beyond this including the generation of complex spreadsheets from scratch, which the authors should compare/contrast their paper against: FrontierFinance (Zhang et al.); Finch / FinWorkBench (Dong et al.)

The whole benchmark was built by four experts, which given how complex the rubrics are, indicates there may not have been sufficient data-quality review (the details of that process are a bit sparse in Appendix C, for instance how were rubrics reviewed).

There doesn’t appear to any validation of the pass-rate / rubric scores against expert ratings of whether a particular agent output is acceptable or not overall.

When validating the Judge in Sec 5.2, it would be good to know the inter-rater agreement rate when different experts grade agent rollouts under the same criteria.

More detail about the in-house harness should be provided. What are the design principles of this harness, why not use e.g. OpenCode or Terminus instead?

Questions for authors

What are the implications of models performing well in your benchmark vs. other existing benchmarks? Are the model rankings different for your benchmark vs. other existing benchmarks or what new insights does your benchmark facilitate? I.e. what model capabilities does your benchmark test beyond existing benchmarks or what construct does the benchmark measure?

How does your grader handle scenarios where the agent made up / hallucinated some extra information in the spreadsheet, which is not a field that is checked by any existing rubric criterion?

In converting rubric scores to pass rates, why do you set k_4 = 2 out of 24 not absolutely crucial? Shouldn’t the agent be allowed to fail more of the not crucial criteria than the highly-important criteria? More generally, the pass-rate thresholds seems a bit arbitrary and I’d like to hear about your reasoning behind them.

How are the error bars / confidence intervals computed? Especially given you only did one rollout per task.

Why do some of your task prompts (eg. task_template_shared_v9) include statements to the agent telling it how it will be graded (which does not reflect realistic usage)?

You will be graded against the rubric below: 12 weighted categories, 132 checks. Build so that every check meets its "Good" standard and avoids its "Bad" standard. Category weights (share of total score): Accuracy 19%, Structure 14%, Error Checks 13%, Formulas 12%, Formatting 9.5%, Potential Dangers 8%, Assumptions 7%, Flexibility 6.5%, Documentation 4%, Model Outputs & Executive Summary 4%, Purpose & Scope 2%, Rounding 1%.

Why don’t the authors report per-agent breakdown of costs and tokens used per run?

Will you be publicly releasing the rollout traces of these agent runs?

4 comments

General Response

We sincerely thank the reviewers for their thoughtful feedback. We are grateful that all three reviewers appreciate our extensive validation of the judge (uYW5, w88T, ndEy), value the scale and originality of our 101 financial modeling tasks (uYW5, w88T, ndEy), and appreciate the breadth of the evaluation across 20+ agents, including commercial Excel AI products (uYW5, w88T, ndEy). We are particularly glad that the reviewers recognize that the rubric goes beyond numerical accuracy to evaluate readability, auditability, and maintainability of workbooks (uYW5, w88T, ndEy), since this is exactly how a finance professional’s work would be evaluated in practice, and a key dimension in which our benchmark differentiates from the prevalent evaluation paradigm. The main shared concerns of the reviewers are on further clarification of our settings and novelty relative to existing spreadsheet benchmarks (uYW5, w88T, ndEy). We are thankful for this opportunity to provide further clarifications and experimental results, and are hopeful that we can satisfactorily address them and other concerns in this round.

On settings and contributions relative to existing benchmarks.

Our central contributions are (a) the frontier setting where agents are tasked to construct spreadsheets from scratch (b) one of the most comprehensive and difficult tasks set in financial modeling, (c) and a carefully designed set of rubrics that are generally applicable.

  • Setting: The agent receives only an input workbook and instructions and must build the complete model from scratch. This is a common theme in frontier benchmarks where agents are increasingly tasked with long-horizon, difficult tasks only expected of professionals (e.g. GDPval [1], ProgramBench [2], Terminal-Bench-Science [3]). We agree with Reviewer w88T that these tasks are much more complex than common workflows, but we view this as an important evaluation choice to understand frontier agent capability. This differentiates us from typical simpler benchmarks on finance and/or spreadsheets. We thank Reviewer uYW5 for raising a few related work that serve as clarifying examples:
    • FrontierFinance (Zhang et al.) is a typical QA benchmark. The tasks therein involve finding and analyzing information from existing documents, but distinctly do not evaluate agent’s capability to construct spreadsheets.
    • Finch (Dong et al.) evaluates based on workflows consisting of retrieval and simple spreadsheet edits (e.g. Example 1, review several sheets, retrieve info, and edit one main sheet with appropriate facts). Only 15 of its tasks involve spreadsheets, and these tasks are prototypical cases where a near-complete spreadsheet is already provided, and the agent is only tasked with minor edits on top, but is not expected to build models from scratch.
  • Task Set Size, Difficulty and Breath: We have created 101 bespoke modeling tasks, which is uniquely large in scale for agentic evaluation (e.g. compared to 50 in BTB, 4 in GDPVal, 38 public in MBABench). Besides sheer scale, the benchmark also covers most financial modeling use cases (in stark contrast to BTB and GDPVal, see Figure 4), and focuses on much more difficult tasks than MBABench.
  • Evaluation: we have carefully designed the rubric to extensively cover aspects of good modeling practices (129 rubrics vs. 17 in MBABench) and to be generally applicable (e.g. we avoid overly task-specific rubrics such as Multiples are formatted with suffix ‘x’ , as in BTB). Our rubrics therefore provides a comprehensive taxonomy for good modeling practices, and its general applicability also allows us to develop and validate the judge harness for each criteria.

The aspects above situate the work substantially different from existing benchmarks, and we appreciate the chance to clarify the uniqueness of our contribution. The reviewers have raised other points regarding pass rule and judge validation, and we have performed additional annotation and experiments (e.g. we verify that pass rule aligns with expert judgement, and provide statistics on inter-rater agreement. See response to reviewers uYW5 and ndEy).

We believe these clarifications and additional experiments have strengthened the paper, and will include them in the camera-ready version with the extra page provided. We thank the reviewers for the discussion that led to these improvements.

[1] Patwardhan et al., GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks. arXiv:2510.04374

[2] Yang et al., ProgramBench: Can Language Models Rebuild Programs From Scratch? arXiv: 2605.03546

[3] Terminal-Bench-Science: Evaluating AI agents on research workflows across scientific domains. https://www.tbench.ai/news/terminal-bench-science-0-1

Individual Response to Reviewer uYW5

We thank the reviewer for the careful and detailed review, and are glad the reviewer appreciates the scale and quality in the task set, rubrics, and evaluation. We are particularly grateful to the reviewer for inspiring the discussion on related work that further differentiates the paper, and hope the general response sufficiently conveys the novelty and contribution of our work. In what follows, we hope to address your remaining concerns:

"... implications of models performing well in your benchmark vs. other existing benchmarks?"

The benchmark evaluates whether agents can construct, from scratch, workbooks that meet the quality standards that would be deemed client-ready. This requires not only correct outputs, but also qualitative attributes that ensure the workbook presents its analyses and conclusions in a way that is easily interpretable and can be readily audited (as reflected in the rubric). Compared to:

  • FrontierFinance, Finch, and many others, our benchmark focuses on the from scratch construction of spreadsheets instead of QA or minor edits to existing spreadsheets.
  • GDPval, BTB, MBABench, our benchmark is greater in number of tasks (e.g. 4 in GDPval) and coverage (e.g. BTB focuses only on investment banking, but ours covers private equity, FP&A, fixed income, etc.), and is much more difficult (e.g. MBABench has only 2 Hard tasks) to properly evaluate frontier agent capabilities.

"The whole benchmark was built by four experts ... for instance how were rubrics reviewed"

We agree with the reviewers that involving more experts would be ideal, but would also like to highlight that scaling experts with sufficient qualification is generally infeasible within our academic budget. To the extent possible, we have involved four more experts, with MBA or 3+ years of industry experience in the annotations and inter-rater agreement below.

Re. task creation, we verify (a) that each golden solution correctly implements its instructions, and (b) that the instructions are unambiguous descriptions of the tasks. For (a), each author revised their task over several rounds with AI assistance, and every draft changed during this process. A task was included only after at least two final AI review passes found no errors (Appendix C). For (b), our finance experts checked every instruction for alternative readings. We verify that agents can indeed arrive at the correct numerical solutions given the task description. At least one of the 20 agents solves 72 of the 101 tasks (6 of them by a single agent only), so these 72 tasks ipso facto contain sufficiently clear instructions. The remaining 29 invalid tasks went through an additional review where several AI models flag possible ambiguities, our experts checked each flag, and revised the tasks wherever a flag was confirmed.

Re. the rubric, we aim for criteria that (a) are grounded in established professional standards and (b) our agentic judge grades consistently. For (a), we collected the criteria from several financial modeling standards (e.g., FMI, ICAEW), removed duplicates, kept only those relevant to the benchmark tasks, and ranked them by priority. Each criterion was then specified with a name, a description, and characterizations of good and bad practice (Appendix A.1). For (b), our finance experts and technical team refined the criteria description over several rounds so that the criteria is not misread by the agentic judge (see Appendix A.1). This clarity of rubric is reflected in the judge consistency results in Section 5.1, and in expert-agreement results in Section 5.2 (see also Appendices A.3 and E.2).

”validation of the pass-rate / rubric scores …

Since pass-rate and rubric scores are downstream quantities of the pass/fail decision on individual rubric, we view the judge validation in Sec. 5 as the central validation for the benchmark. That said, we agree with the reviewer that a separate expert rating would help contextualize the headline numbers. We have since annotated 50 attempts with experts’ binary decisions on whether the workbooks are client-ready, and found 48 decisions out of 50 agrees with the pass-rate decision from the agentic judge. This shows the pass rule indeed reflects experts’ opinions on workbook readiness.

inter-rater agreement…

We thank the reviewer for this suggestion, and have performed additional annotations for inter-rater agreement. Eight agent-judge-graded attempts were reviewed by two different human experts (multiple pairs). Each reviewer labeled every check as one of judge right to flag, judge wrong to flag, judge right to pass, judge wrong to pass. Across the 921 aggregate applicable rubric criteria for those eight attempts, the human reviewers agreed on 98.7% of labels. Agreement on individual attempts ranged from 96.4% to 100%, and six of the eight attempts reached at least 99%.

"More detail about the in-house harness ... why not use e.g. OpenCode or Terminus instead?"

We view general terminal agents such as OpenCode or Terminus as similar harnesses to Claude Code and Codex harnesses that support general tasks via terminal and code. Due to the already financially prohibitive scale of the evaluation, we decided against including similar harnesses. Instead, our in-house harness requires all computation to happen inside the spreadsheet. It exposes 21 spreadsheet-native tools over MCP. These include, but are not limited to, reading, writing and formatting a cell and creating or deleting a worksheet. They are implemented with openpyxl and LibreOffice as the recalculation engine. We will add these design principles to the appendices.

"How does your grader handle scenarios where the agent made up / hallucinated some extra information ..."

Undocumented or fabricated inputs violate several criteria regardless of where they appear. Such hallucinations would violate specifying rubric criteria such as centralized assumptions, clear labeling and references for all assumptions, and no hardcoded inputs (outside of the centralized assumptions). Fabricated values that feed the requested outputs are also caught by the deterministic accuracy check.

"How are the error bars / confidence intervals computed? Especially given you only did one rollout per task." The error bars are ±1 standard error across the 101 tasks. We agree with the reviewer that run-to-run variance is important to characterize. To that end, we additionally ran GPT-6 Astra three times on 30 tasks in both the Codex and in-house harnesses (with the 30 tasks selected to reflect difficulty distribution of the benchmark). The mean score was 89.9, 90.3 and 90.6 for Codex and 81.6, 80.0 and 80.6 in-house. The average standard deviation of a task’s score is 2.7 points for Codex and 4.0 for in-house, similar in scale to the small standard deviation of 3.1 from regrading the same workbook.

"Why do some of your task prompts (eg. task_template_shared_v9) include statements to the agent telling it how it will be graded ..."

We thank the reviewer for raising a fine point of the prompt design. We envision the setting where an agent is given the general description of the criteria it should be aware of, reflecting the realistic usage in practice where individual companies have their own requirements in a client-ready workbook. We also note that making the requirement clear is a common design choice in agentic benchmark [1].

"Why don't the authors report per-agent breakdown of costs and tokens used per run?"

We now report per-agent token usage for all API agents in https://anonymous.4open.science/r/SpreadSheetSmith_Rebuttal_Data-8D24/. We report tokens rather than dollar costs because token counts come directly from the API, whereas costs depend on caching and pricing choices that were not consistent across runs. Token usage is not observable for the GUI products, which run on consumer subscriptions.

"Will you be publicly releasing the rollout traces of these agent runs?"

Yes. We will release all rollout traces, output workbooks, and per-criterion judge decisions with rationales.

[1] Merrill et. al, Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces, https://arxiv.org/abs/2601.11868

Dear Reviewer uYW5, as the end of the discussion comes near, we would greatly appreciate the chance to discuss further and hopefully address your concerns.

We are grateful for the time you've put in reviewing the paper and your thoughtful questions on the uniqueness and implications of the benchmark. Overall, we believe the work is a meaningful and novel contribution in its task setup, scale, difficulty, coverage, and rubric design (as covered in more details in the General Response), and we illustrate this with the two related works you have kindly pointed us to.

If anything remains unclear, we would be more than happy to clarify or run additional analyses within the discussion period. Again, we would like to thank you for your time and effort in reviewing the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.