acceptodds
Under review as a conference paper at ICLR 2027

Spreadsheet-RL: Advancing Large Language Model Agents on Realistic Spreadsheet Tasks via Reinforcement Learning

Abstract

Spreadsheet systems such as Microsoft Excel and Google Sheets are central to modern data workflows, and AI agents that operate them are a promising research direction. Existing spreadsheet agents mostly prompt general-purpose LLMs, which works for simple operations but struggles with the multi-step workflows that dominate real-world use. We introduce Spreadsheet-RL, a reinforcement learning (RL) framework for training specialized spreadsheet agents in a realistic Microsoft Excel environment. It comprises an automated data agent that collects paired initial–final spreadsheets from online forums; Spreadsheet Gym, a multi-turn Excel environment with a spreadsheet-native tool harness and an asynchronous Excel-based verifier; and Domain-Spreadsheet, a new benchmark of 1,660 professional tasks covering finance, supply chain, human resources, sales, and real estate. On SpreadsheetBench, Spreadsheet-RL raises Qwen3-4B-Thinking-2507 from 12.0% to 23.4% Pass@1, with gains from both the spreadsheet-native harness and RL post-training; the gains hold at a larger model size on Qwen3-8B (16.8% to 22.3%) and generalize to Domain-Spreadsheet (8.4% to 17.2%). A human audit of 100 training and 100 Domain-Spreadsheet tasks finds majority-valid oracles for 84% and 98% of tasks, and trajectory analysis shows that RL mainly improves interaction reliability, cutting tool-execution errors from 48.9% to 7.4% of calls. These results highlight Spreadsheet-RL's strong potential for generalization and real-world adoption in spreadsheet automation, and broadly, its promise for advancing LLM-based interactions with data interfaces in everyday work.

Primary area:
foundation or frontier models, including LLMs
Keywords:
llm agent, rl, spreadsheet

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_scXgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Spreadsheet-RL combines automatically constructed initial/final workbook pairs, a spreadsheet-native interaction harness, an asynchronous Excel-backed verifier, and GRPO post-training. It also introduces the 1,660-task Domain-Spreadsheet benchmark.

Critical strengths

The environment and data pipeline address substantive obstacles to spreadsheet-agent training: persistent workbook state, formula recalculation, and scalable outcome verification. Constructing 5,928 training tasks and documenting an asynchronous Excel verifier constitutes a useful contribution beyond applying an RL objective to an existing text-only environment.

Critical weaknesses

The evaluation contract is inconsistent across the paper. Sections 3.3.1 and 5.1 describe formula- or structure-level checks, whereas Appendix A.11 specifies comparison of normalized, post-recalculation cell values only, including rounding numbers to two decimal places. Under the latter rule, a hard-coded value can match an oracle formula, and formatting-only requirements may not be represented by the score. The human audit establishes oracle validity and perceived metric suitability, not the evaluator’s error rate on model outputs, as the authors acknowledge. This uncertainty matters for interpreting the reported gains as successful workbook editing rather than value matching.

Questions for authors

In an audit of existing pre- and post-RL outputs, how often does the evaluator accept a workbook that violates the instruction, or reject a valid alternative? Does either error rate change after RL?

What checks exclude shared source threads, duplicate workbooks, and closely related template variants across training data, SpreadsheetBench, and Domain-Spreadsheet?

How many independent RL runs support the reported gains? What uncertainty is associated with the difference between the harness-only and harness-plus-RL conditions?

4 comments
AuthorsOpenReview ↗

Response to Reviewer scXg (1/2)

We thank Reviewer scXg for the detailed comments and for highlighting our paper's contributions in a realistic environment and data pipeline and scalable Excel-based verification. Below, we provide a detailed response to each concern.

W1 & Q1 (evaluation contract; evaluator errors on model outputs). Sections 3.3.1 and 5.1 and Appendix A.11 describe the same evaluator at different levels of detail: the main text summarizes it in one sentence, and the appendix specifies the value normalization. The revision unifies them into the following precise contract, which gives both the training reward and all evaluation scores. An output (the agent's final workbook for one test case) is accepted only if three checks pass on the oracle's answer region:

  1. values: recalculated, then normalized as in SpreadsheetBench (numbers rounded to two decimals and compared exactly);
  2. formula presence: where the oracle cell holds a formula, the output cell must also hold one (formulas are judged by their recalculated values, so equivalent formulas written differently pass);
  3. formatting: fill and font colors.

In the main-text summary, the “numeric tolerance” is the two-decimal rounding, the “formula-level check” is check 2, and structural edits are assessed through the answer-region values. Checks 2 and 3 make our evaluator stricter than SpreadsheetBench's official comparison, which scores values only. In the revision, Sections 3.3.1 and 5.1 and Appendix A.11 state this contract, and Appendix A.3 and the Limitations are aligned with it.

Hard-coded values. Check 2 rejects a plain value written where the oracle has a formula. Such outputs are rare and become rarer with RL: across the three new runs (Q3; new Appendix A.23), outputs that pass checks 1 and 3 but are rejected only for writing constants make up 0.06% of all SpreadsheetBench outputs before RL and 0.02% after.

Formatting-only requirements. Fill and font colors are scored in every task (check 3); other styles, such as bold or borders, are not. Tasks that would change only such unscored styles are removed during the construction of our training data and Domain-Spreadsheet, since their input already matches the target under this evaluator (Appendix A.1); SpreadsheetBench is used as released, so its scores stay comparable with prior work.

Evaluator errors on model outputs (Q1). In addition to the oracle audit in the paper, we audited model outputs (new Appendix A.23). From one new run (Q3), we randomly sampled 50 SpreadsheetBench outputs that the evaluator accepted and 50 that it rejected, at step 0 and at step 60. Three human auditors, blind to the checkpoint and to the evaluator's verdict, judged whether each output satisfies the instruction; unrequested edits outside the answer region count as violations.

Audited outputsStep 0 (before RL)Step 60 (after RL)95% CI of the difference
Accepted by the evaluator but violate the instruction (false accepts)7/50 (14%)2/50 (4%)−22.5 to +1.8 points
Rejected by the evaluator but satisfy the instruction (false rejects)2/50 (4%)2/50 (4%)−9.9 to +9.9 points

Neither error rate increases after RL: false accepts fall from 7/50 to 2/50, with a 95% CI of the change that rules out an increase of more than 1.8 points, and false rejects stay at 2/50.

  • Value matching in the reviewer's sense, a wrong formula whose values coincide with the oracle's on that test case's data, accounts for only four of the nine false accepts (3 before RL, 1 after); the other five reproduce an error in the SpreadsheetBench reference itself.
  • Three of the four false rejects differ from the reference only in letter case or spacing; in the fourth, the reference departs from the instruction.

Correcting this run's measured Pass@1 (19.4% to 23.9%) with both audited rates (accepted share × (1 − false-accept rate) + rejected share × false-reject rate) gives 19.9% to 26.0%, so the audit gives no indication that evaluator errors inflate the gain.

AuthorsOpenReview ↗

Response to Reviewer scXg (2/2)

Q2 (deduplication across datasets). Before building the training set, we removed 26 crawled threads that overlap with SpreadsheetBench and 188 near-duplicate threads within the crawl; the revision documents this step in Section 5.1 and Appendix A.1.

  • Training data vs. SpreadsheetBench. All crawled ExcelForum threads were compared with SpreadsheetBench by identical attachment files, overlap of distinctive cell values, shared word sequences in the task description, and first-post text similarity. The 26 removed threads are the 14 SpreadsheetBench source threads in our crawl (shared threads can only fall between 2024-01-01, the start of our crawl, and 2024-03-19, SpreadsheetBench's latest source post) and 12 near-duplicates of SpreadsheetBench tasks. None of the 5,928 training tasks comes from them. Because all crawled threads were compared, the content checks also cover a SpreadsheetBench workbook reposted in a later thread.
  • Within the training data. Threads with identical attachments or near-identical first posts were grouped, and only the earliest of each group was kept.
  • Template variants. These are covered from two sides: the cell-overlap check catches variants that share much of their content with a SpreadsheetBench workbook, and the task-description and first-post checks catch the same request posed on a different workbook.
  • Domain-Spreadsheet. Its seeds are domain concepts and professional templates from certification-aligned sources (Section 4), not forum threads, so it shares no source threads with the other two datasets.

Q3 (number of RL runs; uncertainty of the gain). Four independent RL runs support the main gain, and for each of the three new runs the 95% confidence interval of the before-vs-after-RL difference excludes zero (new Appendix A.22). We trained the main configuration (Qwen3-4B-Thinking-2507 with harness and tools) three more times with different seeds, which set the data order and rollout sampling (with deterministic kernels), and report every run at its final step 60, without checkpoint selection; the RL-only and Qwen3-8B configurations, which support secondary results, have one run each. The pre-RL condition (harness and tools) is step 0 of the same run (19.3% in Table 1 for the original run), so each gain is paired within a run. SpreadsheetBench Pass@1 (%):

RunStep 0 (before RL)Step 60 (after RL)Gain95% CI of gain
Original (paper)19.323.4+4.1–
Seed 119.423.9+4.5[+2.9, +6.2]
Seed 218.922.6+3.7[+2.3, +5.2]
Seed 319.224.6+5.4[+3.6, +7.1]
Mean (SD), 4 runs19.2 (0.2)23.6 (0.8)+4.4 (0.7)

The intervals use standard errors clustered by task (t distribution, 911 degrees of freedom), since the three test cases of each of SpreadsheetBench's 912 tasks differ in their workbook data and are not independent. The original run's per-test-case results were not retained, so it has no interval. On Domain-Spreadsheet, the gain is +8.8 points in the original run and +9.7, +11.6 and +14.0 in the three new runs (mean +11.0, SD 2.3).

Q4 (personal and confidential information; redistribution). We agree that the data must be screened at the workbook level before release, and our release process does so. Beyond the URL redaction in thread text (Ethics Statement), we have scanned all 5,928 training tasks at the workbook level. The scan covers the instruction and all text in the input and target workbooks that the model can read, including cell text, string constants in formulas, comments and their authors, headers and footers, document properties such as author and company, and sheet, table and column names. We use Presidio with spaCy named-entity recognition and pattern recognizers for contact details, addresses, and ID, card and bank numbers. Flagged items are mainly person names, most of them in document and comment author fields, plus e-mail addresses, phone numbers, street addresses and IBAN-like numbers in a few tasks; all personal and confidential information is removed or replaced before release. Domain-Spreadsheet does not use forum attachments (Q2), and its seed data contain no personal or confidential information. The revision describes this screening in the Ethics Statement.

For redistribution, we follow the practice of SpreadsheetBench, which also builds on forum workbooks (Appendices A.3 and A.4 of its paper): our crawler strictly respects ExcelForum's robots.txt, and we will release only the processed, screened tasks, not the raw crawled threads or attachments.

We hope these responses have addressed the reviewer's concerns, and we are happy to answer any further questions during the discussion period.

Thank you for the evaluator clarification, output audit, and additional training runs. These substantially address my concerns and strengthen the evidence for RL gains. The remaining issue is comparability with prior results under different evaluators. Please identify which comparisons use the same scoring rules and, where possible, report official values-only scores on the same outputs. Redistribution terms should also be documented as a data-release clarification, rather than a methodological weakness. I maintain 3 (weak acceptance), with increased confidence in the results. If there are subdivisions such as 3.5 or 7, I will consider raising the score.

AuthorsOpenReview ↗

Response to Reviewer scXg’s follow-up

We thank Reviewer scXg for the follow-up and are glad that the evaluator clarification, the output audit, and the additional training runs address the main concerns.

Comparability with prior results. Under SpreadsheetBench's official evaluator, which compares values only, the RL gain grows in every run, and the comparison with reported closed-source results does not depend on our stricter checks.

Which comparisons share scoring rules.

  • Our evaluator (values, formula presence, formatting; Section 3.3.1): all rows marked Ours in Table 1 except o3 with our harness and tools, i.e., the Qwen3 baselines, the base-to-RL progression of Qwen3-4B-Thinking-2507, RL-only, and Qwen3-8B, as well as the seed runs (Appendix A.22) and the output audit (Appendix A.23). Every before-vs-after-RL comparison therefore uses the same rules.
  • Official values-only evaluator: o3 with our harness and tools (52.2%) and the rescored runs below. The revision states this in the Table 1 caption and Appendix A.16.
  • Reported by their sources in their own environments: the other closed-source rows (GPT-4o, o3, ChatGPT agent, Claude Files Opus 4.1, Copilot Agent Mode), which we cannot rescore on the same outputs.

Official values-only scores on the same outputs. For the three runs that we newly trained, we rebuilt each output workbook by replaying the trajectory's tool calls on the test case's input workbook; the rebuilt outputs reproduce each run's scores under our evaluator exactly. We then scored all 2,726 test cases with the official evaluator. SpreadsheetBench Pass@1 (%):

RunStep 0 (before RL): ours / officialStep 60 (after RL): ours / officialGain: ours / official
Seed 119.4 / 20.223.9 / 25.5+4.5 / +5.3
Seed 218.9 / 19.622.6 / 23.8+3.7 / +4.2
Seed 319.2 / 19.924.6 / 26.4+5.4 / +6.5
Mean, 3 runs19.2 / 19.923.7 / 25.2+4.5 / +5.3

Because the official evaluator accepts every output that ours accepts, all scores rise. After RL, the 4B agent reaches 23.8–26.4% under the official rule, so its comparison with the reported o3 (23.3%) and GPT-4o (18.4%) results does not rest on our stricter checks.

Redistribution terms. We agree that these belong in the data-release documentation, and the release will document them.

We hope this response addresses the remaining concerns, and we are happy to answer any further questions during the discussion period.

Reviewer_N5DkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Spreadsheet-RL is a framework for training language-model agents to complete multi-step spreadsheet tasks by editing workbooks. It builds training tasks from public spreadsheet forums, pairing each initial workbook with an automatically generated target workbook. Its Spreadsheet Gym provides spreadsheet-specific tools and uses Microsoft Excel to recalculate workbooks and evaluate the agents’ final results. Using outcome-based reinforcement learning, the method raises a 4B model’s Pass@1 on SpreadsheetBench from 12.0% to 23.4%. It also improves performance on professional-domain tasks, although the remaining low success rates show that complex spreadsheet reasoning is still a major challenge.

Critical strengths

1.An important problem with practical relevance. The paper studies sequential edits to a workbook, rather than the generation of a single formula. This setting more closely reflects real spreadsheet work.

2.A complete training pipeline. Automated task construction, an isolated multi-turn environment, recalculation in Microsoft Excel, and outcome-based rewards work together to make reinforcement learning for spreadsheet agents feasible.

Critical weaknesses
  1. Noise in the training targets. In a human audit of 100 training tasks, only 84 oracle workbooks received a majority judgment of “valid,” while 12 tasks had no majority judgment.

  2. Limited evidence of generalization to professional workflows. Domain-Spreadsheet was constructed by the authors using a similar oracle-generation pipeline. It tests transfer to different task types, but may not adequately represent independently sourced, real-world enterprise workflows.

Questions for authors
  1. What exactly does the final reward check? How does it assess formulas, formatting, worksheet structure, and unintended edits outside the designated answer region? Please clarify the difference between the evaluation descriptions in the main text and the appendix.

  2. How stable are the reported gains? Could you report results across multiple training seeds and provide confidence intervals using the original tasks, rather than similar test-case variants, as the unit of analysis?

  3. How much do incorrect oracle workbooks affect RL training? If training tasks whose oracles were judged invalid or disputed were excluded, how would task accuracy and tool-use reliability change?

3 comments
AuthorsOpenReview ↗

Response to Reviewer N5Dk (1/2)

We thank Reviewer N5Dk for the constructive comments and for highlighting our paper's contributions in a practically important problem setting and a complete RL training pipeline. Below, we provide a detailed response to each concern.

W1 & Q3 (noise in the training targets). Our data agent constructs training tasks without human annotation, and the human audit shows that it already yields high-quality supervision: a strict majority of auditors judged 84% of sampled oracles valid and only 4% invalid, and the remaining 12 have no strict majority either way. Expert review of all 5,928 tasks would be prohibitively expensive (about 26,700 judgments under our protocol), especially for an open-source effort. Much as SWE-bench Verified (OpenAI, 2024) followed SWE-bench (Jimenez et al., 2024), we plan to develop a verified version, and the auditing platform we will release allows the community to contribute to it.

How much incorrect oracles affect training. GRPO's group-normalized advantages limit the effect of noise: a task whose oracle no rollout reproduces gives every rollout zero advantage and contributes no gradient from the reward. Consistent with this, on the 16 audited tasks whose oracles were judged invalid (4) or had no majority (12), the step-0 and step-60 checkpoints of three new runs of the main configuration (Q2 below) receive reward 0, so training did not lead the policy to reproduce a questionable oracle. Training on this data is also effective: all four runs improve Pass@1 on SpreadsheetBench, a previously published benchmark whose tasks and references were built independently of our pipeline (table under Q2 below).

What would change if such tasks were excluded. Based on these results, we expect task accuracy and tool-use reliability to change only marginally: the audited tasks without a majority-valid oracle were never reproduced at either end of training, so under GRPO they contribute little to the policy update, and excluding them would mostly remove tasks that the policy already does not learn from.

W2 (generalization to professional workflows). We would like to clarify the scope of “workflow” in our paper (Section 4): each Domain-Spreadsheet task requires carrying out a domain-specific analysis, such as comparable-company analysis, Value-at-Risk, or debt-service coverage in finance, within an Excel workbook, typically through a dependent chain of operations. These tasks can be viewed as a high-fidelity subset of real-world enterprise workflows: they are built from professional templates and certification-aligned material (e.g., CPA, CFA and FRM for finance, CPIM for supply chain, SHRM for human resources, CCIM for real estate), and in our human audit of 100 randomly sampled tasks, 98 oracles were judged valid by majority and none invalid. Independently sourced enterprise workbooks are rarely available to researchers: they are costly to obtain, few are licensed for release, and frontier labs evaluate on proprietary internal data (e.g., the investment-banking spreadsheet tasks used for ChatGPT Agent). Domain-Spreadsheet is, to our knowledge, the first open-source domain-specific spreadsheet benchmark, and it serves as a strong proxy for professional spreadsheet work:

  • It differs strongly from the training data. In an LLM-judge analysis (Appendix A.5), 88.9% of Domain-Spreadsheet tasks require both domain knowledge and a dependent chain of operations, versus 4.6% of training tasks, and the median instruction is 182 versus 32 words. RL improves the 4B agent on it in all four runs of the main configuration, from 8.0–8.7% to 17.2–22.0% (Q2 below).
  • Our headline result does not depend on it. On SpreadsheetBench, a previously published benchmark whose tasks and references were built independently of our pipeline, Pass@1 rises from 12.0% to 23.4%, and the RL part of this gain holds across seeds (Q2 below).

We will release Domain-Spreadsheet openly, together with the training data, environment, harness, training pipeline, trained models and auditing platform, so that the community can audit its tasks, extend it with externally sourced ones, and evaluate agents under the same environment and evaluator.


References

Jimenez et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

OpenAI (2024). Introducing SWE-bench Verified.

AuthorsOpenReview ↗

Response to Reviewer N5Dk (2/2)

Q1 (what the reward checks; main text vs. appendix). The reward is 1 if three checks pass on the oracle's answer region, and 0 otherwise; the same evaluator gives the training reward and all evaluation scores.

  1. Values. After recalculation in Excel, each cell's value must equal the oracle's, normalized exactly as in SpreadsheetBench: numbers are rounded to two decimals, text is compared exactly, dates to the day and times to the minute, and an empty string equals an empty cell.
  2. Formulas. Where the oracle cell contains a formula, the output cell must also contain one, so a hard-coded value fails. The formula is judged by its recalculated value, so an equivalent formula written differently passes.
  3. Formatting. Fill and font colors (RGB) must match the oracle's. Other styles (bold, borders, font name, size, etc.) are not scored.

Checks 2 and 3 make our evaluator stricter than SpreadsheetBench's official comparison, which scores values only. Worksheet structure is assessed through the answer-region values: inserted or deleted rows, columns or sheets must produce the oracle's values in the region, and a missing answer sheet gives reward 0. Edits outside the answer region are not scored, and they are rare and decrease with RL: across the SpreadsheetBench outputs of the three new runs (Q2 below; new Appendix A.23), outputs with such edits fall from 1.4% to 0.4%, and accepted outputs with such edits from 0.5% to 0.1%, as shares of all outputs. Tasks that change only unscored styles are removed during construction of our datasets, since their input already matches the target under this evaluator.

Main text vs. appendix. Both describe this same evaluator at different levels of detail: Sections 3.3.1 and 5.1 summarize it in one sentence, and Appendix A.11 specifies the value normalization. In the summary, the “numeric tolerance” is the two-decimal rounding, the “formula-level check” is the formula-presence check (formulas are scored by their recalculated values, not by their strings), and “structure-level” refers to structural edits assessed through the answer-region values. To make this fully precise, the revision replaces the one-sentence summary with the three-check description above and adds checks 2 and 3 to Appendix A.11, so that Sections 3.3.1 and 5.1 and Appendix A.11 describe the evaluator identically, and Appendix A.3 and the Limitations are aligned with it.

Q2 (stability across seeds; confidence intervals over tasks). The RL gain is stable across four runs, and for each of the three new runs its 95% confidence interval, computed with the task as the unit, excludes zero (new Appendix A.22). We trained the main configuration (Qwen3-4B-Thinking-2507 with harness and tools) three more times with different seeds, which set the data order and rollout sampling (with deterministic kernels), and report every run at its final step 60, without checkpoint selection. SpreadsheetBench Pass@1 (%):

RunStep 0 (before RL)Step 60 (after RL)Gain95% CI of gain
Original (paper)19.323.4+4.1–
Seed 119.423.9+4.5[+2.9, +6.2]
Seed 218.922.6+3.7[+2.3, +5.2]
Seed 319.224.6+5.4[+3.6, +7.1]
Mean (SD), 4 runs19.2 (0.2)23.6 (0.8)+4.4 (0.7)

As you suggested, the intervals treat the task, not its test cases, as the unit. SpreadsheetBench has 912 tasks, each with three test cases that differ in their workbook data (2,726 test cases evaluated; the remaining 10 are Excel-incompatible or prohibitively slow to recalculate, Section 5.1). Within each new run, we take the per-test-case change from step 0 to step 60, average it over test cases as Pass@1 does, and cluster standard errors by task (t interval). The original run's per-test-case results were not retained, so it has no interval.

The gains are also stable on Domain-Spreadsheet: the three new runs improve Pass@1 from 8.4%, 8.7% and 8.0% to 18.1%, 20.3% and 22.0% (original run: 8.4% to 17.2%), a mean gain of +11.0 points (SD 2.3) over the four runs. The harness-and-tools part of the headline result (12.0% to 19.3%) involves no training, and the step-0 column reproduces its endpoint in every new run (18.9–19.4%).

We hope these responses have addressed the reviewer's concerns, and we are happy to answer any further questions during the discussion period.

AuthorsOpenReview ↗

Follow-up to Reviewer N5Dk

We thank Reviewer N5Dk once again for the constructive suggestions on task-level confidence intervals across seeds, clarifying the exact reward checks, and evaluating oracle noise and professional workflows. With the end of the discussion period approaching, we wanted to briefly check whether our responses, additional seed runs, and unified evaluator specification have addressed the reviewer's questions. We are happy to provide any further clarification before the discussion window closes.

Reviewer_DLAeOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Spreadsheet-RL, an end-to-end reinforcement learning framework designed to train LLM agents for complex, multi-step spreadsheet manipulation tasks. To overcome the lack of training data and the challenges of persistent state mutation, the authors contribute: (1) an automated data collection agent that mines real-world Excel forum threads and uses coding agents to construct initial-final workbook pairs; (2) Spreadsheet Gym, a realistic Microsoft Excel environment featuring an asynchronous, Excel-backed verifier and a spreadsheet-native tool harness that enforces an inspect-modify-verify workflow; and (3) Domain-Spreadsheet, a new benchmark of 1,660 professional tasks. Using GRPO with outcome-based rewards, the framework significantly improves the performance of Qwen3 models on SpreadsheetBench and generalizes well to Domain-Spreadsheet. Trajectory analysis reveals that the RL primarily acts by reducing tool-execution errors and improving interaction reliability, rather than fundamentally upgrading the model's semantic spreadsheet reasoning.

Critical strengths

Comprehensive and realistic infrastructure: The development of Spreadsheet Gym with real Microsoft Excel execution and an asynchronous reward API solves a massive engineering bottleneck for this domain. Unlike synthetic table QA, this environment faithfully captures recalculation semantics, dynamic arrays, and file-state mutation, making the RL training highly grounded in reality.

Excellent diagnostic analysis of RL's impact: The paper goes beyond just reporting Pass@1 metrics. By breaking down the exact behavioral changes (Table 4 and Appendix A.17), the authors correctly identify that RL dramatically cuts down low-level protocol/execution errors (from 48.9% to 7.4%) but leaves semantic formula errors as the main residual failure mode. This provides highly actionable insights for future agent design.

Critical weaknesses

Noise in Automatically Constructed Oracles: The framework relies on coding agents to construct ground-truth final workbooks from forum threads. While the human audit is appreciated, the fact that only 84% of the sampled training tasks had a strict-majority valid oracle (with 4% majority invalid) means the RL reward signal is inherently noisy. Training on noisy rewards without explicit noise-robust RL mechanisms could lead to suboptimal policies or reward hacking.

Brittleness of the Exact-Match Evaluation Metric: The reward function relies on allcellsmatch(). Although it uses small tolerances and formula canonicalization, exact cell matching can penalize semantically correct solutions (e.g., a user using a slightly different but mathematically equivalent formula, or a different but valid layout). The paper acknowledges this limitation, but it constrains the ceiling of the agent's learned behavior.

Several important related studies are missing, particularly on sheet data manipulation with agentic RL, including PET-SQL and SQL-Bench. A comparison with these methods would make the current contribution stronger.

PET-SQL: A Prompt-Enhanced Two-Round Refinement of Text-to-SQL with Cross-Consistency
SQL-Bench: Benchmarking the text-to-sql capability of large language models: A comprehensive evaluation

Questions for authors

Since the training dataset contains roughly 16% of tasks that do not have a perfectly valid oracle, how does this noise affect the GRPO training dynamics? Did you consider using a stronger LLM-as-a-Judge to filter the final workbooks during the automated data generation phase, rather than relying purely on rule-based checks?

The paper notes that semantic reasoning (writing the wrong formula) remains the main bottleneck after RL. Given that RL successfully optimized the tool-use protocol, do you think a two-stage approach—first SFT on high-quality formulas, then RL on the harness protocol—might yield better semantic reasoning, or does RL inherently struggle with deep semantic spreadsheet logic without stronger pretraining?

Regarding the exact-match evaluation, how does your verifier handle dynamic arrays or spilled ranges where the agent's formula might output an array of different dimensions than the oracle, even if the overall logical result is equivalent?

3 comments
AuthorsOpenReview ↗

Response to Reviewer DLAe (1/2)

We thank Reviewer DLAe for the insightful comments and for highlighting our paper’s contributions in comprehensive and realistic infrastructure and excellent RL impact. Below, we provide a detailed response to each concern.

W1 & Q1 (oracle noise and its effect on GRPO). Our data agent constructs training tasks without human annotation, and the human audit shows that it already yields high-quality supervision: a strict majority of auditors judged 84% of sampled oracles valid and only 4% invalid. Expert review of all 5,928 tasks would be prohibitively expensive (about 26,700 judgments under our protocol), especially for an open-source effort. Much as SWE-bench Verified (OpenAI, 2024) followed SWE-bench (Jimenez et al., 2024), we plan to develop a verified version, and the auditing platform we will release allows the community to contribute to it. Oracle noise does not necessarily lead to suboptimal policies or reward hacking: RL on data that is not human-verified, with an imperfect rule-based reward, already trains strong agents, e.g., SWE-RL (Wei et al., 2025) learns from open-source software evolution data with a patch-similarity reward and reaches 41.0% on SWE-bench Verified, and our checks below find no reward hacking. Verified data can further improve the final model.

Effect on GRPO. GRPO's group-normalized advantages limit the effect of noise: a task whose oracle no rollout reproduces gives every rollout zero advantage and contributes no gradient from the reward. Consistent with this, on the 16 audited tasks without a majority-valid oracle (4 majority-invalid, 12 tied), the step-0 and step-60 checkpoints of all three new runs below receive reward 0, so training did not lead the policy to reproduce a questionable oracle.

Robustness of the gain. The RL gain holds across seeds (new Appendix A.22). We trained the main configuration (Qwen3-4B-Thinking-2507 with harness and tools) three more times with different seeds, which set the data order and rollout sampling (with deterministic kernels). We report every run at its final step 60, without checkpoint selection. All gains are measured on SpreadsheetBench, a previously published benchmark whose tasks and references were built independently of our pipeline, so they are not scored against our oracles. SpreadsheetBench Pass@1 (%):

RunStep 0 (before RL)Step 60 (after RL)Gain95% CI of gain
Original (paper)19.323.4+4.1–
Seed 119.423.9+4.5[+2.9, +6.2]
Seed 218.922.6+3.7[+2.3, +5.2]
Seed 319.224.6+5.4[+3.6, +7.1]
Mean (SD), 4 runs19.2 (0.2)23.6 (0.8)+4.4 (0.7)

Every run improves by +3.7 to +5.4 points, and each new run's 95% CI excludes zero. The CIs treat the task as the unit (standard errors clustered by task, since the three test cases of a task are correlated). The original run's per-test-case results were not retained, so it has no interval.

No reward hacking. In addition to the oracle audit in the paper, we audited model outputs (new Appendix A.23). From one new run, we randomly sampled 50 outputs that the evaluator accepted and 50 that it rejected, at step 0 and at step 60. Three auditors, blind to the checkpoint and to the evaluator's verdict, judged whether each output satisfies the instruction.

Audited outputsStep 0 (before RL)Step 60 (after RL)95% CI of the difference
Accepted by the evaluator but violate the instruction (false accepts)7/50 (14%)2/50 (4%)−22.5 to +1.8 points
Rejected by the evaluator but satisfy the instruction (false rejects)2/50 (4%)2/50 (4%)−9.9 to +9.9 points

False accepts fall from 7/50 to 2/50 after RL, and the 95% CI of the change rules out an increase of more than 1.8 points, so RL does not learn to exploit the evaluator.

LLM-based filtering. Our pipeline already uses a strong LLM agent, not only rule-based checks. Each oracle is built by a coding agent that reads the complete [SOLVED] thread, including the proposed solutions and follow-up clarifications. The agent discards threads it cannot turn into a well-posed task (e.g., toy workbooks, macro-dependent requests, or answers that cannot be localized to cells), executes its edits in the workbook, and verifies the result before emitting a sample (Appendix A.1). Rule-based checks and recalculation in Excel then filter further. We expect a separate judge pass over the final workbooks to add limited benefit: oracles that no rollout reproduces contribute no gradient, and none of the 16 audited tasks without a majority-valid oracle was reproduced at the start or end of training in any run.


References

Jimenez et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

OpenAI (2024). Introducing SWE-bench Verified.

Wei et al. (2025). SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution.

AuthorsOpenReview ↗

Response to Reviewer DLAe (2/2)

W2 (brittleness of exact matching). Exact matching is imperfect, but it gives RL a clear, reproducible, and verifiable reward signal, while building a checker that reliably recognizes equivalent spreadsheet solutions remains challenging. Our evaluator already handles the reviewer's first example: it compares the recalculated values in the answer region (and requires a formula wherever the oracle has one), so a mathematically equivalent formula written differently passes. We complement exact matching with a human audit of model outputs (W1 & Q1; new Appendix A.23), which shows that the reward rarely penalizes valid solutions: only 2 of 50 rejected outputs at each checkpoint satisfied the instruction, three of these four differ from the reference only in letter case or spacing, and none is an equivalent formula or an alternative layout. Because this rate is the same before and after RL, it leaves the before-vs-after comparison essentially unchanged. More broadly, a benchmark exists to compare agents fairly and reproducibly under one evaluator: we draw conclusions from such comparisons (e.g., before vs. after RL, or between models evaluated in the same environment), not from absolute Pass@1 levels.

W3 (PET-SQL and SQLBench). Thank you for the pointers; the revision cites both in Related Work (Section 2) as work on mapping natural language to executable operations over tabular data. PET-SQL improves text-to-SQL by prompting LLMs with two-round refinement and cross-consistency, and SQLBench benchmarks LLMs on text-to-SQL sub-tasks under different prompting strategies. Both work at inference time, without training the model. Spreadsheet-RL instead trains the model itself with RL, as a multi-turn agent that edits a persistent workbook through tool calls and is rewarded on the resulting workbook state. Since their tasks and metrics do not carry over to workbook editing, we discuss them as related work rather than as quantitative baselines; their prompt-level techniques are complementary to our model-side training.

Q2 (formula-level SFT before RL). Our results indicate that RL also improves semantic spreadsheet reasoning, not only the tool-use protocol. In the failure analysis (Appendix A.18), trajectories with a wrong formula, value, or output format fall from 63.3% to 55.1% after RL, alongside the larger drop in malformed tool calls or protocols (12.9% to 2.4%), and RL roughly doubles Pass@1 on Domain-Spreadsheet (8.4% to 17.2%), whose tasks require domain knowledge and long dependent chains of operations.

A two-stage recipe is a natural extension, and its benefit depends on the SFT data. Formula-only SFT is narrow. Existing formula data (SpreadsheetCoder, FLAME; see Related Work) consist of single formulas, alone or with a static table, and narrow fine-tuning risks degrading the multi-turn tool use and reasoning that RL builds on. SFT that helps the agent would need multi-turn Excel trajectories, which are costly to collect at scale. Spreadsheet-RL avoids this requirement: the spreadsheet-native harness gives RL a usable initial policy without any SFT warm-up. More broadly, Spreadsheet-RL targets spreadsheet-native, multi-step workflows rather than formula generation alone, and the environment, training data and verifiable labels we will release provide the infrastructure for future training techniques; as higher-quality data become available, SFT stages such as formula-level SFT can complement RL and further raise its ceiling on spreadsheet tasks.

Q3 (dynamic arrays and spills of different dimensions). Our evaluator compares the answer region cell by cell after recalculation in Excel, so it scores the values an array produces rather than its shape: a spilled formula and per-cell formulas that yield the same values in the answer region are scored alike. For arrays whose dimensions differ from the oracle's:

  • Larger: extra cells outside the answer region (or, for a whole-column region such as A:G, cells the oracle leaves empty) are not compared, so an equivalent result with extra rows or columns passes.
  • Smaller or offset: part of the answer region is empty or holds different values, so the output is rejected; the same holds for a blocked spill (#SPILL!).

The formula-presence check (W2) adds one case: when the oracle writes a formula in each cell and the agent fills the same values with a single spilled formula, the spilled cells hold values rather than formulas and the output is rejected. Our harness makes this case rare by filling formula targets with fill_formula, which writes a formula into every target cell (Appendix A.7), and none of the false rejects in our audit (W1 & Q1) arose from it. The reverse cases pass: per-cell formulas against a spilling oracle, and a spilled formula against a multi-cell array formula.

We hope these responses have addressed the reviewer's concerns, and we are happy to answer any further questions during the discussion period.

AuthorsOpenReview ↗

Follow-up to Reviewer DLAe

We thank Reviewer DLAe once again for the insightful feedback on oracle noise in GRPO training, exact-match and dynamic-array evaluation, and semantic spreadsheet reasoning. As the discussion period is coming to a close, we wanted to gently check whether our responses, together with the new multi-seed runs and blind audit of model outputs, have addressed the reviewer's concerns. We are happy to provide any further clarification before the discussion window closes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and the AC

We thank all reviewers and the AC for their time and constructive feedback. We are glad that the reviewers found the problem setting practically important (N5Dk), the environment, data pipeline, and infrastructure realistic and comprehensive (DLAe, scXg), the RL training pipeline complete and the Excel-based verification scalable (N5Dk, scXg), and the diagnostic analysis of RL's impact excellent (DLAe). Reviewer scXg also noted that our responses substantially address their concerns and strengthen the evidence for the RL gains. Below, we summarize the main additions; details are in the individual responses.

New evidence added during the discussion

  • Robustness across seeds (new Appendix A.22). Three additional training runs reproduce the RL gain: across four runs, SpreadsheetBench Pass@1 improves by +3.7 to +5.4 points, with the task-level 95% confidence interval of each new run's gain excluding zero, and Domain-Spreadsheet Pass@1 improves by +8.8 to +14.0 points.
  • Evaluator audit on model outputs (new Appendix A.23). In a blind human audit of 50 accepted and 50 rejected outputs at each checkpoint, false accepts fall from 7/50 to 2/50 after RL (the 95% CI of the change rules out an increase of more than 1.8 points), and false rejects stay at 2/50, so RL does not learn to exploit the evaluator.
  • Comparability across evaluators. Rescored with SpreadsheetBench's official evaluator, which compares values only, the RL gain grows in every new run (mean +5.3 versus +4.5 points), and the post-RL 4B agent reaches 23.8–26.4%, so comparisons with reported closed-source results do not rest on our stricter checks.
  • Training on imperfect oracles. On the 16 audited training tasks without a majority-valid oracle, all three new runs receive reward 0 at the start and end of training, so training did not lead the policy to reproduce a questionable oracle.

Revisions to the paper (marked in blue)

  • Evaluator. Sections 3.3.1 and 5.1 and Appendix A.11 state the same three-check evaluator (values, formula presence, formatting); the Table 1 caption and Appendix A.16 note that the o3 + harness row uses the official values-only evaluator.
  • Data. Section 5.1 and Appendix A.1 describe the deduplication against SpreadsheetBench and within the crawl (26 and 188 threads removed), and the Ethics Statement describes the workbook-level screening of all 5,928 training tasks for personal and confidential information.
  • Scope and related work. Section 4 defines each Domain-Spreadsheet task as a domain-specific analysis within an Excel workbook, Section 2 cites PET-SQL and SQLBench, and the Limitations list formula-level SFT as a direction that can complement RL.

We thank the reviewers and the AC again for their engagement during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.