acceptodds
Under review as a conference paper at ICLR 2027

XLGym: API-Grounded Task Synthesis for Training and Evaluating Spreadsheet Agents

Abstract

Spreadsheet agents act on tables, charts, PivotTables and conditional formats, yet existing benchmarks are dominated by formula, computation and domain-oriented queries, which frontier models are getting increasingly good at. We introduce XLGym, a pipeline that turns Excel API documentation and real workbooks into hard tasks with reference solutions. Each task is grounded in a specific API member, verified by execution in Excel, and hardened until frontier models no longer solve it reliably. Because tasks are seeded per API member and per artifact, coverage and difficulty are set by construction. With XLGym we build XLBench, a 190-task benchmark that covers all eight major artifact types and gives every frontier model its lowest score among the benchmarks we evaluate (Opus 5: 52.8%, GPT-5.6: 35.4%), and XLTrain, a 20.6k-task corpus spanning 1,287 API members. Fine-tuning Qwen3.6-27B on XLTrain yields XLSage-27B, which raises the pass rate from 1.8% to 31.9% on XLBench, from 3.1% to 31.6% on WTM-Bench and from 2.5% to 22.2% on SpreadsheetBench. Because XLGym can regenerate tasks against newer models, XLBench can continually evolve.

Primary area:
datasets and benchmarks
Keywords:
Spreadsheet, Agent, LLM, Data, Training, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ibwsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

this paper introduces XLGym, a task synthesis pipeline that pairs 1287 office.js APIs with real workbooks and run two feedback looks inside Live Excel. From this pipeline, XLBench is built with 190 tasks; and XLTrain w/ over 20k tasks.

Paper is well-written, and they also self-distill Qwen3.6-27B into XLSage-27B avoids out-of-domain forgetting is an interesting emperical contribution.

Since XLGym can regenerate tasks against newer models, XLBench can continually evolve to avoid overfitting on benchmarks.

Critical strengths
  1. nice comparison of openpyxl vs. office.js

  2. showing standard off-policy SFT on GPT-5.6 degrades out-of-domain SpreadsheetBench (11.6 -> 7) when data scales while teacher model scales (16.54 -> 22.19) is very interesting

Critical weaknesses
  1. looks like XRBench and XLTrain share some same files? Backward_EnronFuse.1688-File.xlsx ?

  2. are there any dropped tasks? explain datasets/xlbench/excluded_tasks.jsonl

  3. input workbook summary still uses openpyxl

Questions for authors
  1. clean vs. leaked workbook breakdown, report pass rates with content_leaked = false?

  2. how do xlbench numbers change w/ native_gold_diff

  3. what is human-judge agreement (Cohen's kappa)

  4. what's the failure rate of Qwen3.6-27b's teacher and did you filter out failing rewrites before training?

3 comments

i would also like to know what human-judge agreement would be. i appreciate that the authors explicitly listed lack of human review of the pipeline as an explicit limitation, but this is imo quite a problematic limitation for the contribution type, and constitutes a major weakness, especially when there is no mention of dataset release

Thanks for your feedback.

Data Release

We will be releasing the benchmark and training corpus (on huggingface), along with the framework code (on github), so people can train and benchmark using our data and generate more data for their workbooks and APIs.

This was always the plan. We will make this clearer in the camera ready along with links for the data, code.

Human Alignment

We have started the human alignment study, and hope to complete it soon, we will be adding it to the revised paper.

In the meantime, even without human validation, the paper gives evidence that our design supports learning. WTM-Bench and SpreadsheetBench are graded by deterministic checkers with no LLM involved (L416–418), and a model trained using XLGym improves on both (3.1% to 31.6% and 2.5% to 22.2%, Table 2).

Supplementary material - task overlap

We have also conducted an internal investigation for possible leakage. What we found is:

  1. we split out the benchmark as a held-out set as we scaled training data from 2.5k --> 10.5k --> 22.5k
  2. at each new training data stage, we checked wether the previous benchmark has overlap with the newly synthesized training data. we dropped such tasks from the benchmark, and synthesized new ones.
  3. the final numbers are all with the last benchmark that has no overlap.

The issue in the supplementary material is due to wrong summarising by the coding agent when compiling all the data for supplementary material relase. We will correct this for the public release.

Qwen teacher failure rate

The teacher's error rate is 16.8%. The numbers reported in the paper are without rewrite, so we reject the failed samples. This effectively reduces the training dataset size in exchange for a 'cleaner' dataset. We will highlight this in the camera ready.


I hope this can help with some of your concerns and I'd be grateful if you would consider improving the score. Also happy to adress any other questions you might have.

We will be releasing the benchmark and training corpus (on huggingface), along with the framework code (on github), so people can train and benchmark using our data and generate more data for their workbooks and APIs.

This was always the plan. We will make this clearer in the camera ready along with links for the data, code.

We have started the human alignment study, and hope to complete it soon, we will be adding it to the revised paper.

In the meantime, even without human validation, the paper gives evidence that our design supports learning. WTM-Bench and SpreadsheetBench are graded by deterministic checkers with no LLM involved (L416–418), and a model trained using XLGym improves on both (3.1% to 31.6% and 2.5% to 22.2%, Table 2).

Reviewer_sPKJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes XLGym, a grounded synthesis pipeline that scalable generates spreadsheet tasks from Office.js documentation and real workbooks, with execution-based validation and difficulty-base evolution. It also propose XLBench for diverse spreadsheet capability evaluation, and use XLTrain for training. Fine-tuned Qwen3.6-27B gain great improvements on multiple benchmarks.

Critical strengths
  • The data synthesis pipeline is detailed and easy-to-follow. It include API-grounded task generation, execution-based quality checks, and difficulty-based task evolution to produce high-quality data.
  • XLBench cover the limitation from existing spradsheet benchamrk, which provides more diverse artifacts beyond formulars and basic operations.
  • Experiments shows after SFT on XLTRAIN, model improve the spreadsheet capability across multiple benchmarks. And the self-distilled fine-tuning further improve the gain.
Critical weaknesses
  • The data source of workbook is mainly from two old source released in 2015. And the number of workbook is highly limited compared to the task.
  • The benchmark WTM-Bench shared the same source of workbook used for training data synethesis.
  • The LLM-generated oracle validation is insufficient. Both reference code and expected answer are generated together, and assessed by yet another LLM with executed state. The quality verifier only ensure the consistency but not correctness. And there is not human validation, and no rule-based verifier etc. to further ensure quality.
  • The setup adaption of single-turn harness needs justification. All the data and benchmark run in single-turn setup, while it still shows the model capability, it limits the native agentic capability, especially the potential multi-step agentic loop, which is more important at this time.
Questions for authors

NA

1 comment

We thank the reviewer for the careful review, and recognising the strengths of the paper. We address the folowing points:

1. Workbook sources are from 2015 and few relative to the tasks

Age. A workbook's role is to supply real data and native artifacts for a task to act on (L151–154), not to represent what users ask in 2026. A PivotTable or filter in a 2015 workbook is the same Excel object as one made today. Enron [1] and FUSE [2] is one such large public corpora. The framework is modular to input corpora. Existing benchmarks follow the frequency of real requests and rarely reach the long tail of operations each artifact supports (L042–046, L111–115). XLGym sets that coverage by construction, not by how often an artifact appears in user data (L244–245).

Size of Data. The unit of variety in XLGym is the [API member, workbook] pair, not the workbook (Table 1, L241–251). The corpus uses 853 workbooks and 1,287 API members, giving over 100k artifact-matched pairs. We drew 46,615 seeds; for five of the eight families generation stopped at a budget of about 3.5k kept tasks, not for lack of seeds. A workbook hosts 24.9 tasks on average, each exercising a different attribute. SheetCopilot builds 221 tasks on 28 workbooks [4]. Newer workbooks can be added as-is.

2. WTM-Bench shares the workbook source

WTM-Bench reverse-engineers workbooks (L105–106): the corpus workbook is the target output, and the input is that workbook with its artifacts stripped [3]. XLGym works forward (L151–161): the workbook, artifacts intact, is the input. A WTM-Bench task therefore never appears in XLTrain.

Transfer also does not rest on WTM-Bench: XLSage-27B improves on SpreadsheetBench [5] (2.5% to 22.2%) and Spreadsheet-RL [6] (5.2% to 16.8%) (Table 2), built on forum workbooks unrelated to Enron and FUSE. We will state the shared source in the paper.

3. The verifier checks consistency, not correctness; no human or rule-based check

The query verifier checks both (L156–161). After the reference program runs in Excel, we extract the workbook state before and after the run (values, formulas, formats, object metadata) and the printed output, and the verifier checks that this evidence satisfies the query q and, separately, that it matches the expected answer a*. A reference that is consistent with its own expected answer but does not do what the query asks is rejected. §3.2 (L230–233), Figure 3 list what this catches in practice.

On rule-based checks and Human Validation Our Limitations note the lack of human validation (L463–465). Execution in Excel is itself deterministic and rejects every program that has incorrect syntax (L231), the state extraction is also deterministic, only the final judgement is made by a model. The paper also shows evidence that even without human validation, tasks synthesized this way are useful for learning. WTM-Bench and SpreadsheetBench are graded by deterministic checkers with no LLM involved (L416–418), and a model trained only on XLGym references improves on both (3.1% to 31.6% and 2.5% to 22.2%, Table 2).

On the same model in several roles. This is established practice (e.g. Self-Refine [7] uses one LLM to generate, critique and refine). Each role acts on different inputs: the verifier sees Excel's output, not the generator's code (L158–160), and the solver sees only the query and workbook (L216–217).

4. Justification for the single-step harness

We have justified this in the paper:

  • It isolates the capability studied (L308–309): what the model knows before an error message can correct it. In the multi-step harness, Office.js trajectories hit an execution error 2–3× as often as Python ones and recover from many (Table 5, L800–804), masking this knowledge gap.
  • It preserves the ranking (L309–311). For the two frontier models run in both harnesses, the ranking is identical on all three external benchmarks, with multi-step scores 11–22 points higher (Table 4).
  • Training on single programs transfers to the multi-step agentic loop (§5.1, L419–426; Figure 8). XLSage-27B is trained on single programs only, yet its multi-step pass rate rises from 19.3% to 36.5% on XLBench and from 29.7% to 44.5% on WTM-Bench. SpreadsheetBench is the exception (§5.1).

We agree that multi-step use is where agents are deployed, and will move the harness comparison and multi-step gains into §4.


We hope this addresses the reviewer's concerns and would be grateful if they would consider revising the scores.

References

[1] Hermans, Murphy-Hill. Enron's Spreadsheets and Related Emails. ICSE 2015.

[2] Barik et al. FUSE: An Internet-Scale Corpus of Spreadsheets. MSR 2015.

[3] Uniyal et al. Back to the Future: A Workbook Time Machine. COLM 2026.

[4] Li et al. SheetCopilot. NeurIPS 2023.

[5] Ma et al. SpreadsheetBench. NeurIPS 2024.

[6] Chi et al. Spreadsheet-RL. arXiv:2605.22642.

[7] Madaan et al. Self-Refine. NeurIPS 2023.

Reviewer_YhzaOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors present a pipeline for spreadsheet task synthesis, as well as a training set and evaluation benchmark. They ground their tasks in spreadsheets from the Enron and Fuse corpora, and use LLMs to draft, revise, and evaluate tasks. Coverage of eight different Excel objects is tracked and compared across multiple comparison benchmarks. The authors train Qwen3.6-27b on their training set and find that their model improves in performance across comparison benchmarks as well as their own benchmark

Critical strengths
  • The authors point out and aim to quantify a real and seemingly consequential limitation of many existing related works in this space. The authors' work is built on Office.js, which is more expressive than openpyxl which many similar spreadsheet agents and benchmarks are built on and/or with. Office.js runs in actual Excel workbooks, whereas openpyxl runs "headless" without needing to install Excel and lacks full support for some Excel objects (though the authors' assertion that Python covers only "few native Excel artifacts") seems overstated. Though my licensing concerns stand, it is regardless valuable to understand what limitations in capability and coverage exist in previous openpyxl-based spreadsheet literature
  • Released code and datasets could possible be valuable, if usable (see licensing concerns)
Critical weaknesses
  • The manuscript is extremely difficult to read. While I completely understand that presentation in initial submissions can be less polished than ideal and can even lead to substantial ambiguities, in this case the presentation issues are severe and pervasive enough that it seems inappropriate to entertain acceptance. Notably, the claims and contributions themselves are difficult to understand and evaluate. Non-exhaustively:

    • Critical information is absent. What is a "task"? The authors define a task as "a triple (q, c*, a*): a natural-language query q, a reference Office.js program c* that carries out q, and an expected answer a* that describes the resulting workbook changes or printed output", but do not explicitly specify what kinds of queries or answers are in or out of scope for the pipeline/benchmark-- what kinds of languages are represented in the tasks? do all tasks ask models to manipulate a spreadsheet? do any requested edits call for deletion of data or an object?
    • Critical information is presented in a nonsensical order (e.g. the model(s) used throughout the synthesis pipeline is not named until step 3, when step 2 (which seems like two separate steps in practice) also uses an LLM
    • The paper uses non-standard terms like "API member" throughout the manuscript without appropriate definition
    • The paper mentions "all eight major artifact types" in the abstract without ever providing justification for their particular set of Excel objects as canonical or complete
    • Even in the first paragraph, there is an unsubstantiated, perhaps accidental, claim that research systems for spreadsheet tasks "have followed" proprietary spreadsheet agents. The cited works themselves seem to contradict the implicit claim in that one of the cited works for "research systems" predate the citations for proprietary spreadsheet agents.
  • Even if presentation were to be improved, the complete absence of human validation of the pipeline is problematic. The authors' pipeline uses seemingly (this is also unclear) the same LLM for generation, "query verification," solve attempts, and solution verification. Without separate human validation (at least of resulting tasks in the benchmark) it is difficult to make an assessment of task quality

Questions for authors

N/A, or see above

4 comments
AuthorsOpenReview ↗

Writing, and task verification

We thank the reviewer for the close reading, and for recognising that the Office.js-versus-openpyxl gap is a real limitation of prior work. We answer each point below.

1. Writing

a. What is a task, and what is in scope

A task is defined formally in §3.1 (L147–149). Scope is fixed by construction: every task starts from one Office.js API member and one real workbook that contains the artifact it acts on (§3, L141–142; §3.1 step 1, L151–154), so every query is an Excel request about one of the eight artifact families of Table 1. On the specific questions:

  • Language. All queries are in English.
  • Do all tasks manipulate the workbook? No, by design: queries ask the model to create, modify or inspect an artifact (§4, L313; Figure 4). 66% of XLBench tasks are read-only, 26% create an artifact and 8% modify an existing one. XLTrain has a similar mix (about 65%, 29% and 6%). Read queries dominate because the pass-rate gate keeps the queries/APIs the solver is weakest on.
  • Deletion. Yes. Requests to remove or clear existing content or objects are included, within the modify tasks.

This matches the benchmarks we compare against: SheetCopilot [1] scopes its tasks by six operation categories, and SpreadsheetBench [2] and Spreadsheet-RL [3] define a task as an instruction with an initial and an oracle workbook.

b. Where the model is named

XLGym is model-agnostic: generator, solver and verifiers are roles that any LLM can fill, and the paper's claim that tasks can be regenerated against newer models (L483–484) rests on this. Hence the generic wording ("a frontier LLM, the generator", L079–080); GraphForge [4] likewise names its single model only in its experimental setup (§4.2).

We agree, and will open §3.1 by naming GPT-5.6 Reasoning Sol as generator, solver, query verifier and solution verifier. Using one model in several roles is established practice: Self-Refine [5] uses a single LLM as generator, feedback provider and refiner, and Self-Debug [6] has the model repair its own code from execution feedback. Here each role runs in a separate context with its own prompt and inputs.

c. "API member"

"Member" is the standard term for the methods and properties of a class, and it is the term the Office.js reference itself uses [7].

d. The eight artifact types

The Introduction (L038–042) motivates the eight families as the native objects real workbooks hold alongside formulas: structured tables, charts, PivotTables, conditional formats, filters, sort states and hyperlinks, as documented in the corpus study we cite [8].

e. "Have followed" (L035–036)

Thank you; this was loose chronological wording. We will revise the sentence.

2. Task Verification, LLM Bias

Yes, one model fills all four roles (§1b), and the submission states it has no human validation (L463–465) as a limitation. Two design properties limit LLM bias:

  1. Execution-grounded verification. Every reference program and solver attempt is executed in Excel (compile check) and the verifier is given the workbook state before and after the run (§3.1, L158–160, L220–221). The verifier therefore judges the output workbook, not code the solver/generator wrote. Grading the executed outcome rather than the program follows SheetCopilot [1] and Spreadsheet-RL [3].
  2. Solver isolation. The solver sees only the query and the input workbook (L216–217), not LLM generated code.

Even without human validation, the paper gives evidence that this design supports learning. WTM-Bench and SpreadsheetBench are graded by deterministic checkers with no LLM involved (L416–418), and a model trained only on XLGym references improves on both (3.1% to 31.6% and 2.5% to 22.2%, Table 2).


We hope these clarifications address the reviewer's concerns and would be grateful if the reviewer would consider revising the scores.

References

[1] H. Li, J. Su, Y. Chen, Q. Li, Z. Zhang. SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models. NeurIPS 2023.

[2] Z. Ma et al. SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation. NeurIPS 2024, Datasets and Benchmarks Track.

[3] B. Chi et al. Spreadsheet-RL: Advancing Large Language Model Agents on Realistic Spreadsheet Tasks via Reinforcement Learning. arXiv:2605.22642, 2026.

[4] Q. Su et al. GraphForge: Training Working Agents with Graph-Anchored Workspace Synthesis. arXiv:2609.38923, 2026.

[5] A. Madaan et al. Self-Refine: Iterative Refinement with Self-Feedback. NeurIPS 2023. arXiv:2303.17651.

[6] X. Chen, M. Lin, N. Schärli, D. Zhou. Teaching Large Language Models to Self-Debug. arXiv:2304.05128, 2023.

[7] Microsoft. Excel JavaScript API overview. Microsoft Learn, 2026. https://learn.microsoft.com/en-us/office/dev/add-ins/reference/overview/excel-add-ins-reference-overview

[8] M. Uniyal et al. Back to the Future: A Workbook Time Machine for Spreadsheet Creation Benchmarks. COLM 2026. arXiv:2608.07873.

AuthorsOpenReview ↗

Response to Ethical Concerns

We thank you for highlighting this. We have done a legal audit of the license, have taken appropriate permissions and the work does fall within purview.

Thank you for engaging. Out of respect for authors', reviewers', and AC's time, I will not be requesting further experiments or analyses-- my fundamental concerns have not been addressed, and a revision that fully addresses major concerns would constitute a different submission that warrants a fresh round of review.

I would like to point out that the rebuttal itself contains additional errors and improper comparisons to related work that further damage confidence in the submission's soundness.

"Member" [...] is the term the Office.js reference itself uses [7].

I won't dispute that a class "member" has a standard meaning in objected oriented programming, and that some office.js documentation does mention an "API member" [1], but 1) even so, it would aid readability if it were defined formally for an ML paper; additionally 2) the reference provided is outdated at best (it redirects to a different link) and not a does not mention a "member" anywhere on the page. The page's title does not match the citation's title, either. Regardless, 3) authors do not use the term "API member", even if it is a standard term with casual use, with enough precision to make it clear what it means in context, including in contexts that are relevant to methodology. Do method overloads count as separate members? Are enum values "members"? are api members version-agnostic? is the pipeline itself pinned to a version?

XLGym is model-agnostic: generator, solver and verifiers are roles that any LLM can fill, and the paper's claim that tasks can be regenerated against newer models (L483–484) rests on this.

claims that the pipeline is "model-agnostic" should be supported with empirical evidence. The authors seem to have tested only a single closed model. We have no evidence supporting whether the pipeline and results should hold with an open model, a smaller model, a cheaper model, etc.

Self-Refine

Self-Refine includes human evaluations to support claims that its method results in better model generations than its comparisons. In any case, using a single model for generation, verification, solve attempts, and solve verification is not inherently problematic if the claimed contributions are still validated, but as it is we don't have much of a sense of the current method's limitations or model-specific blind spots or biases. All we know is that training on this benchmark (created with this specific model) can increase performance on some easier benchmarks, and that the pipeline ensures "hardness" by construction. We don't know why exactly the training data is helpful, or what parts of the pipeline are essential or beneficial, or that there aren't issues with data quality, or that there aren't stylistic quirks introduced by the specific model used.

defined formally in 3.1

The snippet the rebuttal quotes from Section 4 ("artifacts it asks to create, modify or inspect, using an LLM that sees only the query text") should have been included in Section 3 and probably elaborated on.

Uniyal et al

The wording in the introduction and in the rebuttal don't make it clear that there are eight, and Uniyal et al themselves do not claim or establish that there are eight. Are formulas meant to be distinguished from the other types (see: "alongside" formulas)? Uniyal et al's Appendix A also mentions and explicitly drops Sparklines as additional artifact type. The current submission's authors might wish to explicitly include some sort of qualification or statement that they are loosely "following previous work" in considering a specific set of eight that is not well-covered by previous works-- but in any case this is just one instance of imprecise language that confuses readers.

Other than imprecision and ambiguity introduced by presentation issues, the manuscript stylistically is much closer to something seen in informal tech blogposts than appropriate for an academic research paper

As a follow-up, I just saw the authors' comment below-- addressing now:

I appreciate the authors’ stated plan to release the dataset and code, as well a human validation effort that is underway. This is welcome and will allow later third-party audits of task quality even if a human validation ends up being limited. However, a planned release is not available to reviewers now and does not substitute for validation in the paper, so it does not change my assessment.

[1] https://learn.microsoft.com/en-us/javascript/api/requirement-sets

Thank you for your comments, the paper can definitely benefit from a stylistic writing change to make it more academically inclined. We will be doing this, thanks for all your detailed feedback on individual writing points.

While the benchmark is available in supplementary material, here are the preliminary results from the human alignment study. The annotator team consists of people with >2 years experience with both Excel and OfficeJS. We acknowledge that these results are on less than < 20% of the data but hope they serve as an indicator for the quality of the dataset.

Annotation Questions

  1. Query ↔ workbook: Does the query refer to things that exist in the input workbook? (Yes / No)
  2. Query → gold: Do the gold workbook and expected answer satisfy the query? (Fully / Partly / No / Cannot verify)
  3. API ↔ workbook: Can the API member be meaningfully used on this workbook? (Yes / Partly / No)
  4. Query ↔ API: Would a reasonable solution need this member, or its family? (Needs this member / Needs the family only / Neither)

Results

Results

QuestionResponseShare
Query ↔ workbookYes100%
Query → goldFully96%
Partly4%
API ↔ workbookYes84%
Partly11%
No5%
Query ↔ APINeeds this member56%
Needs the family only34%
Neither10%

Thank you

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.