- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
this paper introduces XLGym, a task synthesis pipeline that pairs 1287 office.js APIs with real workbooks and run two feedback looks inside Live Excel. From this pipeline, XLBench is built with 190 tasks; and XLTrain w/ over 20k tasks.
Paper is well-written, and they also self-distill Qwen3.6-27B into XLSage-27B avoids out-of-domain forgetting is an interesting emperical contribution.
Since XLGym can regenerate tasks against newer models, XLBench can continually evolve to avoid overfitting on benchmarks.
Critical strengths
-
nice comparison of openpyxl vs. office.js
-
showing standard off-policy SFT on GPT-5.6 degrades out-of-domain SpreadsheetBench (11.6 -> 7) when data scales while teacher model scales (16.54 -> 22.19) is very interesting
Critical weaknesses
-
looks like XRBench and XLTrain share some same files? Backward_EnronFuse.1688-File.xlsx ?
-
are there any dropped tasks? explain datasets/xlbench/excluded_tasks.jsonl
-
input workbook summary still uses openpyxl
Questions for authors
-
clean vs. leaked workbook breakdown, report pass rates with content_leaked = false?
-
how do xlbench numbers change w/ native_gold_diff
-
what is human-judge agreement (Cohen's kappa)
-
what's the failure rate of Qwen3.6-27b's teacher and did you filter out failing rewrites before training?
3 comments
Thanks for your feedback.
Data Release
We will be releasing the benchmark and training corpus (on huggingface), along with the framework code (on github), so people can train and benchmark using our data and generate more data for their workbooks and APIs.
This was always the plan. We will make this clearer in the camera ready along with links for the data, code.
Human Alignment
We have started the human alignment study, and hope to complete it soon, we will be adding it to the revised paper.
In the meantime, even without human validation, the paper gives evidence that our design supports learning. WTM-Bench and SpreadsheetBench are graded by deterministic checkers with no LLM involved (L416–418), and a model trained using XLGym improves on both (3.1% to 31.6% and 2.5% to 22.2%, Table 2).
Supplementary material - task overlap
We have also conducted an internal investigation for possible leakage. What we found is:
- we split out the benchmark as a held-out set as we scaled training data from 2.5k --> 10.5k --> 22.5k
- at each new training data stage, we checked wether the previous benchmark has overlap with the newly synthesized training data. we dropped such tasks from the benchmark, and synthesized new ones.
- the final numbers are all with the last benchmark that has no overlap.
The issue in the supplementary material is due to wrong summarising by the coding agent when compiling all the data for supplementary material relase. We will correct this for the public release.
Qwen teacher failure rate
The teacher's error rate is 16.8%. The numbers reported in the paper are without rewrite, so we reject the failed samples. This effectively reduces the training dataset size in exchange for a 'cleaner' dataset. We will highlight this in the camera ready.
I hope this can help with some of your concerns and I'd be grateful if you would consider improving the score. Also happy to adress any other questions you might have.
We will be releasing the benchmark and training corpus (on huggingface), along with the framework code (on github), so people can train and benchmark using our data and generate more data for their workbooks and APIs.
This was always the plan. We will make this clearer in the camera ready along with links for the data, code.
We have started the human alignment study, and hope to complete it soon, we will be adding it to the revised paper.
In the meantime, even without human validation, the paper gives evidence that our design supports learning. WTM-Bench and SpreadsheetBench are graded by deterministic checkers with no LLM involved (L416–418), and a model trained using XLGym improves on both (3.1% to 31.6% and 2.5% to 22.2%, Table 2).
i would also like to know what human-judge agreement would be. i appreciate that the authors explicitly listed lack of human review of the pipeline as an explicit limitation, but this is imo quite a problematic limitation for the contribution type, and constitutes a major weakness, especially when there is no mention of dataset release