- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a pipeline for synthesizing training data for agents that operate over real files. It combines occupation-grounded task seeds, retrieved workspaces, an evidence graph used to generate tasks and rubrics, execution-conditioned revision, and rubric-based trajectory admission. Supervised fine-tuning on 2,169 trajectories improves two Qwen backbones across three working-agent benchmarks and multiple scaffolds. A subsequent experiment compares anchored, unanchored, and random trajectory selection for rejection fine-tuning (RFT). The practical problem is relevant, and the experiments support the usefulness of the resulting corpus. The remaining questions concern the evidence graph's specific contribution to SFT and the content-level reliability of the training-data admission judge.
Critical strengths
- The work addresses an important training-data bottleneck for agents producing document and spreadsheet deliverables. Grounding tasks in real workspaces is practically appealing.
- The SFT gains are substantial and extend across scaffolds. For example, the 27B model improves from 10.3 to 24.0 on SpreadsheetBench II under Claude Code, and from 56.0 to 63.7 on Workspace-Bench-Lite (Table 2, p. 7).
- The GDPVal contamination audit includes file hashes, text comparisons, and an occupation-based transfer analysis (Table 4 and Appendix B). The RFT comparison also matches query IDs, candidate pools, and optimization budgets, which makes it more informative than an unmatched selection comparison.
- The paper does test the admission judge with controlled evidence perturbations (Table 4(d), p. 9). The clear response to worksheet deletion supports structural sensitivity, and the authors explicitly acknowledge weak responses to finer content corruptions.
Critical weaknesses
. Judge validation is present, but does not yet quantify content-level admission reliability. Table 4(d), p. 9, reports mean target-criterion score changes of -0.377 for worksheet deletion, -0.018 for row corruption, -0.022 for numeric corruption, and -0.013 for citation corruption. The non-target mean absolute change is 0.016. These results support sensitivity to a missing worksheet, while raising a concern about finer content errors, which the authors themselves acknowledge (lines 452-460). However, target-criterion mean changes and non-target mean absolute changes are different summaries; their numerical proximity alone does not establish statistical indistinguishability or a detection error rate. Nor do per-criterion changes determine whether the overall admission thresholds of Q > 0.90 or Q > 0.95 are crossed. The missing evidence is an independently labeled estimate of correctness among admitted trajectories and detection rates for controlled corruptions. This concern applies to the training-data admission judge; it is not a claim that all downstream benchmark evaluators are unvalidated. 2. The evidence graph's specific contribution to SFT is not isolated. Table 2, p. 7, compares the complete GraphForge training pipeline with base models and an independently released NexForge model. This supports the effectiveness of the resulting corpus, but does not distinguish graph-based synthesis from the effects of the teacher, real-file workspaces, task coverage, or training volume. Table 3 does contain an anchoring-related ablation, but all arms start from the same graph-trained SFT checkpoint and vary subsequent RFT trajectory selection; it therefore does not answer whether using a graph improves the SFT corpus itself. A matched SFT comparison between graph-based generation and direct task/rubric generation from the same workspaces, controlling the backbone, teacher, and training-token budget, would address this question. A separate revision ablation would help assess that stage as well. 3. The RFT evidence should be interpreted within its stated scope. The unanchored condition removes explicit evidence anchors and verification instructions together (p. 8, lines 395-398), so it does not isolate anchors alone. All arms share the 462 queries selected by the anchored procedure. As the authors explicitly state (lines 399-400), this is a matched within-query selection experiment, not a comparison of complete admission pipelines. The results favor anchored selection on Workspace-Bench-Lite and SpreadsheetBench II, which is positive evidence and should be credited. On GDPVal, unanchored selection has higher point estimates, while the paper reports that these differences are not statistically resolved. Appendix D supplies conditional intervals relative to a fixed SFT anchor; all reported arm-versus-SFT intervals include zero. This does not establish equivalence between selection methods. Paired uncertainty on the other two benchmarks and a separate anchors-versus-instructions ablation would clarify how broadly the anchoring benefit holds.
Questions for authors
- On an independently annotated sample, what fraction of trajectories admitted at Q > 0.90 and Q > 0.95 are actually correct? For the judge-sensitivity experiment, what are the sample sizes, corruption severities, paired uncertainty, and rates at which corruptions change the overall admission decision?
- Can the authors provide a matched graph-versus-direct-rubric SFT experiment, holding the teacher, workspaces, training tokens, and backbone fixed?
- How much does one-step revision improve executability and factual rubric correctness, measured before and after revision against independent annotations?
- Can anchors and verification instructions be ablated separately, and can RFT differences on the two non-GDPVal benchmarks be accompanied by paired uncertainty estimates?
5 comments
[Q4 - Anchors and instructions ablated separately]: the gains of anchored selection come from both anchors and verification instructions.
We added an instructions-only RFT arm, where the selection judge sees the rubric text and verification instructions but not the explicit evidence anchors, branching from the same 35B SFT checkpoint with the same 462 queries, candidate pools, and optimization budget.
| Model | GDPVal Elo, Codex | Workspace-Bench-Lite, Claude Code | SpreadsheetBench II, Claude Code |
|---|---|---|---|
| 35B SFT | 1384.4 | 59.7 | 19.3 |
| + RFT (random-of-4) | 1374.9 | 60.8 | 19.0 |
| + RFT (unanchored) | 1409.7 | 62.2 | 17.5 |
| + RFT (instructions-only) | 1381.7 | 63.3 | 18.0 |
| + RFT (anchored) | 1395.4 | 63.7 | 20.3 |
The decomposition gives Anchored − Instructions-only = +0.4 [−3.9, +4.6] and Instructions-only − Unanchored = +1.1 [−2.7, +4.7] on Workspace-Bench-Lite, and +2.3 [−1.3, +6.0] and +0.5 [−3.6, +4.5] on SpreadsheetBench II, all positive in point estimate. On GDPVal the unanchored arm attains the highest point estimate. As the paper notes, differences at this scale are noisy rather than decisive, so this ordering is not conclusive. The paired analysis does resolve the anchored-versus-random comparison, with an interval of [+4.2, +57.5] Elo points under Codex that excludes zero.
We are wondering if the responses have addressed your concerns and would appreciate it if you consider raising your final rating. Looking forward to your further feedback!
We sincerely thank the reviewer again for the detailed questions. For your convenience, we briefly summarize the additions in our responses above, including the blinded audit of admitted workspaces, the matched graph-versus-direct SFT comparison, the before-and-after annotation of revision, and the new instructions-only RFT arm with paired intervals. We hope these results address your concerns. We would be grateful if you could take another look at our responses when time permits, and we remain happy to provide any further information.
We thank the reviewer for the detailed and balanced assessment.
[Q1 - Admission correctness under independent annotation]: most admitted artifacts pass independent fact verification, and controlled corruptions measurably reduce admissions.
The audit is human-based with blinded inputs. Annotators verified 120 admitted workspaces, stratified across the four deliverable types, against the source files without seeing any pipeline outputs. Our judge score Q is the artifact-level admission score. Rates use a 10,000-sample stratified bootstrap.
For judge sensitivity we extended Table 4d to admission decisions. We sampled 50 admitted workspaces, corrupted each in one of four ways, deleting the cited worksheet, clearing cited rows, altering cited numbers, or altering cited references, and had the judge re-score the original and corrupted versions in blinded AB/BA order with a hidden condition mapping.
Corruptions flipped 7 of 50 admissions at Q > 0.90 and 10 of 50 at Q > 0.95, with no reverse crossings, so no corrupted artifact newly entered the admitted set. The judge remains more sensitive to structural than to fine-grained factual corruption, consistent with the limitation acknowledged in the paper.
[Q2 - Matched graph-versus-direct SFT]: the graph arm outperforms direct generation on all three benchmarks.
We fine-tuned the same Qwen3.6-27B base on 221 trajectories per arm with identical steps and matched packed-token capacity. The Direct arm compiles tasks and rubrics from the same workspaces without the evidence graph.
The same pairing at construction time admits 68.4% of workspaces with the graph versus 60.2% without it on 500 frozen workspaces, a paired difference of +8.2 percentage points [2.6, 13.6].
[Q3 - Effect of one-step revision]: revision improves rubric factual consistency, admission rate, and downstream performance on all three benchmarks.
We toggled only the revision stage, at construction time on the same 500 frozen workspaces and in the matched SFT comparison of Q2.
Independent annotators also checked 60 tasks before and after revision with blinded presentation. Strict consistency fails a task when any rubric criterion contains a confirmed factual contradiction, even a minor one. Substantive consistency fails a task only when a confirmed factual error affects the substantive correctness of the rubric.
Executability is already guaranteed by the initial rollout probe before revision, so revision leaves it unchanged, while rubric factual consistency improves by +10.0 points under strict checking and +5.0 points under substantive checking.