acceptodds
Under review as a conference paper at ICLR 2027

GraphForge: Training Working Agents with Graph-Anchored Workspace Synthesis

Abstract

Working agents need to read diverse files, coordinate tools, and produce deliverables. Training such agents requires tasks built on many real files with verifiable results, but few pipelines exist to synthesize this kind of data. Existing pipelines either generate files with models, which lack realism and diversity, or build tasks on real files without task-specific verifiers, leaving result quality unchecked. We introduce GraphForge, an evidence-graph based framework for synthesizing working agent training data. For each task seed, GraphForge first crawls the real files for its workspace and builds an evidence graph over the relations among these files. The graph is then compiled into task statements and rubrics with evidence anchors. An initial model rollout tests whether the task is executable, and a revise agent repairs any issues in the task statement and rubrics against the original files before the final trajectory is collected. The evidence anchors tell the judge exactly which files to consult when scoring each rubric. We fine-tune Qwen3.6-27B on 2,169 trajectories from GraphForge, bringing GDPVal to 1445.7 (+65.7), Workspace-Bench-Lite to 63.7 (+7.7), and SpreadsheetBench II to 24.0 (+13.7). Starting from the SFT model, we then apply offline rejection fine-tuning, sampling candidate trajectories, scoring them with the evidence-anchored rubrics, and keeping only the highest-scoring ones for another round of training. This brings further improvements over SFT on all three benchmarks. As the gain comes from training on the model's own rollouts, it suggests that the evidence-anchored rubrics provide useful selection signal.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_vLeMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a pipeline for synthesizing training data for agents that operate over real files. It combines occupation-grounded task seeds, retrieved workspaces, an evidence graph used to generate tasks and rubrics, execution-conditioned revision, and rubric-based trajectory admission. Supervised fine-tuning on 2,169 trajectories improves two Qwen backbones across three working-agent benchmarks and multiple scaffolds. A subsequent experiment compares anchored, unanchored, and random trajectory selection for rejection fine-tuning (RFT). The practical problem is relevant, and the experiments support the usefulness of the resulting corpus. The remaining questions concern the evidence graph's specific contribution to SFT and the content-level reliability of the training-data admission judge.

Critical strengths
  1. The work addresses an important training-data bottleneck for agents producing document and spreadsheet deliverables. Grounding tasks in real workspaces is practically appealing.
  2. The SFT gains are substantial and extend across scaffolds. For example, the 27B model improves from 10.3 to 24.0 on SpreadsheetBench II under Claude Code, and from 56.0 to 63.7 on Workspace-Bench-Lite (Table 2, p. 7).
  3. The GDPVal contamination audit includes file hashes, text comparisons, and an occupation-based transfer analysis (Table 4 and Appendix B). The RFT comparison also matches query IDs, candidate pools, and optimization budgets, which makes it more informative than an unmatched selection comparison.
  4. The paper does test the admission judge with controlled evidence perturbations (Table 4(d), p. 9). The clear response to worksheet deletion supports structural sensitivity, and the authors explicitly acknowledge weak responses to finer content corruptions.
Critical weaknesses

. Judge validation is present, but does not yet quantify content-level admission reliability. Table 4(d), p. 9, reports mean target-criterion score changes of -0.377 for worksheet deletion, -0.018 for row corruption, -0.022 for numeric corruption, and -0.013 for citation corruption. The non-target mean absolute change is 0.016. These results support sensitivity to a missing worksheet, while raising a concern about finer content errors, which the authors themselves acknowledge (lines 452-460). However, target-criterion mean changes and non-target mean absolute changes are different summaries; their numerical proximity alone does not establish statistical indistinguishability or a detection error rate. Nor do per-criterion changes determine whether the overall admission thresholds of Q > 0.90 or Q > 0.95 are crossed. The missing evidence is an independently labeled estimate of correctness among admitted trajectories and detection rates for controlled corruptions. This concern applies to the training-data admission judge; it is not a claim that all downstream benchmark evaluators are unvalidated. 2. The evidence graph's specific contribution to SFT is not isolated. Table 2, p. 7, compares the complete GraphForge training pipeline with base models and an independently released NexForge model. This supports the effectiveness of the resulting corpus, but does not distinguish graph-based synthesis from the effects of the teacher, real-file workspaces, task coverage, or training volume. Table 3 does contain an anchoring-related ablation, but all arms start from the same graph-trained SFT checkpoint and vary subsequent RFT trajectory selection; it therefore does not answer whether using a graph improves the SFT corpus itself. A matched SFT comparison between graph-based generation and direct task/rubric generation from the same workspaces, controlling the backbone, teacher, and training-token budget, would address this question. A separate revision ablation would help assess that stage as well. 3. The RFT evidence should be interpreted within its stated scope. The unanchored condition removes explicit evidence anchors and verification instructions together (p. 8, lines 395-398), so it does not isolate anchors alone. All arms share the 462 queries selected by the anchored procedure. As the authors explicitly state (lines 399-400), this is a matched within-query selection experiment, not a comparison of complete admission pipelines. The results favor anchored selection on Workspace-Bench-Lite and SpreadsheetBench II, which is positive evidence and should be credited. On GDPVal, unanchored selection has higher point estimates, while the paper reports that these differences are not statistically resolved. Appendix D supplies conditional intervals relative to a fixed SFT anchor; all reported arm-versus-SFT intervals include zero. This does not establish equivalence between selection methods. Paired uncertainty on the other two benchmarks and a separate anchors-versus-instructions ablation would clarify how broadly the anchoring benefit holds.

Questions for authors
  1. On an independently annotated sample, what fraction of trajectories admitted at Q > 0.90 and Q > 0.95 are actually correct? For the judge-sensitivity experiment, what are the sample sizes, corruption severities, paired uncertainty, and rates at which corruptions change the overall admission decision?
  2. Can the authors provide a matched graph-versus-direct-rubric SFT experiment, holding the teacher, workspaces, training tokens, and backbone fixed?
  3. How much does one-step revision improve executability and factual rubric correctness, measured before and after revision against independent annotations?
  4. Can anchors and verification instructions be ablated separately, and can RFT differences on the two non-GDPVal benchmarks be accompanied by paired uncertainty estimates?
5 comments

We thank the reviewer for the detailed and balanced assessment.

[Q1 - Admission correctness under independent annotation]: most admitted artifacts pass independent fact verification, and controlled corruptions measurably reduce admissions.

The audit is human-based with blinded inputs. Annotators verified 120 admitted workspaces, stratified across the four deliverable types, against the source files without seeing any pipeline outputs. Our judge score Q is the artifact-level admission score. Rates use a 10,000-sample stratified bootstrap.

CheckPass rate95% CI
Rubric factual consistency95.4%[90.3, 99.1]
Evidence localization91.2%[84.5, 96.7]
Fact verification, all judgeable cases83.0%[73.3, 92.2]
Fact verification, Q > 0.95 band87.8%[75.7, 97.4]
Fact verification, 0.90 < Q ≤ 0.95 band76.0%[59.2, 92.3]

For judge sensitivity we extended Table 4d to admission decisions. We sampled 50 admitted workspaces, corrupted each in one of four ways, deleting the cited worksheet, clearing cited rows, altering cited numbers, or altering cited references, and had the judge re-score the original and corrupted versions in blinded AB/BA order with a hidden condition mapping.

Corruption typeAdmission drop (pp)95% CIn
Worksheet deletion7.9[2.8, 15.0]11
Numeric corruption1.7[0.4, 3.3]12
Row clearing1.5[0.2, 4.1]15
Citation corruption1.1[0.5, 1.8]12

Corruptions flipped 7 of 50 admissions at Q > 0.90 and 10 of 50 at Q > 0.95, with no reverse crossings, so no corrupted artifact newly entered the admitted set. The judge remains more sensitive to structural than to fine-grained factual corruption, consistent with the limitation acknowledged in the paper.

[Q2 - Matched graph-versus-direct SFT]: the graph arm outperforms direct generation on all three benchmarks.

We fine-tuned the same Qwen3.6-27B base on 221 trajectories per arm with identical steps and matched packed-token capacity. The Direct arm compiles tasks and rubrics from the same workspaces without the evidence graph.

SFT dataGDPVal EloWorkspace-Bench-LiteSpreadsheetBench II
Graph + Revision1412.6 [1355.5, 1468.8]64.527.4
Direct + Revision1396.8 [1338.6, 1455.8]60.423.7
Graph w/o Revision1378.2 [1337.1, 1419.0]60.621.5

The same pairing at construction time admits 68.4% of workspaces with the graph versus 60.2% without it on 500 frozen workspaces, a paired difference of +8.2 percentage points [2.6, 13.6].

[Q3 - Effect of one-step revision]: revision improves rubric factual consistency, admission rate, and downstream performance on all three benchmarks.

We toggled only the revision stage, at construction time on the same 500 frozen workspaces and in the matched SFT comparison of Q2.

ArmTasks admitted, 500 workspacesGDPVal EloWorkspace-Bench-LiteSpreadsheetBench II
Graph + Revision68.4%1412.6 [1355.5, 1468.8]64.527.4
Graph w/o Revision66.6%1378.2 [1337.1, 1419.0]60.621.5

Independent annotators also checked 60 tasks before and after revision with blinded presentation. Strict consistency fails a task when any rubric criterion contains a confirmed factual contradiction, even a minor one. Substantive consistency fails a task only when a confirmed factual error affects the substantive correctness of the rubric.

CheckBefore revisionAfter revision
Task executability100.0%100.0%
Rubric strict factual consistency78.3%88.3%
Rubric substantive factual consistency88.3%93.3%

Executability is already guaranteed by the initial rollout probe before revision, so revision leaves it unchanged, while rubric factual consistency improves by +10.0 points under strict checking and +5.0 points under substantive checking.

[Q4 - Anchors and instructions ablated separately]: the gains of anchored selection come from both anchors and verification instructions.

We added an instructions-only RFT arm, where the selection judge sees the rubric text and verification instructions but not the explicit evidence anchors, branching from the same 35B SFT checkpoint with the same 462 queries, candidate pools, and optimization budget.

ModelGDPVal Elo, CodexWorkspace-Bench-Lite, Claude CodeSpreadsheetBench II, Claude Code
35B SFT1384.459.719.3
+ RFT (random-of-4)1374.960.819.0
+ RFT (unanchored)1409.762.217.5
+ RFT (instructions-only)1381.763.318.0
+ RFT (anchored)1395.463.720.3

The decomposition gives Anchored − Instructions-only = +0.4 [−3.9, +4.6] and Instructions-only − Unanchored = +1.1 [−2.7, +4.7] on Workspace-Bench-Lite, and +2.3 [−1.3, +6.0] and +0.5 [−3.6, +4.5] on SpreadsheetBench II, all positive in point estimate. On GDPVal the unanchored arm attains the highest point estimate. As the paper notes, differences at this scale are noisy rather than decisive, so this ordering is not conclusive. The paired analysis does resolve the anchored-versus-random comparison, with an interval of [+4.2, +57.5] Elo points under Codex that excludes zero.

We are wondering if the responses have addressed your concerns and would appreciate it if you consider raising your final rating. Looking forward to your further feedback!

We sincerely thank the reviewer again for the detailed questions. For your convenience, we briefly summarize the additions in our responses above, including the blinded audit of admitted workspaces, the matched graph-versus-direct SFT comparison, the before-and-after annotation of revision, and the new instructions-only RFT arm with paired intervals. We hope these results address your concerns. We would be grateful if you could take another look at our responses when time permits, and we remain happy to provide any further information.

Reviewer_FH2oOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GraphForge builds agent training tasks from real workspace files. It constructs evidence graphs, generates tasks and grading rubrics, and revises them using teacher rollouts. Supervised and reinforcement learning experiments evalaute the resulting data across work-related benchmarks and agent frameworks.

Critical strengths
  1. Grounding tasks and rubrics in workspace files offers a useful approach to generating realistic agent training data.
  2. Supervised training improves performance across the tested backbones and frameworks. The paper also includes controlled reinforcement learning comparisons and a GDPVal overlap audit.
Critical weaknesses
  1. The judge is weak at detecting some factual errors. In Table 4(d), numeric, row, and citation changes receive small penalties close to unrelated score changes. High rubric scores therefore provide limited assurance of factual correctness.
  2. The supervised experiments do not isolate the value of evidence graphs and task revision from the effects of source files, teacher trajectories, and training volume.
Questions for authors
  1. What factual and calculation error rates are found by independently checking accepted trajectories?
  2. How much do evidence graphs and task revision help under matched supervised training budgets?
1 comment

We thank the reviewer for the positive assessment and the constructive questions. We truly appreciate your recognition of the file-grounded design and the controlled comparisons, and we address both questions with new experiments below.

[Q1 - Independent error rates of accepted artifacts]: independent human checking finds that most accepted artifacts are factually correct, and errors concentrate in the lower-Q band.

The audit is human-based with blinded inputs. Annotators verified 120 admitted workspaces, stratified across the four deliverable types, against the source files without seeing any pipeline outputs. Our judge score Q is the artifact-level admission score. Fact verification checks whether factual and calculation claims in the deliverable hold against the source files. Rates use a 10,000-sample stratified bootstrap.

CheckPass rate95% CI
Rubric factual consistency95.4%[90.3, 99.1]
Evidence localization91.2%[84.5, 96.7]
Fact verification, all judgeable cases83.0%[73.3, 92.2]
Fact verification, Q > 0.95 band87.8%[75.7, 97.4]
Fact verification, 0.90 < Q ≤ 0.95 band76.0%[59.2, 92.3]

The higher band passes fact verification more often, which indicates that Q carries useful quality signal.

A major error is a consequential numerical, factual, or calculation mistake that would mislead a user of the deliverable, and a minor error is a visible flaw that does not change the substance. Each case is counted once under its most severe error, and the type tag follows the file suffix of the required deliverables.

Deliverable typeCasesMajor errorsMinor errorsNo problem
Document359917
Mixed3391014
Presentation213162
Spreadsheet312821

On the judge weakness noted in Table 4d, we extended the corruption probe to admission decisions with a blinded and counterbalanced calibration on 50 admitted workspaces. All four corruption types reduce admissions with intervals excluding zero, worksheet deletion by 7.9 points and the fine-grained corruptions by 1.1 to 1.7 points, and no corrupted artifact newly entered the admitted set. The judge remains weaker on fine-grained factual corruption, and we keep this limitation explicit in the paper.

[Q2 - Evidence graphs and revision under matched training budgets]: both components contribute, and the full pipeline attains the best score on every benchmark.

We built 500 frozen workspaces and ran the admission pipeline with and without the evidence graph, pairing every condition on the same workspaces, teacher, and judge.

Construction armAdmittedRate
Graph + Revision342/50068.4%
Graph w/o Revision333/50066.6%
Direct + Revision301/50060.2%

The paired difference between Graph + Revision and Direct + Revision is +8.2 percentage points [2.6, 13.6] from a 10,000-sample paired bootstrap.

We then fine-tuned the same Qwen3.6-27B base on 221 trajectories per arm with identical steps and matched packed-token capacity, so the teacher, workspaces, and training budget are held fixed across arms.

SFT dataGDPVal EloWorkspace-Bench-LiteSpreadsheetBench II
Graph + Revision1412.6 [1355.5, 1468.8]64.527.4
Direct + Revision1396.8 [1338.6, 1455.8]60.423.7
Graph w/o Revision1378.2 [1337.1, 1419.0]60.621.5

Removing either stage also degrades downstream performance on all three benchmarks, and the full pipeline attains the best score in every column.

We hope these additional results address your questions and further support your assessment. Looking forward to your further feedback!

Reviewer_WXXwOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

method: GraphForge use a financial occupation-task specification database to prompt GLM5.2 to crawl online real files to construct a workspace. Then GLM5.2 constructs an evidence graph, upon which tasks and rubrics are generated. GLM5.2 tries to complete the task; if badly specified, retry task specification; if score low on rubric, retry trace generation. Obtain thousands of traces on various tasks.

experiment: small Qwen models SFTed and RFTed on these traces scores higher on well known benchs compared with untuned ones.

contribution: enable diverse task-workspace generation with real files and quality evaluation grounded in the files.

Critical strengths
  1. Ventures into large scale automated workspace&file generation for LLM agent training, with a consideration on being real and quality assurance on eval rubrics.
Critical weaknesses
  1. Lack rigor.
    • The paper mentions previous work EnvCraft and NexForge, but has not given experiment results of fine-tuning on traces generated from them. Thus superiority than previous work is not empirically demonstrated.
    • Ablation hasn't consider removing the evidence graph step to prove whether it is useful.

Thus, the quality of the dataset cannot be determined from the experiment results reliably.

  1. Lack transparency.
    • Abstract says fine-tuned model weights are on an anon link, but the link is not currently given. We cannot verify whether the models really live up to the evals.
    • No data file repo link is given, or representative full detailed examples given in the appendix. We cannot verify the data amount or quality.
    • No pipeline code, so we cannot examine the data generation details or easily replicate the process.
Questions for authors

Please address the critical weaknesses.

5 comments

We sincerely thank the reviewer for the constructive feedback and for recognizing the value of large-scale automated workspace generation with quality-assured evaluation rubrics.

[W1 - Comparison with Prior Pipelines]: We compare against both officially released checkpoints under identical protocols.

We appreciate this suggestion and have expanded the empirical comparison. We respectfully note that NexForge does not publicly release its generated trajectories, so fine-tuning on its traces is not possible. Its officially released model Nex-N2-Mini-35B has been reported in Table 2 of our paper under all scaffolds since submission. For EnvCraft, we evaluate its officially released checkpoint EnvCraft-9B under identical benchmark protocols.

ModelGDPVal (OpenHands)Workspace-Bench-Lite (Claude Code)SpreadsheetBench II (Claude Code)
Nex-N2-Mini-35B (official ckpt)1288.8 [1175.9, 1378.9]33.16.5
EnvCraft-9B (official ckpt)1104.7 [1029.3, 1169.2]37.31.2
GraphForge 27B SFT (Ours)1445.7 [1376.2, 1513.8]63.724.0

GDPVal bootstrap intervals for Nex-N2-Mini-35B and our model are from Appendix C, Table 6 of the paper.

[W1 - Ablations]: Both the graph and revision stages contribute, and the full pipeline is best on all three benchmarks.

All arms were constructed on the same 500 frozen workspaces with the same teacher, then retrained from the same Qwen3.6-27B base model, 221 trajectories per arm with matched optimizer steps and matched packed-token capacity.

ArmQualified of 500Admission RateGDPVal (Elo)Workspace-Bench-LiteSpreadsheetBench II
Graph + Revision (full)34268.4%1412.6 [1355.5, 1468.8]64.527.4
Direct + Revision30160.2%1396.8 [1338.6, 1455.8]60.423.7
Graph w/o Revision333*66.6%*1378.2 [1337.1, 1419.0]60.621.5

*Constructed and admitted through the same pipeline on the same 500 workspaces, with only the revision step removed. For task construction, 10,000 paired bootstrap resamples give a graph admission-rate advantage of +8.2pp [2.6, 13.6], which is statistically significant, and removing revision reduces the admission rate from 68.4% to 66.6%. Removing either stage also degrades downstream performance on all three benchmarks.

[W2 - Transparency]: Checkpoints have been available since submission, and we now release the full SFT corpus and pipeline code.

Thank you for pointing this out. The repository was already linked in the abstract through the phrase "anonymous repository", and the trained checkpoints have been available there since submission. The hyperlink may not be visible in every PDF viewer, so we share the direct URL here for convenience: https://huggingface.co/collections/randomsubmit/graphforge. The revised version will print the URL in plain text.

The HuggingFace collection now also contains the full set of 2,169 SFT trajectories, so the quantity and quality of the corpus can be verified directly. The pipeline code is released at https://anonymous.4open.science/r/graphforge_query_pipeline-1D8C/, so the data generation process can be inspected end to end. Raw workspace files are subject to partner restrictions and cannot be redistributed.

We are wondering if the responses have addressed your concerns and would appreciate it if you consider raising your final rating. Looking forward to your further feedback!

Reviewer_WXXwOpenReview ↗

Query about data replication

Thanks for the authors' detailed response.
The authors report Nex-N2-Mini-35B's GDPVal Elo of 1289 and 1342 under OpenHands and Codex, respectively. However, nex-agi reports Nex-N2-Mini-35B having a GDPVal score of 1402, significantly closer to GraphForge 27B SFT's 1446 score. We would like the authors to clarify on this issue.

We thank the reviewer for raising this point. We have re-examined our evaluation records, and we believe the two numbers are not contradictory.

[Clarification on the Nex-N2-Mini-35B Elo]: the two values use different anchors, pools, and harnesses, so they are not directly comparable.

  • Different anchor. Elo is a relative score measured against an anchor model fixed at a chosen value. We anchor at GLM-5.3 (OpenHands) = 1667.0, while nex-agi report on a scale where GPT-5.5 is 1769.0 and GLM-5.1 is 1535.0, so the same model takes different absolute values on different scales.
  • Different pool. Each Elo value is fit jointly over all evaluated models, so it depends on which models are compared together. The NexForge report itself notes that such pool-internal values should not be read as external leaderboard scores. Across every model reported by both sources, nex-agi's values are consistently 392.0 to 476.0 Elo higher than the official third-party leaderboard.
  • Different harness. The 1402.0 value comes from nex-agi's in-house NexAU scaffold, whose comparison records are not public, while we evaluate the released checkpoint under OpenHands and Codex. The same checkpoint differs by 53.5 Elo between OpenHands and Codex in our runs, and a model post-trained for its own framework is likely best matched to its native setting.
ModelNexForge ReportOfficial GDPval-AA (best setting)Gap
Nex-N2-Pro1585.01109.0476.0
Kimi K2.61481.01046.0435.0
GPT-5.51769.01357.0412.0
GLM-5.11535.01125.0410.0
Claude Opus 4.71753.01361.0392.0
Nex-N2-Mini-35B1402.0ours 1288.8 (OpenHands), 1342.3 (Codex)113.2, 59.7

What we stand behind is the internal consistency of our Table 2, where all models are evaluated under one protocol with pass@1 on the same 220 tasks and the same judge. We will add a note in the revision stating the anchor and the pool-internal nature of our Elo values.

We are wondering if the responses have addressed your concerns and looking forward to your further feedback!

Reviewer_j8C2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GraphForge builds training workspaces from real files and constructs cross-file evidence graphs used to generate tasks and file-anchored rubrics. Initial rollouts probe feasibility, a revision stage revisits the source files, and artifact scores filter trajectories for training. SFT with 2,169 trajectories improves reported results on GDPVal, Workspace Bench Lite, and SpreadsheetBench II. Matched RFT comparisons examine trajectory selection with anchored, unanchored, and random signals, while corruption tests assess the judge's sensitivity to artifact errors.

Critical strengths
  1. A useful, traceable data-production pipeline. Connecting source files, tasks, rubrics, revisions, and trajectory selection provides inspectable evidence links and a practical approach to generating workspace tasks. The resulting SFT model improves GDPVal from 1380 to 1445.7, Workspace Bench Lite from 56.0 to 63.7, and SpreadsheetBench II from 10.3 to 24.0 under the respective reported scaffolds (Table 2). These are distinct metrics, but together support the utility of the resulting training data.

  2. The paper directly probes its selection signal. Fixing the candidate pool, queries, and training budget in the RFT comparisons is more informative than reporting a final score alone. The corruption experiment also exposes an important limitation: the judge detects structural damage much more readily than fine-grained factual damage. Reporting this negative result improves the interpretability of the method.

Critical weaknesses
  1. Evidence traceability is not yet reliable factual verification. Deleting worksheets reduces the score by 0.377 on average, whereas corrupting rows, numbers, or citations changes it by only 0.018, 0.022, or 0.013 (Table 4). This raises a central concern about whether high-scoring selected artifacts are actually correct. Because task generation and rubrics share an evidence representation, omissions or misreadings can also be correlated. Independent human calibration of consequential factual errors is needed to substantiate stronger verification claims.

  2. The causal contribution of graph anchoring remains incompletely established. GDPVal differences in the matched RFT experiment have confidence intervals containing zero; unanchored point estimates are higher there, so the evidence establishes neither method's superiority. Moreover, the shared 462-query pool was first selected by the anchored pipeline from 2,000 queries (Section 4.4). The comparison isolates trajectory selection within those admitted tasks, not the entire admission pipeline. Strong-teacher distillation, real-file tasks, revision, graph structure, and anchoring remain partly bundled, limiting attribution of the gains to the proposed mechanism.

Questions for authors
  1. Among high-scoring retained artifacts, how often do independent human checks find consequential numerical, factual, or citation errors? Please break this down by artifact type and error severity, and distinguish evidence localization from fact verification.
  2. Under matched teachers, tasks, token budgets, and candidate pools, how much benefit remains when removing the graph, anchors, or revision separately? Would a flat evidence list perform similarly?
  3. Can task admission and within-task trajectory selection be evaluated as separate factors over the full candidate query set? Please also provide uncertainty for the Workspace/Spreadsheet comparisons and calibration using an independent model family or human-validated subset.
  4. The reported contamination audit checks exact file hashes and a small high-similarity sample for GDPVal. What checks cover semantically similar tasks, business instances, and the other two benchmarks? Given the stated plan not to redistribute the corpus, what manifests, hashes, collection scripts, license records, and task/rubric artifacts can be released for independent auditing?
3 comments

We thank the reviewer for the careful reading and the constructive questions.

[Q1 - Human verification of admitted artifacts]: most admitted artifacts pass independent fact verification, and higher judge scores pass more often.

The audit is human-based with blinded inputs. Annotators verified 120 admitted workspaces, stratified across the four deliverable types, against the source files without seeing any pipeline outputs. Our judge score Q is the artifact-level admission score. Evidence localization checks whether cited file anchors point to the supporting content, and fact verification checks whether factual claims in the deliverable hold against the source files. Rates use a 10,000-sample stratified bootstrap.

CheckPass rate95% CI
Rubric factual consistency95.4%[90.3, 99.1]
Evidence localization91.2%[84.5, 96.7]
Fact verification, all judgeable cases83.0%[73.3, 92.2]
Fact verification, Q > 0.95 band87.8%[75.7, 97.4]
Fact verification, 0.90 < Q ≤ 0.95 band76.0%[59.2, 92.3]

The higher band passes fact verification more often, which indicates that Q carries useful quality signal.

A major error is a consequential numerical, factual, or citation mistake that would mislead a user of the deliverable, and a minor error is a visible flaw that does not change the substance. Each case is counted once under its most severe error, and the type tag follows the file suffix of the required deliverables.

Deliverable typeCasesMajor errorsMinor errorsNo problem
Document359917
Mixed3391014
Presentation213162
Spreadsheet312821

We also extended the corruption probe in Table 4d to admission decisions. We sampled 50 admitted workspaces, assigned each to one of the four corruption types, and had the judge re-score the original and corrupted versions in blinded AB/BA order with a hidden condition mapping.

Corruption typeAdmission drop (pp)95% CIn
Worksheet deletion7.9[2.8, 15.0]11
Numeric corruption1.7[0.4, 3.3]12
Row clearing1.5[0.2, 4.1]15
Citation corruption1.1[0.5, 1.8]12

Corruptions flipped 7 of 50 admissions at Q > 0.90 and 10 of 50 at Q > 0.95, with no reverse crossings, so no corrupted artifact newly entered the admitted set. The judge remains more sensitive to structural than to fine-grained factual corruption, and we keep this limitation explicit in the paper.

[Q2 - Removing the graph and revision under matched conditions]: both stages contribute, and the flat-evidence Direct arm underperforms the graph arm on every benchmark.

We built 500 frozen workspaces and ran the admission pipeline with and without the evidence graph, pairing every condition on the same workspaces, teacher, and judge. The Direct arm compiles tasks from the same files without the evidence graph, which corresponds to the flat-evidence condition in the question.

Construction armAdmittedRate
Graph + Revision342/50068.4%
Graph w/o Revision333/50066.6%
Direct + Revision301/50060.2%

The paired difference between Graph + Revision and Direct + Revision is +8.2 percentage points [2.6, 13.6] from a 10,000-sample paired bootstrap.

We then fine-tuned the same Qwen3.6-27B base on 221 trajectories per arm with identical steps and matched packed-token capacity.

SFT dataGDPVal EloWorkspace-Bench-LiteSpreadsheetBench II
Graph + Revision1412.6 [1355.5, 1468.8]64.527.4
Direct + Revision1396.8 [1338.6, 1455.8]60.423.7
Graph w/o Revision1378.2 [1337.1, 1419.0]60.621.5

Removing either stage also degrades downstream performance on all three benchmarks, and the full pipeline attains the best score in every column.

[Q3 - Admission versus within-task selection]: task admission and within-task selection are now measured separately, with paired intervals for the Workspace and Spreadsheet gains.

We agree the 462-query experiment in the paper isolates within-task selection only, since its pool was pre-admitted by the anchored pipeline. The Q2 study evaluates admission separately, as its 500 workspaces are built from freshly sampled queries rather than the admitted pool, and the Q1 audit provides the human-validated calibration. Paired bootstrap intervals for the other two benchmarks are below.

ComparisonBenchmarkScaffoldPaired difference95% CI
27B SFT − BaseWorkspace-Bench-LiteClaude Code+7.7[+1.6, +14.2]
27B SFT − BaseWorkspace-Bench-LiteCodex+3.8[−1.7, +9.5]
35B SFT − BaseSpreadsheetBench IIClaude Code+16.7[+12.6, +21.1]
27B SFT − BaseSpreadsheetBench IICodex+8.5[+4.1, +13.0]

The review also notes that the GDPVal intervals in the matched RFT experiment contain zero. A paired bootstrap over the same runs gives [+4.2, +57.5] Elo points for anchored versus random selection under Codex, which excludes zero.

[Q4 - Contamination and independent auditing]: the extended audit covers all three benchmarks with zero hash overlap, and checkpoints, trajectories, and pipeline code are released.

We extended the exact-hash audit to all three benchmarks and added semantic nearest-neighbor checks.

BenchmarkTasksReferences checkedUsable hashesHash hitsSame-skill neighborsUnrelated neighbors
Workspace-Bench-Lite10076066104555
SpreadsheetBench II321321320025467

In total 38,335 usable hashes were compared and 2,105 semantic neighbor pairs were checked, and the 8 text-level candidates flagged for manual review were all noise. Same-skill neighbors share the broad occupational skill but are different business instances. The audit cannot cover the 1,222 references without usable hashes.

For independent auditing we release the checkpoints and the full set of 2,169 SFT trajectories at https://huggingface.co/collections/randomsubmit/graphforge and the pipeline code at https://anonymous.4open.science/r/graphforge_query_pipeline-1D8C/. Unfortunately, raw workspace files are subject to partner restrictions and cannot be redistributed.

We are wondering if the responses have addressed your concerns and would appreciate it if you consider raising your final rating. Looking forward to your further feedback!

We sincerely thank the reviewer again for the constructive questions. For your convenience, we briefly summarize the additions in our responses above, including the blinded human audit of admitted workspaces, paired graph-versus-direct experiments with matched budgets, paired uncertainty intervals, and the extended contamination audit, together with released checkpoints, trajectories, and pipeline code. We hope these results address your concerns. We would be grateful if you could take another look at our responses when time permits, and we remain happy to provide any further information.

Comments on the paper

1 comment

We sincerely thank all reviewers for the constructive feedback. We summarize the key updates below.

(1) Data quality and judge reliability (Reviewers j8C2, FH2o, and vLeM). A blinded human audit of 120 admitted workspaces finds that 95.4% of rubrics are factually consistent with the source files and 83.0% of deliverables pass fact verification. Higher judge scores pass more often, with 87.8% above Q = 0.95 versus 76.0% below. We also damaged 50 admitted workspaces on purpose, and admissions dropped under all four damage types, with no damaged artifact newly admitted.

(2) Value of the graph and revision (Reviewers j8C2, FH2o, vLeM, and WXXw). On 500 frozen workspaces, the evidence graph raises the admission rate by +8.2 pp (95% CI [2.6, 13.6]). When we train the same 27B base model on equal budgets of data built with and without the graph, the graph-trained model scores best on all three benchmarks. In independent annotation on 60 tasks, revision raises rubric factual consistency from 78.3% to 88.3%.

(3) Anchors and instructions both help trajectory selection (Reviewers j8C2 and vLeM).** A new instructions-only RFT arm separates anchors from verification instructions, and both contribute positively on the two non-GDPVal benchmarks. A new paired bootstrap over the existing runs gives [+4.2, +57.5] Elo for anchored versus random selection on GDPVal, excluding zero.

(4) Contamination and transparency (Reviewers j8C2 and WXXw). The hash audit now covers all three benchmarks with zero hits over 38,335 usable hashes plus 2,105 semantic pairs, and we release the checkpoints, all 2,169 SFT trajectories, and the pipeline code.

We would be grateful if the reviewers could take another look at our responses when time permits, and we remain happy to provide any further information.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.