- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies post-training for professional artifact agents: benchmarks like GDPval evaluate agents by complete, usable deliverables rather than isolated answers, yet general instruction corpora rarely demonstrate how a situated request and supporting materials become a finished work product. The authors argue this capability gap is partly a distributional mismatch rather than a reasoning deficit, and propose WorkSynth, a pipeline that synthesizes professional work supervision from situated sources, varying task formulation, artifact genre and structure, and revision-oriented feedback. A tool-using teacher produces reference artifacts and trajectories, which are preflighted, scored by discriminative rubrics, audited at task level, and frozen; the student is then trained with assistant-token behavior-cloning SFT, with no sandbox or judge inside the gradient-update loop. Evidence comes in two deliberately separated branches: a professional-work branch demonstrates construction feasibility and exposes evaluator failures, and a software-artifact branch provides the controlled training study—frozen-trajectory SFT on a 9B student raises SWE-bench Verified resolution from 1.6% to 9.8% with 6,500 random traces, 10.8% with all 16,728, and 14.4% with a historical rule-selected subset (significant by paired bootstrap and McNemar), though the rule fails post-hoc validity checks and reverses on an unmatched 27B run; a 30-step online pilot shows skill-only training dropping GDPval-style score from 0.160 to 0.056 while mixing artifact tasks reaches 0.237.
Critical strengths
S1: Methodologically honest: off-policy is precisely defined as data-collection-level, and the simple SFT objective attributes gains to the supervision distribution.
S2: Clever two-branch design: feasibility diagnostics in professional work, controlled executable evidence in software artifacts, with the adjacency caveat stated explicitly.
S3: Rigorous pipeline auditing: preflight checks, solution-invariance criteria, task-level three-property audit, contamination screens, and a zero-repository-overlap subset.
Critical weaknesses
W1: Title-to-evidence mismatch: the controlled off-policy evidence is entirely on software patches, while the professional-artifact claims remain feasibility-level, as the authors concede.
W2: Small, partially unreliable professional data: only 22% of tasks retained, "scored attempts" do not imply usable artifacts, and rollout launch was not gated on integrity reports.
W3: Limited statistical power: one training seed per arm, token/cohort mismatch between Rule-selected and Random, equal-epoch (not equal-compute) Full, and no Base/Random control in the 27B check.
W4: Selection-bias risk in a headline result: the v1 gain is most plausibly a composition effect of the success gate, the rule fails post-hoc validity checks and reverses at 27B, yet it is presented prominently.
Questions for authors
See Weaknesses.