- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Task–Method–Test (TMT), a coding agent training framework that decomposes SWE tasks into subtasks and constructs, admits, and freezes local tests before implementation. Locally passing records supply selected actions for supervised fine-tuning, while failure records guide the updated model in diagnosing and repairing its harness. The authors evaluate one teacher-assisted update cycle through a model–harness factorial comparison, cross-task coding benchmarks, and harness transfer to frozen external models.
Critical strengths
-
The fine-grained supervision design is reasonable and novel to me. Decomposing SWE tasks into independently verifiable subtasks allows the framework to retain successful work from globally unsuccessful trajectories and identify failed attempts within globally successful ones. This provides a potentially useful improvement over supervision based solely on whole-task outcomes.
-
The evaluation separates model and harness contributions. The Model-Harness comparison and transfer experiments with frozen external models provide useful evidence about the individual effects and reusability of the updates.
Critical weaknesses
-
Missing comparison or discussion with reinforcement learning. Training coding models using executable test feedback is closely related to RL with verifiable rewards, yet the paper provides neither an RL baseline nor a substantive discussion of this relationship. In particular, the proposed local tests could also provide process rewards to address sparse feedback in long trajectories. The authors should explain why TMT-SFT is preferable in their setting and discuss its expected advantages and limitations relative to RL. A controlled comparison using the same model, tasks, and explicitly reported compute budgets would strengthen the paper.
-
Insufficiently specified benchmark protocols. Benchmarks such as MBPP and BigCodeBench are reasonable supplementary tests of cross-task transfer, but offer limited coverage of SWE agent capabilities. The paper does not adequately explain how these tasks are adapted to the agent setting: whether multi-turn tool use is allowed, what execution feedback is exposed, how final code is extracted, and how termination is determined. The initial 27B model solving only 1/50 MBPP tasks particularly warrants investigation of the adaptation and scoring pipeline. The version and selection procedure for the 170-task SWE-bench subset should also be specified.
-
The inference-token savings lack sufficient analysis. Most of the aggregate zero-shot token reduction is driven by xCodeEval, where the joint system’s pass rate simultaneously decreases from 52% to 48%. The joint update also does not consistently reduce tokens across individual benchmarks. These aggregate results therefore do not distinguish more efficient problem solving from earlier termination or reduced exploration. Per-task token distributions, separate statistics for successful and failed tasks, tool-call and retry counts, and a precise token-accounting definition would help. Moreover, fewer supervised tokens do not directly imply proportionally lower training compute.
-
Insufficient dataset reporting undermines the interpretability of the Raw-SFT comparison. The appendix reports 1,021 training windows for TMT-SFT but does not provide the corresponding sample count for Raw-SFT, the number of distinct source tasks represented in either corpus, or the number of retained Raw-SFT trajectories. Sharing the same pool of 1,000 source tasks does not ensure comparable training coverage: TMT can retain locally successful actions from globally failed rollouts, whereas Raw-SFT retains only globally successful trajectories. Moreover, the two pipelines generate different trajectories and use unmatched training-token budgets. Consequently, the comparison cannot distinguish gains from finer-grained supervision from those due to different task coverage, trajectory construction, or training volume. The authors should report these corpus statistics and provide controlled comparisons with matched task coverage and clearly specified training settings.
-
Dependence on a stronger external teacher and unclear applicability to smaller models (minor concern). The first update cycle relies on a stronger external model for task decomposition, test construction, and rollout generation. The authors report that Qwen3.5-9B failed to complete the required steps and subsequently adopt Qwen3.6-27B, leaving the framework’s applicability to smaller models or the scalability unestablished.
Questions for authors
Please refer to weakness.