- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes IterLab, a framework for generating calibrated synthetic ML tasks with measurable improvement opportunities and using successful agent trajectories for supervised fine-tuning. The tasks are calibrated using reference models to measure structural improvement potential and verify that input transformations preserve useful information. The authors then collect and filter complete experimentation trajectories for SFT. Experiments on two Qwen models show improvements across four external benchmarks, including DSBench, MLAgentBench, MLE-bench, and SWE-fficiency.
Critical strengths
- The paper is well written and accompanied by nicely made figures. It was easy to follow the paper.
- The method shows consistent improvements across four external benchmarks and two Qwen models, which provides strong evidence for the usefulness of the collected training data.
Critical weaknesses
- The benefit of calibrated task synthesis is not clearly demonstrated: Although 214/300 tasks have positive structural benefit, only 26/422 selected trajectories come from these tasks. It's unclear whether the improvements mainly come from the proposed task design or simply from SFT on successful trajectories. I would like to see a comparison with ordinary synthetic ML tasks, while keeping the rollout models, number of trajectories, training tokens, and SFT settings the same.
- The paper lacks RL experiments on the generated tasks. Since these synthetic tasks provide automatically verifiable rewards, it would be valuable to evaluate whether they can also improve agents through RL, rather than only SFT on successful trajectories.
- It is unclear how much the new IterLab data contributes to the improvements. Appendix D mentions 413 newly collected IterLab trajectories and 269 supplemental histories. However, no ablation study is provided to distinguish their contributions. Comparing IterLab-only, supplemental-only, and combined SFT would help clarify where the performance gains come from.
- The benefit of training on complete experimental histories is not sufficiently demonstrated. The authors should compare full-history SFT with training only on final successful solutions or selected successful steps. This would help determine whether learning from intermediate experiments, failures, and revisions actually contributes to the improvements.
Questions for authors
Please see the weaknesses section above
2 comments
Response to reviewer (2/2): Full histories, feedback use, and online RL
3. What do the intermediate steps teach?
Q3. Do intermediate experiments, failures, and revisions add value beyond the final successful solutions? (W4)
Full histories add 0.0075 RPG and 0.0379 SR over final solutions. We use the same gold-qualified source trajectories and match supervised tokens and optimizer steps. In one arm we supervise the final solution. In the other we also supervise the assistant's intermediate messages and actions, with tool outputs in context. In Appendix E, for example, a timeout prompts the agent to narrow its search (p. 27, Table 17). The full history records the outcome and the decision it prompted. The comparison tests the value of supervising that process from the same successes.
Supervision comparison (Table 24, seed 42)
| Main-collection supervision | DSBench RPG | SWE SR | SWE correct / 498 |
|---|---|---|---|
| Final solutions | 0.4328 | 0.2782 | 454 |
| Full histories | 0.4403 | 0.3161 | 455 |
On a fixed SWE-fficiency subset, we also show or hide validation scores while keeping the valid starting patch, available budget, execution feedback, and correctness feedback fixed. Final scores come from independent tests. Subtracting the hidden result from the shown result for each checkpoint measures what it gains from receiving the score.
Validation-feedback diagnostic (Table 26)
| Training condition | Feedback hidden | Feedback shown | Gain |
|---|---|---|---|
| Generic tool-use histories | 0.2084 | 0.2176 | 0.0092 |
| Main, final solutions | 0.2431 | 0.2622 | 0.0191 |
| Main, full histories | 0.2731 | 0.3187 | 0.0456 |
The full-history checkpoint gains 0.0265 more from seeing the score than the final-solution checkpoint. This shows how supervision on intermediate decisions helps when a model receives performance feedback on an existing repository solution.
4. What does RL add to fresh online SFT?
Q4. Can verifiable rewards improve agents through RL beyond collecting new successful trajectories for SFT? (W2)
Online RL adds 0.0045 RPG and 0.0242 SR over online SFT. Both methods start from the same seed-42 main full-history checkpoint. They match sampling rounds, training task distributions, generated tokens, and update budgets, and use the same submission-validity check. The comparison therefore tests the type of update at a common starting point and budget. Evaluation tasks are held out from training.
Online SFT trains on newly sampled gold-qualified trajectories. RL instead scores valid terminal submissions with r = clip((u − w)/(a − w), 0, 1), where u is terminal utility, w is weak-reference utility, and a is calibration-anchor utility. Utility is the metric for AUC and macro-F1 and negative RMSE for regression. Invalid submissions receive zero. A valid improvement above the weak reference can earn a positive reward even below gold, giving the reward-based update credit for partial progress.
Online comparison from the same checkpoint (Table 27)
| Training condition | DSBench RPG | SWE SR | SWE correct / 498 |
|---|---|---|---|
| Main full-history SFT checkpoint | 0.4403 | 0.3161 | 455 |
| Online SFT on new successful trajectories | 0.4474 | 0.3318 | 456 |
| IterLab RL | 0.4519 | 0.3560 | 455 |
Relative to the shared starting checkpoint, RL gains 0.0116 RPG and 0.0399 SR. The SWE correctness count remains 455/498, compared with 456/498 for online SFT. The gain over that matched online baseline is in the two continuous scores.
Revision: Blue text in §5.3 (pp. 6–7, Table 3) and Appendix G.1 (p. 32, Table 24) reports the supervision comparison. Appendix G.2 (pp. 32–33, Table 26) gives the feedback diagnostic. Appendix G.3 (p. 33, Table 27 and Eq. 12) reports the online comparison and reward.
Response to reviewer (1/2): Task calibration and the contribution of the new collection
Thank you for proposing these comparisons. We added controls for task design, data source, supervision, and online updates.
1. Does task calibration contribute beyond selecting successful runs?
Calibration improves the resulting agents under the same success filter. We start from the same candidate seeds and mechanism distributions and keep the input transformations and task descriptions fixed. The task-construction changes are parameter repair and task admission. We use the same rollout teachers and sampling budget, and select successes in both groups using the common ordinary-reference threshold
τ = max(s − σ_s, w + 0.3|s − w|). Heresandware ordinary- and weak-reference utility, andσ_sis the ordinary reference's bootstrap standard deviation. Each Qwen3.6-27B SFT group uses 256 distinct histories with matched supervised tokens and optimizer steps.Both groups therefore train on successes selected by the same rule. The difference tests what calibration changes about the tasks supplying those successes. It yields +0.0056 DSBench RPG and +0.0181 SWE SR, with 451 correct SWE patches in each group.
Task calibration under common success filtering (Table 24, seed 42)
The 214/300 figure counts tasks with positive structural benefit. The 26/422 figure is the share of automatic candidates from structural families. Input recovery measures a separate aspect of difficulty. Recovering the inputs improves the reference on 68/86 controls (Table 7).
2. What does the new IterLab collection add?
The new histories add value at a fixed training volume. The original Full SFT combines both sources as shown in Table 15. In the controlled comparison, we train on only main histories, only supplemental histories, or a mixture of 128 histories from each source. The SFT groups match 256 distinct histories, supervised tokens, and optimizer steps.
Original Full SFT data split (Table 15)
Data-source comparison (Table 24, seed 42)
Main-only SFT improves on supplemental-only SFT by 0.0176 RPG and 0.0907 SR. Replacing half the supplemental histories with main histories raises RPG by 0.0256 and SR by 0.1366 over supplemental-only training. This replacement shows the contribution of the new histories while keeping the total training volume fixed.
Revision: Blue text in §5.3 (pp. 6–7, Table 3) and Appendix G.1 (p. 32, Table 24) reports the matched-budget comparisons. Appendix D.1 (p. 25, Table 15) gives the original Full SFT split. §5.6 (p. 9) and Appendix A.1 (p. 18, Table 7) give the coverage breakdown.