acceptodds
Under review as a conference paper at ICLR 2027

IterLab: Training Agents to Experiment and Improve over Long Horizons with Calibrated Synthetic Tasks

Abstract

Training agents to improve solutions over long horizons requires tasks that make experimentation both necessary and productive. We introduce IterLab, a pipeline combining calibrated synthetic tasks with supervision on complete experimental histories. Prediction problems with known mechanisms expose structural gains; paired references set quality targets, and recovery checks assess whether input transformations preserve useful signal. Agents inspect data, run code, interpret feedback, and revise solutions within a shared budget. Across 300 tasks, structural features improve reference performance on 214. Quality and interaction screening retain 413 demonstrations from 6,000 requested trajectories. Supervised fine-tuning on these and supplemental demonstrations improves both Qwen3.6-27B and Qwen3.6-35B-A3B on DSBench, MLAgentBench, MLE-bench, and SWE-fficiency. MLE-bench medal rates increase by 15.56 and 14.67 percentage points, respectively, while software-optimization performance improves alongside correctness. Task analyses distinguish modeling from input-processing demands and identify which challenges yield demonstrations. IterLab turns measurable improvement opportunities into training experience for extended, feedback-driven experimentation. Code is available at https://anonymous.4open.science/r/anonymous_code2-CCB1.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-Horizon Agents, Dynamic Environment Scaling, Synthetic training data, Agent Post-Training, Recursive Self-Improvement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ryqHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes IterLab, a framework for generating calibrated synthetic ML tasks with measurable improvement opportunities and using successful agent trajectories for supervised fine-tuning. The tasks are calibrated using reference models to measure structural improvement potential and verify that input transformations preserve useful information. The authors then collect and filter complete experimentation trajectories for SFT. Experiments on two Qwen models show improvements across four external benchmarks, including DSBench, MLAgentBench, MLE-bench, and SWE-fficiency.

Critical strengths
  • The paper is well written and accompanied by nicely made figures. It was easy to follow the paper.
  • The method shows consistent improvements across four external benchmarks and two Qwen models, which provides strong evidence for the usefulness of the collected training data.
Critical weaknesses
  • The benefit of calibrated task synthesis is not clearly demonstrated: Although 214/300 tasks have positive structural benefit, only 26/422 selected trajectories come from these tasks. It's unclear whether the improvements mainly come from the proposed task design or simply from SFT on successful trajectories. I would like to see a comparison with ordinary synthetic ML tasks, while keeping the rollout models, number of trajectories, training tokens, and SFT settings the same.
  • The paper lacks RL experiments on the generated tasks. Since these synthetic tasks provide automatically verifiable rewards, it would be valuable to evaluate whether they can also improve agents through RL, rather than only SFT on successful trajectories.
  • It is unclear how much the new IterLab data contributes to the improvements. Appendix D mentions 413 newly collected IterLab trajectories and 269 supplemental histories. However, no ablation study is provided to distinguish their contributions. Comparing IterLab-only, supplemental-only, and combined SFT would help clarify where the performance gains come from.
  • The benefit of training on complete experimental histories is not sufficiently demonstrated. The authors should compare full-history SFT with training only on final successful solutions or selected successful steps. This would help determine whether learning from intermediate experiments, failures, and revisions actually contributes to the improvements.
Questions for authors

Please see the weaknesses section above

2 comments
AuthorsOpenReview ↗

Response to reviewer (1/2): Task calibration and the contribution of the new collection

Thank you for proposing these comparisons. We added controls for task design, data source, supervision, and online updates.

1. Does task calibration contribute beyond selecting successful runs?

Q1. Do calibrated tasks produce better training data than ordinary synthetic tasks under the same selection and training rules? (W1)

Calibration improves the resulting agents under the same success filter. We start from the same candidate seeds and mechanism distributions and keep the input transformations and task descriptions fixed. The task-construction changes are parameter repair and task admission. We use the same rollout teachers and sampling budget, and select successes in both groups using the common ordinary-reference threshold τ = max(s − σ_s, w + 0.3|s − w|). Here s and w are ordinary- and weak-reference utility, and σ_s is the ordinary reference's bootstrap standard deviation. Each Qwen3.6-27B SFT group uses 256 distinct histories with matched supervised tokens and optimizer steps.

Both groups therefore train on successes selected by the same rule. The difference tests what calibration changes about the tasks supplying those successes. It yields +0.0056 DSBench RPG and +0.0181 SWE SR, with 451 correct SWE patches in each group.

Task calibration under common success filtering (Table 24, seed 42)

Training conditionDSBench RPGSWE SRSWE correct / 498
Ordinary synthetic + common filtering0.41270.1874451
Calibrated synthetic + common filtering0.41830.2055451

The 214/300 figure counts tasks with positive structural benefit. The 26/422 figure is the share of automatic candidates from structural families. Input recovery measures a separate aspect of difficulty. Recovering the inputs improves the reference on 68/86 controls (Table 7).


2. What does the new IterLab collection add?

Q2. How much do the main collection and supplemental histories contribute when the training volume is held fixed? (W3)

The new histories add value at a fixed training volume. The original Full SFT combines both sources as shown in Table 15. In the controlled comparison, we train on only main histories, only supplemental histories, or a mixture of 128 histories from each source. The SFT groups match 256 distinct histories, supervised tokens, and optimizer steps.

Original Full SFT data split (Table 15)

SourceExportedUsed for updatesLoss monitoring
Main4134085
Supplemental2692681
Total6826766

Data-source comparison (Table 24, seed 42)

Training conditionDSBench RPGSWE SRSWE correct / 498
Base0.39650.0893447
Generic tool-use histories0.41380.2362450
Supplemental only0.42270.2254452
Main, full histories0.44030.3161455
Main + supplemental (128 each)0.44830.3620457

Main-only SFT improves on supplemental-only SFT by 0.0176 RPG and 0.0907 SR. Replacing half the supplemental histories with main histories raises RPG by 0.0256 and SR by 0.1366 over supplemental-only training. This replacement shows the contribution of the new histories while keeping the total training volume fixed.


Revision: Blue text in §5.3 (pp. 6–7, Table 3) and Appendix G.1 (p. 32, Table 24) reports the matched-budget comparisons. Appendix D.1 (p. 25, Table 15) gives the original Full SFT split. §5.6 (p. 9) and Appendix A.1 (p. 18, Table 7) give the coverage breakdown.

AuthorsOpenReview ↗

Response to reviewer (2/2): Full histories, feedback use, and online RL

3. What do the intermediate steps teach?

Q3. Do intermediate experiments, failures, and revisions add value beyond the final successful solutions? (W4)

Full histories add 0.0075 RPG and 0.0379 SR over final solutions. We use the same gold-qualified source trajectories and match supervised tokens and optimizer steps. In one arm we supervise the final solution. In the other we also supervise the assistant's intermediate messages and actions, with tool outputs in context. In Appendix E, for example, a timeout prompts the agent to narrow its search (p. 27, Table 17). The full history records the outcome and the decision it prompted. The comparison tests the value of supervising that process from the same successes.

Supervision comparison (Table 24, seed 42)

Main-collection supervisionDSBench RPGSWE SRSWE correct / 498
Final solutions0.43280.2782454
Full histories0.44030.3161455

On a fixed SWE-fficiency subset, we also show or hide validation scores while keeping the valid starting patch, available budget, execution feedback, and correctness feedback fixed. Final scores come from independent tests. Subtracting the hidden result from the shown result for each checkpoint measures what it gains from receiving the score.

Validation-feedback diagnostic (Table 26)

Training conditionFeedback hiddenFeedback shownGain
Generic tool-use histories0.20840.21760.0092
Main, final solutions0.24310.26220.0191
Main, full histories0.27310.31870.0456

The full-history checkpoint gains 0.0265 more from seeing the score than the final-solution checkpoint. This shows how supervision on intermediate decisions helps when a model receives performance feedback on an existing repository solution.


4. What does RL add to fresh online SFT?

Q4. Can verifiable rewards improve agents through RL beyond collecting new successful trajectories for SFT? (W2)

Online RL adds 0.0045 RPG and 0.0242 SR over online SFT. Both methods start from the same seed-42 main full-history checkpoint. They match sampling rounds, training task distributions, generated tokens, and update budgets, and use the same submission-validity check. The comparison therefore tests the type of update at a common starting point and budget. Evaluation tasks are held out from training.

Online SFT trains on newly sampled gold-qualified trajectories. RL instead scores valid terminal submissions with r = clip((u − w)/(a − w), 0, 1), where u is terminal utility, w is weak-reference utility, and a is calibration-anchor utility. Utility is the metric for AUC and macro-F1 and negative RMSE for regression. Invalid submissions receive zero. A valid improvement above the weak reference can earn a positive reward even below gold, giving the reward-based update credit for partial progress.

Online comparison from the same checkpoint (Table 27)

Training conditionDSBench RPGSWE SRSWE correct / 498
Main full-history SFT checkpoint0.44030.3161455
Online SFT on new successful trajectories0.44740.3318456
IterLab RL0.45190.3560455

Relative to the shared starting checkpoint, RL gains 0.0116 RPG and 0.0399 SR. The SWE correctness count remains 455/498, compared with 456/498 for online SFT. The gain over that matched online baseline is in the two continuous scores.


Revision: Blue text in §5.3 (pp. 6–7, Table 3) and Appendix G.1 (p. 32, Table 24) reports the supervision comparison. Appendix G.2 (pp. 32–33, Table 26) gives the feedback diagnostic. Appendix G.3 (p. 33, Table 27 and Eq. 12) reports the online comparison and reward.

Reviewer_cmFZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces IterLab, a framework which train agents to do experiment and iteratively refine solutions. The authors generate synthetic tabular prediction problems with known data-generating mechanisms, using paired ordinary and structure-enhanced references to quantify structural improvement headroom and input recovery checks to decouple feature-engineering demands from predictive modeling. Agents explore these tasks within a fixed budget. Full interaction histories exceeding an empirical gold target are screened and used for sft. Across external evaluations, IterLab improves agent performance on DSBench, MLAgentBench, MLE-bench, and the repository-level software optimization benchmark SWE-fficiency.

Critical strengths

By comparing ordinary baselines against structure-enhanced references and validating feature-processing reversibility via recovery checks, the framework quantitatively measures task improvement headroom and separates representation hurdles from predictive complexity.

Critical weaknesses
  1. The synthetic training environment is strictly restricted to small-scale tabular prediction tasks, whereas the primary evaluation tasks involve complex multimodal/textual competitions and large-scale Python code repositories. The paper does not provide sufficient empirical analysis to show whether the transfer stems from generalized scientific exploration strategies or merely from basic instruction format alignment and prolonged tool-use persistence.

  2. The final SFT relies on only 413 main-collection trajectories, and only 6.2% originate from tasks with measurable structural benefits , with 0% success on parity tasks. Tuning 27B and 35B models on a few hundred trajectories where agents mostly solved baseline-level tasks raises valid concerns about sample efficiency, coverage of the intended structural mechanisms, and overfitting.

  3. The empirical evaluation strictly compares fine-tuned checkpoints against their raw base models without benchmarking against relevant agent training and self-improvement methods. Furthermore, the paper lacks essential ablations—such as fine-tuning on uncalibrated synthetic data or supervising only final successful outcomes without intermediate failure histories—making it difficult to attribute the reported gains specifically to task calibration and full-history supervision.

Questions for authors

Given the substantial domain gap between small-scale synthetic tabular data and repository-level engineering or multimodal ML competitions, how do the authors verify that the empirical gains stem from generalizable scientific experimentation behaviors rather than superficial format alignment?

2 comments
AuthorsOpenReview ↗

Response to reviewer (1/2): Does feedback use transfer from tabular tasks to repositories?

Thank you for pressing us to explain what transfers across domains.

1. Feedback-driven revision on repository tasks

Q1. Do the gains reflect a transferable experimental behavior beyond instruction-format alignment or prolonged tool use? (W1; Question 1)

Full-history training improves how much agents gain from a numerical validation score on repository tasks. Table 2 establishes end-to-end transfer across four benchmarks. On SWE-fficiency, both models improve speedup and correctness after Full SFT (Tables 2 and 18).

ModelSWE speedup score, Base → Full SFTCorrect patches, Base → Full SFT
27B0.0893 → 0.5317447/498 → 463/498
35B-A3B0.2841 → 0.4562413/498 → 444/498

The control in Table 26 adds a test of one behavior. We compare each checkpoint with itself on a fixed SWE-fficiency subset, once with the validation performance score available and once with it hidden. The two conditions share the same valid starting solution, remaining execution budget, and execution and correctness feedback. The score is the information we vary. Independent test cases determine the final score. We count every task, keep the initial solution for failed attempts, and score valid regressions as submitted.

The last column measures the benefit of revealing the score. The hidden-score result captures what each checkpoint can achieve with its existing training, instruction format, and ordinary tool feedback. Subtracting this result measures the additional improvement from the score for that same checkpoint. The shared starting patch makes this a test of revising an existing solution.

Training conditionScore hiddenScore shownGain from seeing the score
Generic tool-use histories0.20840.21760.0092
Main, final solutions0.24310.26220.0191
Main, full histories0.27310.31870.0456

For full histories, that difference is 0.3187 − 0.2731 = 0.0456. It is 0.0265 larger than for final-solution SFT and 0.0364 larger than for generic tool-use SFT. The final-solution and full-history models use the same successful training trajectories and match supervised tokens and update steps. Full-history SFT also supervises the assistant's intermediate decisions, with tool outputs retained as context. In the tabular trajectory in Appendix E, for example, the agent narrows its search after a command timeout. The full history records both the result and the response to it.

This is evidence for transfer of feedback use on the SWE-fficiency subset. The available budget is matched. Agents decide how to spend it, so the measured benefit includes changes in the choice and number of attempts after receiving feedback. For the other benchmarks, Table 2 reports end-to-end transfer.


Revision: The blue feedback-use paragraph in §5.3 (p. 7) gives the main result. Appendix G.2 (pp. 32–33, Table 26) gives the shared initial solution, score-visibility conditions, and independent test results. Appendix G.1 (p. 32) details the matched training conditions. The intermediate decisions are illustrated in Appendix E (p. 27, Table 17).

AuthorsOpenReview ↗

Response to reviewer (2/2): Transfer at fixed data volume, coverage, and ablations

2. Transfer with few demonstrations

Q2. Will a small collection with sparse structural coverage transfer to held-out tasks? (W2)

256 main histories improve held-out SWE speedup at each of three training seeds. Full SFT uses 408/413 main and 268/269 supplemental histories for updates. In the new Qwen3.6-27B comparison, we replace 256 generic tool-use histories with 256 main full histories, pairing by task ID and matching supervised tokens and update steps. DSBench and SWE-fficiency measure transfer to external tasks at this fixed data volume.

Per-seed evaluation (Table 25). Correct counts are out of 498 SWE-fficiency tasks.

SeedTraining conditionDSBench RPGSWE SRSWE correct
42Generic tool-use histories0.41380.2362450
42Main, full histories0.44030.3161455
43Generic tool-use histories0.38880.2319450
43Main, full histories0.42670.3023455
44Generic tool-use histories0.43310.2355453
44Main, full histories0.41900.2888455

SWE SR gains are +0.0799, +0.0704, and +0.0533, with a mean of +0.0679. DSBench RPG gains +0.0168 on average, with a −0.0141 change at seed 44. At seed 42, both groups solve 425 SWE tasks correctly, main alone solves 30, generic alone solves 25, and neither solves 18. These are held-out results at the tested budget of 256 histories.

Structural coverage is sparse. The 6.2% is 26/422 automatic candidates from structural families. Those families supply 25/413 retained histories, and parity supplies zero. The 408 main histories used for updates span 100 tasks. Retained histories pass the quality and completion checks. H quantifies the reference gain from mechanism features, and equals zero on controls because both references see the same inputs. Recovery improves the reference on 68/86 controls, identifying a separate opportunity for input processing.

Coverage by family (Table 7)

FamilyTasksPositive HAutomatic candidates
Polynomial380173
Threshold290115
Saturating190108
Hidden-periodic11211216
Rotated-radial606010
Parity42420
Total300214422

3. Task calibration and full-history supervision

Q3. Which controls separate task calibration and full histories from general agent tuning or training on successes? (W3)

The two ablations change different parts of training. For task calibration, ordinary and calibrated groups use the same candidate seeds, mechanisms, transformations, descriptions, teachers, sampling budgets, and success filter. Changing parameter repair and task admission jointly adds 0.0056 RPG and 0.0181 SR. For supervision, final-only and full-history SFT start from the same gold-qualified trajectories and match supervised tokens and updates. Both use successful solutions. The full-history objective also supervises the assistant's earlier decisions and actions, with tool outcomes as context. Its gains of 0.0075 RPG and 0.0379 SR measure the value of including that process under the same success selection.

Matched-budget SFT (Table 24, seed 42). Generic tool-use SFT controls for fine-tuning on tool histories. The supplemental-only and 128+128 mixed groups separate the contribution of the new collection from that of the existing data.

Training conditionDSBench RPGSWE SRSWE correct / 498
Base0.39650.0893447
Generic tool-use histories0.41380.2362450
Supplemental only0.42270.2254452
Ordinary synthetic + common filtering0.41270.1874451
Calibrated synthetic + common filtering0.41830.2055451
Main, final solutions0.43280.2782454
Main, full histories0.44030.3161455
Main + supplemental (equal shares)0.44830.3620457

Online self-improvement (Table 27). Starting from the same full-history checkpoint, we match online SFT and terminal-reward RL on sampling rounds, task distribution, generated tokens, and updates. RL adds 0.0045 RPG and 0.0242 SR over SFT on newly sampled successes.

UpdateDSBench RPGSWE SRSWE correct / 498
Shared starting checkpoint0.44030.3161455
Online SFT0.44740.3318456
Online RL0.45190.3560455

Revision: Blue text in §5.3 (pp. 6–7, Table 3) and Appendices G.1 and G.3 (pp. 32–33, Tables 24, 25, and 27) reports the comparisons. §5.6 (p. 9) and Appendix A.1 (p. 18, Table 7) report coverage. Appendix D.1 (p. 25, Table 15) gives the Full SFT data split.

Reviewer_aQi6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary: Problem. To train an agent to experiment, you need tasks where experimenting pays. But an agent's score can't tell you whether a task was hard to model, thin on evidence, or just badly formatted. Existing task generators give no way to separate these. Approach. IterLab generates prediction tasks with known generating mechanisms, so the headroom is measurable. Stage A calibrates the problem; Stage B degrades the inputs while recording how to undo it. Agents see only the degraded data. All calibration stays private. Successful full histories become SFT data.

Contributions: The paper makes four contributions.

  1. A calibration procedure for synthetic prediction tasks. The same predictor portfolio is fitted with and without features derived from the known generating mechanism, giving a per-task measure of structural benefit (H). Separately, each input transformation is applied with its inverse recorded, and the portfolio is refitted on both the delivered and recovered inputs, giving a measure of representation loss. Admission requires recovered performance to remain within a fixed tolerance of the original.
  2. A per-instance quality target. The target is derived from the stronger of the two fitted references, adjusted by its bootstrap dispersion and floored relative to a weak baseline, so that the bar reflects the headroom measured on each task rather than a fixed global threshold.
  3. A controllable task generator and a 300-task collection. Six mechanism families, three of which act as controls, are crossed with thirteen statistical and representation parameters, with bounded parameter repair and rejection of infeasible configurations. Task descriptions are generated by a separate model with no access to the data or calibration state.
  4. A pipeline for training on complete experimental histories. Supervision is applied to assistant text and tool actions, with prompts and tool results masked but retained in context, and intermediate command failures preserved. Screening separates prediction validity from training eligibility, and the resulting corpus is analysed by structural benefit to characterise which generated tasks contribute supervision.

Validation:

  1. External evaluation covers four benchmarks under their published protocols: DSBench (74 tasks, one run), MLAgentBench (13 tasks, eight runs), MLE-bench (all 75 competitions, three runs, 24-hour limit), and SWE-fficiency (all 498 tasks, pass@1, matched environments and correctness checks). Both models improve on all four. MLE-bench medal rates rise by 15.56 and 14.67 points, MLAgentBench success by 12.50 and 14.42, DSBench relative progress by 0.084 and 0.161, and SWE-fficiency speedup score by 0.443 and 0.172 with correctness rising 3.21 and 6.22 points. Appendix F reports success counts against effective denominators and confirms no missing or excluded evaluations.
  2. The calibration is validated directly. Structure-enhanced references exceed ordinary references on 214 of 300 tasks, recovery improves over delivered inputs on 232 of 300, median recovery loss is zero within each metric, and all 300 tasks pass the retention gate. Appendix A.4 reports a worked instance in which recovery restores the ordinary reference to within 0.000008 AUC.
  3. Supervision yield is reported in full: 6,000 requested episodes, 3,881 valid scores, 916 gold passes, 448 passing the score and completion checks, 422 automatic candidates, 413 retained after review, with per-model and per-metric breakdowns and wall-time exit counts. Agent success is reported by mechanism family and committed parameter setting, with requested episodes retained in the denominator.
  4. Statistical treatment uses three rollouts per internal task, repeats averaged within task before pairing, and task-bootstrap intervals from 10,000 paired draws. Training uses a single seed, which the authors note limits the intervals to task-sampling uncertainty. A separate matched-budget experiment compares one continuous 30-minute run against twe held-out tasks.
Critical strengths
  1. The recovery check is a sound, verified, reusable instrument. Unlike structural benefit, recovery loss is well-posed: the same portfolio on original versus restored features, no privileged information in the comparison. It separates input-processing difficulty from modelling difficulty, and the paper verifies it rather than asserting it, recovery returns the reference to within 0.000008 AUC on the worked example, median loss is zero within each metric, all 300 tasks pass the gate. Consequential because it solves a real problem for anyone building obfuscated tabular agent tasks: knowing the obfuscation preserved the signal. It is the component I can verify is correct, and it stands independently of the training claims.

  2. The SWE-fficiency study has a very large sample size and is adequately powered. The speedup was 0.089 to 0.532 and 0.284 to 0.456, while the correctness also increased (447 to 463 and 413 to 444 out of 498). All 498 tasks, nine repositories, pass@1, matched environments. Speed and correctness are generally inversely related. This is significant because the transfer distance is long, from synthetic tabular prediction to repository optimization, and this is the only evaluation in this paper with a sample size that is large enough to show an effect that is clearly above the level of noise. If the attribution issue below is resolved, then this will be sufficient for acceptance.

Critical weaknesses
  1. Gold is based on a privileged-information reference; therefore, on high-H tasks, the bar cannot be seen to be attainable from the data provided to the agent. Equation 1 bases gold on the structure-enhanced reference when H > 0. The reference is created using mechanism-derived features such as the rotation, the hidden projection, and the parity subset; however, none of these features will ever be seen by the agent. In the worked example, gold is 0.132 AUC higher than what the ordinary portfolio achieves on correctly parsed inputs. This reframes the primary empirical finding of the paper. The 1.6% gold rate, the 6.2% candidate share, and zero passes across 840 parity attempts are consistent with weak agents, but they are also consistent with a target set higher than what the public data supports. Section 3.2 specifies that H measures what the fitting procedure gains from privileged features; however, Equation 1 then uses H to establish the agent's bar. Appendix A.2 also reports a Bayes diagnostic over 60,000 Monte Carlo samples and states that it "does not determine gold or trigger repair," so an achievability estimate exists and is rejected for exactly this reason. This is significant because calibration is listed as the first contribution and Section 5.4's difficulty characterization is presented as the headline result. If gold is partially unattainable by definition, that result measures the target rather than the agents.

  2. There is nothing that separates the method from the supplemental data or from generic agentic fine-tuning, and the internal evaluation is null. Of 682 training histories, 269 are not from the main collection, including 258 from an earlier corpus that contained a row-order signal. Both runs use all of it with no separating ablation. The only comparison is against Base checkpoints, with no arm trained on an equal amount of generic agentic histories, which is important considering how low the Base models begin. Meanwhile the 30 held-out IterLab tasks, the cohort most similar to the training distribution, moved by only -1.67 and +1.67 points with intervals spanning zero, while external benchmarks moved by 12 to 44 points. The paper mentions the discrepancy in a single sentence. This is significant because, combined with the first weakness, the headline table cannot be attributed to the method at any level. Both fixes require new training runs rather than re-analysis.

Questions for authors
  1. Is the H ≤ 0 stratum identical to the control families? - Appendix A.1 lists 86 control tasks (38 polynomial, 29 threshold, 19 saturating), which is exactly the number of H ≤ 0 tasks listed in Table 3. The 396 candidates listed in Appendix A.1 also match the number of candidates in this stratum, while the other 26 candidates are from periodic and radial tasks. Since Appendix A.2 states that controls may not have any privileged features, is H effectively zero by design for these tasks? Please provide a breakdown of each family by H. If the identity is true, then the 6.2% result is mainly due to a comparison between control and structural families rather than a separate measure of the relationship between headroom and demonstration yield.
  2. Does test-based estimator selection introduce bias into H and gold? - Section 3.2 selects the best test score of a single predictor and stacking ensemble in each bootstrap replicate. This introduces upward selection bias, which may differ between ordinary and structure-enhanced references. How sensitive are H, the 214 positive-H tasks, and gold to this choice? Please report results using each estimator individually. Also, is the test split used for reference selection the same private split used to grade agents?
  3. Are the structural-family gold targets attainable from public features? - Appendix A.2 computes a Bayes diagnostic but does not use it to determine gold. Could the authors report this diagnostic along with gold for parity, hidden-periodic, and rotated-radial tasks? In the worked periodic example, gold exceeds the recovered ordinary reference by 0.132 AUC. Without evidence that these targets are attainable from public inputs within the available data and budget, it is difficult to distinguish agent limitations from overly demanding calibration.
  4. What proportion of the external gains is attributable specifically to IterLab? - Full SFT uses 413 main-collection histories and 269 supplemental histories, but only Base versus Full SFT is evaluated. Please report a main-collection-only ablation and, ideally, a matched-volume baseline using existing screened agentic trajectories. Otherwise, the contributions of calibrated task generation, supplemental data, and generic agentic SFT remain confounded.
  5. Why does held-out IterLab performance remain unchanged despite large external gains?- Internal gold success changes by only −1.67 and +1.67 percentage points, with intervals including zero, despite much larger improvements on external benchmarks. Could the authors report internal results by mechanism family, including valid-submission and ordinary-reference attainment rates? Is this a floor effect caused by parity and periodic tasks, or do the external gains primarily reflect improved general tool use rather than better experimentation?
  6. How robust are the gold definition and admission thresholds? - Equation (1) uses max(a − σa, w + 0.3|a − w|), making gold partly dependent on reference-estimation variance. What motivated the 0.3 coefficient, and how sensitive are results to it? Similarly, the admission threshold of 0.05 uses native metric units across AUC, macro-F1, and RMSE. Could the authors report sensitivity to scale-normalized thresholds, particularly for regression tasks?
  7. How are the reported training steps reconciled with the recipe?- Appendix D reports 21.4M training tokens, eight epochs, packing lengths of 131,072/98,304, and final steps of 664/440 for 27B/35B-A3B. These counts cannot be reconstructed from the stated settings alone. Please clarify effective global batch sizes, packing efficiency, data-parallelism, and tokens consumed per optimizer step.
  8. How do trajectory provenance and shortcut screening affect the training set?- Of the 413 retained main histories, 305 were generated by 27B and 108 by 35B-A3B. Were both models trained on this same mixture, and how does performance change with own-model-only supervision? Additionally, 258 supplemental histories came from an earlier corpus with a known row-order signal. Were these retrospectively screened using the same criteria as the main collection, and how many were flagged or excluded?
3 comments
AuthorsOpenReview ↗

Response to reviewer (1/2): Gold attainment, reference sensitivity, and structural evaluation

Thank you for the questions about calibration and evaluation.

1. Gold attainability

Q1. Can a solver reach gold using public inputs, and how does gold compare with Bayes? (W1, Question 3)

A public-input solver exceeds gold on 10/214 structural tasks. It uses public task descriptions and training data, three-fold validation for model selection, and one locked submission scored by an isolated grader. Each task gets 30 minutes, 4 CPUs, and 16 GiB. Timeouts and invalid submissions count in the denominator.

Gold comes from fitted references. The Bayes diagnostic uses the known mechanism and approximates observation noise. Passes strictly exceed each task's original gold in utility, with higher AUC and macro-F1 and lower RMSE preferred.

Public-input gold attainment (Table 11)

FamilyMetricMedian goldMedian BayesPasses/tasks
Hidden-periodicAUC0.805420.812380/14
Hidden-periodicMacro-F10.286530.338874/45
Hidden-periodicRMSE1.574081.227503/53
Rotated-radialAUC0.794900.833340/16
Rotated-radialMacro-F10.288820.350102/24
Rotated-radialRMSE2.408402.092571/20
ParityAUC0.768130.791410/42
Total10/214

The solver passes seven hidden-periodic tasks and three rotated-radial tasks. It passes 0/6 AUC tasks whose gold exceeds the approximate Bayes value. Counting only valid above-gold first attempts within 30 minutes, 27B passes 3/214 and 35B-A3B passes 1/214.


2. Reference selection and thresholds

Q2. How sensitive are structural benefit and gold to reference selection and thresholds? (W1, Questions 2 and 6)

All three alternative rules give 214 positive-H tasks while gold pass counts vary. The original rule picks the higher-utility single model or stack on bootstrap rows from the same private test split used to grade agents. We recompute references and gold on the original tasks and rescore archived submissions. Gold shifts are normalized by the original weak-to-anchor utility gap and summarized by the median. RMSE utility is negative RMSE.

Reference selection sensitivity (Table 12)

SelectionH > 0 tasks / 300Median normalized gold shiftPasses / 6,000 requests
Original private-test selection2140916
Single model only214−0.0213977
Stacking ensemble only214−0.0147964
Training-set internal validation214−0.0133959

The gold coefficient was fixed at 0.3 before collection, requiring at least 30% of the weak-to-anchor improvement. At 0.4 and 0.5, passes fall to 905 and 893, with changes confined to regression. Replacing the native-unit 0.05 admission threshold with a normalized 5% threshold retains 100/100 AUC, 100/100 macro-F1, and 81/100 RMSE tasks. Classification normalizes by the gap from the weak baseline to the maximum score. Regression uses relative RMSE reduction from the weak baseline.


3. Structural coverage and internal evaluation

Q3. What does 6.2% structural coverage mean, and how should we read the internal results? (W1, W2, Questions 1 and 5)

H ≤ 0 covers the 86 control tasks. Their paired reference fits use identical inputs, giving H = 0. Structural families supply 26/422 candidates (6.2%) after automatic screening and before final review, 25/413 reviewed main histories, and 25/408 main histories used for updates. Input recovery improves the reference on 68/86 controls.

Structural benefit and candidate coverage (Table 7)

FamilyTasksH > 0Automatic candidates
Polynomial380173
Threshold290115
Saturating190108
Hidden-periodic11211216
Rotated-radial606010
Parity42420
Total300214422

All 30 held-out internal tasks are structural with gold above the ordinary reference. There are two runs per task. Attainment uses all requests. Arrows show Base → Full SFT.

Internal evaluation by family (Table 20)

ModelFamilyRequests/conditionValidOrdinary passesGold passes
27BHidden-periodic2828→2719→93→2
27BRotated-radial2222→2116→131→1
27BParity1010→103→40→0
35B-A3BHidden-periodic2822→245→100→1
35B-A3BRotated-radial2215→219→121→1
35B-A3BParity108→103→50→0

Ordinary-reference passes change from 38 to 26 for 27B and 17 to 27 for 35B-A3B. Gold passes change from 4 to 3 (−1.67 pp, paired 95% CI [−5.00, 0.00]) and from 1 to 2 (+1.67 pp, [0.00, 5.00]). Both intervals include zero. Q5 in our second reply tests feedback use on SWE-fficiency.


Revision: Blue changes cover gold attainment (§5.4, pp. 7–8, A.5, pp. 21–22, Table 11), the shared split (§3.2, p. 4, A.2, p. 20), sensitivity (A.6, p. 22, Table 12), coverage (§5.6, p. 9, Table 4, A.1, p. 18, Table 7), and internal results (§5.2, p. 6, F.2, pp. 28–29, Table 20).

AuthorsOpenReview ↗

Response to reviewer (2/2): Training gains, feedback use, and reproducibility

4. Matched-budget SFT

Q4. What do the main collection, task calibration, and full histories contribute? (W2; Question 4)

The main collection outperforms generic and supplemental-only training at matched budgets. Each Qwen3.6-27B SFT group uses 256 distinct histories with matched supervised tokens and optimizer steps. We pair the primary main-versus-generic comparison by task ID. The mixed group uses 128 histories from each source. Evaluation uses the original 74 DSBench and 498 SWE-fficiency tasks.

Matched-budget SFT (Table 24, seed 42)

Training conditionDSBench RPG ↑SWE SR ↑SWE correct
Base0.39650.0893447/498 (89.76%)
Generic tool-use histories0.41380.2362450/498 (90.36%)
Supplemental only0.42270.2254452/498 (90.76%)
Ordinary synthetic + common filtering0.41270.1874451/498 (90.56%)
Calibrated synthetic + common filtering0.41830.2055451/498 (90.56%)
Main, final solutions0.43280.2782454/498 (91.16%)
Main, full histories0.44030.3161455/498 (91.37%)
Main + supplemental, equal shares0.44830.3620457/498 (91.77%)

Replacing half the supplemental histories with main histories raises RPG and SR by 0.0256 and 0.1366. Calibration adds 0.0056 and 0.0181 with teachers, sampling budget, and filtering fixed. On the same gold-qualified trajectories, full histories add 0.0075 and 0.0379 over final solutions.

Main versus generic histories across seeds (Table 25)

SeedRPG genericRPG mainSR genericSR mainCorrect / 498 genericCorrect / 498 main
420.41380.44030.23620.3161450455
430.38880.42670.23190.3023450455
440.43310.41900.23550.2888453455

SWE SR improves on all three seeds (+0.0799, +0.0704, +0.0533). Mean gains are +0.0168 RPG and +0.0679 SR. Seed 44 shows −0.0141 RPG.


5. Feedback use on SWE-fficiency

Q5. Do the external gains reflect better use of feedback? (W2; Question 5)

The full-history group gains most from validation feedback on the fixed SWE-fficiency subset. We vary access to validation scores with the same valid initial solution, remaining budget, and execution and correctness feedback. Independent test cases score every task. Failed attempts retain the initial solution, and valid regressions count.

Validation-feedback diagnostic (Table 26)

Training conditionFeedback hiddenFeedback shownGain
Generic tool-use histories0.20840.21760.0092
Main (final solutions)0.24310.26220.0191
Main (full histories)0.27310.31870.0456

Full histories gain 0.0364 more from feedback than generic histories and 0.0265 more than final solutions.


6. Teacher sources and row order

Q6. Who generated the histories, and how were earlier row-order shortcuts checked? (Question 8)

Both models use the same teacher mixture. The 27B and 35B-A3B teachers generated 305 and 108 main histories, with 301 and 107 used for updates. At matched tasks and supervised tokens, 27B's own trajectories lower RPG and SR by 0.0040 and 0.0131 relative to the mixture.

Full SFT data split (Table 15)

SourceExportedTrainingLoss monitoring
Main4134085
Supplemental2692681
Total6826766

After row reordering, 254/258 earlier trajectories give the same predictions and four differ. Manual review traced the sole row-index flag to data alignment and kept that trajectory eligible.


7. Packing and optimizer steps

Q7. How do packing and batch sizes yield 664 and 440 optimizer steps? (Question 7)

Eight epochs give 664 optimizer steps for 27B and 440 for 35B-A3B. Both use 8 GPUs, microbatch size 1, and gradient accumulation 1. The final 27B batch in each epoch is smaller.

Packing and optimizer steps (Table 16)

Setting27B35B-A3B
Packing length131,07298,304
Sequence / data parallelism4 / 22 / 4
Packed sequences per epoch165220
Packed sequences per full batch24
Optimizer steps per epoch⌈165/2⌉ = 83220/4 = 55
Optimizer steps over 8 epochs664440
Packing utilization≈99.05%≈99.05%
Context tokens per epoch21,420,75621,420,756
Supervised tokens per epoch11,352,12611,352,126
Approx. context tokens per full batch259,646389,468

Revision: Changes are in blue. §5.3 (pp. 6–7, Table 3 Panel A) and Appendix G.1 (p. 32, Tables 24–25) report training comparisons. Appendix G.2 (pp. 32–33, Table 26) gives the feedback diagnostic. Appendix D.1 (pp. 25–26, Table 15) gives data provenance and row-order review. Appendix D.2 (p. 26, Table 16) details packing and optimizer steps.

Thanks for the additional experiments to clarify my concerns. I would like to maintain my score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal and Revision Summary: Core Contribution and Further Validation

We thank all reviewers for their time and helpful feedback.

The original submission already demonstrates that IterLab constructs tasks with measurable room for improvement and that training on complete experimental histories improves performance on external benchmarks. Paired references quantify structural benefit, and recovery checks quantify input-processing difficulty (original §§3.2–3.3 and Figure 3). Mechanism features improve reference performance on 214/300 tasks, and input recovery improves the ordinary reference on 232/300. In the original Table 2, Full SFT combines main and supplemental histories and improves both Qwen3.6 models on all four external benchmarks. On SWE-fficiency, speedup scores rise from 0.0893 to 0.5317 for 27B and 0.2841 to 0.4562 for 35B-A3B.

The additional experiments address questions about attribution and scope. They test which training choices contribute to the observed gains, how agents use validation feedback on repository tasks, and when a solver can reach gold from public inputs.

  • Calibration and full-history supervision each improve held-out scores. With 256 histories per group and matched supervised tokens and updates, 27B main-only SFT outperforms generic tool-use and supplemental-only SFT at seed 42. With the same teachers, sampling budget, and success filter, task calibration adds 0.0056 DSBench RPG and 0.0181 SWE speedup score. On the same successful trajectories, full histories add 0.0075 and 0.0379 over final-solution SFT.
  • The feedback experiment measures a specific behavior on repository tasks. On a fixed SWE-fficiency subset, we show or hide validation scores with the initial solution and available budget held fixed. Revealing the score improves independent-test results by 0.0456 for full histories, 0.0191 for final solutions, and 0.0092 for generic tool-use histories. The generated tasks also support online RL, which raises DSBench RPG and SWE speedup relative to matched online SFT.
  • The calibration checks characterize the empirical gold targets. Three alternative reference rules each yield 214 tasks with positive structural benefit. A fixed solver exceeds gold on 10/214 structural tasks using public inputs under a 30-minute budget per task, with 0/42 on parity. This test examines attainment on the generated tasks. The transfer results use separate external benchmarks.

Changes in the Revised Manuscript

The main additions are marked in blue.

  • Training ablations and repeated runs. §5.3 and Table 3, Panel A (pp. 6–7), compare data sources, calibrated and ordinary synthetic tasks, and full histories and final solutions. Appendix G.1 (Tables 24–25, p. 32) gives the matched-budget setup and the three-seed comparison of main-only and generic tool-use histories.
  • Feedback use and online RL. §5.3 (p. 7) summarizes the score-visibility experiment. Appendix G.2 (Table 26, pp. 32–33) reports results with validation scores shown and hidden. Table 3, Panel B (p. 6), and Appendix G.3 (Table 27, p. 33) compare online RL with matched online SFT from the same checkpoint.
  • Gold attainment and sensitivity. §3.2 (p. 4) clarifies that reference estimation and agent grading use the same private test split. §5.4 (pp. 7–8) and Appendix A.5 (Table 11, pp. 21–22) give the public-input solver protocol and attainment by mechanism and metric. Appendix A.6 (Table 12 and accompanying text, p. 22) tests alternative reference-selection rules, gold coefficients, and admission thresholds.
  • Coverage by task family. §5.6 (p. 9) clarifies structural and input-processing coverage alongside the demonstration-yield results in Table 4. Appendix A.1 adds a family-level breakdown of task counts, positive structural benefit, and automatically screened candidates (Table 7, p. 18).
  • Training data and recipe. Appendix D.1 (Table 15, pp. 25–26) details the main and supplemental data split, teacher sources, and row-order review. Appendix D.2 (Table 16, p. 26) reports packing, effective batch sizes, and optimizer steps.
  • Held-out internal results. §5.2 (p. 6) clarifies the composition of the held-out cohort. Appendix F.2 (pp. 28–29) adds valid-submission counts, ordinary-reference attainment, and gold attainment by mechanism family (Table 20).
AuthorsOpenReview ↗

Looking Forward to Your valuable Feedback — A Brief Follow-Up

Thank you again for taking the time to review our paper. If time permits, we’d be grateful for any feedback on our rebuttal.

With less than one day remaining in the discussion period, we’d be happy to address any remaining questions or concerns before it closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.