- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether language agents can learn control responsibilities that are initially handled by external harnesses. The authors introduce Harness Annealing Training (HAT), which combines explicit control supervision with a curriculum over trajectories collected under weaker harness configurations. The results show that selected intermediate checkpoints improve performance under tools-only deployment.
Critical strengths
- I think the paper addresses an interesting and important question: whether external control mechanisms can become unnecessary as models improve through harness-supported experience. Also, the evaluation of each checkpoint under all four harness configurations is very informative.
Critical weaknesses
- The experiments do not distinguish this mechanism from the effects of additional training, changes in data distribution, or differences in supervision construction. Each successive checkpoint receives additional optimization steps and different training mixtures.
- The primary evaluation uses a reference-based judge that observes only the final answer, without access to the interaction trajectory. I don't think the evidence convincingly establish that the model has internalized the targeted control decision.
Questions for authors
- Can the authors isolate the benefits of the annealing curriculum from additional training and data exposure?
- Could the authors evaluate without teacher-forced reasoning, such as some instances where an initial answer is unsupported or additional investigation is required?
5 comments
Thank you for the response. Supervision construction is clear to me now, and the new trajectory analysis addresses my Q2. However, my Q1 is still not fully addressed. Non-monotonic performance does not rule out the effect of extra training. For example, performance may first improve and then drop due to overfitting; later stages use different data mixtures, which could also explain why performance drops... Also, Table 4 compares models trained from Base with HAT checkpoints that receive additional training and different data mixtures. Without matching these factors, it is not clear to me whether the gains come from annealing.
Thank you for clarifying this concern.
To address this more directly, we conducted additional controlled experiments with the 9B model. All three conditions start from the same H4-trained checkpoint:
- Ordered (HAT): the H2-targeted checkpoint reported in the paper.
- Continue-H4: continues training on H4-source data, matching the number of optimizer updates and training records.
- Shuffled: pools the original H3-targeted and H2-targeted stage datasets, retaining replay and cue-dropout examples, and shuffles the records before splitting them into two training stages.
The tools-only deployment results on SWE-QA are:
| Training condition | SWE-QA, H1 |
|---|---|
| Ordered | 77.00 |
| Continue-H4 | 70.98 |
| Shuffled | 72.25 |
All three conditions are evaluated on the same complete set of 720 SWE-QA questions using GPT-5.5 and the full reference-based judge prompt reproduced in our paper appendix.
On SWE-QA under tools-only deployment, Ordered exceeds Continue-H4 by 6.02 points and Shuffled by 4.75 points. This provides supporting evidence for the ordered curriculum in this specific setting.
We thank the reviewer for raising this question. We show below that additional training, changes in the data distribution, and differences in supervision construction each fail to explain, on its own, the performance gains of HAT.
Additional training. If the gains came mainly from accumulated optimization steps, performance would increase monotonically with training, but Table 3 does not show this pattern. For example, on SWE-QA with the 9B model, H3-targeted reaches 77.41 under H1 after only a single annealing stage, outperforming the later, more extensively trained stages. Performance does not grow with training; instead, gains appear at specific stages. This indicates that the gains come from the training signal introduced at each stage, not from additional optimization steps.
Changes in the data distribution. The independently trained Layered checkpoints in Table 4 are each trained from Base on data from a single source harness. This separates exposure to a given harness level's data from going through the annealing procedure. The ablation shows:
Exposure to weaker-harness data alone does not improve performance under weak support. Models trained only on H2- or H3-source data perform below the H4 starting checkpoint under tools-only deployment. Aligning the training and deployment distributions does not reproduce the gains of HAT either. H2-targeted never sees tools-only-source data during training, yet under tools-only deployment it outperforms the model trained directly on that source. This advantage holds consistently across both model scales and both benchmarks.
Therefore, the gains of HAT come neither from more weak-harness data nor from matching the training and deployment distributions. They come from the procedure of progressively withdrawing support during training.
Differences in supervision construction. All stages of the annealing chain use the same Layered supervision construction, so performance differences among checkpoints within the chain are unrelated to supervision construction. The effect of supervision construction itself is isolated by the Direct vs. Layered ablation in Table 4. Under tools-only deployment, Layered supervision consistently and substantially outperforms Direct supervision across all source levels, model scales, and benchmarks. This shows that the effect of Layered supervision is independent and robust, and that the annealing gains add to it rather than being confounded with it.
In summary, the weak-support gains of HAT cannot be attributed to additional training, data exposure, or differences in supervision construction.
We thank the reviewer for this suggestion. To address this, we analyze the existing tools-only (H1) evaluation trajectories. These trajectories are generated freely by each model without teacher-forced reasoning, and no external state, workflow, or verification controller is present. We use gpt-5.5 as judge to annotate the control-related behaviors in each trajectory.
On the same questions, the fraction of trajectories that exhibit subgoal tracking and integration rises from 0.2667 for Base to 0.5000 for H3-targeted. Under H1, the task organization normally supplied by the workflow layer is entirely absent. H3-targeted is the stage that targets workflow control, so this increase is consistent with the model assuming the organizational responsibility previously provided by that layer. Appendix G illustrates this behavior: with workflow control removed, H4-trained repeats an unresolved search until the loop guard terminates the run, whereas H3-targeted switches its search strategy and returns an answer.
Overall, these free-running trajectories provide behavioral evidence of partial harness internalization that complements answer-level evaluation.