acceptodds
Under review as a conference paper at ICLR 2027

Harness Annealing: Learning to Act with Less External Control

Abstract

Language agents rely on external harnesses to track state, organize workflows, and verify answers. Beyond providing tools and information, these harnesses supply control decisions about what to investigate, whether to revise, and when to stop. Training on successful harness-supported trajectories can improve task performance while leaving these decisions dependent on runtime intervention. We ask whether harness-supported experience can also teach the model to make these decisions, allowing the division of control to change as the model learns. We call this objective harness internalization: learning to assume specified control responsibilities while retaining task performance after the corresponding support is withdrawn. We introduce HARNESS ANNEALING TRAINING (HAT), which combines explicit control supervision with a curriculum over teacher trajectories collected under progressively weaker harnesses. Experiments with 9B and 35B models on SWE-QA and SWE-QA-Pro evaluate every checkpoint under four deployment harnesses. Selected annealed checkpoints operating with tools alone achieve scores close to those of their respective starting checkpoints deployed with the full harness. The benefits vary with model scale and deployment configuration, and further annealing does not uniformly improve performance. These findings suggest that harness-supported experience can help reduce the runtime control required by a trained agent.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM Agent, Harness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1gFjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether language agents can learn control responsibilities that are initially handled by external harnesses. The authors introduce Harness Annealing Training (HAT), which combines explicit control supervision with a curriculum over trajectories collected under weaker harness configurations. The results show that selected intermediate checkpoints improve performance under tools-only deployment.

Critical strengths
  1. I think the paper addresses an interesting and important question: whether external control mechanisms can become unnecessary as models improve through harness-supported experience. Also, the evaluation of each checkpoint under all four harness configurations is very informative.
Critical weaknesses
  1. The experiments do not distinguish this mechanism from the effects of additional training, changes in data distribution, or differences in supervision construction. Each successive checkpoint receives additional optimization steps and different training mixtures.
  2. The primary evaluation uses a reference-based judge that observes only the final answer, without access to the interaction trajectory. I don't think the evidence convincingly establish that the model has internalized the targeted control decision.
Questions for authors
  1. Can the authors isolate the benefits of the annealing curriculum from additional training and data exposure?
  2. Could the authors evaluate without teacher-forced reasoning, such as some instances where an initial answer is unsupported or additional investigation is required?
5 comments

Response to W1 & Q1

We thank the reviewer for raising this question. We show below that additional training, changes in the data distribution, and differences in supervision construction each fail to explain, on its own, the performance gains of HAT.

  1. Additional training. If the gains came mainly from accumulated optimization steps, performance would increase monotonically with training, but Table 3 does not show this pattern. For example, on SWE-QA with the 9B model, H3-targeted reaches 77.41 under H1 after only a single annealing stage, outperforming the later, more extensively trained stages. Performance does not grow with training; instead, gains appear at specific stages. This indicates that the gains come from the training signal introduced at each stage, not from additional optimization steps.

  2. Changes in the data distribution. The independently trained Layered checkpoints in Table 4 are each trained from Base on data from a single source harness. This separates exposure to a given harness level's data from going through the annealing procedure. The ablation shows:

    Exposure to weaker-harness data alone does not improve performance under weak support. Models trained only on H2- or H3-source data perform below the H4 starting checkpoint under tools-only deployment. Aligning the training and deployment distributions does not reproduce the gains of HAT either. H2-targeted never sees tools-only-source data during training, yet under tools-only deployment it outperforms the model trained directly on that source. This advantage holds consistently across both model scales and both benchmarks.

    Therefore, the gains of HAT come neither from more weak-harness data nor from matching the training and deployment distributions. They come from the procedure of progressively withdrawing support during training.

  3. Differences in supervision construction. All stages of the annealing chain use the same Layered supervision construction, so performance differences among checkpoints within the chain are unrelated to supervision construction. The effect of supervision construction itself is isolated by the Direct vs. Layered ablation in Table 4. Under tools-only deployment, Layered supervision consistently and substantially outperforms Direct supervision across all source levels, model scales, and benchmarks. This shows that the effect of Layered supervision is independent and robust, and that the annealing gains add to it rather than being confounded with it.

In summary, the weak-support gains of HAT cannot be attributed to additional training, data exposure, or differences in supervision construction.

Response to W2 & Q2

We thank the reviewer for this suggestion. To address this, we analyze the existing tools-only (H1) evaluation trajectories. These trajectories are generated freely by each model without teacher-forced reasoning, and no external state, workflow, or verification controller is present. We use gpt-5.5 as judge to annotate the control-related behaviors in each trajectory.

On the same questions, the fraction of trajectories that exhibit subgoal tracking and integration rises from 0.2667 for Base to 0.5000 for H3-targeted. Under H1, the task organization normally supplied by the workflow layer is entirely absent. H3-targeted is the stage that targets workflow control, so this increase is consistent with the model assuming the organizational responsibility previously provided by that layer. Appendix G illustrates this behavior: with workflow control removed, H4-trained repeats an unresolved search until the loop guard terminates the run, whereas H3-targeted switches its search strategy and returns an answer.

Overall, these free-running trajectories provide behavioral evidence of partial harness internalization that complements answer-level evaluation.

Thank you for the response. Supervision construction is clear to me now, and the new trajectory analysis addresses my Q2. However, my Q1 is still not fully addressed. Non-monotonic performance does not rule out the effect of extra training. For example, performance may first improve and then drop due to overfitting; later stages use different data mixtures, which could also explain why performance drops... Also, Table 4 compares models trained from Base with HAT checkpoints that receive additional training and different data mixtures. Without matching these factors, it is not clear to me whether the gains come from annealing.

Thank you for clarifying this concern.

To address this more directly, we conducted additional controlled experiments with the 9B model. All three conditions start from the same H4-trained checkpoint:

  • Ordered (HAT): the H2-targeted checkpoint reported in the paper.
  • Continue-H4: continues training on H4-source data, matching the number of optimizer updates and training records.
  • Shuffled: pools the original H3-targeted and H2-targeted stage datasets, retaining replay and cue-dropout examples, and shuffles the records before splitting them into two training stages.

The tools-only deployment results on SWE-QA are:

Training conditionSWE-QA, H1
Ordered77.00
Continue-H470.98
Shuffled72.25

All three conditions are evaluated on the same complete set of 720 SWE-QA questions using GPT-5.5 and the full reference-based judge prompt reproduced in our paper appendix.

On SWE-QA under tools-only deployment, Ordered exceeds Continue-H4 by 6.02 points and Shuffled by 4.75 points. This provides supporting evidence for the ordered curriculum in this specific setting.

Reviewer_p4N6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether agents can internalize selected control responsibilities that are usually supplied by an external harness. It introduces Harness Annealing Training (HAT), which combines explicit supervision of control-oriented reasoning with a curriculum over teacher trajectories collected under progressively weaker harnesses. The harness hierarchy retains tools throughout while progressively removing state tracking, workflow organization, and answer verification. The authors train 9B and 35B Qwen models on repository-level QA. The main finding is that selected intermediate annealed checkpoints, especially the H2-targeted checkpoint for 9B, improve tools-only performance and reduce sensitivity to the deployment harness.

Critical strengths
  1. This paper studies a timely problem: as agentic systems increasingly rely on sophisticated external harnesses for planning, state tracking, verification, and repair, it is interesting to investigate whether some of these control responsibilities can be transferred into the model.
  2. The HAT-trained checkpoints outperform the base model when evaluated under the same deployment harness on both repository-level QA test benchmarks.
Critical weaknesses
  1. The paper lacks comparisons with strong baselines.
  2. The empirical scope is limited mainly to repository-level QA, with only one OOD evaluation in the banking domain. More diverse tasks and harness designs are needed to establish the method’s generality. Moreover, the OOD gain is small: Base solves 5/97 instances, the best H2-targeted model solves 8/97, while Direct SFT already solves 7/97.
Questions for authors

n/a

3 comments

Response to W1

We thank the reviewer for raising this point. Our current comparisons already include two baselines: trajectory-level SFT on complete harness-supported trajectories (Direct SFT), which follows the standard recipe of FireAct and AgentTuning, and Layered models trained independently on each source harness level (Table 4). We agree that stronger baselines would further clarify the contribution of HAT. We will make every effort to add them during the remainder of the rebuttal period and will include the results in the final version. We would also greatly appreciate it if the reviewer could suggest specific baselines they would like to see compared, so that we can prioritize them.

Response to W2

We thank the reviewer for raising this question.

To examine whether HAT generalizes beyond repository QA, we additionally evaluate the same checkpoints trained by HAT on Terminal-Bench 2.0. All checkpoints are fine-tuned only on SWE-QA-Pro trajectories and evaluated under the same settings as in Figure 3 without any target-domain training.

CheckpointBaseH4-trainedH3-targetedH2-targetedH1-targeted
Pass rate (%)9.29.07.911.210.1

H2-targeted achieves the highest pass rate, followed by H1-targeted, and both exceed Base. This mirrors the banking domain of τ³-Bench (Figure 3), where H4-trained also matches Base (5/97) while H2-targeted performs best (8/97). This consistency suggests that the control behaviors acquired through annealing, rather than task-specific knowledge from full-harness training, transfer to unseen long-horizon agentic tasks. Given its relatively small parameter count, the 9B model naturally achieves low absolute scores on such a challenging benchmark. We will add these results to the revised manuscript.

We hope this message finds you well. We would like to respectfully follow up on our responses posted, in case they were missed during this busy period. To summarize: (1) regarding W1, our current comparisons include Direct SFT (following the standard recipe of FireAct and AgentTuning) and Layered models trained independently on each source harness level (Table 4); we are actively working on additional stronger baselines during the remainder of the rebuttal period and would greatly appreciate any suggestions on which baselines you would like to see prioritized; (2) regarding W2, we added new evaluations on Terminal-Bench 2.0, where annealed checkpoints outperform Base (H2-targeted: 11.2% vs. 9.2%) under a long-horizon multi-turn setting without target-domain training, complementing the existing τ³-Bench OOD results. As the discussion period will end shortly, we would be grateful if you could kindly take a look and let us know whether these results address your concerns or if anything remains unclear. Thank you very much for your time and consideration.

Thank you for the clarifications and the additional experiments. These responses address my concerns.

Reviewer_3QFmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

he paper proposes to internalize some control functions of the harness into the model weights via a strong-to-weak curriculum training method, Harness Annealing Training (HAT). Specifically, it transfers state tracking, workflow organization, and answer verification into the model, while tools and the execution infrastructure remain external. HAT implements this via Layered (layer-wise) control supervision on constructed OBS/THINK/ACT targets, together with a strong-to-weak annealing curriculum that progressively removes verification, then workflow, then state support from the training sources so the model must learn those decisions itself. Extensive experiments are performed on Qwen3.5-9B/35B with SWE-QA and SWE-QA-Pro. The authors evaluate every checkpoint under all four deployment harnesses; selected annealed models raise tools-only (H1) scores substantially over the H4-trained start, and Layered supervision avoids the weak-support collapse seen with Direct trajectory SFT. Gains and preferred checkpoints vary with scale, and further annealing is not uniformly better.

Critical strengths

S1: The paper cleanly frames harness internalization as transferring state tracking, workflow organization, and answer verification into the model while keeping tools external, which separates this goal from plain trajectory SFT and from optimizing the external harness. S2: The evaluation design matches the claim: every checkpoint is tested under all four deployment harnesses, and the red cells of Table 3 specifically check whether performance holds after the targeted control layer is withdrawn. S3: The Direct versus Layered comparison shows that treating control decisions as supervised outputs matters under weak support, and the authors report that further annealing is not uniformly helpful and that preferred checkpoints vary with model scale.

Critical weaknesses

W1: The main experiments are merely performed on repository QA (SWE-QA / SWE-QA-Pro) with a fixed nested harness ladder, so it remains unclear whether the same internalization holds for the longer-horizon, multi-turn agent settings that the community currently cares more about, or under a differently implemented harness. W2: The causal role of annealing is not cleanly isolated: results are essentially single-seed, and later stages see more data and updates, so it is hard to tell curriculum order from simply training longer—an issue the authors also note in the Limitations.

Questions for authors

Q1. Does internalization transfer to a differently implemented harness (not just weaker nested levels of the same HarnessX-style ladder)? Q2. Can you report the same checkpoint-by-harness withdrawal matrix on longer-horizon multi-turn settings such as Terminal-Bench?

3 comments

Response to W1 & Q2

We thank the reviewer for raising this question.

To examine whether HAT generalizes beyond repository QA, we additionally evaluate the same checkpoints trained by HAT on Terminal-Bench 2.0. All checkpoints are fine-tuned only on SWE-QA-Pro trajectories and evaluated under the same settings as in Figure 3 without any target-domain training.

CheckpointBaseH4-trainedH3-targetedH2-targetedH1-targeted
Pass rate (%)9.29.07.911.210.1

H2-targeted achieves the highest pass rate, followed by H1-targeted, and both exceed Base. This mirrors the banking domain of τ³-Bench (Figure 3), where H4-trained also matches Base (5/97) while H2-targeted performs best (8/97). This consistency suggests that the control behaviors acquired through annealing, rather than task-specific knowledge from full-harness training, transfer to unseen long-horizon agentic tasks. Given its relatively small parameter count, the 9B model naturally achieves low absolute scores on such a challenging benchmark. We will add these results to the revised manuscript.

Due to time constraints, we have not yet completed the full checkpoint-by-harness withdrawal matrix on Terminal-Bench. We will make every effort to provide these results during the remainder of the rebuttal period and will include them in the final version.

Response to W2

We thank the reviewer for raising this question. We show below that additional training, changes in the data distribution, and differences in supervision construction each fail to explain, on its own, the performance gains of HAT.

  1. Additional training. If the gains came mainly from accumulated optimization steps, performance would increase monotonically with training, but Table 3 does not show this pattern. For example, on SWE-QA with the 9B model, H3-targeted reaches 77.41 under H1 after only a single annealing stage, outperforming the later, more extensively trained stages. Performance does not grow with training; instead, gains appear at specific stages. This indicates that the gains come from the training signal introduced at each stage, not from additional optimization steps.

  2. Changes in the data distribution. The independently trained Layered checkpoints in Table 4 are each trained from Base on data from a single source harness. This separates exposure to a given harness level's data from going through the annealing procedure. The ablation shows:

    Exposure to weaker-harness data alone does not improve performance under weak support. Models trained only on H2- or H3-source data perform below the H4 starting checkpoint under tools-only deployment. Aligning the training and deployment distributions does not reproduce the gains of HAT either. H2-targeted never sees tools-only-source data during training, yet under tools-only deployment it outperforms the model trained directly on that source. This advantage holds consistently across both model scales and both benchmarks.

    Therefore, the gains of HAT come neither from more weak-harness data nor from matching the training and deployment distributions. They come from the procedure of progressively withdrawing support during training.

  3. Differences in supervision construction. All stages of the annealing chain use the same Layered supervision construction, so performance differences among checkpoints within the chain are unrelated to supervision construction. The effect of supervision construction itself is isolated by the Direct vs. Layered ablation in Table 4. Under tools-only deployment, Layered supervision consistently and substantially outperforms Direct supervision across all source levels, model scales, and benchmarks. This shows that the effect of Layered supervision is independent and robust, and that the annealing gains add to it rather than being confounded with it.

In summary, the weak-support gains of HAT cannot be attributed to additional training, data exposure, or differences in supervision construction.

Response to Q1

We thank the reviewer for this question. To test whether internalization depends on the specific HarnessX implementation, we evaluate our checkpoints with an independently implemented, Codex-style repository-QA agent. This agent provides only Read, Grep, Glob, and Bash tools. It shares no prompts, state representation, workflow, or verification components with HarnessX, and it differs from it in system prompt, tool interface, and agent loop. It therefore serves as a differently implemented tools-only harness that the models never encountered during training.

CheckpointBaseH4-trainedH3-targetedH2-targeted
Score78.0177.7078.8381.80

Under this unseen harness, H4-trained, which is fine-tuned only on full-harness trajectories, does not improve over Base (77.70 vs. 78.01). This suggests that training on full-harness experience alone ties the gains to the training harness. In contrast, performance rises steadily along the annealing chain. H2-targeted reaches 81.80, which is 3.79 points above Base and 4.10 points above H4-trained. Because none of the HarnessX control components or interfaces are present in this harness, these gains cannot be explained by adaptation to HarnessX-specific formats. Instead, they indicate that the control responsibilities internalized through annealing transfer to a differently implemented harness. We have added these results to the revised manuscript.

We hope this message finds you well. We wanted to kindly bring your attention to our responses posted, in case they may have escaped your notice. In brief: (1) for Q1, we evaluated our checkpoints under an independently implemented, Codex-style repository-QA agent that shares no prompts, state representation, workflow, or verification components with HarnessX, and the annealed checkpoints transfer well (H2-targeted reaches 81.80, vs. 77.70 for H4-trained and 78.01 for Base); (2) for Q2, we added Terminal-Bench 2.0 results, where annealed checkpoints (H2-targeted: 11.2%) outperform Base (9.2%) without any target-domain training; (3) for W2, we conducted additional controlled experiments isolating the annealing curriculum from additional training and data exposure. As the discussion period is drawing to a close, we would greatly appreciate it if you could take a moment to review these results. Any further questions or feedback would be most welcome, and we are happy to run additional experiments if needed. Thank you very much for your time.

Comments on the paper

1 comment

We thank all reviewers for their constructive feedback. Our responses address four main concerns:

  1. Generality beyond repository QA and HarnessX (3QFm-W1/Q1/Q2, p4N6-W2).

    • On an independently implemented Codex-style harness that shares no components with HarnessX, H2-targeted reaches 81.80, compared with 78.01 for Base and 77.70 for H4-trained.
    • Zero-shot on Terminal-Bench 2.0, a long-horizon multi-turn task, H2-targeted achieves the highest pass rate (11.2%, vs. 9.2% for Base). This is consistent with our τ³-Bench banking results.
    • In both settings, training on full-harness trajectories alone (H4-trained) does not improve over Base, whereas annealing does.
  2. Attribution of the gains (3QFm-W2, 1gFj-W1/Q1). Using the existing results in Table 3 and the ablations in Table 4, we show that none of the three factors raised by the reviewers explains, on its own, the gains of HAT:

    • Additional training: performance does not increase monotonically with the number of training steps.
    • Data exposure: training only on weaker-harness data, or directly on tools-only data, does not reproduce the gains.
    • Supervision construction: the Layered construction is held constant throughout the annealing chain.
  3. Evidence of internalization beyond final answers (1gFj-W2/Q2). We analyze freely generated tools-only trajectories, with no teacher-forced reasoning and no external controllers. The fraction of trajectories exhibiting subgoal tracking and integration rises from 0.2667 (Base) to 0.5000 (H3-targeted). This provides behavioral evidence of partial harness internalization that complements answer-level evaluation.

  4. Stronger baselines (p4N6-W1). Our current comparisons include trajectory-level SFT (Direct SFT, following FireAct/AgentTuning) and Layered models trained independently on each source level. We are adding stronger baselines and welcome specific suggestions from the reviewers.

All new results will be included in the revised manuscript. We are also working on the full withdrawal matrix on Terminal-Bench and the additional baselines, and will post these results during the remainder of the rebuttal period. We are happy to address any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.