acceptodds
Under review as a conference paper at ICLR 2027

Harness-Aware Distillation for Small Language Model Agents

Abstract

Language model agents are deployed with a harness, the software around the model that manages its context, tools, and feedback. When such an agent is distilled into a smaller one, the harness stays in place, so the student mainly needs the teacher-specific abilities that the harness cannot provide, such as acting correctly on harness information. Standard distillation, however, imitates the teacher's full outputs and treats the harness as part of the input. We propose Harness-Aware Distillation (HAD), which focuses distillation on what the teacher adds beyond the harness. HAD complements on-policy distillation with two components: an action preference that contrasts the same teacher's actions with and without the harness information, scored after the student's own reasoning, and a validity check that drops preference pairs whose preferred action contradicts the harness records. We show that the contrast gives the student information that imitating the teacher alone cannot provide, and HAD needs no task rewards, success labels, or future information. Across multiple long-horizon agent benchmarks and models, HAD outperforms on-policy distillation baselines with the same harness, and on one benchmark its student even surpasses the larger teacher. Our analysis shows that HAD enters fewer unproductive loops and recovers from errors more often than the baselines, and suggests that it adaptively keeps learnable feedback in its weights while reading state information from the harness.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Large language models, distillation, agent harness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RZ2aOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies a gap in on-policy distillation (OPD) of language-model agents: agents are deployed inside a harness (software that manages context, tools, and feedback. The student therefore doesn't need to learn what the harness already provides; it needs the teacher's ability to act on what the harness reports. Standard OPD imitates the teacher's full outputs and treats the harness as part of the input, leaving the teacher's use of the harness implicit and failing to separate harness-provided information from teacher-added ability.

Critical strengths

The observation that distilling a harness-equipped agent is a problem between two systems sharing the same harness, so the student needs the teacher's ability to act on harness information, not the harness information itself is, to my knowledge, original and directly relevant to real deployments (where harnesses like OpenHands, SWE-agent, etc. are standard).

Strong gains on ALFWorld, cross-family transfer, and meaningful behavioral analysis. The ALFWorld gains are large (+16.4 SR over best baseline on unseen, +11.4 on seen) and the student surpasses the 8B teacher (63.4 vs 51.5).

Critical weaknesses

The core assumption that the harness-equipped teacher action is the better action is neither tested nor guaranteed, and the validity filter covers only a narrow class of failures. The paper states this as an assumption ("Assuming that harness information generally helps the teacher choose better actions") but never measures the rate at which it holds.

No variance reporting in any main or ablation table, and the smaller-benchmark gains may be noise. Tables all report point estimates with no standard deviations, no seed count, and no confidence intervals. The WebShop SR gain is +0.6 (19.0→19.6) and the ScienceWorld SR gain is +1.4 (8.05→9.40) are small enough that they could easily be within single-run noise for agentic benchmarks where episode-level variance is high.

Questions for authors

At states where the two teacher actions differ, what fraction of the time does the harness-equipped action lead to better task outcomes (e.g., higher eventual success or progress) than the stripped action?

How many seeds were used for the main results?

What fraction of preference pairs are filtered out by the validity checker (m_t = 0)? Among the unfiltered pairs where the two teacher actions differ, what fraction have the harness-equipped action actually correct (by some task-progress measure) vs. wrong-but-valid?

The students are 0.6B–1.7B. Have you tested whether HAD helps when the student is larger (e.g., 4B or 7B), where the student may already use harness information effectively without explicit contrast supervision? Does the gain shrink with student scale?

HAD is evaluated with one hand-designed harness. How sensitive are the gains to the harness design? If the harness were much richer (e.g., also suggesting next actions) or much sparser (only budget tracking), would the contrast mechanism still help?

5 comments
AuthorsOpenReview ↗

Official Response (1/2)

[W1-1 & Q1] (The harness-contrast assumption): We thank the reviewer for this question. The harness-equipped action is the better one, both over whole episodes and at single states.

  • Whole episodes. The Teacher row of Table 2 reports the same teacher on the same tasks with and without the harness information. With the harness, its success rate and score are higher in all three environments. Table 5 shows the same for a second model family. The two runs differ only in the harness information, so the harness makes the teacher choose better actions.
  • Single states (Q1). Following the reviewer's suggestion, we measured the eventual success after each action on ALFWorld. We took the 825 states in the test episodes of the HAD student where the two teacher actions differ. At each state, we executed each action and let the student finish the episode 8 times, so the two branches differ in this one action only. Table R.5 shows that the harness-equipped action leads to success more often than the stripped action.
  • How the harness helps. Table 3 reports prevention, the share of episodes without a stall or a repeated failed action, and recovery, the share of stalls that the agent leaves. Table R.6 adds the teacher without the harness. Without the harness, the teacher avoids stalls in fewer episodes and rarely leaves a stall. The harness-equipped action is therefore the better one because it prevents stalls and recovers from them.

Table R.5. Eventual success after each teacher action | ALFWorld (SR)

SR
After the harness-equipped action31.7
After the stripped action26.4

Table R.6. Teacher with and without the harness | ALFWorld (%)

PreventionRecovery
Teacher with the harness (Table 3)64.647.0
Teacher without the harness55.116.7

[W1-2 & Q3] (The validity check): The validity check is narrow by design, and the harness-equipped action is rarely wrong in the pairs that remain.

  • Design of the validity check. HAD uses no task reward or future information, so it cannot know whether a valid action is wrong. The validity check therefore removes only the actions that contradict the harness records at the current state (Section 3.2).
  • Requested fractions (Q3). Table R.7 reports them at the states of Table R.5. The validity check removes a small part of the pairs. A kept pair is wrong-but-valid when the episode ends worse after the harness-equipped action than after the stripped action. Few such pairs remain after the validity check.
  • Effect of the validity check. In Table 6, removing the validity check lowers the success rate of HAD, so the check helps even though it is narrow.

Table R.7. Fractions requested in Q3 | ALFWorld, 8 continuations per pair

Out ofShare
Removed by the validity checkall pairs13.6%
Wrong-but-validkept pairs7.6%

[W2 & Q2] (Variance across random seeds):

  • Number of seeds (Q2). We repeated the main comparison of HAD and OPD over four random seeds in all three environments, each with an independent training run, and Table R.2 reports the mean and standard deviation. HAD is above OPD in every row.
  • WebShop and ScienceWorld. On WebShop, HAD improves both the score and the SR. The standard deviations overlap slightly, but the SR gain with four seeds is larger than the single-run gain that the reviewer cites. On ScienceWorld, the SR counts only episodes with a full score, which a 0.6B student rarely reaches, so it stays low for every method. The score shows a clear gain.

Table R.2. Four random seeds (mean ± std) | with the harness

OPDHAD
ALFWorld unseen SR44.8±3.060.0±4.6
ALFWorld seen SR41.7±1.849.0±4.7
WebShop score45.8±3.650.3±2.8
WebShop SR19.2±1.522.3±2.7
ScienceWorld score−6.2±7.69.6±4.9
ScienceWorld SR6.9±1.98.1±1.3
AuthorsOpenReview ↗

Official Response (2/2)

[Q4] (Larger scale experiments): Following the reviewer's suggestion, we trained HAD and OPD with a 4B student on ALFWorld, with the same recipe. HAD still helps a larger student.

  • 4B student. Table R.8 uses the 8B teacher of the paper and a larger 32B teacher. The zero-shot 4B student is already close to the 8B teacher, and OPD from this teacher does not improve it. On seen tasks, HAD also stays near the zero-shot student. On unseen tasks, HAD is far above OPD and above the teacher with both teachers, so the gain does not vanish with a larger student.
  • Stronger teacher. HAD also helps when OPD is already strong. With a Qwen3-4B teacher trained with RL (GiGPO) on the ALFWorld training tasks with our harness, OPD reaches a high success rate, and HAD still improves it (Table R.9).

Table R.8. HAD with a 4B student | ALFWorld, with the harness

Unseen SRSeen SR
Teacher 8B (Table 2)51.550.0
Teacher 32B61.256.4
Student 4B (zero-shot)48.550.0
OPD, 8B teacher47.847.1
HAD, 8B teacher61.950.0
OPD, 32B teacher55.248.6
HAD, 32B teacher64.248.6

Table R.9. HAD with an RL-trained teacher | ALFWorld, 1.7B student (three seeds)

Unseen SRSeen SR
Teacher RL-4B97.095.7
OPD87.1±1.787.6±0.8
HAD93.0±2.393.1±0.4

[Q5] (Sensitivity to the harness design): Following the reviewer's suggestion, we tested HAD with a richer and a sparser harness on ALFWorld. HAD helps in both cases.

  • Richer harness. Table R.10 uses a harness designed by others (Life-Harness; Xu et al., 2026), which also gives procedural guidance for the task. We replaced our harness lines in the prompt with its text and changed nothing else. With this harness, OPD only reaches the level of the teacher, and HAD goes above the teacher.
  • Sparser harness. Table R.11 removes parts of our harness step by step. State tracking only removes behavior monitoring (Table 1). Budget only keeps just the step budget of state tracking, as the reviewer suggested. Training and evaluation use the same reduced harness. HAD is above OPD in every row, by a wide margin on unseen tasks.

Table R.10. HAD with a richer external harness | ALFWorld, 1.7B student (three seeds)

Unseen SRSeen SR
Student 1.7B (zero-shot)63.469.3
Teacher 8B (zero-shot)77.680.0
OPD76.9±2.080.5±2.2
HAD84.8±0.485.2±2.3

Table R.11. HAD with a sparser harness | ALFWorld (unseen SR / seen SR)

OPDHAD
Budget only32.1 / 27.943.3 / 42.9
State tracking only (includes the budget)47.0 / 40.756.7 / 41.4
Full harness (Table 2)47.0 / 40.063.4 / 51.4

Dear Reviewer RZ2a,

Thank you for your valuable comments and suggestions. We hope that our responses and the revised PDF have addressed your concerns, and we would be happy to answer any remaining questions. We would be grateful if you could take them into account in your final evaluation.

Thank you for your time and consideration.

Reviewer_U5d9OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the research gap in SLM agent distillation under the Harness setting. The authors propose Harness-Aware Distillation (HAD), which focuses distillation on what the teacher adds beyond the harness. The paper evaluates HAD on three datasets against four baselines. The method achieves the best task performance and harness utilization.

Critical strengths
  1. The paper is well structured, with sufficient explanation of the correspondence among the shortcomings of existing distillation methods, the research motivation, and the method design. The illustrative diagrams aid understanding of the core idea.

  2. By comparing the same teacher's actions with and without harness information, the method constructs action preference supervision under the student's own reasoning context. The design is concise and well targeted.

  3. The method is validated on three agent benchmarks and two model families, providing relatively comprehensive evidence of its effectiveness.

Critical weaknesses
  1. The method lacks theoretical novelty. HAD is essentially on-policy generation of preference data by the teacher model followed by DPO in the Harness setting. The work is an engineering effort that transfers an already mature method to the Harness setting rather than a theoretical innovation.

  2. The experiments don't include multiple random seeds and confidence intervals, and the teacher model's temperature is set relatively high. The results on WebShop (19.00–19.60) and ScienceWorld (8.05–9.40), where the improvements are small, are not sufficiently convincing.

Questions for authors
  1. Can the authors further clarify the theoretical innovation of the method?
  2. Can the authors supplement multi-seed experiments and confidence intervals for the results?
4 comments

[W1 & Q1] (Novelty of the method): We thank the reviewer for the positive comments on the design of our method. HAD uses a pairwise preference loss from the same family as DPO (Section 2). However, HAD clearly differs from DPO and from existing OPD. We clarify its novelty and its theoretical innovation below.

  • Novelty. HAD is new in what it distills. The natural approach is OPD with the harness, which distills only the teacher with the harness, P+. In Figure 2, this improves the harness utilization only slightly and does not reliably improve the performance, while HAD improves both. HAD queries the same teacher again without the harness, P−, and also distills the ratio P+/P−, that is, which actions the harness makes more likely. Unlike pairs built from answer correctness (Section 2), this pair needs no task reward or success label.
  • Theoretical innovation. The paper analyzes this supervision and derives each design choice of HAD from it (Sections 3.2 and 3.3, Appendix A). First, the optimum of the preference term is log(P+(a)/P−(a)) up to a constant (Proposition A.1), and response distillation does not identify this ratio (Observation A.1). Second, a direct transfer of DPO would compare full responses. The optimum then also contains a contrast between the two reasonings (Corollary A.1), which can reverse which action is preferred (Proposition A.2). HAD therefore scores only the actions. Third, scoring after the student's own reasoning keeps the target P+/P− (Corollary A.2), while scoring after the teacher's reasoning changes it (Corollary A.3).
  • Ablations. The experiments confirm these choices. Removing the contrast lowers both the success rate and the harness utilization (Table 6), and a full-response contrast or a teacher reference model is below HAD (Table 15). The gain therefore comes from these parts, not from the loss.

[W2-1 & Q2] (Multiple seeds):

  • Four seeds. We repeated the main comparison of HAD and OPD over four random seeds in all three environments, each with an independent training run, and Table R.2 reports the mean and standard deviation. HAD is above OPD in every row.
  • WebShop and ScienceWorld. On WebShop, HAD improves both the score and the SR. The standard deviations overlap slightly, but the SR gain with four seeds is larger than the single-run gain that the reviewer cites. On ScienceWorld, the SR counts only episodes with a full score, which a 0.6B student rarely reaches, so it stays low for every method. The score shows a clear gain.

Table R.2. Four random seeds (mean ± std) | with the harness

OPDHAD
ALFWorld unseen SR44.8±3.060.0±4.6
ALFWorld seen SR41.7±1.849.0±4.7
WebShop score45.8±3.650.3±2.8
WebShop SR19.2±1.522.3±2.7
ScienceWorld score−6.2±7.69.6±4.9
ScienceWorld SR6.9±1.98.1±1.3

[W2-2] (Teacher temperature): We thank the reviewer for this point. The teacher temperature of 1.0 follows from how HAD builds its pair, and a lower temperature does not give a better result.

  • Why 1.0. The preference term compares the teacher's action distributions with and without the harness (Section 3.2). At temperature 1.0, the two teacher actions are drawn from these distributions as they are (Appendix B.4), while a lower temperature sharpens both. Guided-OPD and SOPD also use a temperature of 1.0 for their rollouts.
  • Lower teacher temperature. We also trained HAD with a teacher temperature of 0.5 and with a greedy teacher (Table R.4). HAD stays above OPD in both cases, so its gain does not come from the high temperature, and 1.0 gives the best result of the three.

Table R.4. HAD with a lower teacher temperature | ALFWorld, with the harness

Unseen SRSeen SR
OPD (Table 2)47.040.0
HAD, teacher temperature 057.547.1
HAD, teacher temperature 0.561.942.9
HAD, teacher temperature 1.0 (Table 2)63.451.4

Dear Reviewer U5d9,

Thank you for your valuable comments and suggestions. We hope that our responses and the revised PDF have addressed your concerns, and we would be happy to answer any remaining questions. We would be grateful if you could take them into account in your final evaluation.

Thank you for your time and consideration.

Thank you for your response. The response has adequately addressed my concerns regarding W2. I will maintain my original score.

Dear Reviewer U5d9,

Thank you for your reply. We are glad that W2 has been resolved. As your score still leans toward rejection, we would like to understand the remaining concern in W1 more specifically, so that we can improve the paper. To this end, building on our response [W1 & Q1], we would like to ask two specific questions.

(1) "The method lacks theoretical novelty."

Your review describes the design of HAD, comparing the same teacher with and without the harness, as concise and well targeted. It also describes our evaluation on three agent benchmarks and two model families as relatively comprehensive evidence. The paper further provides a theoretical analysis of HAD, from which each design choice is derived (Section 3, Appendix A). Given these points, could you clarify what further theoretical novelty you would expect for HAD?

(2) "HAD is essentially on-policy generation of preference data by the teacher model followed by DPO in the Harness setting. The work is an engineering effort that transfers an already mature method to the Harness setting ..."

HAD differs from DPO in each component that distinguishes DPO from other preference losses. HAD has no reference model, forms its pairs from the same teacher with and without the harness rather than from preference judgments, and scores only the action after the student's own reasoning, as an auxiliary term to response distillation (Equations 6 and 7). Comparing full responses or using a reference model (here, the teacher) was weaker than HAD (Table 15). The Bradley–Terry form that HAD shares with DPO is common to many preference losses, so it does not make HAD an application of DPO. Could you clarify which component of HAD you regard as DPO?

Your answers would help us greatly in improving the paper.

Best, The Authors

Reviewer_U21jOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HAD targets small agents that run inside a fixed harness, which adds its own lines to the page the agent sees. At each state the student visits, the teacher is asked twice: once with the harness lines and once without. The student learns the teacher's with-harness response by cross-entropy, as in the SOPD baseline. It also learns to prefer the with-harness action over the without-harness action, both scored after the student's own reasoning. A check drops pairs whose preferred action conflicts with the harness records. On ALFWorld (Qwen3-1.7B from 8B), HAD gets 63.4/51.4 unseen/seen success.

Critical strengths
  1. The ALFWorld result with Qwen3 is large, and the paper's own controls do not remove it. HAD solves 85 of 134 unseen tasks, while the best baseline solves 63 and the 8B teacher 69 (Table 2). Giving OPD and SOPD twice the teacher queries does not close the gap (46.7 and 46.8 vs 57.4, Table 7).

  2. Each state costs just one more text-only teacher call. The released trainer works without a reward, environment resets or teacher log-probs, so a black-box teacher is enough.

Critical weaknesses
  1. The code and the text disagree. The paper says a rejected pair loses only its preference term. In train.py (contrast_rows), the whole state is dropped, cross-entropy target included. The λ=0 row of Table 6 (43.5) matches the SOPD baseline exactly. So part of HAD's gain over it (57.4) may come from filtering the cross-entropy data.

  2. The statistics are thin, and it is unclear how ρ was chosen. No seeds or intervals are reported. The ρ sweep runs on the test split, and unseen success falls from 63.4 to 49.3 between ρ=0.5 and ρ=1.0. App. A.1 says ρ and β were tuned on a held-out split, but neither Table 8 nor the supplement shows one.

Questions for authors
  1. App. B.4 says the teacher samples at temperature 1.0, but train.py decodes greedily (temperature 0.0). For ScienceWorld, run_had.sh also uses temperature 0 and 30 turns, unlike Sec. 4.1 and Table 8. Which settings produced Table 2?
  2. What happens if SOPD gets the same validity filter on its cross-entropy targets?
2 comments

[W1 & Q2] (Validity check and the source of the gain): We thank the reviewer for the careful reading of our code, and we apologize for the confusion. The gain of HAD does not come from filtering the cross-entropy data.

  • Released code. Our code has several filtering variants for the ablations, and the contrast_rows function in the supplementary material is one of them. In all runs in the paper, a rejected pair loses only its preference term and keeps its cross-entropy target, as the paper describes. We will update the released code with a clearer README and explicit options for these variants, so that the setting of each experiment is clear.
  • λ = 0 row of Table 6. HAD keeps every cross-entropy target, so it differs from this row only in the preference term. The gain over this row therefore comes from the preference term, not from filtering.
  • SOPD with the validity filter (Q2). Following the reviewer's suggestion, we trained SOPD with the validity filter applied to its cross-entropy data, under the same settings. In Table R.1, the filter does not help SOPD. It only removes cross-entropy targets, so SOPD is trained on fewer states, and its success rate is slightly lower. HAD remains far above it, so filtering alone cannot explain the gain of HAD.

Table R.1. SOPD with the validity filter on its cross-entropy data | ALFWorld, with the harness

Unseen SRSeen SR
SOPD (Table 2)45.541.4
SOPD + validity filter44.040.0
HAD (Table 2)63.451.4

[W2-1] (Multiple seeds):

  • Four seeds. We repeated the main comparison of HAD and OPD over four random seeds in all three environments, each with an independent training run, and Table R.2 reports the mean and standard deviation. HAD is above OPD in every row.
  • WebShop and ScienceWorld. On WebShop, HAD improves both the score and the SR. The standard deviations overlap slightly, but the SR gain with four seeds is larger than in the single run of the paper. On ScienceWorld, the SR counts only episodes with a full score, which a 0.6B student rarely reaches, so it stays low for every method. The score shows a clear gain.

Table R.2. Four random seeds (mean ± std) | with the harness

OPDHAD
ALFWorld unseen SR44.8±3.060.0±4.6
ALFWorld seen SR41.7±1.849.0±4.7
WebShop score45.8±3.650.3±2.8
WebShop SR19.2±1.522.3±2.7
ScienceWorld score−6.2±7.69.6±4.9
ScienceWorld SR6.9±1.98.1±1.3

[W2-2] (Selection of ρ): We thank the reviewer for raising this issue. ρ and β were selected on a validation split that does not overlap with the test sets (Appendix A.1).

  • Validation split. Table R.3 lists this split. The pool sizes in Table 8 include it, and we will release it with the code.
  • Sensitivity to ρ (Table 16). Table 16 was not used for the selection. Even its lowest result is above every baseline in Table 2, on both unseen and seen tasks. HAD therefore outperforms the baselines for every ρ in this range.
  • One value for all settings. We use the same ρ and β in all three environments and for both model families (Table 12, Appendix B.1). One value works in every setting, so ρ does not need to be tuned for each environment or model.

Table R.3. Validation split used to select ρ and β | held out from the training pool

ALFWorldWebShopScienceWorld
Validation tasks355500358
Sampling10% per task typerandom10% per task type

[Q1] (Settings of Table 2): We thank the reviewer for checking the released scripts. Table 2 was produced with the settings stated in the paper. The values that the reviewer found are default values in the released scripts.

  • Temperature. During training, the student rollouts and the teacher queries both use temperature 1.0 (Appendix B.4). All evaluations use temperature 0.4 (Section 4.1).
  • ScienceWorld. ScienceWorld uses a step limit of 15 (Table 8).

We will make sure to set the defaults of the released scripts to the settings of the paper.

Dear Reviewer U21j,

Thank you for your valuable comments and suggestions. We hope that our responses and the revised PDF have addressed your concerns, and we would be happy to answer any remaining questions. We would be grateful if you could take them into account in your final evaluation.

Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their valuable and constructive comments. They found the problem well motivated and practically relevant (R-U5d9, R-RZ2a), the method simple and practical (R-U21j, R-U5d9), and the results well supported by experiments (R-U21j, R-U5d9, R-RZ2a). We summarize the new results below.

  • Four random seeds. Over four random seeds in all three environments, HAD is above OPD in every row (Table R.2).
  • Larger-scale experiments. With a 4B student, HAD is far above OPD on unseen tasks, with both an 8B and a 32B teacher (Table R.8). With a stronger teacher trained with RL, OPD already reaches a high success rate, and HAD still improves it (Table R.9).
  • Existing external harness. With a harness designed by others (Life-Harness; Xu et al., 2026), HAD goes above the teacher, while OPD only reaches it (Table R.10).

We have uploaded a revised PDF and updated code, and we kindly ask the reviewers to check them. All new content in the revised PDF is shown in green, and the new results are in Sections 4.4 to 4.6 and Appendix D.

This revision focuses on the new results. Repeating main experiments with more random seeds is not feasible within the rebuttal period, and we will include these results in the final version. After the discussion with the reviewers, we will also revise the writing based on the points raised, including the novelty of HAD.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.