acceptodds
Under review as a conference paper at ICLR 2027

Harness Policy Distillation: Aligning Agent Policies with Runtime Semantics

Abstract

Deploying a general-purpose model as an agent means running it inside a product-specific runtime harness that decides which actions are executable, how state advances, and when the loop may terminate. Ordinary trajectory supervision shows which action was taken but leaves the runtime semantics behind that action implicit. We introduce Harness Policy Distillation (HPD), which makes these runtime control semantics an explicit training target alongside actions to align the harness policy. HPD replays reference trajectories through the harness to recover the runtime semantics that are implicit in ordinary trajectory data, and represents them as control documents at each model-call boundary. These documents provide privileged information for a teacher to generate actions, while harness verification ensures that the resulting targets are executable under the runtime contract. We jointly trains a shared model on the runtime semantics recovered from control-state privileged information and native history, thereby internalizing these semantics into the model policy. At deployment, the trained model uses the original harness interface without requiring the control document or any additional supplement. On TRAJECT-Bench with the Pi harness, Qwen3-8B trained with HPD reaches 22.98% autonomous task success and 49.23% runtime compliance, against 3.09% and 19.90% for the unadapted model. Harness-verified teacher actions substantially improve the native deployment policy, while joint control-state supervision provides further gains in task completion and runtime compliance. HPD turns executable runtime semantics into dense supervision for learning harness-compatible actions at each decision boundary.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Agent Harness, Policy Distillation, Runtime Semantics, Agent Alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BzUgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Harness Policy Distillation (HPD), a framework designed to align LLM agent policies with the runtime semantics of their execution harnesses. HPD replays reference trajectories to extract runtime control states and constructs control documents that guide a teacher model to generate harness-verified actions. It then jointly trains a shared model through three supervision views: native-context action prediction, control-document-conditioned action prediction, and runtime-state reconstruction. Experiments on TRAJECT-Bench demonstrate that HPD significantly improves task success and runtime compliance over existing SFT, RL, and distillation baselines. The results also show promising generalization across tasks, model architectures, and agent harnesses.

Critical strengths

Strengths

  1. Novel Perspective: The paper introduces an interesting perspective by explicitly incorporating harness runtime semantics into agent policy training, rather than relying solely on action-level supervision.

  2. Effective Method: The proposed three-view joint training framework is well-designed. The ablation studies demonstrate the effectiveness of combining action supervision with runtime-state supervision.

  3. Comprehensive Experiments: The paper provides extensive comparisons with SFT, RL, and distillation baselines. It also evaluates generalization across different benchmarks, model architectures, and agent harnesses, demonstrating the effectiveness and robustness of the proposed method.

Critical weaknesses

I do not have many concerns about this work.

The only weakness is that the author does not demonstrate their method by using more backbone models

Questions for authors

None

2 comments

Thank you for the positive assessment. Regarding backbone coverage, the submission evaluates HPD on three mainstream model families and nine backbones in total, each trained and evaluated under the same procedure and compared with its own Base model:

BackboneBase TSHPD TSSource
Llama3-8B0.0020.07Figure 4
Gemma4-12B4.6321.10Figure 4
Qwen3-30B-A3B (MoE)4.1216.81Figure 4
Qwen3 dense, 0.6B–32B (6 sizes)0.00–6.004.46–19.90Table 8

HPD improves task success and final-answer F1 for every backbone, including dense and mixture-of-experts architectures and model sizes from 0.6B to 32B parameters (Section 4.6; Appendix D.4).

Many thanks.

It addresses my concerns.

Reviewer_kEt9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Harness Policy Distillation replays reference trajectories through an agent harness, constructs privileged control documents, and generates verifier-selected action targets. One shared model learns three views: native-context action prediction (T), document-conditioned action prediction (I), and document reconstruction (R). Deployment uses only the native interface. On TRAJECT-Bench/Pi, full HPD reports 22.98% task success, versus 15.61% for OPD and 20.58% for HPD-T.

Critical strengths

Relevant problem and useful deployment design. Agent performance depends on obligations, scheduling, permissions, and termination conditions. Explicit supervision of these structures is well motivated. Retaining the original interface without an additional inference call is a practical advantage. Helpful experimental breadth and disclosure. The paper includes component and scheduling ablations, several backbones, separately adapted harnesses, and cross-benchmark transfer. Appendix B.2 explicitly distinguishes observed state from reference-derived annotations, and Appendix A acknowledges replay limitations.

Critical weaknesses

W1. The central component ablation confounds auxiliary supervision with training budget

Table 2 matches exposure per active view, rather than total optimization. Appendix C.2 specifies:

ConfigurationOptimizer updates
T only20k
T+I or T+R40k
T+I+R60k

Consequently, the improvement from 20.58% to 22.98% task success cannot be attributed specifically to I/R supervision. Appendix D.1 shows that full HPD continues improving with additional optimization. At 20k optimizer updates, full HPD reaches 16.12%, below HPD-T's 20.58%.

That comparison does not establish that auxiliary views are ineffective: full HPD has fewer T updates at that budget. It establishes that the current experiments do not separate supervision content from optimization allocation.

W2. The experiments do not distinguish runtime-state supervision from privileged action hints

Appendix B.2 states that control documents combine:

  • Prefix-causal observations from the harness.
  • Reference-stage progress labels.
  • Witness-derived labels from the current recorded action.

The last two are explicitly non-prefix-causal. Appendix B.3 further requires teacher candidates to match the recorded control decision, tool names, and normalized arguments.

These are legitimate training-time privileged labels; their presence is not evidence of test leakage. However, they leave an alternative explanation: HPD benefits from next-action hints and structured reference supervision, rather than specifically from observing runtime semantics.

The closest conceptual comparison is also important. The cited π-Distill work already jointly trains privileged and unprivileged policies with shared parameters. HPD's potential distinction lies in its harness-derived supervision and verification pipeline. Table 1 does not directly establish the benefit of that distinction.

Questions for authors
  1. Matched training budgets: Does full HPD outperform T-only training at matched total budgets? Can you provide performance curves for the different configurations at common optimizer-update counts, together with processed token counts and training costs?
  2. Attribution of supervision benefits: How much benefit remains when the reference-ledger and action-witness fields are removed? How do observed-state-only, reference-action-only, and full control documents compare?
2 comments

Q1 / W1. Matched training budgets.

We trained T-only for 60k optimizer updates using the same seed, initialization, data, and optimization settings as the main experiments, and evaluated its 20k, 40k, and 60k checkpoints under the same Pi-runtime protocol (TS).

Configuration20k40k60k
T-only20.5818.3519.73
Full HPD16.1219.5522.98

At matched total updates (60k), Full HPD exceeds T-only by 3.25 points, even though T-only receives three times as many action-view updates. T-only does not improve beyond its 20k result, and its 60k action loss is lower than Full HPD’s action-view loss (0.218 vs. 0.2415; Appendix D.2), so action-only optimization does not recover the gain. At equal 20k action-view updates, Full HPD is also higher (22.98 vs. 20.58). The revised Figure 8 in Appendix D reports both curves.

Regarding cost over 60k updates, T-only processes 229.9M tokens and full HPD 268.9M (1.17×). Measured training time is about 23.1 and 27.6 A800 GPU-hours, respectively.

Q2&W2. Attribution of supervision benefits.

To separate runtime, action supervision from sampling and verification, we ran both ablations in the same HPD-T setting; only the control document used for target construction changes.

Control documentTSF1Step-SRRC
Full document20.5839.9659.2543.74
w/o reference-action fields (observed-state-only)9.6134.3333.7924.70
w/o harness-runtime fields (reference-action-only)17.8441.5056.8242.36

Removing the harness-runtime reference lowers TS from 20.58% to 17.84%. Removing the reference-action fields, leaving only pre-action information, lowers TS to 9.61%, with RC 24.70% and Step-SR 33.79%. The decrease of sampling efficiency means less proposals match the reference, so most are rejected and the retained targets become sparse. The reference-action fields therefore make verified target construction feasible, while the harness-runtime fields govern conformance to the runtime contract.

W2 (second part). Direct comparison with π-Distill.

Table 1 now includes a direct comparison. π-Distill uses the same data split, deployment protocol, evaluation settings, privileged control document, training budget, and validation-best checkpoint as Full HPD. How that information becomes a supervision signal is left to each algorithm.

MethodTSF1Step-SRRCExactAction-Acc
π-Distill20.2439.9558.5543.4015.4461.21
Full HPD22.9841.2263.7949.2318.7064.95

The two algorithms use the privileged context differently. π-Distill trains a privileged and an unprivileged policy with shared parameters, so the control document conditions the teacher’s action distribution but is never itself a prediction target. HPD additionally supervises it: the same document is reconstructed from native history and supplies the condition for document-conditioned action prediction. Under identical privileged information, full HPD exceeds π-Distill by 2.74 task-success points.

Dear Reviewer kEt9,

Thank you for your careful review and constructive feedback. Following your suggestions, we trained T-only for 60k optimizer updates and compared it with full HPD at matched total budgets, together with processed token counts and training costs, added control-document ablations comparing observed-state-only, reference-action-only, and full documents, and included a direct comparison with π-Distill under identical privileged information in Table 1.

We hope these additions address your remaining concerns and appreciate your support for our work. If any questions or concerns remain, please do not hesitate to let us know.

Reviewer_KN2zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Harness Policy Distillation (HPD). It replays recorded trajectories through the harness and creates a “control document” at each model call. The document combines facts available from the current run with labels taken from the reference trajectory, including whether the next step should be a tool call or a final answer. A frozen copy of the base model uses this document to propose actions. A verifier keeps candidates whose control decision, tool name, and tool arguments match the reference. The authors then train one model to predict the action from ordinary history, predict it with the document supplied, and predict the document from history. At test time, the model uses ordinary history alone.

On TRAJECT-Bench with the Pi harness, full HPD raises task success from 3.09% for the base Qwen3-8B model to 22.98%. HPD-T uses the same document-guided, verified targets but trains only the ordinary-history action task; it reaches 20.58%. A task counts as successful only if the agent completes the required steps, gives an adequate answer, and exits cleanly.

Critical strengths

The paper tackles a practical agent failure: a model can choose a plausible tool call at the wrong time, repeat a completed step, or answer before the required work is done. I like that the evaluation reports both task completion and compliance with the harness rules. Those measures tell different stories here. For instance, OPD has higher runtime compliance than HPD (66.21% versus 49.23%), while HPD completes more tasks (22.98% versus 15.61%; Table 1).

The verified action targets look useful on their own. HPD-T reaches 20.58% task success using the base 8B model to generate its targets, compared with 15.61% for OPD, which uses a 32B teacher (Table 1; Appendix C.3). The paper also reports gains when the method is applied separately to other model families and harnesses (Figure 4) and across seven Qwen3 sizes (Table 8). The basic training recipe is interesting even before considering the document tasks.

Critical weaknesses

The experiment does not show whether the document tasks help more than additional action training. HPD-T receives 20,000 optimizer updates. Full HPD alternates among three training tasks and receives 60,000 updates in total (Appendix C.2). Its task success is 22.98%, compared with 20.58% for HPD-T (Table 1). At 20,000 and 40,000 total updates, full HPD reaches 16.12% and 19.55%, respectively (Table 7; Table 6). Those checkpoints have also seen fewer action-training updates than HPD-T, so they do not resolve the comparison. The missing control is HPD-T trained for 60,000 updates under the same evaluation protocol, ideally across several seeds. Without it, I cannot attribute the final gain to the two document tasks.

The SFT baseline is hard to interpret. Its action accuracy is 2.06% on a test that ranks the reference action against one alternative, compared with 53.41% for the base model. Its final-answer F1, which measures overlap with the reference answer, also falls from 26.82 to 2.40, although its runtime compliance rises from 19.90% to 55.92% (Table 1; Appendix E). The compliance gain suggests the training changed the model, but the sharp drop in the other measures needs an explanation. I would like to see a check of the training and evaluation formats, along with a supervised baseline whose learning rate and training budget have been chosen for that method. Appendix C.3 reports a shared learning rate of 2 × 10⁻⁶ but does not give comparable total training costs for the methods.

The source of HPD’s gain is still unclear. The control document includes fields derived from the current reference action, such as the next control decision and ready tool calls (Appendix B.2). The frozen model sees those fields when it proposes targets; the verifier then requires the proposed control decision, tool, and full arguments to match the reference, and selection favors the remaining candidate closest to the reference (Section 3.3; Appendix B.3). Sampling and filtering may therefore explain much of the strong HPD-T result. An ablation that removes the reference-action fields, plus a version that samples without a document and uses the same verifier, would show what the harness-state supervision adds. It would also make the distinction from the related π-Distill method clearer.

The BFCL transfer result is encouraging but small. It covers 92 tasks, and full HPD’s 21.74% task success is four successful tasks above OPD’s 17.39% (Section 4.3; Appendix C.6). I would temper the claim of “superior generalization” until it is tested on more tasks or repeated runs. Figure 4 shows gains after training a separate adapter for each harness; it supports portability of the training procedure. Appendix C.5 also reports a significance test against OPD, but Table 5 does not show the OPD runs used for that test.

Questions for authors

What task success does HPD-T reach after 40,000 and 60,000 optimizer updates? Please compare it with full HPD at the same update counts and, if possible, across the same seeds.

What explains SFT’s 2.06% action accuracy and 2.40 final-answer F1? Were its target and output formats checked against the Pi evaluation format? How were the learning rate, training duration, and total training cost chosen for each baseline?

How much performance remains if the control document contains only information available before the next action, with no fields derived from that action? How does HPD-T compare with sampling from the same base model without a document and applying the same verifier?

Which OPD runs and task counts were used for the significance test against HPD in Appendix C.5? Table 5 does not include OPD results.

2 comments

Q1&W1. We trained HPD-T for 60k optimizer updates in a single run with the same settings as the main experiments, and evaluated its 20k, 40k and 60k checkpoints under the same protocol (TS).

Optimizer steps20k40k60k
HPD-T20.5818.3519.73
HPD16.1219.5522.98

At 60k updates, HPD-T has received three times as many action-view updates as full HPD, yet reaches 19.73% versus 22.98%. Further fitting of the action target does not translate into task completion. Conversely, at matched action-view exposure (20k action updates each), full HPD also exceeds HPD-T (22.98% vs. 20.58%), while Full HPD trails at 20k total updates because it has then received only about 6.7k action-view updates. We have added the HPD-T curve to Appendix D in the revision.

Q2&W2. The SFT baseline is trained on the original TRAJECT-Bench target/output annotations. We provide the Pi format and retains a generic-call fallback parser, so SFT outputs in the original format are parsed and executed rather than rejected. The low scores therefore do not come from a format mismatch at evaluation. They reflect the gap between what mismatch supervision teaches and how the harness operates. SFT learns to emit Tool calls but not when to return control to the harness, wait for dependent tool results, or terminate. By checking the test logs, SFT successfully invoked a tool but failed to fully leverage its feedback in 91.8% of agent turns, compared with only 13.6% for HPD. Final-answer F1 depends on using the tool results, and an answer submitted before the dependent results arrive is rarely correct, which explains the F1 of 2.40. The same misalignment appears at the control decision: SFT assigns a mean probability of only 0.1296 to the reference control entry, i.e. whether to continue with a tool call or to stop (0.6242 for Base, 0.9962 for HPD). When it generates complete actions from its own prefix, only 7.40% are state-valid, versus 81.48% for HPD (Section 4.5, Figure 3). All trained baselines, including SFT, use the configuration in Table 4 (Appendix C.3): AdamW with a constant learning rate of 2×10−6 within 20k train-step updates.

Q3&W3. To separate runtime, action supervision from sampling and verification, we ran both ablations in the same HPD-T setting. only the control document used for target construction changes.

Control documentTSF1Step-SRRC
Full document20.5839.9659.2543.74
w/o action reference9.6134.3333.7924.70
w/o runtime reference17.8441.5056.8242.36

Removing the harness-runtime reference, while keeping the reference-action fields, the same verifier and the same selection rule, lowers task success from 20.58% to 17.84%, with lower Step-SR and RC. In the other direction, when only pre-action information is available to the teacher, task success falls to 9.61%, below every distillation baseline in Table 1. The two variants degrade different aspects of the generated action. The decrease of sampling metric means less proposals match the reference, so most are rejected and the retained targets become sparse, which degrades target construction itself. We therefore read the two field classes as playing distinct roles — the reference-action fields make verified target construction feasible, while the harness-runtime fields govern conformance to the runtime contract. Besides, thanks for your review noting that these ablations would also clarify the distinction from π-Distill. Which reaches 20.24% task success, against 22.98% for full HPD. The full comparison is in our reply W2 (part-2). Direct comparison with π-Distill. to Reviewer kEt9.

Q4&W4. Over all 583 test tasks, the OPD baseline here is the classic on-policy distillation run of Table 1, not other variants. For the revised statistical analysis, we included OPD under the same five seeds, 715–719, as the other entries; each run covers all 583 test cases. Table 5 now reports OPD at 14.79 ± 0.93% task success, 41.17 ± 0.25 final-answer F1, 38.81 ± 1.43 Step-SR, and 66.00 ± 0.45 runtime compliance. We have added the OPD row to Table 5 and updated Appendix C.5 and Section 4.2 accordingly.

W4(part-2). Following the suggestion to test on more tasks, we evaluated the same checkpoints on further task collections to scope of the transfer evidence under the same evaluation settings:

MethodBFCLACEBenchToolHopτ-bench
Base13.0444.760.005.31
OPD17.3923.080.005.31
HPD-T13.0437.761.695.31
Full HPD21.7450.356.786.19

Full HPD achieves the highest TS on all 4 collections and improves over HPD-T on each of them. Across the 466 tasks, it completes 52 more tasks than OPD and 33 more than HPD-T. We will add these results to Section 4.3, and replace "superior generalization capabilities" with this scoped statement of task transfer.

Dear Reviewer KN2z,

Thank you for your detailed review and constructive feedback. Following your suggestions, we trained HPD-T for 60k optimizer updates and compared it with full HPD at matched computing budget, clarified the cause of the SFT baseline's low scores and the shared baseline training configuration. We have added ablations that remove the reference-action and harness-runtime fields from the control document, and reported OPD under the same five seeds in Table 5. We also extended the transfer evaluation to ACEBench, ToolHop, and τ-bench and scoped the generalization claim accordingly.

We hope these additions address your remaining concerns and appreciate your support for our work. If any questions or concerns remain, please do not hesitate to let us know.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.