acceptodds
Under review as a conference paper at ICLR 2027

Latent Harness: Compiling Agent Behaviors into Post-Block Activation Shifts

Abstract

Harness engineering and self-evolving agents improve language-model agents quickly, yet their gains live outside the model: the harness is re-sent and re-paid on every call, and the standard route to internalization is a weight update that is costly to iterate and fixed once merged. We propose compiling the harness instead. Latent Harness distills an external condition, either an explicit process-level behavior instruction or the behavioral contract of an agentic harness, into a rank-8 activation shift applied at the output of every decoder block of a frozen backbone (0.03–0.05% of parameters). At inference the compiled bundle carries no harness text, adds zero tokens, and exposes three runtime controls: injection strength, injection phase, and shift-space mixing. Compiled behaviors recover the source-harness gain in 8 of 10 preregistered cells, and where the explicit source pays the heaviest per-call token tax the compiled form surpasses it outright; they transfer zero-shot across tasks (5 of 7 pairs positive), compose in shift space while stacked text instructions interfere destructively, and lift budget-matched harness search by 1.45. On three agentic benchmarks the same operator aligns policy to the harness contract: under matched data, capacity, and effective strength it outperforms same-capacity LoRA, which needs 11.1 the parameters to reach the same band, and it regresses nothing where the base policy is already strong. Across 1B-to-8B models the method skeleton is invariant while gain locations move with the model-task pair, which delimits where compiled harnesses should be deployed. Our code is available at https://anonymous.4open.science/r/latentharness-C574.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent harness, activation steering, Parameter-Efficient Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MWp2OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Latent Harness, a parameter-efficient approach to internalizing behaviors induced by explicit instructions or agent harnesses. The method freezes the backbone and trains rank-8 adapters at the output of each decoder block, using instruction-conditioned teacher outputs or decision pairs extracted from successful agent trajectories. The learned modules support runtime adjustment of intervention strength, injection phase, and behavior composition; instruction compilation additionally removes the need to repeat the source instruction at inference. Experiments across reasoning, code generation, and tool-use benchmarks examine behavioral fidelity, cross-task transfer, composition, and policy alignment. The authors report recovering at least half of the explicit instruction gains in eight of ten preselected settings and outperforming capacity-matched LoRA in selected agent evaluations. Additional analyses investigate behavioral markers and failure modes, showing that effectiveness depends on the model, task, and intervention configuration.

Critical strengths
  1. The paper addresses a clear and practical problem: internalizing instruction- and harness-induced behaviors through lightweight adaptation.
  2. The method is parameter-efficient and supports flexible control of intervention strength, injection phase, and module composition.
  3. Behavioral markers and carefully designed controls provide evidence of behavioral transfer beyond accuracy improvements.
  4. Experiments cover diverse tasks and models, with gains over capacity-matched LoRA in selected settings.
Critical weaknesses
  1. Limited evaluation scope. The experiments only consider models with up to 8B parameters, leaving effectiveness on larger, more capable models unclear. The selected benchmarks also provide limited evidence of applicability to more complex, long-horizon agent tasks.

  2. Small evaluation sets for agent tasks. ToolHop, SearchQA, and API-Bank are evaluated on only 95, 50, and 34 samples, respectively. Several reported improvements therefore correspond to only a few additional solved tasks. Although repeated evaluations help assess variability, the small task pools limit confidence in the robustness and generalizability of these gains.

  3. Limited methodological novelty. The proposed adapter closely resembles existing bottleneck adapters and representation-finetuning methods such as LoReFT and DiReFT. In particular, its linear variant closely matches a bias-free DiReFT intervention. Without direct comparisons under matched training data and parameter budgets, the benefits of the proposed adaptation design over these existing methods remain unclear.

Questions for authors
  1. Could you evaluate the method on larger LLMs and more challenging agent benchmarks, such as SWE-bench Verified or Terminal-Bench, to assess its applicability beyond the current settings?

  2. Could you directly compare the method against LoReFT and DiReFT using the same backbone and training data, with matched parameter and hyperparameter-tuning budgets, to clarify the contribution of the proposed adaptation design?

Reviewer_SZzXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes latent harness, that complie extrnal agent instructions and harness behaviors into low-rank activation shift while keeping the backbone model frozen. The method is parameter-efficient and the resulting behaviors require no addtional prompt tokens in inference. Experimental results across multiple scale models shows that the method preserves much of the original harness benefits, and outperformances LoRA on a range of benchmarks.

Critical strengths
  1. The motivation to simplify the harness is interesting and practical; currently, the settings of agent harness are expensive as it add a lot of context and prompt tokens.
  2. The proposed method is prameter efficient, and only a tiny part of the backbone parameters are modified to match the performance of LoRA.
  3. Insightful analysis. The experiments provide evidence that the learned activation shifts can transfer across tasks.
Critical weaknesses
  1. The evaluaiton sets are relatively small, and 95 ToolHop, 50 SearchQA, and 34 API-bank.
  2. The evaluation lack real-world agent harness, like multi-agent, long-horizon agent harness.
Questions for authors

Have the authors evaluated with large backbone models (>30B)?

Reviewer_RpEyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether you can distill instructions and harness-instilled behaviors into the weights of a model, but it's unclear what the motivation for distilling instructions into the model is. The method is to shift the output of every decoder block in a learned direction. The paper validates the method in 4 ways: recovery of the explicit-harness gain in 8 of 10 preregistered instruction cells, behavior-level attribution through preregistered markers, zero-shot cross-task transfer, and and policy alignment on ToolHop, SearchQA and API-Bank.

Critical strengths

none

Critical weaknesses

The main weakness of the paper is the method is not well motivated: instructions and harness code are so much easier to modify and adapt to new tasks than the method proposed in the paper and cost barely anything more at inference time. If the authors can motivate the method better, I'll reconsider my score.

Questions for authors
  1. [clarification] What were ∆L and R for the two M2×BoolQ cells missing from Table 2, and does their failure locate the limit of recovery where the base policy is already strong?

  2. [clarification] §3.4 states that parameter-matched soft prompts are "weaker everywhere they can be compared", but Table 2 has them above the compiled form in M1×ARC×COMPUTE (0.347 vs. 0.322), M1×ARC×CONCISE (0.376 vs. 0.362) and M3×BoolQ×CONCISE (0.833 vs. 0.774). What does "can be compared" exclude?

  3. [clarification] The 1.45× search gain is one cell (M2×MBPP) and one point estimate per arm at T = 1300. Was it replicated across seeds, and why does the full loop (+0.610) trail the plain library arm (+0.624) on test while tying it on validation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.