acceptodds
Under review as a conference paper at ICLR 2027

Differentiable Harness

Abstract

An agent’s harness defines the instructions, tools, memory, context management, and control flow surrounding its language model. Existing methods improve harnesses by modifying discrete prompt text or agent code, which precludes gradient-based optimization. During generation, however, the model accesses harness text through the internal cache created during prefill. We introduce a differentiable harness: rather than modifying prompt text, we directly optimize the cache it induces, keeping model weights, visible prompt tokens, and tool implementations fixed. We train the cache using teacher-forced replay with cross-entropy and reverse KL regularization against the base distribution, augmented with optional clipped token unlikelihood on error steps. At deployment, an inference engine loads the learned cache artifact at registered token positions without altering the prompt. Across 25 task × harness configurations spanning 16 benchmarks and three frozen model groups (dense, MoE, and hybrid attention/linear-recurrent architectures), differentiable harnesses improve task success rate by up to 20.8% over base agents (with relative gains up to 2.5×) and consistently match or surpass handwritten harnesses. Beyond benchmark accuracy, action-level audits show that the learned cache reduces step-level tool execution errors by 30%, cuts step budget exhaustion by 60% on OSWorld, and avoids the conversational forgetting that degrades weight-adapted baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Agent Harness Optimization, Recursive Self Improvement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_69KMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Differentiable Harness, which adapts an agent by optimizing the internal cache states associated with selected harness segments, by optimizing through teacher-forced trajectory replay with cross-entropy and reverse-KL regularization. The resulting artifacts are injected into the inference cache at registered positions and reused across queries.

Critical strengths
  1. The paper turns the prefilled state induces by an existing harness into a persistent, trainable artifact that can be reused across associated requests, also enabling multiple adapted behaviors on a shared model.

  2. Adaptation to variant attention architectures, including recurrent-state models.

  3. Broad evaluation on agent execution covers SWE, general tool use, conversational agents and computer use workloads, with practical implementation details ablated comprehensively.

Critical weaknesses
  1. Critical Mismatch: the reported Coder-30B configuration specifies 48 layers and a KV width of 512, yet the official configuration has 64 layer with KV width 1024. This mismatch matters for the reported trainable params to be 2x1024x512x48 \approx 50.3M, similar with estimated in the paper, but the actual config should instead yield: 2x1024x1024x64 \approx 134.2M. Consequently, the baseline budget can also be chosen incorrectly based on the estimated magnitude of params (Table 3 ,4 and 16).

  2. Differentiable harness is trained primarily to reproduce successful trajectories, but the attribution of performance gain to harness-level improvement and conceptual benefits of this choice remain unclear.

  3. An alternative modular design could be using human designed harness with task-oriented adapters together, where both components can operate in their own rights. Table 4 do not clarify whether the use of LoRA is combined with rich written harness, nor does it show how performance scales with training compute across adapter-based methods.

Questions for authors

In the result analysis of Section 3.2, there contains excessive numerical values (like LLM-genearted text), which makes it visually cluttered. Please present the data with tables.

Reviewer_NuyJOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Differentiable Harness (DH): optimizing the internal cache induced by an agent’s harness while keeping model weights, visible prompts, and tools fixed. It covers KV, compressed-latent, and recurrent-state representations, with per-request serving-time injection. Training primarily uses trajectory replay with cross-entropy and reverse-KL regularization. Experiments span 15 benchmarks, 22 task–harness configurations, and three model groups.

Critical strengths
  1. The method connects agent adaptation to a concrete serving interface and evaluates diverse workloads. Comparisons against written harnesses, rather than only unassisted agents, strengthen the practical relevance of the reported improvements.

  2. Informative controls and technical detail. The study includes objective, initialization, and placement ablations, relevant continuous-adaptation baselines, and warm-prefix-cache comparisons.

Critical weaknesses
  1. The blanket claim in §1 that prefix tuning cannot be applied to agent harnesses is too strong. Activation-based initialization is already explored in Prefix-Tuning (§7.4), and text-derived KV initialization in Cartridges (§3.2); Table 4’s evaluated text-initialized prefix baseline is also effective. This does not negate DH’s placement, state, or serving extensions, but the paper should distinguish these advances more precisely from established components and clarify the consequential differences between the compared implementations.

  2. Table 5 reports 1.5–2.0-point SWE improvements without uncertainty intervals for the comparison differences. Repeating the same 50 tasks improves decoding precision but does not expand task coverage; run-level SDs reported elsewhere do not directly characterize these contrasts. The paper should distinguish fixed-cohort precision from broader task generalization and report uncertainty appropriate to its evaluation design.

Questions for authors

N/A

Reviewer_usdCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper claims that a frozen LLM agent harness consisting of system instructions, tools, memory, context management, and control flow maintains a steady KV-cache. Subsequent harness optimization prohibits gradient-based methods, and resorts to prompt or code changes. The paper posits that both discrete and continuous prompt changes are sub-optimal, due to either representation collapse by random initialization, or poor steering by adding abstract vectors at the beginning. To overcome these limitations, this work presents Differentiable Harness (DH), which directly targets the cache rather than prompt optimization. The key idea involves parameterizing the cache under four architecture-specific forms, (i) post-/ (ii) pre-RoPE K/V (the latter enabling mid-prompt splicing), (iii) Multi-head Latent Attention (MLA) compressed latents, and (iv) a gated-DeltaNet recurrent state. The parameters are initialized from the real harness prefill activations to avoid the collapse seen with random continuous prompts. Training is done via teacher-forced replay with the joint objective of cross-entropy on successful trajectories and KL-divergence between captured base distribution + optional unlikelihood on error steps. Experiments are conducted across various SWE, coding, tool-calling and conversational agentic benchmarks where DH outperforms base agent by up to 20.8 percentage points. Compared to hand-authored harness, the proposed approach reaches parity or minor gains on the Coder 30B model (other models are not implemented as a comparison to DH). Other supporting evidence includes matched-capacity comparisons against LoRA/soft-prompt/prefix-tuning/Cartridges/GEPA (Table 4), ablations on objective, capacity, and authentic-vs-Gaussian initialization.

Critical strengths
  • The approach is supported by strong theoretical foundations of a transformer model, not just end-task evidence. Four different cache types are studied - post-/pre-RoPE K/V, MLA compressed latents, and a gated-DeltaNet recurrent state - and the proposed method shows consistent gains on dense, MoE, and hybrid attention/recurrent backbones.
  • The experiments are fairly detailed. Crucially, the paper does not stop at task-success deltas: it provides a causal/mechanistic story via (i) the Key-vs-Value decomposition showing ~92.7% of the gain lives in Value representations with Keys, and (ii) a targeted intervention showing that masking rejection-and-recovery turns during training collapses τ\tau-bench performance (43.5% → 32.2%, −11.3 pp).
  • The experimental design is well-scoped and ablation analysis is thorough. Specifically, comparison against LoRA/soft-prompt/prefix-tuning/Cartridges/GEPA at comparable parameter budget is the right control for the method's close relatives.
  • The limitations section (Appendix B) is forthright about the ceiling imposed by the frozen model, artifact staticness, model/engine coupling, and the confidentiality finding.
Critical weaknesses
  • The experimental setting is not balanced as a comparison between DH, BA and RH. Specifically, DH is trained on dedicated train trajectories, and even evaluated on a held-out set, providing in-domain privileged information which other comparative baselines may be lacking.
  • Furthermore, DH is initialized from the prefill from RH, and thus comparison with RH is slightly unfair. A better comparison would be iterative improvement of RH on the same training data vs DH.
  • In all honesty, the claim is not completely backed by the experiments. The genuinely new elements like authentic initialization, pre-RoPE mid-prompt splicing, and paged zero-overhead serving etc. are valuable but incremental engineering, yet the paper markets the result as optimizing "the harness," a term it defines (Introduction) to include tools, control flow, memory, and stopping rules, none of which are actually optimized.
  • As acknowledged by the authors, the thorough comparative baselines are compared with only on one backbone, while comparison with less optimized methods is done on three. Therefore, the claim of +20.8 percentage point improvements in the highlights should be taken with some grains of salt.
Questions for authors
  • It would be good to get some analysis of how much the gain between DH and RH is attributed to the continuous representation versus access to in-domain training dataset enhancement which RH is lacking.
  • Can the analysis between DH and other comparative baselines be extended to other model families also?
  • Practical value proposition. Given that the artifact is static, tokenizer-/weight-/engine-coupled, non-transferable across models, and must be recaptured and retrained whenever the harness, weights, or tokenizer change (Appendix B), in which deployment regime does a ~1.5–3 pp gain over an editable, portable text harness justify this maintenance and per-model training cost?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.