- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes Differentiable Harness, which adapts an agent by optimizing the internal cache states associated with selected harness segments, by optimizing through teacher-forced trajectory replay with cross-entropy and reverse-KL regularization. The resulting artifacts are injected into the inference cache at registered positions and reused across queries.
Critical strengths
-
The paper turns the prefilled state induces by an existing harness into a persistent, trainable artifact that can be reused across associated requests, also enabling multiple adapted behaviors on a shared model.
-
Adaptation to variant attention architectures, including recurrent-state models.
-
Broad evaluation on agent execution covers SWE, general tool use, conversational agents and computer use workloads, with practical implementation details ablated comprehensively.
Critical weaknesses
-
Critical Mismatch: the reported Coder-30B configuration specifies 48 layers and a KV width of 512, yet the official configuration has 64 layer with KV width 1024. This mismatch matters for the reported trainable params to be 2x1024x512x48 \approx 50.3M, similar with estimated in the paper, but the actual config should instead yield: 2x1024x1024x64 \approx 134.2M. Consequently, the baseline budget can also be chosen incorrectly based on the estimated magnitude of params (Table 3 ,4 and 16).
-
Differentiable harness is trained primarily to reproduce successful trajectories, but the attribution of performance gain to harness-level improvement and conceptual benefits of this choice remain unclear.
-
An alternative modular design could be using human designed harness with task-oriented adapters together, where both components can operate in their own rights. Table 4 do not clarify whether the use of LoRA is combined with rich written harness, nor does it show how performance scales with training compute across adapter-based methods.
Questions for authors
In the result analysis of Section 3.2, there contains excessive numerical values (like LLM-genearted text), which makes it visually cluttered. Please present the data with tables.