- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper identifies an important failure of agentic SFT could improve performance on a specific agent harness but degrade performance on another harness. The authors called this failure as harness entanglement. In the paper, the authors introduce ARCHER (Across-task Randomized Consistency for Harness Entanglement Reduction), and demonstrated the effectiveness of their approach in transferring to other harnesses.
Critical strengths
- This paper identifies an important problem of harness entanglement, which is important to multi-harness training. Training on one harness could degrade performance on another has always been a issue for agentic SFT. The paper presents a method ARCHER that could potentially mitigate this problem.
- The authors demonstrated some gains in cross-harness transfer using ARCHER.
Critical weaknesses
- The authors only evaluated on one model, which weakens the generalization of their method. It would be great if the authors could at least test on another model.
- In table 1, the improvement on Claude Code only shows a 0.7% gain, which might not be statistically significant, weakening the claim of effectiveness of ARCHER. Similarly, for Table 2, ARCHER demonstrated similar performance compared to shuffled random mask, further weakening ARCHER's importance.
Questions for authors
- The authors study the problem of harness entanglement under the agentic SFT setting, but why not in the agentic RL setting?
- It is still a bit unclear to me intuitively and theoretically why ARCHER works?