acceptodds
Under review as a conference paper at ICLR 2027

When Coding Skills Bind to Their Harness: Understanding and Mitigating Cross-Harness Failures in Agentic SFT

Abstract

Agentic supervised fine-tuning (SFT) can improve an agent on the harness that generated its trajectories while degrading performance after deployment through another harness. We identify this failure in Qwen3.5-9B: SFT on Claude Code trajectories raises native success but lowers success on the held-out DeepSeek Harness by 9.8 percentage points. We call this failure harness entanglement. Controlled interventions reveal sensitivity to tool presentation and identifiers. We introduce ARCHER (Across-task Randomized Consistency for Harness Entanglement Reduction), a cross-task shuffled-pair objective that trains different harness renderings with position-level loss consistency. On SWE-bench Verified, ARCHER reaches 62.2% and 63.2% on the held-out DeepSeek Harness and Pi, respectively, compared with 50.6% and 49.9% for the base model, while matching the base model on Claude Code and improving OpenHands. Ablations favor the shuffled construction and show higher transfer scores with position-level consistency. These results show that native-harness scores can hide deployment failures and that randomized consistency can reduce this dependence during SFT.

Primary area:
foundation or frontier models, including LLMs
Keywords:
coding agents, agentic fine-tuning, cross-harness generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_F2v7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies an important failure of agentic SFT could improve performance on a specific agent harness but degrade performance on another harness. The authors called this failure as harness entanglement. In the paper, the authors introduce ARCHER (Across-task Randomized Consistency for Harness Entanglement Reduction), and demonstrated the effectiveness of their approach in transferring to other harnesses.

Critical strengths
  • This paper identifies an important problem of harness entanglement, which is important to multi-harness training. Training on one harness could degrade performance on another has always been a issue for agentic SFT. The paper presents a method ARCHER that could potentially mitigate this problem.
  • The authors demonstrated some gains in cross-harness transfer using ARCHER.
Critical weaknesses
  • The authors only evaluated on one model, which weakens the generalization of their method. It would be great if the authors could at least test on another model.
  • In table 1, the improvement on Claude Code only shows a 0.7% gain, which might not be statistically significant, weakening the claim of effectiveness of ARCHER. Similarly, for Table 2, ARCHER demonstrated similar performance compared to shuffled random mask, further weakening ARCHER's importance.
Questions for authors
  • The authors study the problem of harness entanglement under the agentic SFT setting, but why not in the agentic RL setting?
  • It is still a bit unclear to me intuitively and theoretically why ARCHER works?
Reviewer_AYtqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether agentic SFT binds coding behavior to the harness that produced the trajectories. The authors use DeepSeek v4 Flash to collect trajectories from Claude Code, OpenHands, and DeepSeek Harness, fine-tune Qwen3.5-9B, and compare Claude Code, OpenHands, DeepSeek Harness, and Pi on SWE-bench Verified. They call the case in which the native interface does not drop and the other interfaces drop harness entanglement, and they propose ARCHER, which adds position-level loss consistency on two renderings shuffled across tasks.

Critical strengths

The cross-harness comparison is run on the same set of SWE-bench Verified tasks. Tables 1 and 2 give the means of the four interfaces and the standard deviations of three evaluations, and the drop of single-harness fine-tuning on the DeepSeek Harness that did not take part in training can be seen directly.

Critical weaknesses

The understanding and the mitigation in the title are not separated by the experiments. SFT-CC is 51.4±1.7 on Claude Code and the baseline is 50.6±0.8, and the native half falls within the standard deviation. On DeepSeek Harness it goes from 50.6±0.6 to 40.8±0.8, and these 9.8 points are real, but they only show that single-interface fine-tuning damages the other interfaces. ARCHER and the random-position control use the same packed data and are only 1.8 and 1.6 points higher on the held-out set, and the main text also states that whether the tokens match is not required for this gain. After consistency is added to the content-preserving group, Pi drops from 55.6 to 53.8. When the skin is changed, the system prompt, tool schema, message history, result wrapping, and context length change together, and the second renaming set also changes the size and the sign of the effect. All results come from one Qwen3.5-9B, and the trajectories come from the same teacher.

ARCHER does not compare two harness implementations of the same segment of coding behavior. The second rendering is shuffled onto other tasks, and the loss is aligned only at positions where the target token ids on the two sides happen to be the same. The main text states that shared punctuation or digits can also enter this set and do not correspond to the same coding action. The decoupling in the title is therefore not realized by this objective. Across ordinary mixing, content preservation, and the shuffled construction, the pairing method, the retained samples, and the comparison window also change together, and these comparisons do not separate cross-task co-occurrence from packing.

Questions for authors
  1. The harness entanglement in the title requires that native performance does not drop while transfer becomes worse. SFT-CC is only 0.8 points higher than the baseline on Claude Code, and that difference falls within the reported standard deviation. Which result separates "the skill is bound to the training harness" from "single-interface fine-tuning damaged the other interfaces"?
  2. ARCHER and the random-position control use the same packed data and are only 1.8 and 1.6 points higher on the held-out set. Please provide the paired differences and intervals for these three evaluations. If the interval covers 0, the position-level consistency in the title is not the source of the improvement.
  3. In the position set MABM_{AB}, what share is made up of punctuation, digits, and other tokens that do not correspond to tool calls? If these positions make up the majority, the position-level consistency does not constrain coding actions.
Reviewer_bKCXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates harness entanglement, where agentic SFT improves performance on the training harness but degrades generalization to other harnesses. Through several diagnostic experiments, the authors identify dependencies on harness-specific interfaces and tool identifiers as potential causes. To address this, they propose ARCHER, which randomly pairs trajectories from different tasks and applies token-level NLL consistency regularization. Experiments on SWE-bench Verified show that ARCHER substantially improves performance on held-out harnesses compared to standard SFT and several baselines.

Critical strengths
  1. Important and practical problem. The paper studies harness entanglement, an important but underexplored issue in agentic SFT. Understanding why improvements on one harness may not transfer to others is highly relevant to real-world agent deployment.

  2. Insightful diagnostic experiments. The paper provides a valuable analysis of why harness entanglement occurs. The interface-skin interventions, tool identifier renaming, and tool-call failure analysis offer useful evidence of how models become dependent on specific harness interfaces.

Critical weaknesses
  1. Unclear motivation and mechanism behind ARCHER. The connection between the proposed consistency objective and the diagnostic findings is not well established. It is unclear why cross-task pairing is necessary or why matching token-level NLLs across unrelated trajectories should reduce harness entanglement. The ablations do not adequately answer these questions, especially given the strong performance of random-position consistency. Overall, the paper does not convincingly identify what actually drives the improved generalization.

  2. Concerns about the SFT setup. SFT on Claude Code trajectories improves native performance by only 0.8 points, compared to much larger gains from OpenHands and DeepSeek Harness trajectories. This raises questions about the training data quality and optimization setup. In particular, the learning rate of 1e-4 seems relatively high, and the paper lacks learning-rate ablations. It remains unclear whether some of the observed transfer degradation could be attributed to suboptimal SFT rather than harness entanglement itself.

Questions for authors
  1. What is the motivation for applying token-level NLL consistency across unrelated tasks, and how does this connect to the harness-specific dependencies identified in the diagnostic experiments? In particular, why should matching token-level losses promote harness-invariant behavior, given that random-position consistency achieves similar results?

  2. Have the authors evaluated ARCHER on larger models or additional benchmarks beyond SWE-bench Verified? Given that the motivation behind the proposed consistency objective remains unclear, I find it difficult to assess whether the observed improvements reflect a generalizable method or are specific to the current experimental setting. Additional evidence across model scales and tasks would significantly strengthen the claims.

  3. Why does SFT on Claude Code trajectories improve native performance by only 0.8 percentage points, while SFT on OpenHands and DeepSeek Harness yields much larger gains? Could the authors clarify the quality and selection criteria of the teacher trajectories? Additionally, given the relatively high learning rate (1e-4), could the authors provide lower-learning-rate SFT baselines to rule out suboptimal optimization as a contributing factor to the observed transfer degradation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.