acceptodds
Under review as a conference paper at ICLR 2027

HINT: Learning the Harness Protocol Interface-Invariant Training for SWE Agents

Abstract

Large language model (LLM) agents now resolve real GitHub issues by exploring a repository, editing files and running tests. A model does this through a harness such as Claude Code, Codex CLI, OpenCode or OpenHands SDK: the scaffold that decides which tools it sees, what they are called, and how calls and results are formatted. These details differ across harnesses and they matter. With model and tasks fixed, changing only the harness moves SWE-bench Verified accuracy by up to 9.0 points, and fine-tuning on trajectories from one harness teaches the interface along with the task: the gain is largest on the source harness and absent on another, widening the spread across harnesses from 3.4 to 11.7 points. Examining 578,856 real tool calls from four production harnesses, we find that they all reduce to the same 12 semantic actions. We formalize this shared core as the Harness Protocol and use it to separate a harness into two layers: the protocol trajectory, the actions and observations every harness must carry, and a surface of tool names, documentation, output and turn formatting. Rendering a protocol trajectory into any surface and parsing it back recovers the same actions (1.87M checks, no mismatches). HINT (Harness-INvariant Training) exploits this: it parses each real trajectory into the protocol and re-renders it under many sampled surfaces, some with meaningless tool names, so that one harness's data trains an agent for any harness with no new task data. Fine-tuning Qwen3.6-35B-A3B on Claude Code trajectories, HINT matches a step-matched control on Claude Code itself, leads under every unseen production harness by up to 8.7 points, narrows the SWE-bench Multilingual accuracy range across five harnesses from 7.4 to 1.6 points, and is on a par with training on real trajectories from four harnesses.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Software Engineering Agents, Harness, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_DLuLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the sensitivity of software engineering agents to different tool harnesses. It introduces a unified Harness Protocol consisting of 12 semantic actions and proposes HINT, which transforms trajectories collected from a single harness into diverse tool-interface formats for supervised fine-tuning. Experiments on SWE-bench Verified and Multilingual show that HINT improves performance on unseen harnesses and reduces cross-harness performance variation compared with standard single-harness fine-tuning.

Critical strengths
  1. The paper addresses an important practical problem: the performance of SWE agents can vary significantly across different harnesses, even with the same model and tasks.
  2. HINT provides a simple and effective approach to improving cross-harness generalization by augmenting existing trajectories with diverse tool interfaces, without requiring additional task data or architectural changes.
  3. The experiments demonstrate consistent improvements across different harnesses and benchmarks. The controlled comparisons and behavioral analyses further support the effectiveness of the proposed method.
Critical weaknesses
  1. The claimed semantic equivalence across harnesses is not fully convincing. The round-trip verification primarily establishes syntactic consistency, while differences in tool execution constraints, error handling, and runtime behavior may not be preserved.
  2. The methodological novelty is somewhat limited, as HINT largely builds on domain randomization and data augmentation. More detailed ablations are needed to distinguish the contributions of counterfactual rendering and interface anonymization from simpler tool-interface randomization strategies.
  3. The generalization evaluation is limited to one main training backbone and one source harness, with a single fine-tuning run per model. Moreover, some tool names from the unseen harnesses appear in the training data, making the extent of generalization to genuinely novel interfaces unclear.
Questions for authors
  1. Could the authors clarify to what extent the proposed Harness Protocol preserves operational semantics beyond syntactic round-trip consistency? How are differences in tool execution constraints and runtime behavior handled?
  2. Could the authors provide additional ablations comparing counterfactual rendering alone, interface anonymization alone, and simpler tool-name or schema randomization baselines to better establish the contribution of each component?
  3. Could the authors evaluate HINT with additional source harnesses or model backbones, and test its generalization to entirely unseen tool vocabularies to further support the claimed interface invariance?
Reviewer_k94bOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

When training llm models using trajectories from one harness, the model may rely too much on the harness and have performance drop when adopted to other harnesses. This paper found that there are around 580k tool calls can be categorized into 12 semantic actions from four production harness. And to mitigate the problem, they proopse HINT, which maps the real trajectories into Harness protocol and then rerender to different synthetic harness surfaces. These rerendered trajectories are used to do SFT. The HINT could help to increase performance on SWE bench by around 8.7% on unseen production harness.

Critical strengths

1.The paper analyzes ~580k real tool calls across 4 production harness and show that they can be largely categorized into 12 semantic actions. This provides some interesting empirical support

2.The method tries to augment the training data without requiring modification to the model architecture

3.The HINT could improve performance across unseen harness configurations

Critical weaknesses

1.The method seems conceptually similar to existing work on interface-invariant evaluation and scaffold-independent trajectory representations. Although the paper claims to be the first to synthesize surfaces by re-rendering real trajectories, the underlying problem-solving process (trajectories) remains unchanged. The novelty of this approach is not entirely clear to me.

2.Are tool names from other production harnesses used during re-rendering? If yes, the synthetic trajectories reuse tool names from other production harnesses while preserving the original actions and observations from Claude Code. However, tools with similar names may behave differently across harnesses. Could this introduce inconsistencies in the training data and potentially confuse the model when deployed in those harnesses?

3.How is CHAMELEON built? Is it implemented from scratch or based on an existing framework? How closely does it reflect real production harnesses?

4.The experiments mainly focus on SWE-bench tasks and Qwen models. It would be helpful to evaluate the method on more diverse tasks and model families.

5.Section 4.6 analyzes different surface axes at test time. It would be interesting to see how varying individual axes during training affects performance, rather than only evaluating their effects at test time.

Questions for authors

please see Weakness

Reviewer_DmjTOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Who has not suffered the credit spent and redoing due to the use of their own harness only to realize that the native harness is the best one to use. This is the paper in my pile that relay addresses the pain for many who struggle to use a seamless workflow in building tools and benchmarking. HINT provide a mechanism to use any harness for the benchmarks, a much needed service for many benchmarks.

Critical strengths

The paper identifies an important problem of being able to use a harness of the user choice when using a benchmark. This is a much needed common problem to anyone who does not want to switch the development to use bench specific harnesses. They also provide detailed experiments showing the effectiveness of HINT.

Critical weaknesses

SWE bench is a very good bench but is that adequate to be the prime driver of the work and experiments?

Questions for authors

Where were your challenges and where were the places you found improvements was not possible? Do you anticipate this approach being widely useable for other benches? In other words, is there a spiral plan to show this will be useful beyond, SWE, terminal, Tau, and Job-bench?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.