- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the sensitivity of software engineering agents to different tool harnesses. It introduces a unified Harness Protocol consisting of 12 semantic actions and proposes HINT, which transforms trajectories collected from a single harness into diverse tool-interface formats for supervised fine-tuning. Experiments on SWE-bench Verified and Multilingual show that HINT improves performance on unseen harnesses and reduces cross-harness performance variation compared with standard single-harness fine-tuning.
Critical strengths
- The paper addresses an important practical problem: the performance of SWE agents can vary significantly across different harnesses, even with the same model and tasks.
- HINT provides a simple and effective approach to improving cross-harness generalization by augmenting existing trajectories with diverse tool interfaces, without requiring additional task data or architectural changes.
- The experiments demonstrate consistent improvements across different harnesses and benchmarks. The controlled comparisons and behavioral analyses further support the effectiveness of the proposed method.
Critical weaknesses
- The claimed semantic equivalence across harnesses is not fully convincing. The round-trip verification primarily establishes syntactic consistency, while differences in tool execution constraints, error handling, and runtime behavior may not be preserved.
- The methodological novelty is somewhat limited, as HINT largely builds on domain randomization and data augmentation. More detailed ablations are needed to distinguish the contributions of counterfactual rendering and interface anonymization from simpler tool-interface randomization strategies.
- The generalization evaluation is limited to one main training backbone and one source harness, with a single fine-tuning run per model. Moreover, some tool names from the unseen harnesses appear in the training data, making the extent of generalization to genuinely novel interfaces unclear.
Questions for authors
- Could the authors clarify to what extent the proposed Harness Protocol preserves operational semantics beyond syntactic round-trip consistency? How are differences in tool execution constraints and runtime behavior handled?
- Could the authors provide additional ablations comparing counterfactual rendering alone, interface anonymization alone, and simpler tool-name or schema randomization baselines to better establish the contribution of each component?
- Could the authors evaluate HINT with additional source harnesses or model backbones, and test its generalization to entirely unseen tool vocabularies to further support the claimed interface invariance?