- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper investigates how different harnesses affect agent training and cross-harness transfer. The authors distinguish between task competence, which concerns the ability to solve the underlying task, and interface competence, which concerns the ability to interact with a particular harness. They evaluate single-harness and multi-harness distillation across MedAgentBench, Reasoning Gym, BFCL, and AppWorld. Their results show that single-harness SFT can improve performance on the training harness while degrading performance on other harnesses, whereas multi-harness SFT generally mitigates this interference. The authors further propose harness rehearsal, which uses synthetically generated trajectories to train models to operate a target harness. They find that combining harness rehearsal with task supervision from other harnesses can substantially improve performance on the target harness. Finally, they investigate GRPO following multi-harness SFT and find that single-harness RL can improve performance on other harnesses in some settings.
Critical strengths
- The paper provides a systematic empirical study of training and transfer across harnesses, covering multiple benchmarks, interfaces, and training configurations. The results demonstrate that the choice of harness can substantially affect agent performance, including performance on held-out harnesses.
- Harness rehearsal provides a concrete intervention for investigating the roles of task supervision and interface training. By comparing task supervision alone, rehearsal alone, and their combination, the authors demonstrate a complementary effect between the two training signals. The additional control experiments in the appendix further address alternative explanations related to training data volume and template leakage.
- The finding that single-harness RL can improve performance on other interfaces after multi-harness SFT is interesting. It suggests that cross-harness transfer may depend on the training stage and optimisation method, and that training on multiple harnesses may not always be necessary during the RL stage to improve performance across interfaces.
Critical weaknesses
- The main findings are largely consistent with established concepts in machine learning, which limits the paper's conceptual novelty. The degradation on other harnesses following single-harness training can be understood as negative transfer under differences between training and evaluation distributions. The mitigation of this interference through multi-harness training is also consistent with the expected benefits of learning from multiple sources of supervision. Similarly, the gains from combining harness rehearsal with task supervision can be interpreted in terms of complementary training signals. Although the paper provides a systematic empirical analysis of these phenomena, it does not sufficiently establish what additional explanatory power the distinction between task competence and interface competence provides beyond existing concepts such as distribution shift, negative transfer, and complementary supervision, or what novel predictions follow from this distinction.
- The experiments use harness rehearsal to investigate the distinction between task competence and interface competence, but the evidence for separating these two forms of competence remains indirect. The authors use task trajectories from other harnesses as task supervision and rehearsal trajectories on the target harness as interface supervision, and the 2×2 experiments demonstrate that combining these signals can substantially improve performance. However, these results establish the complementarity of the two training signals more directly than they establish the existence of two independently identifiable forms of competence. Rehearsal trajectories may still convey task-related information, while ordinary task trajectories also involve interface operations. The authors acknowledge that they do not directly measure task competence and interface competence, leaving the proposed decomposition insufficiently validated.
- The cross-harness transfer observed with single-harness RL is an interesting finding, but its scope and underlying mechanism remain unclear. On MAB, single-harness GRPO improves performance on multiple interfaces, including the held-out XML harness. Some transfer is also observed on Reasoning Gym, although the results are not consistent across all settings. The authors acknowledge that these results do not establish single-harness RL as a general alternative to multi-harness RL. They also identify starting RL directly from the base model as a direction for investigating how much transfer depends on interface competence established through SFT. The current experiments leave open which factors determine whether single-harness RL transfers across harnesses and how much this transfer depends on the initial model and training configuration.
- Each main training condition is evaluated using a single training run. The reported confidence intervals capture uncertainty from task and evaluation sampling, but do not account for variation across training seeds. This limits the reliability of comparisons involving relatively small differences, particularly in the RL experiments where some confidence intervals overlap. Multiple independent training runs would help establish the robustness of these findings.
Questions for authors
- What explanatory or predictive advantages does the distinction between task competence and interface competence offer over existing concepts such as distribution shift, negative transfer, and complementary supervision? Can this decomposition generate and validate predictions that cannot be readily derived from these existing concepts?
- The combination of harness rehearsal and task supervision from other harnesses produces substantial gains. How do the authors determine whether these gains primarily reflect improvements in interface competence, given that rehearsal trajectories may retain task-related information and may introduce other differences in the training distribution?
- Single-harness RL improves performance on other harnesses in some settings. To what extent does this transfer depend on interface competence acquired through multi-harness SFT, and to what extent can RL itself induce knowledge that transfers across harnesses?
6 comments
Comment 2/4 to Reviewer MRa3
W2 & Q2: "Rehearsal trajectories may still convey task-related information, while ordinary task trajectories also involve interface operations."
We agree that rehearsal trajectories can contain task-related information, so the key question is whether the gains mainly come from that task content or from learning how to use the target harness. To test this, we use failed rehearsal traces as a control.
Failed rehearsal (Table A) addresses the concern that rehearsal may pass on task competence. Keeping only verifier-passing traces is a simple way to control for the quality of harness use, but it also selects for correct task completion. If rehearsal helps mainly by teaching the interface, then failed traces should have a similar effect when they use the harness well, and should fail when the interaction itself is poor. The results closely follow this pattern. On Reasoning Gym the failed traces are mostly valid and differ mainly in the final answer, and keep about 80% of the gain. On BFCL-MT fewer than half are fully valid and they keep less than half. On MAB they are long broken sessions and keep almost none (Table H). This strengthens the case that the gain comes from demonstrating the interface, rather than from incidental leakage of task knowledge.
Table A. Failed-rehearsal control: target-harness score (%)
| BFCL-MT (fc) | RG easy (Terminus-2) | RG hard (Terminus-2) | MAB (Terminus-2) | |
|---|---|---|---|---|
| Task supervision only | 23.6 ± 4.7 | 18.7 ± 3.7 | 13.5 ± 3.3 | 8.6 ± 4.3 |
| Task + rehearsal (verifier-passing traces) | 56.8 ± 5.6 | 41.4 ± 6.8 | 30.2 ± 6.0 | 35.2 ± 8.4 |
| Task + rehearsal (verifier-failing traces) | 37.4 ± 4.9 | 36.8 ± 6.6 | 26.5 ± 5.9 | 10.8 ± 4.6 |
| Share of the gain kept (derived) | 42% | 80% | 78% | 8% |
Table H. Rehearsal traces, failed / successful
| BFCL-MT (fc) | RG (Terminus-2) | MAB (Terminus-2) | |
|---|---|---|---|
| Traces with every turn valid | 47% / 100% | 68% / 87% | 6% / 9% |
| Traces with no valid turn | 29% / 0% | 0% / 0% | 0% / 0% |
| Median assistant turns | 4 / 4 | 2 / 2 | 56 / 17 |
Non-target rehearsal (Table B) asks whether cross-harness traces already involve relevant target interface operations. We paired task supervision with the same amount of rehearsal through a non-target training harness (json on BFCL-MT, fc on MAB). Non-target rehearsal falls below task supervision alone on both benchmarks, indicating that the benefit comes from practice on the target interface.
Table B. Rehearsal through a non-target harness: target-harness success (%)
| BFCL-MT, target fc | MAB, target Terminus-2 | |
|---|---|---|
| Task supervision only | 23.6 ± 4.7 | 8.6 ± 4.3 |
| Task supervision + target-harness rehearsal | 57.5 ± 5.6 | 35.2 ± 8.4 |
| Task supervision + the same amount of rehearsal through a non-target training harness | 13.5 ± 4.0 | 2.5 ± 2.0 |
Two further controls for task leakage, a clean demonstrator that never saw target-harness solutions (Table J) and a template-level leakage analysis on MAB (Table I), are reported in our reply to Reviewer xLUT (Q3.1 and Q3.2).
Tables A and B are default-seed controls (RG: default seed plus seeds 2 and 3). Each cell reports mean ± 95% task-bootstrap CI. In Table A, share kept = (verifier-failing − task only) / (verifier-passing − task only).
We use the same table lettering across all reviewer responses, so each letter refers to the same result throughout. As a result, tables may not appear in alphabetical order within an individual response; missing letters correspond to tables shown in our replies to other reviewers.
Comment 3/4 to Reviewer MRa3
W3 & Q3: scope of single-harness RL transfer, and its dependence on multi-harness SFT
A correction. While preparing additional RL experiments for the rebuttal, we found an error in the GRPO rows of Table 5: reported MAB and RG GRPO rows were inadvertently run using the Terminus-2 single-harness setting. We apologise for this error and report corrected, verified results below, together with the requested multi-seed and from-base experiments. Our main findings are preserved, with some qualifications to the earlier single- vs. multi-harness GRPO comparison. We will update the manuscript accordingly.
We thank the reviewer for this question. Our RL experiments ask whether the cross-harness patterns seen in the distillation (SFT) experiments persist with RL, particularly the interference pattern where single-harness training often hurts performance on held-out harnesses. We find that multi-harness competence learned during distillation carries through RL but, as the reviewer notes, leave open how much of this comes from the RL setup itself versus the prior multi-harness SFT. To address this, we extend our RL experiments in the following ways: (1) multiple training seeds across settings, (2) a second single-harness GRPO condition using a different harness, and (3) single-harness GRPO directly from the base model, which helps separate the effect of prior multi-harness SFT from the effect of RL itself. Table K reports all conditions.
Table K. single- and multi-harness GRPO from 3H-SFT checkpoint and base model: success (%)
MedAgentBench
| parity | Terminus-2 | fc | xml (held out) | |
|---|---|---|---|---|
| Base model | 36.1 ± 9.2 | 5.0 ± 2.8 | 24.3 ± 8.4 | 50.1 ± 11.0 |
| + parity-only GRPO | 59.8 ± 10.9 | 5.0 ± 3.2 | 25.1 ± 8.1 | 57.2 ± 11.5 |
| 3H SFT | 68.3 ± 10.1 | 27.3 ± 7.2 | 56.4 ± 7.2 | 71.8 ± 9.1 |
| + parity-only GRPO | 74.7 ± 9.2 | 31.9 ± 7.6 | 61.4 ± 6.7 | 72.5 ± 9.1 |
| + Terminus-2-only GRPO | 70.9 ± 9.6 | 53.1 ± 10.4 | 66.9 ± 7.2 | 69.5 ± 9.6 |
| + 3H GRPO | 77.6 ± 8.4 | 50.8 ± 9.7 | 85.7 ± 6.2 | 78.0 ± 8.1 |
Reasoning Gym, easy / hard
| Terminus-2 | mini-swe | Goose | Aider (held out) | |
|---|---|---|---|---|
| Base model | 18.8 / 16.4 | 58.7 / 44.1 | 24.6 / 20.7 | 20.8 / 17.6 |
| + Terminus-2-only GRPO | 41.1 / 32.0 | 59.9 / 47.9 | 28.7 / 21.7 | 22.0 / 20.7 |
| 3H SFT | 55.3 / 43.3 | 63.0 / 51.0 | 60.8 / 46.0 | 19.0 / 9.8 |
| + Terminus-2-only GRPO | 59.3 / 45.3 | 66.3 / 51.1 | 63.2 / 43.8 | 18.4 / 11.8 |
| + mini-swe-only GRPO (one seed) | 56.9 / 41.5 | 66.0 / 52.3 | 63.3 / 45.4 | 17.5 / 6.3 |
| + 3H GRPO | 56.9 / 43.0 | 64.1 / 50.7 | 64.7 / 45.4 | 18.3 / 11.4 |
Two seeds training runs for GRPO row unless marked, 4 attempts per task. ± is a 95% task-bootstrap CI (on RG, ±4 to ±8, omitted for space).
Table K shows that GRPO is much more robust to training on a single harness than SFT. We do not see a significant drop on another harness in any setting, whether that harness was seen during SFT or held out. The same pattern appears when training from the base model: single-harness SFT can reduce performance on other harnesses below the base model, whereas single-harness GRPO through the same harness leaves those scores at or above base. Starting from 3H SFT, single-harness GRPO can also improve harnesses used only during SFT. On MAB, for example, fc rises from 56.4 to 61.4 with parity and to 66.9 with Terminus-2. The multi-harness GRPO pattern differs across benchmarks. With the corrected experimental setting, we find that 3H GRPO is strongest overall on MAB, which qualifies our earlier observations on single-harness GRPO relative to multi-harness GRPO. On RG, single-harness and multi-harness GRPO perform comparably after 3H SFT. Overall, the multiple training runs, second single-harness RL condition, and RL-from-base experiments further support our main finding that multi-harness competence learned during SFT carries through single-harness RL.
Comment 1/4 to Reviewer MRa3
We thank the reviewer for the thoughtful comments and address the points raised below.
W1 & Q1: "What explanatory or predictive advantages does the distinction between task competence and interface competence offer over existing concepts"
We agree that our findings can be described in these terms. Training on one harness and evaluating on another is a distribution shift, and the collapse on held-out harnesses is a form of negative transfer. The mitigation of this interference through multi-harness training is also consistent with the expected benefits of learning from multiple sources of supervision.
First, we see our contribution as grounding these concepts in a specific, practical setting: agentic training across harnesses. Practitioners may mix trajectories from several harnesses without a principled view of what each one contributes. As we show, this can cause sharp collapses on other harnesses.
Second, the concept of negative transfer does not say what transfers negatively. A naive reading may be simply that distillation from other harnesses hurts on held-out harness. Our decomposition qualifies this in a practically meaningful way: task knowledge from other harnesses can transfer, while what fails is the interface. On BFCL-MT, task supervision through other harnesses lowers fc from 43.8 (base) to 12.8. Adding rehearsal through fc, which contains no new task supervision, raises it to 56.8, well above the base model. The same supervision that appears harmful in isolation becomes useful once the target interface is learned with cheap rehearsal trajectories.
This leads to several predictions that we test empirically:
We also see a practical implication. If the inference-time harness is not known during training, these results suggest that task knowledge can still transfer across harnesses, with relatively small amounts of target-harness rehearsal used later to establish interface competence. Our failed-trace experiments provide further evidence that the rehearsal benefit comes from harness use rather than simply from exposing the model to the correct task solution.