acceptodds
Under review as a conference paper at ICLR 2027

Learning What vs. Learning How: Decomposing Task and Interface Competence in Agentic Training

Abstract

LLM agents operate through harnesses that define system prompts, action formats, feedback loops, and resource constraints. Although a model may possess the same underlying task knowledge, successful execution does not necessarily follow under every harness. This raises basic questions: when a policy fails under a given harness, is it missing task knowledge, or does it simply lack the competence to operate the interface? Can harness-conditioned training improve task competence while degrading interface competence, or vice versa? To address these, we formulate a two-component view of agentic competence: task competence, the knowledge and reasoning required to solve a task, and interface competence, the ability to act on that knowledge through a given harness. We demonstrate that harness-conditioned training can yield vastly different outcomes depending on how these competencies interact. While multi-harness distillation aggregates task knowledge learned from different harness trajectories into a shared policy, this knowledge is only realised if the training distribution preserves or improves interface competence for the target harness. Otherwise, harness diversity can induce interference in interface competence, sometimes catastrophically collapsing performance on a held-out harness despite improvements in underlying task competence. To further distinguish the effects of task and interface supervision, we introduce harness rehearsal: distilling the model on synthetic task trajectories designed to carry minimal task knowledge while exercising the target harness’s interface. We find that rehearsal alone raises performance under the target harness, and when recombined with task knowledge acquired under different harnesses, produces gains far larger than either alone. We show that the effect holds across four reasoning and agentic benchmarks and persists through the subsequent reinforcement learning stage.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Agent Harness, Harness-Conditioned Training, Multi-Harness Training, Agent Failure, Distillation, Supervised Finetuning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MRa3OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper investigates how different harnesses affect agent training and cross-harness transfer. The authors distinguish between task competence, which concerns the ability to solve the underlying task, and interface competence, which concerns the ability to interact with a particular harness. They evaluate single-harness and multi-harness distillation across MedAgentBench, Reasoning Gym, BFCL, and AppWorld. Their results show that single-harness SFT can improve performance on the training harness while degrading performance on other harnesses, whereas multi-harness SFT generally mitigates this interference. The authors further propose harness rehearsal, which uses synthetically generated trajectories to train models to operate a target harness. They find that combining harness rehearsal with task supervision from other harnesses can substantially improve performance on the target harness. Finally, they investigate GRPO following multi-harness SFT and find that single-harness RL can improve performance on other harnesses in some settings.

Critical strengths
  • The paper provides a systematic empirical study of training and transfer across harnesses, covering multiple benchmarks, interfaces, and training configurations. The results demonstrate that the choice of harness can substantially affect agent performance, including performance on held-out harnesses.
  • Harness rehearsal provides a concrete intervention for investigating the roles of task supervision and interface training. By comparing task supervision alone, rehearsal alone, and their combination, the authors demonstrate a complementary effect between the two training signals. The additional control experiments in the appendix further address alternative explanations related to training data volume and template leakage.
  • The finding that single-harness RL can improve performance on other interfaces after multi-harness SFT is interesting. It suggests that cross-harness transfer may depend on the training stage and optimisation method, and that training on multiple harnesses may not always be necessary during the RL stage to improve performance across interfaces.
Critical weaknesses
  • The main findings are largely consistent with established concepts in machine learning, which limits the paper's conceptual novelty. The degradation on other harnesses following single-harness training can be understood as negative transfer under differences between training and evaluation distributions. The mitigation of this interference through multi-harness training is also consistent with the expected benefits of learning from multiple sources of supervision. Similarly, the gains from combining harness rehearsal with task supervision can be interpreted in terms of complementary training signals. Although the paper provides a systematic empirical analysis of these phenomena, it does not sufficiently establish what additional explanatory power the distinction between task competence and interface competence provides beyond existing concepts such as distribution shift, negative transfer, and complementary supervision, or what novel predictions follow from this distinction.
  • The experiments use harness rehearsal to investigate the distinction between task competence and interface competence, but the evidence for separating these two forms of competence remains indirect. The authors use task trajectories from other harnesses as task supervision and rehearsal trajectories on the target harness as interface supervision, and the 2×2 experiments demonstrate that combining these signals can substantially improve performance. However, these results establish the complementarity of the two training signals more directly than they establish the existence of two independently identifiable forms of competence. Rehearsal trajectories may still convey task-related information, while ordinary task trajectories also involve interface operations. The authors acknowledge that they do not directly measure task competence and interface competence, leaving the proposed decomposition insufficiently validated.
  • The cross-harness transfer observed with single-harness RL is an interesting finding, but its scope and underlying mechanism remain unclear. On MAB, single-harness GRPO improves performance on multiple interfaces, including the held-out XML harness. Some transfer is also observed on Reasoning Gym, although the results are not consistent across all settings. The authors acknowledge that these results do not establish single-harness RL as a general alternative to multi-harness RL. They also identify starting RL directly from the base model as a direction for investigating how much transfer depends on interface competence established through SFT. The current experiments leave open which factors determine whether single-harness RL transfers across harnesses and how much this transfer depends on the initial model and training configuration.
  • Each main training condition is evaluated using a single training run. The reported confidence intervals capture uncertainty from task and evaluation sampling, but do not account for variation across training seeds. This limits the reliability of comparisons involving relatively small differences, particularly in the RL experiments where some confidence intervals overlap. Multiple independent training runs would help establish the robustness of these findings.
Questions for authors
  • What explanatory or predictive advantages does the distinction between task competence and interface competence offer over existing concepts such as distribution shift, negative transfer, and complementary supervision? Can this decomposition generate and validate predictions that cannot be readily derived from these existing concepts?
  • The combination of harness rehearsal and task supervision from other harnesses produces substantial gains. How do the authors determine whether these gains primarily reflect improvements in interface competence, given that rehearsal trajectories may retain task-related information and may introduce other differences in the training distribution?
  • Single-harness RL improves performance on other harnesses in some settings. To what extent does this transfer depend on interface competence acquired through multi-harness SFT, and to what extent can RL itself induce knowledge that transfers across harnesses?
6 comments
AuthorsOpenReview ↗

Comment 1/4 to Reviewer MRa3

We thank the reviewer for the thoughtful comments and address the points raised below.


W1 & Q1: "What explanatory or predictive advantages does the distinction between task competence and interface competence offer over existing concepts"

We agree that our findings can be described in these terms. Training on one harness and evaluating on another is a distribution shift, and the collapse on held-out harnesses is a form of negative transfer. The mitigation of this interference through multi-harness training is also consistent with the expected benefits of learning from multiple sources of supervision.

First, we see our contribution as grounding these concepts in a specific, practical setting: agentic training across harnesses. Practitioners may mix trajectories from several harnesses without a principled view of what each one contributes. As we show, this can cause sharp collapses on other harnesses.

Second, the concept of negative transfer does not say what transfers negatively. A naive reading may be simply that distillation from other harnesses hurts on held-out harness. Our decomposition qualifies this in a practically meaningful way: task knowledge from other harnesses can transfer, while what fails is the interface. On BFCL-MT, task supervision through other harnesses lowers fc from 43.8 (base) to 12.8. Adding rehearsal through fc, which contains no new task supervision, raises it to 56.8, well above the base model. The same supervision that appears harmful in isolation becomes useful once the target interface is learned with cheap rehearsal trajectories.

This leads to several predictions that we test empirically:

  • Interface competence is harness-specific, whereas task competence transfers. BFCL-MT reaches 57.5 with fc rehearsal, but only 13.5 and 11.8 with json and xml rehearsal. MAB reaches 35.2 vs. 2.5. Rehearsal through a non-target training harness falls below task supervision alone on both benchmarks (Table B, in our next comment).
  • The two signals are not merely additive. On MAB, task-only adds +1.2 over the base model and rehearsal-only adds +7.3, while the combination adds +35.5 (Table C, in our reply to W4).
  • Rehearsal needs correct interaction, not necessarily correct task knowledge. Failed traces retain ~80% of the gain on Reasoning Gym, where the interaction is sound. They retain 42% on BFCL-MT and 8% on MAB, where the interaction itself breaks down (Tables A and H, in our next comment).

We also see a practical implication. If the inference-time harness is not known during training, these results suggest that task knowledge can still transfer across harnesses, with relatively small amounts of target-harness rehearsal used later to establish interface competence. Our failed-trace experiments provide further evidence that the rehearsal benefit comes from harness use rather than simply from exposing the model to the correct task solution.

AuthorsOpenReview ↗

Comment 2/4 to Reviewer MRa3

W2 & Q2: "Rehearsal trajectories may still convey task-related information, while ordinary task trajectories also involve interface operations."

We agree that rehearsal trajectories can contain task-related information, so the key question is whether the gains mainly come from that task content or from learning how to use the target harness. To test this, we use failed rehearsal traces as a control.

Failed rehearsal (Table A) addresses the concern that rehearsal may pass on task competence. Keeping only verifier-passing traces is a simple way to control for the quality of harness use, but it also selects for correct task completion. If rehearsal helps mainly by teaching the interface, then failed traces should have a similar effect when they use the harness well, and should fail when the interaction itself is poor. The results closely follow this pattern. On Reasoning Gym the failed traces are mostly valid and differ mainly in the final answer, and keep about 80% of the gain. On BFCL-MT fewer than half are fully valid and they keep less than half. On MAB they are long broken sessions and keep almost none (Table H). This strengthens the case that the gain comes from demonstrating the interface, rather than from incidental leakage of task knowledge.

Table A. Failed-rehearsal control: target-harness score (%)

BFCL-MT (fc)RG easy (Terminus-2)RG hard (Terminus-2)MAB (Terminus-2)
Task supervision only23.6 ± 4.718.7 ± 3.713.5 ± 3.38.6 ± 4.3
Task + rehearsal (verifier-passing traces)56.8 ± 5.641.4 ± 6.830.2 ± 6.035.2 ± 8.4
Task + rehearsal (verifier-failing traces)37.4 ± 4.936.8 ± 6.626.5 ± 5.910.8 ± 4.6
Share of the gain kept (derived)42%80%78%8%

Table H. Rehearsal traces, failed / successful

BFCL-MT (fc)RG (Terminus-2)MAB (Terminus-2)
Traces with every turn valid47% / 100%68% / 87%6% / 9%
Traces with no valid turn29% / 0%0% / 0%0% / 0%
Median assistant turns4 / 42 / 256 / 17

Non-target rehearsal (Table B) asks whether cross-harness traces already involve relevant target interface operations. We paired task supervision with the same amount of rehearsal through a non-target training harness (json on BFCL-MT, fc on MAB). Non-target rehearsal falls below task supervision alone on both benchmarks, indicating that the benefit comes from practice on the target interface.

Table B. Rehearsal through a non-target harness: target-harness success (%)

BFCL-MT, target fcMAB, target Terminus-2
Task supervision only23.6 ± 4.78.6 ± 4.3
Task supervision + target-harness rehearsal57.5 ± 5.635.2 ± 8.4
Task supervision + the same amount of rehearsal through a non-target training harness13.5 ± 4.02.5 ± 2.0

Two further controls for task leakage, a clean demonstrator that never saw target-harness solutions (Table J) and a template-level leakage analysis on MAB (Table I), are reported in our reply to Reviewer xLUT (Q3.1 and Q3.2).

Tables A and B are default-seed controls (RG: default seed plus seeds 2 and 3). Each cell reports mean ± 95% task-bootstrap CI. In Table A, share kept = (verifier-failing − task only) / (verifier-passing − task only).


We use the same table lettering across all reviewer responses, so each letter refers to the same result throughout. As a result, tables may not appear in alphabetical order within an individual response; missing letters correspond to tables shown in our replies to other reviewers.

AuthorsOpenReview ↗

Comment 3/4 to Reviewer MRa3

W3 & Q3: scope of single-harness RL transfer, and its dependence on multi-harness SFT

A correction. While preparing additional RL experiments for the rebuttal, we found an error in the GRPO rows of Table 5: reported MAB and RG GRPO rows were inadvertently run using the Terminus-2 single-harness setting. We apologise for this error and report corrected, verified results below, together with the requested multi-seed and from-base experiments. Our main findings are preserved, with some qualifications to the earlier single- vs. multi-harness GRPO comparison. We will update the manuscript accordingly.

We thank the reviewer for this question. Our RL experiments ask whether the cross-harness patterns seen in the distillation (SFT) experiments persist with RL, particularly the interference pattern where single-harness training often hurts performance on held-out harnesses. We find that multi-harness competence learned during distillation carries through RL but, as the reviewer notes, leave open how much of this comes from the RL setup itself versus the prior multi-harness SFT. To address this, we extend our RL experiments in the following ways: (1) multiple training seeds across settings, (2) a second single-harness GRPO condition using a different harness, and (3) single-harness GRPO directly from the base model, which helps separate the effect of prior multi-harness SFT from the effect of RL itself. Table K reports all conditions.

Table K. single- and multi-harness GRPO from 3H-SFT checkpoint and base model: success (%)

MedAgentBench

parityTerminus-2fcxml (held out)
Base model36.1 ± 9.25.0 ± 2.824.3 ± 8.450.1 ± 11.0
+ parity-only GRPO59.8 ± 10.95.0 ± 3.225.1 ± 8.157.2 ± 11.5
3H SFT68.3 ± 10.127.3 ± 7.256.4 ± 7.271.8 ± 9.1
+ parity-only GRPO74.7 ± 9.231.9 ± 7.661.4 ± 6.772.5 ± 9.1
+ Terminus-2-only GRPO70.9 ± 9.653.1 ± 10.466.9 ± 7.269.5 ± 9.6
+ 3H GRPO77.6 ± 8.450.8 ± 9.785.7 ± 6.278.0 ± 8.1

Reasoning Gym, easy / hard

Terminus-2mini-sweGooseAider (held out)
Base model18.8 / 16.458.7 / 44.124.6 / 20.720.8 / 17.6
+ Terminus-2-only GRPO41.1 / 32.059.9 / 47.928.7 / 21.722.0 / 20.7
3H SFT55.3 / 43.363.0 / 51.060.8 / 46.019.0 / 9.8
+ Terminus-2-only GRPO59.3 / 45.366.3 / 51.163.2 / 43.818.4 / 11.8
+ mini-swe-only GRPO (one seed)56.9 / 41.566.0 / 52.363.3 / 45.417.5 / 6.3
+ 3H GRPO56.9 / 43.064.1 / 50.764.7 / 45.418.3 / 11.4

Two seeds training runs for GRPO row unless marked, 4 attempts per task. ± is a 95% task-bootstrap CI (on RG, ±4 to ±8, omitted for space).

Table K shows that GRPO is much more robust to training on a single harness than SFT. We do not see a significant drop on another harness in any setting, whether that harness was seen during SFT or held out. The same pattern appears when training from the base model: single-harness SFT can reduce performance on other harnesses below the base model, whereas single-harness GRPO through the same harness leaves those scores at or above base. Starting from 3H SFT, single-harness GRPO can also improve harnesses used only during SFT. On MAB, for example, fc rises from 56.4 to 61.4 with parity and to 66.9 with Terminus-2. The multi-harness GRPO pattern differs across benchmarks. With the corrected experimental setting, we find that 3H GRPO is strongest overall on MAB, which qualifies our earlier observations on single-harness GRPO relative to multi-harness GRPO. On RG, single-harness and multi-harness GRPO perform comparably after 3H SFT. Overall, the multiple training runs, second single-harness RL condition, and RL-from-base experiments further support our main finding that multi-harness competence learned during SFT carries through single-harness RL.

Reviewer_xLUTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how the agent harness (system prompt, action format, feedback and budgets) shapes agentic training. It splits agentic competence into task competence, the knowledge needed to solve a task, and interface competence, the ability to act through a given harness. On MedAgentBench, Reasoning Gym, AppWorld and BFCL v3, Qwen3-32B trajectories from three training harnesses are distilled into Qwen3-4B, and a fourth harness is held out. Single-harness distillation improves its own harness but can collapse others, while distilling all three harnesses avoids the collapse. Harness rehearsal, i.e., target-harness traces from instructions that hand over the solution, combined with task traces from other harnesses raises MedAgentBench success on Terminus-2 from 3.7 to 34.4, far above either source alone.

Critical strengths
  1. The question is timely, and the 2×2 rehearsal design varies the sources of supervision instead of claiming to measure latent competence.

  2. The finding that single-harness distillation can collapse unseen harnesses is useful for anyone building agentic SFT data.

Critical weaknesses
  1. The 2×2 is not size-matched. In every benchmark, the combined arm concatenates the two single-arm pools. It therefore has more data than either arm (MAB: 365 vs. 258 and 107 rows; RG: 389 vs. 269 and 120; BFCL multi-turn: 1,070 vs. 802 and 268; Table A6). The paper itself notes that the BFCL combination "is not a size-matched replacement" (App. B.1), and Table A5 shows that data volume explains part of the 3H gain in the same setting. Without arms of equal size, part of the super-additive effect may simply come from training on more data.

  2. The complementarity is not robust. With FC instead of Terminus-2 as the MAB target, the effect disappears (base 24.3, task 25.0, rehearsal 39.6, both 36.6; Table A15). The task + failed-rehearsal control designed for BFCL is never reported (App. B.1). Every condition is a single training run, and the test sets are small (61 MAB tasks, 96 RG tasks per tier), with 95% intervals of up to about ±10 points. The central claim therefore rests on one run per benchmark, and its pattern does not hold for another target harness.

Questions for authors
  1. Can you add size-matched arms, and an arm that pairs task supervision with the same amount of rehearsal through a non-target harness, to show that the gain requires practice on the target interface?

  2. Why does the effect disappear with FC as the MAB target, and can you report the task + failed-rehearsal control on all three benchmarks?

  3. Can you report the leakage-detector results cited in Appendix C, and repeat the 2×2 with rehearsal traces from a demonstrator that never saw target-harness solutions?

3 comments
AuthorsOpenReview ↗

Comment 1/3 to Reviewer xLUT

We thank the reviewer for the detailed review and for identifying a concrete set of experiments that would strengthen the paper. We addressed all seven experiments and clarifications raised in the review, most of them on all three benchmarks, and organise the response below by request (W = Weakness, Q = Question):

PointSuggestionAnswerComment
W1, Q1.1Size-matched 2×2Table E1
Q1.2Non-target-harness rehearsalTable B1
W2.1Multiple training runs per armTable C2
W2.2, Q2.1Clarification of results with FC as the MAB targetTable G2
W2.3, Q2.2Failed rehearsal distillation on all three benchmarksTables A, H2
Q3.1Leakage-detector resultsTable I3
Q3.2Clean rehearsal demonstratorTable J3

We use the same table lettering across all reviewer responses, so each letter refers to the same result throughout. As a result, tables may not appear in alphabetical order within an individual response; missing letters correspond to tables shown in our replies to other reviewers.

Across these additional runs and controls, the main findings are preserved, and we believe the additional experiments provide more robust evidence for them. The new experiments also sharpen where the claims hold and where they should be qualified, which we will reflect in the revision.


W1 & Q1.1: size-matched 2×2

We ran the size-matched 2×2 in both directions on all three benchmarks: shrinking the combined pool to the task-supervision arm's size, and repeating task supervision up to the combined size.

Table E. Size- and budget-matched 2×2: target-harness score (%)

BFCL-MT (fc)MAB (Terminus-2)RG easy (Terminus-2)RG hard (Terminus-2)
Task supervision only23.6 ± 4.78.6 ± 4.318.7 ± 3.713.5 ± 3.3
Rehearsal only33.6 ± 5.112.9 ± 7.218.5 ± 4.514.6 ± 4.1
Task supervision + rehearsal57.5 ± 5.635.2 ± 8.441.4 ± 6.830.2 ± 6.0
Combined, shrunk to task size, all rehearsal rows kept56.2 ± 5.633.6 ± 8.440.2 ± 6.427.9 ± 6.2
Combined, shrunk to task size, same mixture57.1 ± 5.232.0 ± 7.642.2 ± 6.730.7 ± 6.2
Task only, repeated to combined size7.8 ± 2.88.6 ± 4.121.2 ± 3.716.4 ± 3.1
  • At equal size the combination keeps its full gain. Shrunk to exactly the task-supervision arm's size, the combined pool scores within about three points of the full combined pool on every benchmark, under both shrinking schemes.
  • More task data does not reproduce the effect. Repeating task supervision to the combined size gives no gain on any benchmark.

Q1.2: non-target-harness rehearsal control

We paired task supervision with the same amount of rehearsal through a non-target training harness (json on BFCL-MT, fc on MAB):

Table B. Rehearsal through a non-target harness: target-harness success (%)

BFCL-MT, target fcMAB, target Terminus-2
Task supervision only23.6 ± 4.78.6 ± 4.3
Task supervision + target-harness rehearsal57.5 ± 5.635.2 ± 8.4
Task supervision + the same amount of rehearsal through a non-target training harness13.5 ± 4.02.5 ± 2.0

Non-target rehearsal falls below task supervision alone on both benchmarks, indicating that the benefit comes from practice on the target interface.


Across our responses to Reviewer xLUT. Each cell reports mean ± 95% task-bootstrap CI. Table C equally pools four training runs per cell (three new seeds plus the reported paper run). Runs use 4 attempts per task. Tables A, B, E, and J are default-seed controls (RG: default seed plus seeds 2 and 3, for robustness where compute allowed). In Table A, share kept = (verifier-failing − task only) / (verifier-passing − task only). Table G uses the single default-seed runs reported in the paper.

AuthorsOpenReview ↗

Comment 2/3 to Reviewer xLUT

W2.1: "Every condition is a single training run, and the test sets are small"

We retrained every 2×2 arm with three additional seeds on all three benchmarks:

Table C. Rehearsal 2×2: target-harness score (%)

BFCL-MT (fc)RG easy (Terminus-2)RG hard (Terminus-2)MAB (Terminus-2)
Base model43.818.8 ± 5.316.4 ± 5.55.0 ± 2.8
Task supervision only12.8 ± 2.920.2 ± 3.611.9 ± 3.36.2 ± 2.6
Rehearsal only31.1 ± 4.619.7 ± 4.714.3 ± 4.112.3 ± 5.3
Task supervision + rehearsal56.8 ± 5.039.1 ± 6.528.8 ± 5.540.5 ± 7.7

Across all runs and benchmarks, the combined arm is consistently better than either single arm. We also report the distillation experiments with three additional seeds. See Table D in our reply to Reviewer hahH.


W2.2 & Q2.1: "Why does the effect disappear with FC as the MAB target"

The fc column of Table A15 shows that rehearsal alone reaches 39.6, while adding task supervision gives 36.6. We interpret this as a setting where the main limitation is interface use rather than task knowledge, so rehearsal alone is already sufficient to recover most of the available gain. Table G compares rehearsal alone with two models trained with both task and target-interface supervision: the target-harness specialist and 3H SFT.

Table G. How much of the full gain rehearsal alone reaches: target-harness score (%)

Rehearsal onlyTask + rehearsalSpecialist3H SFTRehearsal / specialistRehearsal / 3H SFT
BFCL-MT, fc32.558.855.259.059%55%
RG easy, Terminus-220.0 ± 5.737.0 ± 7.154.7 ± 7.855.3 ± 7.937%36%
MAB, Terminus-29.5 ± 5.734.4 ± 8.221.7 ± 6.416.9 ± 5.944%56%
MAB, fc39.6 ± 9.636.6 ± 9.845.1 ± 9.455.3 ± 8.288%72%

Where the interaction appears, rehearsal alone reaches only about 35 to 60% of these models: it gives the model enough interface competence to use the harness, but leaves substantial task skill for task supervision to provide. On MAB-fc, rehearsal alone already reaches 72–88% of the performance of models trained with both task and interface supervision. This suggests that rehearsal alone already unlocks much of the base model’s task competence on that target harness, leaving less additional task skill for cross-harness supervision to contribute.


W2.3 & Q2.2: task + failed-rehearsal control on all three benchmarks

Failed rehearsal (Table A) addresses the concern that rehearsal may pass on task competence. Keeping only verifier-passing traces is a simple way to control for the quality of harness use, but it also selects for correct task completion. If rehearsal helps mainly by teaching the interface, then failed traces should have a similar effect when they use the harness well, and should fail when the interaction itself is poor. The results closely follow this pattern.

Table A. Failed-rehearsal control: target-harness score (%)

BFCL-MT (fc)RG easy (Terminus-2)RG hard (Terminus-2)MAB (Terminus-2)
Task supervision only23.6 ± 4.718.7 ± 3.713.5 ± 3.38.6 ± 4.3
Task + rehearsal (verifier-passing traces)56.8 ± 5.641.4 ± 6.830.2 ± 6.035.2 ± 8.4
Task + rehearsal (verifier-failing traces)37.4 ± 4.936.8 ± 6.626.5 ± 5.910.8 ± 4.6
Share of the gain kept (derived)42%80%78%8%

To understand the differences across benchmarks, we compare the trace quality of the verifier-passing and verifier-failing rehearsal datasets (Table H).

Table H. Rehearsal traces, failed / successful

BFCL-MT (fc)RG (Terminus-2)MAB (Terminus-2)
Traces with every turn valid47% / 100%68% / 87%6% / 9%
Traces with no valid turn29% / 0%0% / 0%0% / 0%
Median assistant turns4 / 42 / 256 / 17

The verifier-failing rehearsal pools contain all failed traces after the per-task cap: 38 for BFCL-MT, 94 for RG, and 107 for MAB. For MAB and BFCL-MT, we subsample the passing pool to match the failed-pool size. On RG, the failed pool is 22% smaller than the passing pool, yet still preserves about 80% of the gain.

The improvement over cross-harness task-only supervision is consistent with the quality of harness interaction in the failed traces. The RG failed-rehearsal data consist mostly of valid interactions that differ from successful traces mainly in the final answer, and retain about 80% of the gain. On BFCL-MT, fewer than half are fully valid, and they retain less than half the gain. MAB rehearsal failures are long, broken sessions that often time out and retain almost none of the gain. This suggests that success filtering mainly removes poor interface demonstrations, rather than helping because successful rehearsal traces contain the correct answer.

AuthorsOpenReview ↗

Comment 3/3 to Reviewer xLUT

Q3.1: Leakage-detector results

We report the cited Appendix C leakage analysis here. MAB has 10 task templates, all represented in the test set, and rehearsal was built from 4 of them, so six of the test templates were never rehearsed (templates are shared across splits, but no training task appears in the test set). If the gain mainly came from leaked task knowledge, it should concentrate on the rehearsed templates. Instead, we see comparable gains on the unrehearsed templates:

Table I. Leakage analysis on MedAgentBench: success (%) on rehearsed vs never-rehearsed templates

Rehearsed templates (task2/3/4/8)Never-rehearsed templates (task1/5/6/7/9/10)
Task supervision only5.9 ± 4.08.6 ± 4.8
Task supervision + rehearsal42.5 ± 15.642.6 ± 8.7
Gain of the combined arm (derived)+36.6+34.0

Across seeds, the combined arm gains +24–42 points over task supervision on unrehearsed templates and +33–40 on rehearsed ones, so the effect is not concentrated on templates seen during rehearsal. The two rehearsed write templates (task3 and task8), whose payload schema is the only content shared with evaluation, are solved in at most 5 of 48 trials in any arm and seed. Rehearsal also uses training tasks only, with no test-task overlap.


Q3.2: Clean-demonstrator rehearsal control

We repeated the 2×2 with rehearsal from a demonstrator that never saw target-harness solutions (Qwen3-32B base):

Table J. Clean demonstrator (Qwen3-32B base, never trained with target-harness solutions): target-harness score (%)

Rehearsal only, 3H demonstratorRehearsal only, clean (32B) demonstratorTask + rehearsal, 3H demonstratorTask + rehearsal, clean demonstrator
BFCL-MT (fc)33.6 ± 5.130.5 ± 5.257.5 ± 5.654.2 ± 5.6
MAB (Terminus-2)12.9 ± 7.214.4 ± 6.635.2 ± 8.432.0 ± 9.0
RG easy (Terminus-2)21.3 ± 5.748.8 ± 7.843.7 ± 7.544.4 ± 7.6
RG hard (Terminus-2)17.7 ± 5.136.6 ± 8.130.4 ± 6.532.6 ± 7.1

With the clean demonstrator, the combined arm keeps its gain: it stays within about three points of the original-demonstrator arm on BFCL-MT and MAB (MAB: 32.0 in a new run, 21.7 in the paper's run in Table A15), and is equal or higher on RG. On RG, rehearsal alone from the 32B demonstrator is already strong (48.8 / 36.6), reaching about 85 to 88% of 3H SFT, similar to the fc target in Table G, so there is little additional gain from task supervision.

Reviewer_hahHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how agent training affects task-solving ability and the ability to operate through a particular harness. It compares distillation from one harness with distillation from several, evaluating both training and unseen harnesses across four benchmarks. Its central intervention, harness rehearsal, supplies explicit instructions for completing training tasks so that demonstrations emphasize interface execution rather than discovering the solution. Combining rehearsal with demonstrations from other harnesses produces substantially better target-harness performance than either source alone. Additional experiments examine whether cross-harness gains survive subsequent reinforcement learning.

Critical strengths
  1. The rehearsal experiment addresses a clear and consequential question. The paper goes beyond showing that different harnesses produce different scores. It changes the supervision to test whether interface practice helps a model use information learned elsewhere. On MedAgentBench, cross-harness supervision and rehearsal alone achieve 2.0% and 9.5% on the target harness, while their combination reaches 34.4%. Reasoning Gym shows a similar pattern. I found these interactions the most convincing contribution: they suggest a practical way to make demonstrations collected under another interface useful.

  2. The study examines interference alongside improvement. The comparisons across training and held-out harnesses reveal substantial regressions that would be missed by evaluating only the training interface. For example, BFCL training on native function calls reduces performance through JSON from 33.5% to 0.4%, while multi-harness training largely avoids this failure. This is useful evidence that improvements under one execution setup can give a misleading picture of an agent’s broader capabilities.

Critical weaknesses
  1. The competence decomposition remains an interpretation rather than an identified mechanism. The authors appropriately acknowledge that they do not directly measure task and interface competence. However, providing the correct procedure removes solution discovery without necessarily removing all transferable task skills from a demonstration. Conversely, demonstrations from other harnesses can teach general interaction skills as well as task knowledge. The experiment therefore establishes complementary supervision more clearly than it establishes two separable underlying competencies. This distinction matters because the decomposition is the paper’s central scientific claim.

  2. The robustness of the central interaction is insufficiently established. Each condition in the main rehearsal experiment is a single training run; the confidence intervals describe evaluation uncertainty rather than variation across training runs. The large effects are encouraging, but the combined condition also changes the amount and composition of training. The multi-harness comparisons include volume controls, yet those do not by themselves settle the interpretation of the rehearsal interaction. Moreover, Figure 4 shows mixed transfer to unseen harnesses, so the practical conclusion should remain narrower than a general recipe for portable agents.

Questions for authors
  1. Clarification: What evidence distinguishes improved interface execution from improved task solving in the rehearsal experiments? For example, do the gains primarily come from fewer invalid calls, incorrect submissions, or termination failures, rather than better solution selection?

  2. Evidence needed for the central claim: Does the Table 4 interaction persist across independent training runs and against controls that use the same training budget for additional or repeated cross-harness demonstrations? This would materially strengthen the claim that the combination works because the supervision sources are complementary.

5 comments
AuthorsOpenReview ↗

Comment 1/4 to Reviewer hahH

We thank the reviewer for the thoughtful and constructive review. Below, we address the two main concerns and report results from additional experiments, which we will incorporate into the manuscript revision.


W1 & Q1: competence decomposition and mechanism — does rehearsal carry task skill, and do cross-harness demonstrations still teach interface skill?

Our 2×2 experiments are designed to test this empirically: on the target harness each source alone gives little or no gain, and the two together do. As this is central to the paper, we ran one control for each direction of possible leakage.

Failed rehearsal (Table A) addresses the concern that rehearsal may pass on task competence. Keeping only verifier-passing traces is a simple way to control for the quality of harness use, but it also selects for correct task completion. If rehearsal helps mainly by teaching the interface, then failed traces should have a similar effect when they use the harness well, and should fail when the interaction itself is poor. The results closely follow this pattern. On Reasoning Gym the failed traces are mostly valid and differ mainly in the final answer, and keep about 80% of the gain. On BFCL-MT fewer than half are fully valid and they keep less than half. On MAB they are long broken sessions and keep almost none (Table H, reply to Reviewer xLUT). This strengthens the case that the gain comes from demonstrating the interface, rather than from incidental leakage of task knowledge.

Table A. Failed-rehearsal control: target-harness score (%)

BFCL-MT (fc)RG easy (Terminus-2)RG hard (Terminus-2)MAB (Terminus-2)
Task supervision only23.6 ± 4.718.7 ± 3.713.5 ± 3.38.6 ± 4.3
Task + rehearsal (verifier-passing traces)56.8 ± 5.641.4 ± 6.830.2 ± 6.035.2 ± 8.4
Task + rehearsal (verifier-failing traces)37.4 ± 4.936.8 ± 6.626.5 ± 5.910.8 ± 4.6
Share of the gain kept (derived)42%80%78%8%

Non-target rehearsal (Table B) asks whether cross-harness traces already supply general interaction skill. We paired task supervision with the same amount of rehearsal through a non-target training harness (json on BFCL-MT, fc on MAB). Non-target rehearsal falls below task supervision alone on both benchmarks, indicating that the benefit comes from practice on the target interface.

Table B. Rehearsal through a non-target harness: target-harness success (%)

BFCL-MT, target fcMAB, target Terminus-2
Task supervision only23.6 ± 4.78.6 ± 4.3
Task supervision + target-harness rehearsal57.5 ± 5.635.2 ± 8.4
Task supervision + the same amount of rehearsal through a non-target training harness13.5 ± 4.02.5 ± 2.0

Across our comments to Reviewer hahH. Each cell reports mean ± 95% task-bootstrap CI. Tables C and D equally pool four training runs per cell (three new seeds plus the reported paper run). Runs use 4 attempts per task, except the new MAB and RG ladder seeds in Table D, which use 1. Tables A, B, and E are default-seed controls (RG: default seed plus seeds 2 and 3, for robustness where compute allowed). In Table A, share kept = (verifier-failing − task only) / (verifier-passing − task only).

AuthorsOpenReview ↗

Comment 2/4 to Reviewer hahH

W2 & Q2.1: robustness across independent training runs

Because the paper involves several stages of training—distillation, rehearsal, and GRPO on multiple harnesses—each reported condition was originally a single training run. We therefore reran every arm of the main rehearsal 2×2 and the distillation experiments with three additional seeds on all three benchmarks, using the same data, training recipe, and evaluation protocol, and pool these with the paper run.

Table C. Rehearsal 2×2: target-harness score (%)

BFCL-MT (fc)RG easy (Terminus-2)RG hard (Terminus-2)MAB (Terminus-2)
Base model43.818.8 ± 5.316.4 ± 5.55.0 ± 2.8
Task supervision only12.8 ± 2.920.2 ± 3.611.9 ± 3.36.2 ± 2.6
Rehearsal only31.1 ± 4.619.7 ± 4.714.3 ± 4.112.3 ± 5.3
Task supervision + rehearsal56.8 ± 5.039.1 ± 6.528.8 ± 5.540.5 ± 7.7

Across all runs and benchmarks, the combined condition is consistently higher than either single-source condition. We will replace the single-run cells in Table 4 with these multi-run results.

Table D. Distillation results over four training runs: score (%) per evaluation harness (held-out harness last)

MedAgentBench

parityTerminus-2fcxml (held out)
parity-only45.8 ± 10.37.6 ± 3.418.6 ± 7.846.3 ± 9.3
Terminus-2-only28.8 ± 7.725.3 ± 7.221.7 ± 6.828.5 ± 7.2
fc-only28.5 ± 11.20.0 ± 0.048.3 ± 7.955.5 ± 8.0
3H68.0 ± 10.325.1 ± 6.652.4 ± 7.268.7 ± 10.0

Reasoning Gym, easy

Terminus-2mini-sweGooseAider (held out)
Terminus-2-only52.9 ± 7.957.1 ± 7.626.8 ± 5.32.4 ± 2.0
mini-swe-only20.7 ± 4.048.8 ± 7.949.9 ± 7.90.0 ± 0.0
Goose-only40.1 ± 6.853.0 ± 7.861.3 ± 8.40.8 ± 0.9
3H56.6 ± 7.561.4 ± 7.363.8 ± 8.313.1 ± 4.3

Reasoning Gym, hard

Terminus-2mini-sweGooseAider (held out)
Terminus-2-only40.2 ± 7.641.5 ± 7.823.8 ± 5.61.3 ± 1.0
mini-swe-only15.6 ± 4.235.0 ± 7.631.7 ± 7.20.0 ± 0.0
Goose-only28.0 ± 6.141.1 ± 7.245.4 ± 8.20.3 ± 0.4
3H39.5 ± 7.148.1 ± 8.045.8 ± 8.07.0 ± 3.1

BFCL multi-turn

fcjsonpycallxml (held out)
fc-only53.8 ± 4.90.1 ± 0.214.2 ± 3.01.3 ± 0.4
json-only23.4 ± 3.947.7 ± 4.830.7 ± 3.915.8 ± 3.3
pycall-only31.5 ± 4.333.8 ± 4.647.7 ± 4.917.8 ± 3.4
3H60.2 ± 5.150.2 ± 5.249.1 ± 4.820.5 ± 3.7

The same pattern holds for the distillation experiments. On held-out harnesses, 3H continues to avoid the sharp regressions seen in single-harness specialists: on Reasoning Gym's Aider interface, the specialists fall to roughly 0–2 while 3H remains at 13 (19 in the paper run). On BFCL-MT's held-out XML interface, the specialists score 1.3–17.8, compared with 20.5 for 3H and 20.6 for the base model.

AuthorsOpenReview ↗

Comment 3/4 to Reviewer hahH

Q2.2: Matched training-budget controls

We ran the control in both directions: shrinking the combined pool to the task-supervision arm's size, and repeating task supervision up to the combined size.

Table E. Size- and budget-matched 2×2: target-harness score (%)

BFCL-MT (fc)MAB (Terminus-2)RG easy (Terminus-2)RG hard (Terminus-2)
Task supervision only23.6 ± 4.78.6 ± 4.318.7 ± 3.713.5 ± 3.3
Rehearsal only33.6 ± 5.112.9 ± 7.218.5 ± 4.514.6 ± 4.1
Task supervision + rehearsal57.5 ± 5.635.2 ± 8.441.4 ± 6.830.2 ± 6.0
Combined, shrunk to task size, all rehearsal rows kept56.2 ± 5.633.6 ± 8.440.2 ± 6.427.9 ± 6.2
Combined, shrunk to task size, same mixture57.1 ± 5.232.0 ± 7.642.2 ± 6.730.7 ± 6.2
Task only, repeated to combined size7.8 ± 2.88.6 ± 4.121.2 ± 3.716.4 ± 3.1
  • At equal size, the combination keeps essentially its full gain. When reduced to the task-supervision arm's size, it stays within about three points of the full combined pool on every benchmark under both sampling schemes.
  • More task supervision does not reproduce the effect. Repeating it to the combined size gives no gain on any benchmark.

Comments on the paper

4 comments

We thank the reviewer for the follow-up and are glad the new experiments resolve the concerns on data size, the non-target harness and training seeds. We address the three remaining points below, with the per-seed results in a second additional comment.


Leakage test on MAB: per-template results

Table M. MedAgentBench success (%) per test template, Terminus-2, mean of three training seeds

TemplateRehearsedTask onlyRehearsal onlyTask + rehearsalGain
task2yes (read)5.665.186.1+80.5
task4yes (read)18.28.976.4+58.2
task3yes (write)0.04.31.4+1.4
task8yes (write)0.01.46.9+6.9
task1no (read)1.49.740.3+38.9
task5no (read + conditional write)33.326.173.6+40.3
task6no (read)10.80.053.0+42.2
task7no (read)0.04.222.6+22.6
task9no (read + conditional write)4.213.148.6+44.4
task10no (read + conditional write)1.41.415.3+13.9
Never rehearsed, mean8.59.142.2+33.7

Gain: task + rehearsal minus task only. 6 test tasks per template (task6: 7), 4 attempts each. The mean row weights templates equally, Table I weights tasks.

Table M directly addresses the concern that the improvement may be driven by leaked task knowledge. All six templates that were never seen during rehearsal improve substantially, with gains ranging from +13.9 to +44.4. This includes the conditional-write templates task5 (+40.3) and task9 (+44.4). On these never-rehearsed templates, task supervision alone (8.5) and rehearsal alone (9.1) both remain weak, whereas their combination reaches 42.2. Thus, the complementarity between task supervision and rehearsal is fully present even on task templates that the model never rehearsed, which is difficult to reconcile with an explanation based on task-template leakage.

The rehearsed templates do not change this picture. The two read templates gain more (+80.5 and +58.2), as expected when rehearsal practises the same request patterns the test tasks use (rehearsal alone reaches 65.1 on task2). They are also among the easiest templates overall, so part of this gain reflects task difficulty. The two write templates gain little (+1.4 and +6.9), but as noted, they are the hardest templates on the Terminus-2 target. Taken together, the per-template results argue against the simplest leakage explanation: the main effect does not depend on having rehearsed the evaluation template itself. We will add Table M to the appendix and describe this per-template pattern in the text.


When complementarity holds: "Please state this condition in the main text."

We agree and will state this condition in the main text. The explanation stays consistent with our clarification on the MAB-fc results (Table G). The 2×2 is designed to restrict additional task supervision to cross-harness sources, which gives the cleanest test that task knowledge transfers once the target interface is learned. This does not mean the base student model starts with no task competence of its own. In both settings without complementarity with cross-harness task supervision, rehearsal alone already reaches most of the score of a model trained with both task and interface supervision. We will mention these two cases, along with their interpretation, in the main text.

Run-to-run variance: per-seed results

Table N. Rehearsal 2×2 per training run: target-harness score (%)

BenchmarkRunTask onlyRehearsal onlyTask + rehearsal
BFCL-MT (fc)s08.5 ± 2.832.5 ± 5.158.8 ± 5.7
s18.6 ± 2.932.2 ± 4.854.9 ± 5.2
s226.0 ± 4.928.9 ± 5.056.1 ± 5.4
s38.2 ± 3.130.8 ± 5.157.5 ± 5.4
RG easy (Terminus-2)s019.8 ± 5.920.0 ± 5.737.0 ± 7.1
s124.9 ± 5.824.8 ± 5.938.9 ± 7.5
s220.0 ± 5.216.5 ± 5.040.2 ± 7.3
s316.1 ± 4.617.7 ± 4.840.1 ± 7.8
RG hard (Terminus-2)s015.1 ± 4.914.7 ± 5.028.5 ± 6.1
s16.6 ± 2.916.4 ± 5.226.7 ± 6.2
s212.6 ± 4.012.8 ± 4.328.5 ± 6.4
s313.3 ± 4.313.3 ± 4.831.6 ± 7.1
MAB (Terminus-2)s02.0 ± 1.69.5 ± 5.734.4 ± 8.2
s17.4 ± 3.912.9 ± 7.243.9 ± 9.4
s210.7 ± 4.516.1 ± 5.937.9 ± 9.4
s34.6 ± 3.310.7 ± 4.445.9 ± 8.6

Mean ± 95% task-bootstrap CI, 4 attempts per task.

The combined arm is the best arm in every run on every benchmark, by at least 10 points. The variance the reviewer points to is concentrated in BFCL-MT task supervision only: the four runs in Table N score 8.5 / 8.6 / 26.0 / 8.2, and a fifth run, the same-seed reference run of Tables A and E, scores 23.6. The instability is itself an interface effect. In the two low runs we checked (8.6 and 8.2), about 40% of assistant turns write the tool call as text, in the json or pycall format seen in training, instead of making a native fc call, and 26–30% of trials never make a native call. In the two high runs (26.0 and 23.6), about 20% of turns do so, and 6–12% of trials. This is the interference the paper describes, seen within one condition: a model trained without the target interface remains brittle in how it uses it, and the outcome varies across training runs. With target-harness rehearsal this variation largely disappears: the combined arm stays at 54.9–58.8 across all four runs, and in every combined run with traces available, text calls fall to about 1% of turns. Each control in Tables A and E is compared with its own same-seed reference, so their gains are internally valid, and Table C's four-run mean (12.8) is the more representative task-only value. We will add Table N to the appendix and note this instability in the text.


We thank the reviewer again for the detailed feedback. Over the rebuttal we have addressed the following points raised:

  • Data size and training budget (W1, Q1.1): size- and budget-matched 2×2 on all three benchmarks (Table E).
  • Non-target-harness rehearsal (Q1.2): rehearsal through a non-target harness falls below task supervision alone (Table B).
  • Training seeds (W2.1): four training runs per arm (Table C), now also reported per run with confidence intervals (Table N).
  • MAB-fc and when complementarity holds (W2.2, Q2.1, follow-up): rehearsal alone against fully trained models (Table G), and the condition now stated in the main text.
  • Failed rehearsal (W2.3, Q2.2): verifier-failing rehearsal traces on all three benchmarks (Tables A, H).
  • Leakage (Q3.1, Q3.2, follow-up): rehearsed vs never-rehearsed templates (Table I), now per template (Table M), and a clean demonstrator that never saw target-harness solutions (Table J).

We hope these results resolve the reviewer's remaining concerns. Together with our first three comments, we believe that these results clarify and strengthen the evidence supporting our central claims. We will incorporate all of these additions into the revised manuscript. We would be grateful if the reviewer could take them into account when reconsidering the score and overall assessment.

We sincerely thank the reviewer for the detailed follow-up and are glad that the additional experiments and analyses resolved the remaining concerns. We appreciated the reviewer’s engagement throughout the rebuttal and have incorporated the resulting revisions into the manuscript. We also appreciate the reviewer adjusting the score in light of our rebuttal.

AuthorsOpenReview ↗

Rebuttal Discussion Summary

We thank all three reviewers for their positive and constructive feedback. We addressed each point raised by the reviewers with new experiments and analyses, including several questions that were shared across reviews. These additions are now incorporated into the revised manuscript. Below, we summarise the main points from each review, how we addressed them, and where the corresponding additions appear in our replies and in the revised paper:

Reviewer hahH

  • The reviewer highlighted the rehearsal experiment as addressing "a clear and consequential question" and found the harness rehearsal 2×2 experiments "the most convincing contribution: they suggest a practical way to make demonstrations collected under another interface useful". The reviewer also viewed the joint study of improvement and interference as "useful evidence that improvements under one execution setup can give a misleading picture of an agent's broader capabilities".
  • The reviewer asked whether rehearsal carries task skill and cross-harness demonstrations still teach interface skill (W1, Q1), and whether the interaction is robust across training runs and under matched budgets (W2, Q2).
  • We addressed these points with failed-rehearsal and non-target-rehearsal controls (Tables A, B), four training runs per arm (Tables C, D), size- and budget-matched arms (Table E), and a failure-type analysis of 3,618 trials using automatic labels and three human annotators (Tables F, L).

Reviewer xLUT

  • The reviewer saw the question as "timely" and noted that the 2×2 rehearsal design "varies the sources of supervision instead of claiming to measure latent competence". The reviewer also found the collapse result "useful for anyone building agentic SFT data".
  • The reviewer asked about size matching, non-target rehearsal, training seeds, the MAB-fc case, the failed-rehearsal control, leakage, and a clean demonstrator, followed by per-template results, the complementarity condition, and per-run variance.
  • We addressed these points with Tables E, B, C, G, A and H, I and M, J, and N. The reviewer confirmed that these additions resolved the concerns and raised the score.

Reviewer MRa3

  • The reviewer highlighted the "systematic empirical study of training and transfer across harnesses, covering multiple benchmarks, interfaces, and training configurations", the use of rehearsal as "a concrete intervention" whose control experiments "address alternative explanations related to training data volume and template leakage", and highlighted the finding that "cross-harness transfer may depend on the training stage and optimisation method" as interesting.
  • The reviewer asked about the explanatory role of the decomposition (W1, Q1), the indirectness of the evidence supporting it (W2, Q2), the scope of single-harness RL and whether it depends on prior multi-harness SFT (W3, Q3), and the use of single training runs (W4).
  • We addressed these points by clarifying the predictions made by the decomposition beyond negative transfer (first reply), adding the same mechanism controls (Tables A, B, H), re-running RL with verified multi-harness rollouts and two seeds per condition together with single-harness GRPO from the base model (Table K), and adding multiple runs (Tables C, D).

How we addressed each point

AdditionRebuttalRevised manuscript
Multiple training runs, pooled and per runTables C, D (hahH 2, xLUT 2, MRa3 4) · Table N (xLUT follow-up)Table 2 (four runs, three for AppWorld) · Sec. 4 run protocol · App. E.1, Tables 21 to 23
Size- and budget-matched 2×2Table E (hahH 3, xLUT 1)App. E.2, Table 24
Failed-rehearsal and non-target-rehearsal controlsTables A, B (hahH 1, xLUT 1 and 2, MRa3 2) · Table H (xLUT 2, MRa3 2)App. E.3 and E.4, Tables 25 to 27
When task and interface supervision are complementaryTable G (xLUT 2 and follow-up)Sec. 5.3, condition stated · App. E.5, Table 28
Leakage, rehearsed vs never-rehearsed templatesTable I (xLUT 3) · Table M (xLUT follow-up)App. E.6, Tables 29 and 30
Clean 32B demonstratorTable J (xLUT 3)App. E.7, Table 31
Failure types, automatic and humanTables F, L (hahH 4)App. E.8, Tables 32 and 33 · Sec. 5.3
RL from the 3H checkpoint and from the base modelTable K (MRa3 3)Sec. 5.4, Table 4 · App. D.4 for Gemma

We thank the reviewers again for the thoughtful discussion and for the suggestions that led to these additional experiments and analyses. We believe these additions have strengthened our paper. We hope this summary provides the Area Chair with a concise overview of how the points raised during discussion were addressed and incorporated into the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.