acceptodds
Under review as a conference paper at ICLR 2027

Capable Teacher, Myopic Scorer: Reward-Free Teacher Repair for Agentic On-Policy Distillation

Abstract

Agentic on-policy distillation (OPD) trains a student agent on its own trajectories while a teacher scores every token. A natural explanation of where it goes wrong is compounding error: the student's mistakes carry its trajectories beyond the region where the teacher is reliable, so recent methods keep training inside that region, through episode-length curricula, teacher-guided turns, prefix replay, or teacher intervention where the two disagree. We show this explanation is incomplete. From the states where the student departs most from the teacher, the teacher's own takeovers still finish the task, yet the student trained on its token scores keeps running out of turns. We attribute this to the myopia of next-token scoring. The teacher rates each token without seeing where the episode ends, so its scores let through the safe, delaying moves that outcome-based RL would punish with zero reward, a penalty agentic OPD lacks. We call this difference between what the teacher does and what it scores the rollout–scorer gap. We propose Rollout-to-Scorer Distillation (R2S), which repairs the scorer before distillation by mining high-disagreement states from the untrained student, running teacher takeovers from them, keeping the states where enough takeovers pass a reward-free check such as consensus or an LLM judge, and fine-tuning the teacher to give its takeover-informed scores on every assistant token from the divergence turn onward, without the takeover in context. The repaired teacher is frozen as the sole scorer for standard OPD, requiring no task reward during repair or training. Across three environments, two teachers and four students, R2S outperforms three state-of-the-art agentic-OPD methods in all twelve teacher–student–environment cells, by 12.8 success points on average over the best of them in each cell. We release our code in the supplementary material.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
On-Policy Distillation, Agent, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PVKhOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies agentic on-policy distillation (OPD), in which a student agent acts in a live environment while a frozen teacher scores every token it produces, so the teacher's next-token distribution on the student's own histories is the entire training signal. The paper argues that the usual 'compounding error' reading of OPD's failures is incomplete. Its analysis (Section 3; Qwen3.5-9B teacher, Qwen3.5-4B student) takes the turns where the untrained student's thinking departs most from the teacher's, including late in episodes, and shows that the teacher's own takeovers from exactly those states still finish the task more often than the student did from the same states (Table 1), so the student has not drifted beyond the teacher's competence. What differs between the two roles is that scoring is myopic: the teacher rates each token on a history it did not write, without seeing where the episode ends, so its scores let 'safe, delaying' moves (one more look, one more search) pass, which outcome-based RL would punish with zero reward. The authors call the divergence between what the teacher does and what it scores the rollout-scorer gap, and show that the same teacher's probability of the student's hedging and searching words falls two to three times more than that of the rest of its thinking only once one of its own finishing takeovers is in context. Method. Rollout-to-Scorer Distillation (R2S) mines one high-disagreement divergence turn per episode of the untrained student, runs K=8 teacher takeovers from each mined state, keeps the states where at least m takeovers pass a reward-free check (consensus among takeovers, or an LLM judge; ground truth as an upper reference), selects the shortest passing takeover as the reference, and then fine-tunes a copy of the teacher to reproduce the frozen teacher's takeover-conditioned distributions q+ on every assistant token of the student's own suffix from the divergence turn onward, without the takeover in context. The repaired teacher is refrozen and used as the sole scorer for otherwise ordinary OPD. No task reward enters repair or training. Validation. Three text-agent benchmarks (ALFWorld, WebShop, ScienceWorld), two teachers (Qwen3.5-9B; Qwen3-30B-A3B-Instruct-2507), four students (Qwen3.5-2B/4B; Qwen3-4B/8B) give twelve cells under one prompt, estimator and evaluation protocol, plus a self-distillation case in which the teacher is the student's own base model. Baselines are vanilla OPD, TCOD-F2B, Guided-OPD and FutureBridge. R2S is reported best in all twelve cells, by 12.8 points on average over the best of the three OPD baselines and by 17.8 / 9.8 over vanilla OPD; it also exceeds the teacher acting alone in every cell. Ablations vary the selection rule (consensus / judge / ground truth), the components (takeovers alone via a label-SFT control, versus takeovers plus informed targets), the loss estimator (sampled vs top-16), and the supervised span (divergence turn / prefix / suffix). A long-horizon analysis reports lower out-of-turns rates, fewer searching turns at about equal task turns, and fewer cumulative observed-failure turns for R2S students, together with target-fidelity measurements showing the repaired teacher is closer to q+ than the original teacher and than the label-SFT teacher.

Critical strengths
  1. The diagnosis is clean, general, and established with the right experiment. The rollout-scorer gap is a distinction the OPD literature has not made explicit, and Section 3 supports it rather than asserting it: from the states where the student diverges most, the teacher's takeovers finish more often than the student did (Table 1), the teacher's scores of the student's hedging and searching words fall by 8.2/6.2, 8.3/7.7 and 13.0/16.6 percent against 3.5/4.2/5.0 percent for the rest of its thinking once a finishing takeover is in context, and the paper checks the robustness of that reading (Tables 9, 10, 13). This matters because it explains why a whole family of otherwise sensible interventions - shorter episodes, teacher-controlled turns, prefix replay, bridge validation - can plateau at similar levels: they all keep the same scorer. The finding also converts into a design principle (move rollout competence into the scores OPD consumes, then freeze) rather than another curriculum, which is what makes the contribution more than incremental.
  2. The empirical scope is unusually wide and the validation is instrumented at the right level of granularity. Twelve teacher-student-environment cells under a shared protocol, spanning two model generations, a dense teacher and a mixture-of-experts teacher; a self-distillation setting in which the teacher is the student's own base model, which removes the 'a bigger model's knowledge is doing the work' confound and is the most informative single result in the paper (Table 3); a reward-free selection ablation that holds up against the ground-truth rule (Table 4); and a long-horizon readout (out-of-turns shares, searching versus task turns, cumulative observed-failure turns; Tables 7, 11, 12, 21, 22) that measures exactly the behavior the mechanism claim is about. The label-SFT control in Table 20, which separates a teacher that acts better from a teacher that scores better, is a thoughtful piece of experimental design.
Critical weaknesses
  1. The reported uncertainty does not support the headline claim, and two tables give inconsistent values for the same cell. Table 2 reports one training run per cell, with error bars that are the standard deviation over three decodes of a single final checkpoint - decoding noise within one trained model, not run-to-run variance - while the claim is a uniform win over three strong baselines in all twelve cells. Table 4 then reports, for what should be the same configuration (the sentence introducing the ablations says they vary one part of R2S in the Qwen3.5-9B-to-4B cells of Table 2, and Table 4's vanilla row matches Table 2 exactly at 65.8 / 34.6 / 34.6), a 'Consensus vote' row of ALFWorld 81.9, WebShop 63.1, ScienceWorld 39.5 - whereas Table 2's R2S row for that cell is 81.9, 55.9, 47.8. Only ALFWorld coincides; the Table 2 R2S row matches none of the three Table 4 rows on WebShop (63.1 / 55.5 / 65.0) or on ScienceWorld (39.5 / 50.3 / 49.0). Either the rows are the same run and the two tables contradict each other, or they are different runs of the same configuration and the pipeline's run-to-run spread is roughly seven to eight points on those two environments - three to five times the reported decode standard deviation. In the second case the all-twelve-cells claim is not established, because several of the 30B-A3B margins over the best baseline are only +3.8 to +7.4 points (WebShop 4B +3.8, WebShop 8B +7.0, ScienceWorld 4B +7.4, ALFWorld 8B +5.9). Figure 3 does add three training runs per method for the 9B-to-4B cells on all three environments, so those three of the twelve cells are covered at run level and are reassuring there; the remaining nine cells rest on a single run each. Figure 3's R2S band is also computed against a single shared repaired teacher (Appendix F.2), so it reflects student-training variance only, not repair variance. This is consequential because a uniform win across cells is the paper's central empirical claim.
Questions for authors
  1. The ablation credits most of the gain to a simpler mechanism than the one the method is built on, and the control that would isolate the paper's actual contribution is missing. In Table 5 the label-SFT arm - fine-tuning the teacher on the selected takeovers as ordinary SFT data, with no informed targets - already recovers 11.0 of R2S's 16.1 points on ALFWorld, 12.1 of 21.3 on WebShop and 6.4 of 13.2 on ScienceWorld, that is 68%, 57% and 48% of the total. The text nevertheless states that the takeovers 'help most as context for the teacher's scores of the student's own tokens, not as SFT data', which is contradicted on two of three environments and should be rewritten or supported. More importantly, nothing in the paper separates takeover-informed targets from a generic teacher-adaptation effect. The natural control - fine-tune the teacher on the same student suffix (or on held-out student-distribution text) toward its own closed-book targets q0, with no takeover in context - is not run. That control matters because Table 20 shows teacher fine-tuning alone moves acting performance by 15 to 20 points (63.1 to 78.5-82.9 on ALFWorld at 2B), so it is plausible that part of R2S's gain comes from adapting the teacher to the student's text distribution rather than from the information the takeover carries. Without that arm, the marginal value of Eq. (3), which is the paper's proposal, is not isolated from the extra fine-tuning step that surrounds it. For related-work positioning, the authors could discuss Ning Yang et al., Token-Importance Guided Direct Preference Optimization (ICLR 2026; https://openreview.net/forum?id=cMEnMVvMw9). Its semantic token-importance weighting provides a conceptual comparison for modifying local optimization signals, while R2S repairs the teacher that supplies those signals. This comparison would help clarify the mechanism without treating importance-weighted DPO as a direct agentic OPD baseline.
3 comments
AuthorsOpenReview ↗

Response to Reviewer PVKh (1/2): W1

We thank you for the detailed and insightful review. We greatly appreciate that you found the diagnosis "clean, general, and established with the right experiment", the rollout–scorer gap "a distinction the OPD literature has not made explicit", and the empirical scope "unusually wide". We reply to each point below.


W1: Do the results hold across training runs, and why do Tables 2 and 4 differ?

A: Table 2 mixed takeover-selection rules across environments, which caused the differences from Table 4 that you point out. Table 2 used:

  • ALFWorld: consensus vote
  • WebShop: LLM judge. Table 2's entry and Table 4's LLM-judge row are two decoding batches of one run (55.9 vs. 55.5 for 9B→4B).
  • ScienceWorld: completion check, which keeps a takeover that ends with the simulator's completion message. It is neither rule of Table 4, which is why Table 2's ScienceWorld column (e.g., 47.8 for 9B→4B) matches neither the consensus vote (39.5) nor the LLM judge (50.3).

In the final version, Table 2 will use the LLM judge in every cell, and the consensus-vote results will move to the appendix. Below we report all twelve cells under both rules:

Setup

  • Models: Qwen3.5-9B → Qwen3.5-2B/4B and Qwen3-30B-A3B-Instruct-2507 → Qwen3-4B/8B
  • Best prior method: the best of TCOD-F2B, Guided-OPD and FutureBridge in each cell, from Table 2

Results (student success, %)

Teacher → studentMethodALFWorldWebShopScienceWorld
9B → 2BBest prior method63.6±2.236.3±1.028.0±2.6
R2S, consensus vote77.9±1.363.2±1.224.2±2.4
R2S, LLM judge78.6±1.557.9±1.634.1±1.1
9B → 4BBest prior method61.6±2.035.5±1.332.2±1.1
R2S, consensus vote81.9±2.463.1±1.639.5±2.9
R2S, LLM judge82.6±0.955.5±3.050.3±1.5
30B → 4BBest prior method42.6±1.123.3±1.638.4±0.9
R2S, consensus vote54.5±1.424.6±0.343.0±3.8
R2S, LLM judge53.3±2.228.5±0.645.7±3.5
30B → 8BBest prior method48.8±1.823.4±0.737.1±2.0
R2S, consensus vote54.7±1.015.8±1.446.5±3.3
R2S, LLM judge52.4±1.430.4±1.242.8±2.9

Key insights

  • With the LLM judge, R2S beats the best prior method in all 12 cells. It leads by 3.6 to 21.6 points, +11.8 on average.
  • With the vote, R2S beats it in 10 of 12 cells. It falls 7.6 points short on WebShop 30B→8B, where only a quarter of the vote's references succeed, and 3.8 points short on ScienceWorld 9B→2B.

To measure run-to-run variance directly, we compare three training runs per method in every cell.

Results (student success, %)

Teacher → studentMethodALFWorldWebShopScienceWorld
9B → 2BBest prior method61.0±4.034.1±1.826.1±1.0
R2S78.2±3.256.2±1.637.1±0.7
9B → 4BBest prior method60.9±1.334.8±1.731.7±0.3
R2S80.2±1.457.4±2.845.8±4.8
30B → 4BBest prior method42.2±0.421.4±3.439.3±1.7
R2S54.4±0.728.7±0.644.7±1.5
30B → 8BBest prior method45.9±2.422.5±1.336.3±3.5
R2S55.9±0.233.9±0.650.4±1.6

Key insights

  • In all 12 cells, every R2S run beats every run of the best prior method. The closest pair, in 30B→4B ScienceWorld, is 1.7 points apart.

The three R2S runs of a cell share one repaired teacher. To include the variance of the repair itself, we repaired the 9B teacher three times independently.

Setup

  • Model: Qwen3.5-9B→Qwen3.5-4B

Results (success at update 150, %; mean ± SD over three runs)

MethodALFWorldWebShopScienceWorld
Best prior method60.9±1.334.8±1.731.7±0.3
R2S, three repairs80.1±1.255.3±1.445.3±4.5

Key insights

  • Redoing the repair adds little variance. The SD over three repairs is no larger than over three student runs on one repaired teacher (1.4, 2.8 and 4.8, above), and the means move by at most 2.1 points.
  • Every repair still beats every run of the best prior method. The closest pair, on ScienceWorld, is 9.8 points apart.
AuthorsOpenReview ↗

Response to Reviewer PVKh (2/2): Q1

Q1: Does the gain come from the takeovers or from fine-tuning the teacher?

A: Two points read differently from what we intended, so we clarify them before reporting the control you propose.

  • Label-SFT is not a separate, simpler mechanism: it runs the R2S pipeline except its last step. It uses the same mined states, takeovers and selected references, and only the fine-tuning target differs: label-SFT fits the reference's own tokens by standard SFT, while R2S fits the teacher's scores of the student's own tokens with the reference in context (Eq. (3)). Its 68%, 57% and 48% therefore come from the takeovers that R2S mines and selects, as the preceding sentence states ("the takeovers carry signal of their own"). The sentence you quote compares the two uses of these takeovers: at the reported doses, using them as context adds 5.1, 9.2 and 6.8 points over SFT.
  • Table 20 makes the opposite point: acting better does not make a teacher a better scorer. On ALFWorld 9B→2B, the label-SFT teachers act 2.9 and 7.3 points above the repaired teacher, yet their students match the R2S student (77.7 and 77.6 vs. 77.9). On WebShop, label-SFT teachers that act only 2.5 and 2.8 points below the repaired teacher train students 8.2 and 9.2 points below the R2S student.

Your control tests what remains: that fine-tuning helps by adapting the teacher to the student's text. Since q0 is the teacher's own closed-book distribution, fitting it brings in nothing from the takeovers. We ran it as proposed.

Setup

  • Model: Qwen3.5-9B→Qwen3.5-4B
  • T_q0: the teacher fine-tuned toward q0 with R2S's recipe, on the same student suffixes and positions as the R2S teacher in the last row (one of the repairs in W1)
  • T_q0-heldout: the same on held-out student text, with as many positions
  • Students: vanilla OPD against each teacher

Results (student success, %)

TeacherALFWorldWebShopScienceWorld
Original (Table 2)65.8±2.634.6±2.934.6±3.5
T_q062.0±0.336.1±0.632.9±1.3
T_q0-heldout64.0±1.137.1±1.030.7±6.2
R2S, same suffixes81.3±1.056.4±0.546.9±3.3

Key insights

  • Adapting the teacher to the student's text does not help the student. Both q0 students stay within 4 points of vanilla OPD and 14.0 to 20.3 points below R2S on the same suffixes.
  • Nor does it make the teacher a better actor. The q0 teachers act within 4 points of the untuned teacher, unlike the label-SFT teachers of Table 20, which fit the takeovers.

On related work, thank you for the pointer to TI-DPO (Yang et al., ICLR 2026). It reweights token-level preference-optimization signals by their semantic importance; R2S instead keeps the OPD objective unchanged and repairs the teacher that supplies the token-level signal. We will add this comparison to the related work.


We hope these replies address your concerns. If so, we would be grateful if you would consider raising your score, which would give us a better chance to present to the community the rollout–scorer gap, a distinction you note the OPD literature has not made explicit. We welcome any further questions.

Dear Reviewer PVKh,

Thank you again for your time and effort in reviewing our paper. As the discussion period is coming to a close, we would like to kindly follow up to check whether our rebuttal has adequately addressed your concerns. If there are any remaining questions, we would be happy to provide further clarification. If you feel your concerns have been resolved, we would sincerely appreciate it if you could kindly reconsider your score.

Best regards,

The Authors of Submission 2079

Reviewer_PQwqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies a rollout–scorer gap in agentic on-policy distillation (OPD), where a teacher capable of completing tasks may still provide ineffective token-level supervision that encourages unnecessary hesitation and searching. To address this issue, the authors propose Rollout-to-Scorer Distillation (R2S), which uses selected teacher takeovers to repair the teacher's scoring behavior without requiring task rewards. Experiments across three agentic benchmarks and multiple teacher–student configurations demonstrate consistent improvements over existing OPD baselines.

Critical strengths
  1. Interesting insight. The paper highlights an important distinction in agentic on-policy distillation: a capable teacher is not necessarily an effective scorer. Beyond strong task-solving capabilities, the teacher must also provide informative token-level supervision that reflects long-horizon task outcomes. This perspective is insightful and motivates the proposed approach.

  2. Comprehensive empirical evaluation. The authors evaluate their method across multiple environments, teacher–student configurations, and competitive baselines. The ablation studies are well designed and provide useful evidence for the effectiveness of the proposed components.

  3. Clear presentation. The paper is generally well written and organized, with a clear motivation, an intuitive methodological explanation, and a coherent experimental analysis.

Critical weaknesses
  1. Unclear consistency between Tables 2 and 4. The R2S results in Table 2 do not appear to match any of the takeover-selection configurations reported in Table 4. For example, the consensus-based results differ substantially on WebShop (55.9 vs. 63.1) and ScienceWorld (47.8 vs. 39.5). It is unclear whether these differences arise from different experimental settings or random seeds. The authors should clarify which selection strategy is used in the main experiments and explain these discrepancies.

  2. Fairness and attribution of improvements. Table 2 compares R2S against baselines using the original teacher, while Appendix G.1 shows that the repaired teacher itself achieves substantially stronger standalone performance. Although teacher improvement is part of the proposed method, this raises questions about how much of the gain comes from improved token-level supervision versus the additional teacher fine-tuning and data. The Label-SFT controls partially address this concern, but they are evaluated only with vanilla OPD. A more systematic comparison in which other OPD baselines also use similarly fine-tuned teachers would strengthen the conclusions.

  3. Potential trade-off between reducing hesitation and preserving exploration. The paper attributes much of the improvement to suppressing unnecessary hedging and searching behaviors. However, prior work (Kaur et al., 2026, Rethinking On-Policy Self-Distillation for Thinking Models) suggests that privileged-context distillation can suppress useful exploration and self-correction, particularly in mathematical reasoning. While the settings differ, it would be interesting to investigate whether R2S exhibits similar trade-offs in tasks where exploration and backtracking are essential. I do not consider this experiment necessary, but such an analysis could provide deeper insights into the applicability and limitations of the proposed approach.

The third point is intended as a constructive suggestion rather than a requirement. I would be willing to raise my score if the authors satisfactorily address the first two concerns, even if the third remains unaddressed.

[1] Kaur, Simran, et al. "Rethinking On-Policy Self-Distillation for Thinking Models." arXiv preprint arXiv:2607.05184 (2026).

Questions for authors

Please see the first two points in the Weaknesses section.

I would be willing to raise my score if the authors satisfactorily address these two concerns.

The third point is intended as a constructive suggestion for future work rather than a requirement for a higher score.

6 comments
AuthorsOpenReview ↗

Response to Reviewer PQwq (1/4): W1

We sincerely thank you for the thoughtful and constructive feedback. We greatly appreciate that you found an "interesting insight" in the point that "a capable teacher is not necessarily an effective scorer", and that you credit the "comprehensive empirical evaluation", the "well designed" ablations and the "clear presentation". We reply to each point below.


W1: Which selection rule does Table 2 use, and why do Tables 2 and 4 differ?

A: Table 2 mixed takeover-selection rules across environments, which caused the differences from Table 4 that you point out. Table 2 used:

  • ALFWorld: consensus vote
  • WebShop: LLM judge. Table 2's entry and Table 4's LLM-judge row are two decoding batches of one run (55.9 vs. 55.5 for 9B→4B).
  • ScienceWorld: completion check, which keeps a takeover that ends with the simulator's completion message. It is neither rule of Table 4, which is why Table 2's ScienceWorld column (e.g., 47.8 for 9B→4B) matches neither the consensus vote (39.5) nor the LLM judge (50.3).

In the final version, Table 2 will use the LLM judge in every cell, and the consensus-vote results will move to the appendix. Below we report all twelve cells under both rules:

Setup

  • Models: Qwen3.5-9B → Qwen3.5-2B/4B and Qwen3-30B-A3B-Instruct-2507 → Qwen3-4B/8B
  • Best prior method: the best of TCOD-F2B, Guided-OPD and FutureBridge in each cell, from Table 2

Results (student success, %)

Teacher → studentMethodALFWorldWebShopScienceWorld
9B → 2BBest prior method63.6±2.236.3±1.028.0±2.6
R2S, consensus vote77.9±1.363.2±1.224.2±2.4
R2S, LLM judge78.6±1.557.9±1.634.1±1.1
9B → 4BBest prior method61.6±2.035.5±1.332.2±1.1
R2S, consensus vote81.9±2.463.1±1.639.5±2.9
R2S, LLM judge82.6±0.955.5±3.050.3±1.5
30B → 4BBest prior method42.6±1.123.3±1.638.4±0.9
R2S, consensus vote54.5±1.424.6±0.343.0±3.8
R2S, LLM judge53.3±2.228.5±0.645.7±3.5
30B → 8BBest prior method48.8±1.823.4±0.737.1±2.0
R2S, consensus vote54.7±1.015.8±1.446.5±3.3
R2S, LLM judge52.4±1.430.4±1.242.8±2.9

Key insights

  • With the LLM judge, R2S beats the best prior method in all 12 cells. It leads by 3.6 to 21.6 points, +11.8 on average.
  • With the vote, R2S beats it in 10 of 12 cells. It falls 7.6 points short on WebShop 30B→8B, where only a quarter of the vote's references succeed, and 3.8 points short on ScienceWorld 9B→2B.
AuthorsOpenReview ↗

Response to Reviewer PQwq (2/4): W2

W2: Is the comparison fair, and does the gain come from a better actor or from better scores?

A: Following your suggestion, we trained the three baselines with the label-SFT teacher as well as the repaired teacher, so the label-SFT control now covers every OPD method, not only vanilla OPD.

Setup

  • Model: Qwen3.5-9B→Qwen3.5-4B
  • Teachers: the original teacher; the label-SFT teacher, as in Table 5; and the repaired teacher that R2S uses in Table 2
  • Rows from the paper (Tables 2 and 5): every Original row, vanilla OPD's Label-SFT row and R2S, which is vanilla OPD with the repaired teacher; one training run each, ± over three decodes
  • New rows: the three baselines with the Label-SFT or Repaired teacher; three training runs each, mean ± SD over runs

Results (student success, %)

MethodTeacherALFWorldWebShopScienceWorld
Vanilla OPDOriginal65.8±2.634.6±2.934.6±3.5
Label-SFT76.8±3.546.7±1.341.0±3.0
R2SRepaired81.9±2.455.9±1.447.8±3.0
TCOD-F2BOriginal59.6±1.535.1±0.930.6±0.7
Label-SFT78.8±2.146.9±0.840.1±2.3
Repaired84.4±2.156.9±0.146.0±5.4
Guided-OPDOriginal61.6±2.035.5±1.332.2±1.1
Label-SFT79.4±2.545.3±1.340.5±2.1
Repaired85.0±0.755.9±1.044.7±4.4
FutureBridgeOriginal58.4±3.733.9±1.730.7±1.3
Label-SFT80.4±1.247.1±1.833.5±2.0
Repaired84.5±0.256.9±2.440.3±5.8

Key insights

  • The repaired teacher also lifts the other OPD methods. It adds 20.4 to 26.1 points on ALFWorld and WebShop, and 9.6 to 15.4 on ScienceWorld.
  • With it, the baselines reach R2S's level. The gain therefore comes with the repaired teacher, not with R2S's student-side training, which is plain OPD.
  • The label-SFT teacher lifts every method too, but less than the repaired teacher. With it, the three baselines gain 9.8 to 22.0 points on ALFWorld and WebShop and 2.8 to 9.5 on ScienceWorld. Yet each stays below its result with the repaired teacher, by 4.1 to 5.6 points on ALFWorld, 9.8 to 10.6 on WebShop and 4.2 to 6.8 on ScienceWorld, close to vanilla OPD's 5.1, 9.2 and 6.8.
  • The extra fine-tuning and data explain part of the gain; fitting the informed scores explains the rest, whichever OPD method uses the teacher. Both teachers use the same selected takeovers and one epoch of fine-tuning; only the target differs: label-SFT fits the takeovers' own tokens, while repair fits the teacher's scores of the student's tokens with the takeover in context.
AuthorsOpenReview ↗

Response to Reviewer PQwq (3/4): W2 (continued)

W2 (continued)

As we understand it, you also ask whether a teacher's acting and its scoring can be separated. We think this is an excellent and fundamental question. The repaired teacher also acts better, and the two cannot be fully separated: in vanilla OPD the teacher never acts during training, so whatever it can do reaches the student only through its scores on the student's tokens. The same problem arises in work on SFT before RL, which studies how SFT should prepare a model for later RL (Kang et al. [1]; Zhang et al. [2]).

SFT before RLRepair before OPD
Prepared modelSFT checkpointRepaired teacher
Own performanceSFT scoreActing alone
What we wantSuccess after RLIts student's success
Why they differRL uses its own rollouts, not SFT dataOPD uses the student's states, not its own

In both, the model's own performance and what we want come from the same weights, so a gain in the next stage could be credited to either. Those studies cannot separate the two either; instead, they show that the two come apart: after identical RL, stronger SFT checkpoints can end up worse than weaker ones. We run the same test on teachers: if acting determined teaching, a teacher that acts about as well as the repaired one would teach about as well. The label-SFT teacher above is one, and Table 20 reports how it acts and how it scores.

Setup

  • Models: Qwen3.5-9B→Qwen3.5-2B/4B, from Table 20
  • Label-SFT: one epoch, as in Table 5
  • KL to q⁺: divergence from q⁺, the original teacher's scores with the reference in context, at the student's thinking tokens on held-out states
  • Students: OPD against each teacher, one run each; ± over three decodes

Results (success, %, and KL to q⁺)

CellTeacherActing aloneKL to q⁺Its student
ALFWorld, 9B → 2BLabel-SFT78.5±1.90.09977.7±2.6
Repaired75.6±1.60.04977.9±1.3
WebShop, 9B → 4BOriginal33.5±0.30.12334.6±2.9
Label-SFT45.7±0.20.12846.7±1.3
Repaired48.5±1.90.07455.9±1.4

Key insight

  • A teacher's own success does not determine how well it teaches. On ALFWorld 9B→2B, the label-SFT teacher acts 2.9 points above the repaired teacher, but its scores are twice as far from q⁺ (0.099 vs. 0.049), and its student only matches the R2S student; on WebShop 9B→4B, it acts 2.8 points below, but its student is 9.2 points below.

References

[1] Feiyang Kang, Michael Kuchnik, Karthik Padthe, Marin Vlastelica, Ruoxi Jia, Carole-Jean Wu, and Newsha Ardalani. Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead. ICLR 2026.

[2] Dylan Zhang, Yufeng Xu, Haojin Wang, Qingzhi Chen, and Hao Peng. Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learning. ICML 2026.

Reviewer_6C9cOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the mismatch between teacher rollout competence and token-level supervision in agentic on-policy distillation (OPD). The authors identify a rollout–scorer gap, where teachers can complete tasks from student states but fail to provide effective token-level guidance. To address this, they propose R2S, which uses selected teacher takeovers to repair the teacher's scoring behavior before standard OPD. Experiments across three environments and twelve teacher–student–environment combinations demonstrate consistent improvements over strong baselines.

Critical strengths
  1. The paper presents an insightful observation about agentic OPD. The rollout–scorer gap offers a fresh perspective on distillation failures beyond compounding errors and distribution shift.

  2. R2S is novel and well-motivated. It transfers information from teacher rollouts into token-level supervision without requiring task rewards or modifying the downstream OPD objective.

  3. The experimental results are impressive. Consistent improvements across diverse settings, together with self-distillation experiments, ablations, and long-horizon behavioral analyses, provide strong evidence of the method's effectiveness.

Critical weaknesses
  1. The reliability of reward-free takeover selection deserves further investigation. Although Table 4 demonstrates improvements with different selection criteria, the actual success rate of selected references is not reported. In particular, the consensus criterion requires agreement from only two of eight takeovers, which does not guarantee task success. Moreover, selected takeovers are presented as successful expert demonstrations even when they may have failed. The impact of such incorrect references remains unclear.

  2. R2S introduces substantial computational overhead, including up to 19,200 teacher takeovers per setting and an additional teacher fine-tuning stage. While the downstream training budgets are matched, the total computational costs are not. A compute-matched comparison would help clarify the efficiency of R2S relative to existing methods.

Questions for authors
  1. What is the actual success rate of takeovers selected by the reward-free criteria? How sensitive is R2S to incorrect reference trajectories?

  2. Could the authors provide a breakdown of the total computational cost and compare R2S with existing methods under a matched compute budget?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 6C9c (1/2): W1 & Q1

We sincerely thank you for the positive assessment and for two precise questions; both led to new analyses. We greatly appreciate that you found the rollout–scorer gap "an insightful observation" that "offers a fresh perspective on distillation failures", R2S "novel and well-motivated", and the experimental results "impressive". We reply to each point below.


W1 & Q1: How often do the selected references succeed, and how sensitive is R2S to incorrect ones?

A: We report the success rate of the selected references under the two reward-free rules of Table 4: the m=2 consensus vote you point to and the LLM judge.

Setup

  • Model: Qwen3.5-9B→Qwen3.5-4B
  • LLM judge: Qwen3.5-9B with reasoning mode on; temperature 0; sees neither the reward nor the completion flag

Results (selected references that succeeded)

Selection ruleALFWorldWebShopScienceWorld
Consensus vote77.4%65.3%60.3%
LLM judge93.0%75.1%81.8%

Key insights

  • Most selected references are correct. Under either rule, more than 60% of the selected references succeed in every environment.
  • The LLM judge is more precise. It is 15.6, 9.8 and 21.5 points above the consensus vote, which only checks that the takeovers agree on how they end.

To further study how much the success of the references matters, we mixed correct and failed references at fixed ratios in the same cells. As in the paper, every reference keeps the header that presents it as a successful expert completion (Appendix C), so the failed ones here are exactly the case you raise.

Setup

  • Correct ratio (by the ground truth): 100%, 75%, 50%, 25%, 0%. We keep the data amount the same and randomly select the correct and wrong data
  • Model: Qwen3.5-9B→Qwen3.5-4B
  • Training Recipe: same as the main paper; three independent training runs at 75%, 50%, 25% and 0%, reported as mean ± SD
  • 100% column: the ground-truth row of Table 4; vanilla OPD column: Table 2; as in the paper, each is one training run, with ± over three decodes

Results (student success, %)

Correct references100%75%50%25%0%Vanilla OPD
ALFWorld85.9±1.283.8±0.578.0±2.367.7±2.662.8±2.465.8±2.6
WebShop65.0±1.460.5±0.553.5±2.547.3±0.640.5±1.234.6±2.9
ScienceWorld49.0±1.147.6±2.245.1±0.943.3±1.836.6±2.334.6±3.5

Key insights

  • With the amount of data held fixed, the precision of the references matters. In all three environments, student success rises monotonically with the share of correct references, by 23.1, 24.5 and 12.4 points from 0% to 100%.
  • Even with only 25% correct references, R2S matches vanilla OPD on ALFWorld and beats it on WebShop and ScienceWorld. OPD aligns the student's token distribution with the teacher's and never uses the final reward, so a reference helps through the text it puts in the teacher's context, and even a failed one can show the teacher useful states and steps.
AuthorsOpenReview ↗

Response to Reviewer 6C9c (2/2): W2 & Q2

W2 & Q2: Total cost and a compute-matched comparison

A: We first give the extra cost of R2S and of the other methods, then compare R2S with vanilla OPD at matched compute.

Setup

  • Model: Qwen3.5-9B→Qwen3.5-4B
  • Stages: Takeover covers the untrained student's episodes, mining the divergence states and the teacher's takeovers; Fine-tuning covers the informed targets and the teacher's fine-tuning
  • API prices: EmpirioLabs and Together AI list prices, November 2026

Results (extra cost of R2S, summed over the three environments)

CostTakeoverFine-tuningTotal
GPU-hours17.541.3218.86
API prices w/ caching (USD)28615301
API prices w/o caching (USD)50515520

Key insights

  • R2S adds about 19 GPU-hours, or USD 301 at API prices with caching, for the three environments. That is about 6 GPU-hours or USD 100 per environment, spent once before R2S's own OPD run. Vanilla OPD's own 150 steps take 43.2 GPU-hours (12.4, 20.2 and 10.6 per environment); R2S runs the same 150 steps after its repair, so its total is 62.1 GPU-hours
  • Almost all of it is the takeover stage; fine-tuning the teacher is cheap. The takeover stage accounts for 93% of the GPU-hours and 95% of the cost at API prices with caching (97% without); computing the informed targets and fine-tuning the teacher take 1.32 GPU-hours and USD 15 in total.

We also measured the GPU-hours of every method.

Setup

  • Model: Qwen3.5-9B→Qwen3.5-4B
  • GPU-hours: summed over the three environments; as in the paper, every method trains for 150 steps
  • R2S: the consensus vote of Table 4

Results (extra GPU-hours relative to vanilla OPD, summed over the three environments; student success, %, from Tables 2 and 4)

MethodExtra GPU-hoursALFWorldWebShopScienceWorld
TCOD-F2B−0.2659.6±1.535.1±0.930.6±0.7
FutureBridge+2.9658.4±3.733.9±1.730.7±1.3
Guided-OPD+10.3461.6±2.035.5±1.332.2±1.1
R2S, consensus vote+18.8681.9±2.463.1±1.639.5±2.9

Key insights

  • Guided-OPD and FutureBridge also spend teacher generation inside training: Guided-OPD hands part of the turns to the teacher, and FutureBridge trains the student on validated teacher bridges. TCOD-F2B, which trains the student on shortened episodes that lengthen over training, costs about as much as vanilla OPD.
  • R2S's extra is its one-time repair. Extra GPU-hours alone do not lift the student: Guided-OPD and FutureBridge never end much above vanilla OPD, while R2S ends above it in all three environments.

To compare at matched compute, we trained vanilla OPD for 250 steps instead of 150. At its measured cost per step, that is 72.0 GPU-hours over the three environments, more than R2S's total of 62.1.

Setup

  • Model: Qwen3.5-9B→Qwen3.5-4B
  • R2S: the consensus vote of Table 4; vanilla OPD: Table 2; as in the paper, each is one training run, with ± over three decodes
  • Vanilla OPD + compute-matched: 250 steps in each environment; three independent training runs, reported as mean ± SD

Results (student success, %)

ArmALFWorldWebShopScienceWorld
R2S, consensus vote81.9±2.463.1±1.639.5±2.9
Vanilla OPD65.8±2.634.6±2.934.6±3.5
Vanilla OPD + compute-matched67.2±2.135.0±2.333.0±2.7

Key insight

  • Even with more GPU-hours than R2S, vanilla OPD stays below it. The extra GPU-hours make little difference. Training longer does not change the teacher, and its plain scores let the student's delay pass.

We hope these answers reinforce your positive assessment. If so, we would be grateful if you would consider raising your score, which would help us bring a new perspective to the community: a capable teacher can still be a myopic scorer. We would be glad to answer any further questions.

Dear Reviewer 6C9c,

Thank you again for your time and effort in reviewing our paper. As the discussion period is coming to a close, we would like to kindly follow up to check whether our rebuttal has adequately addressed your concerns. If there are any remaining questions, we would be happy to provide further clarification. If you feel your concerns have been resolved, we would sincerely appreciate it if you could kindly reconsider your score.

Best regards,

The Authors of Submission 2079

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal and Discussion

We sincerely thank all reviewers for their insightful and constructive feedback, and the ACs, SACs and PCs for their efforts throughout the review process. As the discussion period closes in less than a day, we provide a summary below for the convenience of the AC and reviewers.


Summary

A capable teacher can still be a myopic scorer.

Failures of agentic on-policy distillation (OPD) are usually attributed to the student drifting beyond the teacher's reach. We show that from those very states, the teacher can still finish the task, yet its next-token scores do not say so; we call this the rollout–scorer gap. Existing methods control where the student trains, while R2S repairs how the teacher scores, so the two are complementary. Without any task reward, R2S outperforms three state-of-the-art agentic-OPD methods in all 12 teacher–student–environment cells.

All three reviewers recognize the rollout–scorer gap as a new perspective for the agentic-OPD community. Reviewer 6C9c finds that it "offers a fresh perspective on distillation failures beyond compounding errors and distribution shift". Reviewer PQwq calls the distinction that "a capable teacher is not necessarily an effective scorer" an "interesting insight". Reviewer PVKh notes that it is "a distinction the OPD literature has not made explicit" and that it "explains why a whole family of otherwise sensible interventions [...] can plateau at similar levels: they all keep the same scorer".

The reviewers agree just as clearly on the evidence: "The experimental results are impressive" (6C9c); the paper offers a "comprehensive empirical evaluation" whose ablations "are well designed" (PQwq); and "the empirical scope is unusually wide", with the self-distillation setting "the most informative single result in the paper" (PVKh).

During the rebuttal, we addressed every concern with more than ten new experiments and analyses. Reviewer PVKh's weak rejection is the only remaining negative rating, and we have not yet heard back from this reviewer. We believe the additional evidence resolves every substantive concern raised in that review.


Reviewer PVKh's Concerns

Raised by both Reviewers PQwq and PVKh. Both reviewers asked why Tables 2 and 4 differ, and whether the gain comes from the repaired scores or from extra teacher fine-tuning. We thank them for these questions, which led to two of the rebuttal's central additions. In brief: Table 2 had mixed selection rules, and with the LLM judge throughout, R2S leads the best prior method in all 12 cells; with the same takeovers, the repaired teacher beats the label-SFT teacher under every OPD method. Reviewer PQwq has kindly confirmed that "the authors' thorough response and additional experiments have addressed my concerns".

Raised only by Reviewer PVKh. With three training runs per method in all 12 cells, every R2S run beats every run of the best prior method, and the same holds across three independent repairs of the teacher. In the control Reviewer PVKh proposed, fine-tuning the teacher without the takeovers leaves students at the level of vanilla OPD, far below R2S. We will also add the related work they suggested.


Reviewer 6C9c's Concerns

Reliability of reward-free selection. Most selected references succeed under both selection rules. When correct and failed references are mixed at fixed ratios, student success rises monotonically with the share of correct ones, and even when most references are wrong, R2S still matches or beats vanilla OPD in every environment.

Computational cost. The repair is a one-time cost. Under a matched compute budget, with vanilla OPD trained longer so that it uses at least the same total compute as R2S, vanilla OPD still clearly trails R2S in every environment.


Conclusion

We believe the rebuttal and additional evidence resolve the substantive concerns raised in the reviews and support acceptance of the paper. We hope to share with the community the perspective recognized by all three reviewers: a capable teacher can still be a myopic scorer. We thank the AC and all three reviewers for their time and for questions that made the paper stronger.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.