- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies agentic on-policy distillation (OPD), in which a student agent acts in a live environment while a frozen teacher scores every token it produces, so the teacher's next-token distribution on the student's own histories is the entire training signal. The paper argues that the usual 'compounding error' reading of OPD's failures is incomplete. Its analysis (Section 3; Qwen3.5-9B teacher, Qwen3.5-4B student) takes the turns where the untrained student's thinking departs most from the teacher's, including late in episodes, and shows that the teacher's own takeovers from exactly those states still finish the task more often than the student did from the same states (Table 1), so the student has not drifted beyond the teacher's competence. What differs between the two roles is that scoring is myopic: the teacher rates each token on a history it did not write, without seeing where the episode ends, so its scores let 'safe, delaying' moves (one more look, one more search) pass, which outcome-based RL would punish with zero reward. The authors call the divergence between what the teacher does and what it scores the rollout-scorer gap, and show that the same teacher's probability of the student's hedging and searching words falls two to three times more than that of the rest of its thinking only once one of its own finishing takeovers is in context. Method. Rollout-to-Scorer Distillation (R2S) mines one high-disagreement divergence turn per episode of the untrained student, runs K=8 teacher takeovers from each mined state, keeps the states where at least m takeovers pass a reward-free check (consensus among takeovers, or an LLM judge; ground truth as an upper reference), selects the shortest passing takeover as the reference, and then fine-tunes a copy of the teacher to reproduce the frozen teacher's takeover-conditioned distributions q+ on every assistant token of the student's own suffix from the divergence turn onward, without the takeover in context. The repaired teacher is refrozen and used as the sole scorer for otherwise ordinary OPD. No task reward enters repair or training. Validation. Three text-agent benchmarks (ALFWorld, WebShop, ScienceWorld), two teachers (Qwen3.5-9B; Qwen3-30B-A3B-Instruct-2507), four students (Qwen3.5-2B/4B; Qwen3-4B/8B) give twelve cells under one prompt, estimator and evaluation protocol, plus a self-distillation case in which the teacher is the student's own base model. Baselines are vanilla OPD, TCOD-F2B, Guided-OPD and FutureBridge. R2S is reported best in all twelve cells, by 12.8 points on average over the best of the three OPD baselines and by 17.8 / 9.8 over vanilla OPD; it also exceeds the teacher acting alone in every cell. Ablations vary the selection rule (consensus / judge / ground truth), the components (takeovers alone via a label-SFT control, versus takeovers plus informed targets), the loss estimator (sampled vs top-16), and the supervised span (divergence turn / prefix / suffix). A long-horizon analysis reports lower out-of-turns rates, fewer searching turns at about equal task turns, and fewer cumulative observed-failure turns for R2S students, together with target-fidelity measurements showing the repaired teacher is closer to q+ than the original teacher and than the label-SFT teacher.
Critical strengths
- The diagnosis is clean, general, and established with the right experiment. The rollout-scorer gap is a distinction the OPD literature has not made explicit, and Section 3 supports it rather than asserting it: from the states where the student diverges most, the teacher's takeovers finish more often than the student did (Table 1), the teacher's scores of the student's hedging and searching words fall by 8.2/6.2, 8.3/7.7 and 13.0/16.6 percent against 3.5/4.2/5.0 percent for the rest of its thinking once a finishing takeover is in context, and the paper checks the robustness of that reading (Tables 9, 10, 13). This matters because it explains why a whole family of otherwise sensible interventions - shorter episodes, teacher-controlled turns, prefix replay, bridge validation - can plateau at similar levels: they all keep the same scorer. The finding also converts into a design principle (move rollout competence into the scores OPD consumes, then freeze) rather than another curriculum, which is what makes the contribution more than incremental.
- The empirical scope is unusually wide and the validation is instrumented at the right level of granularity. Twelve teacher-student-environment cells under a shared protocol, spanning two model generations, a dense teacher and a mixture-of-experts teacher; a self-distillation setting in which the teacher is the student's own base model, which removes the 'a bigger model's knowledge is doing the work' confound and is the most informative single result in the paper (Table 3); a reward-free selection ablation that holds up against the ground-truth rule (Table 4); and a long-horizon readout (out-of-turns shares, searching versus task turns, cumulative observed-failure turns; Tables 7, 11, 12, 21, 22) that measures exactly the behavior the mechanism claim is about. The label-SFT control in Table 20, which separates a teacher that acts better from a teacher that scores better, is a thoughtful piece of experimental design.
Critical weaknesses
- The reported uncertainty does not support the headline claim, and two tables give inconsistent values for the same cell. Table 2 reports one training run per cell, with error bars that are the standard deviation over three decodes of a single final checkpoint - decoding noise within one trained model, not run-to-run variance - while the claim is a uniform win over three strong baselines in all twelve cells. Table 4 then reports, for what should be the same configuration (the sentence introducing the ablations says they vary one part of R2S in the Qwen3.5-9B-to-4B cells of Table 2, and Table 4's vanilla row matches Table 2 exactly at 65.8 / 34.6 / 34.6), a 'Consensus vote' row of ALFWorld 81.9, WebShop 63.1, ScienceWorld 39.5 - whereas Table 2's R2S row for that cell is 81.9, 55.9, 47.8. Only ALFWorld coincides; the Table 2 R2S row matches none of the three Table 4 rows on WebShop (63.1 / 55.5 / 65.0) or on ScienceWorld (39.5 / 50.3 / 49.0). Either the rows are the same run and the two tables contradict each other, or they are different runs of the same configuration and the pipeline's run-to-run spread is roughly seven to eight points on those two environments - three to five times the reported decode standard deviation. In the second case the all-twelve-cells claim is not established, because several of the 30B-A3B margins over the best baseline are only +3.8 to +7.4 points (WebShop 4B +3.8, WebShop 8B +7.0, ScienceWorld 4B +7.4, ALFWorld 8B +5.9). Figure 3 does add three training runs per method for the 9B-to-4B cells on all three environments, so those three of the twelve cells are covered at run level and are reassuring there; the remaining nine cells rest on a single run each. Figure 3's R2S band is also computed against a single shared repaired teacher (Appendix F.2), so it reflects student-training variance only, not repair variance. This is consequential because a uniform win across cells is the paper's central empirical claim.
Questions for authors
- The ablation credits most of the gain to a simpler mechanism than the one the method is built on, and the control that would isolate the paper's actual contribution is missing. In Table 5 the label-SFT arm - fine-tuning the teacher on the selected takeovers as ordinary SFT data, with no informed targets - already recovers 11.0 of R2S's 16.1 points on ALFWorld, 12.1 of 21.3 on WebShop and 6.4 of 13.2 on ScienceWorld, that is 68%, 57% and 48% of the total. The text nevertheless states that the takeovers 'help most as context for the teacher's scores of the student's own tokens, not as SFT data', which is contradicted on two of three environments and should be rewritten or supported. More importantly, nothing in the paper separates takeover-informed targets from a generic teacher-adaptation effect. The natural control - fine-tune the teacher on the same student suffix (or on held-out student-distribution text) toward its own closed-book targets q0, with no takeover in context - is not run. That control matters because Table 20 shows teacher fine-tuning alone moves acting performance by 15 to 20 points (63.1 to 78.5-82.9 on ALFWorld at 2B), so it is plausible that part of R2S's gain comes from adapting the teacher to the student's text distribution rather than from the information the takeover carries. Without that arm, the marginal value of Eq. (3), which is the paper's proposal, is not isolated from the extra fine-tuning step that surrounds it. For related-work positioning, the authors could discuss Ning Yang et al., Token-Importance Guided Direct Preference Optimization (ICLR 2026; https://openreview.net/forum?id=cMEnMVvMw9). Its semantic token-importance weighting provides a conceptual comparison for modifying local optimization signals, while R2S repairs the teacher that supplies those signals. This comparison would help clarify the mechanism without treating importance-weighted DPO as a direct agentic OPD baseline.
3 comments
Response to Reviewer PVKh (2/2): Q1
Q1: Does the gain come from the takeovers or from fine-tuning the teacher?
A: Two points read differently from what we intended, so we clarify them before reporting the control you propose.
- Label-SFT is not a separate, simpler mechanism: it runs the R2S pipeline except its last step. It uses the same mined states, takeovers and selected references, and only the fine-tuning target differs: label-SFT fits the reference's own tokens by standard SFT, while R2S fits the teacher's scores of the student's own tokens with the reference in context (Eq. (3)). Its 68%, 57% and 48% therefore come from the takeovers that R2S mines and selects, as the preceding sentence states ("the takeovers carry signal of their own"). The sentence you quote compares the two uses of these takeovers: at the reported doses, using them as context adds 5.1, 9.2 and 6.8 points over SFT.
- Table 20 makes the opposite point: acting better does not make a teacher a better scorer. On ALFWorld 9B→2B, the label-SFT teachers act 2.9 and 7.3 points above the repaired teacher, yet their students match the R2S student (77.7 and 77.6 vs. 77.9). On WebShop, label-SFT teachers that act only 2.5 and 2.8 points below the repaired teacher train students 8.2 and 9.2 points below the R2S student.
Your control tests what remains: that fine-tuning helps by adapting the teacher to the student's text. Since q0 is the teacher's own closed-book distribution, fitting it brings in nothing from the takeovers. We ran it as proposed.
Setup
- Model: Qwen3.5-9B→Qwen3.5-4B
- T_q0: the teacher fine-tuned toward q0 with R2S's recipe, on the same student suffixes and positions as the R2S teacher in the last row (one of the repairs in W1)
- T_q0-heldout: the same on held-out student text, with as many positions
- Students: vanilla OPD against each teacher
Results (student success, %)
| Teacher | ALFWorld | WebShop | ScienceWorld |
|---|---|---|---|
| Original (Table 2) | 65.8±2.6 | 34.6±2.9 | 34.6±3.5 |
| T_q0 | 62.0±0.3 | 36.1±0.6 | 32.9±1.3 |
| T_q0-heldout | 64.0±1.1 | 37.1±1.0 | 30.7±6.2 |
| R2S, same suffixes | 81.3±1.0 | 56.4±0.5 | 46.9±3.3 |
Key insights
- Adapting the teacher to the student's text does not help the student. Both q0 students stay within 4 points of vanilla OPD and 14.0 to 20.3 points below R2S on the same suffixes.
- Nor does it make the teacher a better actor. The q0 teachers act within 4 points of the untuned teacher, unlike the label-SFT teachers of Table 20, which fit the takeovers.
On related work, thank you for the pointer to TI-DPO (Yang et al., ICLR 2026). It reweights token-level preference-optimization signals by their semantic importance; R2S instead keeps the OPD objective unchanged and repairs the teacher that supplies the token-level signal. We will add this comparison to the related work.
We hope these replies address your concerns. If so, we would be grateful if you would consider raising your score, which would give us a better chance to present to the community the rollout–scorer gap, a distinction you note the OPD literature has not made explicit. We welcome any further questions.
Dear Reviewer PVKh,
Thank you again for your time and effort in reviewing our paper. As the discussion period is coming to a close, we would like to kindly follow up to check whether our rebuttal has adequately addressed your concerns. If there are any remaining questions, we would be happy to provide further clarification. If you feel your concerns have been resolved, we would sincerely appreciate it if you could kindly reconsider your score.
Best regards,
The Authors of Submission 2079
Response to Reviewer PVKh (1/2): W1
We thank you for the detailed and insightful review. We greatly appreciate that you found the diagnosis "clean, general, and established with the right experiment", the rollout–scorer gap "a distinction the OPD literature has not made explicit", and the empirical scope "unusually wide". We reply to each point below.
A: Table 2 mixed takeover-selection rules across environments, which caused the differences from Table 4 that you point out. Table 2 used:
In the final version, Table 2 will use the LLM judge in every cell, and the consensus-vote results will move to the appendix. Below we report all twelve cells under both rules:
Setup
Results (student success, %)
Key insights
To measure run-to-run variance directly, we compare three training runs per method in every cell.
Results (student success, %)
Key insights
The three R2S runs of a cell share one repaired teacher. To include the variance of the repair itself, we repaired the 9B teacher three times independently.
Setup
Results (success at update 150, %; mean ± SD over three runs)
Key insights