- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors proposed OSD, training students on teacher continuations from prefixes generated by the evolving student. It uses cross-entropy on teacher tokens and supports text-only teachers. Experiments on reasoning and interactive tasks report improved performance, reduced training time with asynchronous execution, and better retention and sequential learning.
Critical strengths
- The agent experiments show meaningful gains over the tested OPD variants, especially on ScienceWorld and TextCraft.
Critical weaknesses
-
Novelty appears incremental. The cited Revisiting DAgger work already uses online student prefixes followed by teacher continuations. The benefit of OSD's changes is not clearly established.
-
The experiments do not isolate the claimed benefit of evolving student states. In Section 5.2, offline SFT reuses teacher trajectories while OSD obtains fresh continuations. State adaptation is therefore mixed with data freshness and potentially different teacher budgets. Fixed-prefix sampling with fresh teacher continuations, at matched budgets, would clarify the mechanism.
-
The manuscript contains repeated reporting errors. Figure 6 labels avg@8 results as pass@8, and Figure 11 still calls the method ROSE. Its 12.5% pass@4 also cannot arise from a single evaluation of 50 equally weighted tasks with four attempts each without an unstated aggregation scheme. Table 2 contradicts the claim of consistently outperforming existing distillation methods (BabyAI: 67.50 vs GuidedOPD 67.78). These errors affect interpretation of the results and indicate insufficient checking of the manuscript.
Questions for authors
-
What is the main contribution beyond the cited Revisiting DAgger work?
-
How do the existing results separate the benefit of evolving student states from fresh teacher continuations and different teacher budgets in Figure 7? Please clarify teacher calls, token counts and error bars.
-
How were k and the OPD baseline settings selected? Was OPD independently tuned, given the different learning rates and top-16 KL approximation?
-
Please clarify the metric labels in Figure 6 and the evaluation and averaging in Figure 11. How was 12.5% pass@4 obtained from 50 tasks with four attempts each?
3 comments
[Part 2/2]
Takeaway for a fixed teacher budget: our results so far suggest spending it on the current student's states (OSD 20.25 vs 15.75 for the same number of continuations on the initial student's states). The same holds in continual learning: starting from a student already trained with offline distillation, further training with OSD keeps improving it (39.1%), while more offline SFT does not (at most 27.3%). As for where to hand off within a trajectory, the prefix length k was not specifically tuned for OSD. The paper already contains a handoff ablation (App., Fig. 9), where varying how many turns the student plays before the teacher takes over shows that an intermediate handoff works best. To further address the reviewer's concern, we additionally ran a prefix-length ablation on RLVE with the teacher budget fixed: every prefix length we tried clearly improves over the original student, and pass@8 varies by less than four points across them, which suggests that OSD is not overly sensitive to where the teacher takes over. Please refer to Appendix C.1 of the revised manuscript for details.
W3 / Q4. "Clarify the metric labels in Figure 6 and the evaluation and averaging in Figure 11"
We thank the reviewer for the careful reading. These are typos, and we have corrected them in the revision. In Fig. 6, the caption should read avg@8 rather than pass@8; the bars are the same avg@8 values as in Fig. 5. In Fig. 11, "ROSE" is an earlier internal name of OSD, and the reported numbers are avg@4 rather than pass@4: 12.5% corresponds to 25 successes out of 200 episodes (50 tasks × 4 attempts). For Table 2, we have rewritten the sentence on consistent superiority so that it describes the table precisely in the revised manuscript.
Since Fig. 11 concerns terminal agents, we have added a larger study on TMax, a suite of long-horizon terminal tasks in which the agent works inside a real sandboxed shell over up to 60 turns. We distill Qwen3.5-27B into Qwen3.5-9B. We compare OSD with the same alternatives as in the paper, all trained on the same tasks from the same teacher: supervised fine-tuning on the teacher's own trajectories, and OPD. In OSD the student starts every episode on its own and the teacher takes over partway through, so the teacher's actions are learned in the situations the student actually reaches. We report results on the 80 TMax tasks closest to the training distribution, three runs per model (success rate, %; 95% CI from a task-level bootstrap):
| success rate | 95% CI | |
|---|---|---|
| Qwen3.5-9B (original) | 64.17 | [54.58, 73.75] |
| Teacher-SFT | 67.08 | [57.50, 76.67] |
| OPD | 68.75 | [58.75, 78.33] |
| OSD | 72.08 | [62.08, 81.67] |
OSD reaches the highest success rate, suggesting that learning the teacher's actions in the situations the student itself reaches carries over to long-horizon terminal agents at a larger scale. We have included this study in the revised manuscript (Appendix C.3).
Q3. "Was OPD independently tuned?"
Its configuration follows the standard recipe of Li et al. (2026c), as stated in the original manuscript. Its configuration, including the learning rate, is also the standard choice in recent on-policy distillation work (Ye et al., 2025; Zhong et al., 2026; Xu et al., 2026; Xin et al., 2026; Hao et al., 2026; He et al., 2026). We have included a detailed comparison across OPD settings in the manuscript revision (Appendix D).
References
- Hao, Dongze, et al. "Stabilizing On-Policy Distillation for MLLM Reasoning with Global Normalization." arXiv preprint arXiv:2606.09091 (2026).
- Ye, Tianzhu, et al. "Black-box on-policy distillation of large language models." arXiv preprint arXiv:2511.10643 (2025).
- Li, Yaxuan, et al. "Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe." arXiv preprint arXiv:2604.13016 (2026).
- Xin, Haoran, et al. "Escaping the KL Agreement Trap in On-Policy Distillation." arXiv preprint arXiv:2606.09471 (2026).
- Zhong, Qiyong, et al. "SOD: Step-wise on-policy distillation for small language model agents." arXiv preprint arXiv:2605.07725 (2026).
- Xu, Yuanda, et al. "Beyond GRPO and on-policy distillation: An empirical sparse-to-dense reward principle for language-model post-training." arXiv preprint arXiv:2605.12483 (2026).
- He, Qiangqiang, Jin Li, and MingCai Chen. "Calibrating Teacher--Student Discrepancy for On-Policy Distillation." arXiv preprint arXiv:2609.21619 (2026).
Dear Reviewer J5nH,
Thank you again for your careful review. As the discussion period is drawing to a close, we wanted to kindly follow up on our response. Following your comments, we have clarified OSD’s contribution relative to existing literature and added substantial new experiments: a frozen-prefix control with fresh continuations and a matched teacher budget, continued training from an SFT-initialized student, an ablation of the OPD baseline’s settings, and a larger study on long-horizon terminal tasks (TMax), where OSD outperforms Teacher-SFT and OPD. We have also corrected the labels in Fig. 6 and Fig. 11 and the Table 2 sentence. All changes are marked in blue in the revised manuscript.
We would be grateful to know whether these address your concerns, and we are happy to clarify anything further.
Best Regards,
Authors
[Part 1/2]
TLDR:
W1 / Q1. "What is the main contribution beyond the cited Revisiting DAgger work?"
Revisiting DAgger interpolates student and teacher turn by turn and supervises the student with teacher labels at the visited states. OSD rests on two different choices. First, it is continuation-only: the student writes a contiguous prefix, the teacher writes one continuation from that state, and the loss is applied only to the teacher's tokens. Second, the prefixes are regenerated from the evolving student throughout training, so the supervision always follows the states the student currently reaches, which helps beyond more offline data. Starting from an SFT-initialized Qwen3-1.7B (DeepSeek-V4.1-Flash teacher) on ScienceWorld, avg success rate (%):
We have included this experiment in the revised manuscript (Appendix F).
W2 / Q2. "Separate the benefit of evolving student states from fresh teacher continuations and different teacher budgets"
Thank you for raising this. To separate these factors, we designed a frozen-prefix control on ScienceWorld (Qwen3-1.7B student, GPT-5.4-mini teacher) that holds the teacher budget fixed and changes only one thing: whether the teacher intervenes on the states of the student being trained or on those of the initial student. Both methods receive fresh teacher continuations, in the same number, so data freshness and teacher budget are matched by construction. At the same budget, supervising the current student's states outperforms supervising the initial student's, which stays at the level of offline SFT: OSD aligns its supervision with the states of the evolving student.
Whose states to supervise. Frozen-prefix control: the teacher writes fresh continuations, as in OSD, but always from prefixes of the frozen initial student, with the same total number of continuations as 4 OSD epochs. Avg success rate (%) at the data volume of 4 epochs (± = std over 4 evaluation passes):
Fresh continuations alone perform on par with offline SFT, which suggests that the gain comes mainly from supervising the current student's states. We have included this control in the revised manuscript (Appendix E).