- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the issue of noisy teacher signals in On-Policy Distillation (OPD) when the teacher is conditioned on student-generated prefixes. To address this issue, the authors propose Prep-OPD, which fine-tunes the teacher using RLVR on student-generated prefixes before distillation. Experimental results show that Prep-OPD improves accuracy over existing baselines.
Critical strengths
- The proposed approach is simple and intuitive while achieving strong empirical performance.
- The paper provides strong and rigorous empirical results, convincingly demonstrating that Prep-OPD outperforms existing baselines while maintaining better training efficiency.
Critical weaknesses
- My main concern is that while Prep-OPD improves training efficiency during student training, it introduces additional computational costs for teacher fine-tuning. This overhead could be substantial, especially for larger teacher model. Especially in the context of OPD, potentially undermining one of the main motivations for using OPD instead of RLVR: achieving competitive performance at a substantially lower training cost.
- The paper also appears somewhat incremental. The issue of teacher failures on student-generated prefixes has been explored in prior work, including findings that teachers become less accurate when continuing from student-generated prefixes [3]. Furthermore, the idea of learning from suffixes is not new [1,2].
Questions for authors
- Could the authors report the total training compute (teacher fine-tuning + student training) and compare it against other baselines?
- Could the authors also provide the training dynamics during RL fine-tuning of the teacher? Specifically, does the teacher achieve higher rewards when conditioned on student-generated prefixes, and if so, to what extent?
[1] Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixes. ICML 2026.
[2] Learning to Generate Better than your Large Language Models.
[3] Pass the Baton: Trajectory-Relayed On-Policy Distillation.
Thank you for your response. Could the authors clarify why Standard OPD, Relay-OPD, FastOPD, and IW-OPD all have exactly 55.54 hours of training? These methods do not require RLVR training on student-generated prefixes, so it is unclear why their exists computational costs. Furthermore, why does Prep-OPD require only approximately two additional hours compared to these baselines? Given that the teacher model is substantially larger than the student model, I would expect the teacher's computational cost to be significantly higher. Could the authors provide a more detailed breakdown of the computational costs and explain these results? Additionally reporting training FLOPs could also be helpful.