- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates the failure modes of on-policy distillation (OPD), particularly in the large-to-small distillation setting. The authors first observe that when the teacher is an RL-optimized variant of the student model, the distillation loss can decrease to nearly zero, allowing the student to almost fully recover the teacher's performance. However, when the teacher is a larger model than the student, the distillation loss quickly plateaus, hindering further learning and preventing the student from effectively closing the performance gap. The authors then provide detailed mathematical derivations to explain these observations, and further support their analysis with empirical evidence from weight updates and Centered Kernel Alignment (CKA) experiments.
Critical strengths
-
The studied problem is important, large-to-small distillation is a very practical and promising technique in building small foundation models, while the pipeline does not work well currently.
-
The theoretical and empirical analysis are well-conducted and self-contained.
Critical weaknesses
-
My first concern is about the contribution and insights. The conclusions and the empirical analysis (e.g., weight update analysis) in this paper are well-known. The theoretical analysis provides some new insights (I did not check the math derivations carefully).
-
Although the theoretical analysis is novel, I would have expected the authors to propose a method to mitigate the identified problem. Understanding the underlying mechanisms, while valuable, seems insufficient on its own to constitute a complete contribution. A more compelling study would build on these insights to develop and evaluate a targeted solution to address the identified limitations of large-to-small OPD.
Questions for authors
See weakness above.
Response to Reviewer Tnvd
Dear Reviewer Tnvd,
We greatly appreciate your thoughtful feedback and constructive suggestions. Below, we provide detailed responses and clarifications. If our responses address your concerns, we would be grateful if you would consider updating your score.
W1. The conclusions and the empirical analysis, such as the weight-update analysis, are well known
Earlier work reported that OPD from a larger teacher can stall and that post-training changes weights little. The paper characterizes when OPD stalls and how, and explains why it works with self-RL teachers. The new results strengthen each part.
DeepScaleR-1.5B is a same-size public model RL-trained from our initial student on other data (response to Reviewer C7ft, W1.1). On the weight-update analysis, see "Small parameter and representation changes (Section 6)" in general response (2/2): the size of the change does not decide the outcome; its direction does.
W2. The paper should propose and evaluate a method that mitigates the problem
We thank the reviewer for raising this. The paper does not offer a mitigation. Its contribution is a diagnosis of when OPD stalls, how it stalls, and why it works with self-RL teachers. We did try the natural remedies, and none removes the plateau: SFT on the teacher's responses; SFT followed by OPD (Section 4.3); a full-vocabulary objective and more rollouts per prompt (Appendix H); and, in this rebuttal, a larger learning rate, longer training and relaxed clipping (general response 1/2). We therefore think that simple changes to the training recipe are unlikely to solve it: how far OPD gets is set mainly by the teacher and the student themselves. Consistently, changing the teacher gains 4.3 to 6.9 accuracy points, against about one at most for any optimizer setting.