- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates why on-policy distillation (OPD) can fail to effectively transfer knowledge from a strong teacher to a student, sometimes resulting in repetitive generation and degraded performance. The authors analyze OPD learning dynamics at three levels: local token distributions, cross-position gradient interactions, and subsequent on-policy sampling trajectories. Their analysis suggests that sampled-token updates may provide insufficient correction when the student assigns low probability to teacher-preferred tokens, while gradients from mismatched positions can interfere with learning elsewhere and alter future sampling behavior. Based on these findings, the authors explain the benefits of supervised fine-tuning (SFT) initialization and propose state-dependent KL routing, which selectively applies forward KL supervision at states exhibiting substantial teacher–student mismatch. Experiments on three Qwen3 student model scales, with both Base and SFT initialization, demonstrate consistent improvements over standard sampled-token OPD on mathematical reasoning benchmarks.
Critical strengths
-
This paper provides some new insights to the OPD community. The paper goes beyond identifying empirical failure modes and provides a structured investigation of OPD from local token-distribution updates to cross-position gradient interactions and subsequent on-policy sampling. I appreciate the deep analysis made in the paper to help the community better understand the learning dynamics of OPD.
-
Beyond pure analysis, the authors also propose a new method named state-dependent KL routing, which makes the whole story complete. I also appreciate the detailed ablation analysis and empirical results provided in the paper.
Critical weaknesses
-
My major concern is about the limited generalizability of the empirical validation. The experiments are mainly conducted in small-sized Qwen models. I wonder whether the conclusions and the insights can be generalized to larger-scaler and non-Qwen families. Also, the authors only conduct experiments on math tasks, which are insufficient.
-
The proposed method state-dependent KL routing seems to only bring limited improvement over the baselines. Also, the authors mainly discuss in the sampled-token OPD setting throughout the entire paper. However, full vocabulary OPD (e.g., DeepSeek-V4) is a more advanced setting. Do the conclusions remain useful in the full vocabulary OPD setting?
Questions for authors
See above.
We thank the reviewer for the encouraging comments and the valuable questions.
S1&S2: This paper provides some new insights to the OPD community...Beyond pure analysis..
Response
Thank you for the encouraging comments on both the analysis and the routing method that follows from it. We are glad to bring some new insight to the community.
W1. My major concern is about the limited generalizability of the empirical validation.
Response
Thank you for raising this concern. We address the three points in turn.
Tasks beyond math. We added a coding task. We train on CodeR1-12k and evaluate on HumanEval+, MBPP+ and LiveCodeBench v6, with Qwen3-8B as the teacher and three Base students. The table shows mean Avg@8 over the three benchmarks at step 200; the full table is in our response to Reviewer Laj6.
Our method has the best mean Avg@8 at all three scales, improving over sampled-token OPD by 2.92 / 3.19 / 3.44 points.
Model scale. Our experiments cover students from 1.7B to 8B. Within this range, the gains over sampled-token OPD are consistent in both math (Table 1) and coding. Due to computational resource limit, we do not have results for larger models.
Model families. The main results use Qwen3 models. The diagnostic experiments in Section 4 also use DeepSeek-R1-Distill-Qwen-1.5B and JustRL-1.5B. Our analysis itself does not depend on the model family: Proposition 1 only assumes a softmax output, and the cross-position decomposition in Section 4.2 holds for any model with shared parameters.
In the revision, we added the coding results and discussed these limitations.
W2. The proposed method state-dependent KL routing seems to only bring limited improvement over the baselines.
Response
Thank you for these two points. The gains are moderate but consistent, and our analysis still apply to full-vocabulary OPD.
On the improvement. We agree that the absolute gains are moderate. However, our contribution is not only the algorithm. It also includes the learning-dynamics analysis of OPD at three levels: local distributions, cross-position effects, and on-policy sampling. Moreover, the improvement is consistent across three student sizes, on math reasoning with both Base and SFT initialization, and on a newly added coding task.
On full-vocabulary OPD. The full-vocabulary update is the expectation of Eq. (4) over the sampled token, which gives
Δθb=ηπθ(b∣s)(A(s,b)−Aˉ(s)),Aˉ(s)=Eu∼πθ[A(s,u)].
The update of each token is still scaled by its student probability, so tokens that the teacher prefers but the student rarely produces still receive little correction. This comes from reverse KL itself, not from sampling. The advantage of full-vocabulary OPD is that it reduces the noise from sampling, which is the benefit reported in DeepSeek-V4, at the cost of a large GPU memory overhead for storing the full-vocabulary logits of student and teacher models over the whole sequence.
To test this, we ran OPD with a top-K approximation of the full-vocabulary reverse KL on Qwen3-1.7B-Base with the Qwen3-8B teacher. The same failure appears. On math, the mean response length grows from 1.2k to 4.1k tokens over 200 steps, and the fraction of rollouts hitting the 4,096-token cap rises from 14% to 100%. On code, the length grows from 0.8k to 4.0k tokens, and the cap-hit rate rises from 5% to 95%. The outputs also become repetitive: the validation loop rate peaks at 81% on math and 52% on code, and ends at 42% and 30%. Increasing K to 32 does not help: the loop rate peaks at 88% on math and ends at 42% and 32%. So removing the sampling noise does not remove the failure.
We added this discussion and the top-K experiment above to the revision (Appendix D).