acceptodds
Withdrawn from ICLR 2027

Gains and Collapse in On-Policy Distillation: A Reinforcement Learning Perspective

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

On-policy distillation (OPD) has become an important approach to language model post-training. However, despite its performance gains, OPD can also collapse into excessively long and repetitive generation, and the mechanism underlying these divergent outcomes remains poorly understood. We explain these outcomes through a reinforcement learning perspective: the teacher implicitly rewards student behaviors, even those it rarely exhibits itself. From this perspective, our experiments show that OPD improves performance without expanding the student’s capabilities. When the implicit reward model is reliable, OPD makes correct responses easier to sample. In contrast, when the preference misaligns with quality, reward hacking happens: the implicit reward model amplifies overlong, repetitive student rollouts, even though it rarely generates such text itself. Guided by this diagnosis, we find that masking unhealthy responses during training and using SFT initialization can each effectively mitigate the collapse. Together, these findings show that OPD amplifies student behaviors favored by the teacher's implicit feedback, shifting the focus from how well the teacher generates to how reliably it evaluates student rollouts.

Primary area:
reinforcement learning
Keywords:
On-policy Distillation, Post-training, Reinforcement Learning, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5kbZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates an interesting question in on-policy distillation (OPD): why can it improve students’ task performance while also causing excessively long and repetitive generation? The authors treat the teacher’s probability signals on student-generated content as training feedback and examine both successful and deteriorating training runs. Through repeated sampling and per-problem analysis, they find that successful training primarily makes students more likely to produce correct solutions they were already capable of generating. By tracking response preferences before and after training, the paper further shows how pathological responses can be progressively amplified by favorable training signals. The authors propose filtering excessively long responses and initializing students through supervised fine-tuning to mitigate this behavior.

Critical strengths

The paper provides a clear and coherent analysis of generation degradation. The authors find that certain repetitive responses already receive relatively favorable signals before training, after which the student increasingly favors them. They also show that excluding excessively long responses from the training loss can mitigate the degradation. This finding offers useful insight into understanding and diagnosing distillation training.

Critical weaknesses
  1. The main insights are close to those of existing work, making the additional contribution primarily confirmatory. Prior studies have already discussed how repetitive outputs can receive favorable signals and become progressively amplified, and have used repeated sampling to analyze how distillation improves sampling efficiency. Although this paper tracks initial responses and subsequent preference changes in greater detail, its results mainly reinforce existing explanations rather than establishing an equally distinctive new mechanism. This limits the paper’s novelty as an analysis study.

  2. The claims about capability changes are broader than the experimental coverage supports. The analysis relies on a single 1.5B student that already possesses reasoning ability, while both teachers are obtained by further reinforcement learning from that same model. Therefore, the results provide limited support for the general claim in the abstract that “distillation does not expand the student’s capabilities.

Questions for authors

Please see the weakness section.

Reviewer_LKrWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper observes that reverse KL can be written as a negative cross-entropy plus an entropy term. It also observes that OPD sometimes diverges, and they diagnose it and find that this is because the wrong reinforced answers are actually highly rewarded by the teacher, so the divergence is an example of reward hacking. They mitigate this by filtering out very long answers and also adding an SFT warmup phase.

Critical strengths
  • Clearly written, with clear empirical results useful for future researchers' reference.
  • Figure 7 is very insightful, indicating how different teachers reward different rollouts. This can motivate future work to fine-tune/select teachers suitable to be used as a reward model.
Critical weaknesses

The main claims are covered by previous work:

  • The reward interpretation is trivial and exists in previous work [1].
  • The fact that the teacher in OPD assigns a high advantage to degenerate outputs is observed in previous work [2].

[1] Y. Gu, L. Dong, F. Wei, and M. Huang, “MiniLLM: Knowledge Distillation of Large Language Models,” Nov. 21, 2025, arXiv: arXiv:2306.08543.

[2] Y. Fu, H. Huang, K. Jiang, Y. Zhu, and D. Zhao, “Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes,” Mar. 26, 2026, arXiv: arXiv:2603.25562.

Questions for authors
  • Your paper mentions [3] observing collapse, however, for the same setting of Qwen3-4B as teacher and Qwen3-1.7B as student, they just observe that it performs worse than SFT. I noticed you use 1e-5 as the learning rate, whereas they use 1e-6. Could this simple change be causing the collapse in your setup?

  • [3] suggests that the reason for collapse is a mismatch between the distribution of teacher and student. Your paper mentions that the cause of collapse is that the teacher assigns high advantage to the degenerate rollouts. Aren't these different sides of the same coin since the teacher assigns high advantage to the degenerate rollouts because they are out of the distribution of the teacher, i.e. they are because of the mismatch in the distribution of the student and the teacher?


[3] Y. Li et al., “Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe,” Apr. 15, 2026, arXiv: arXiv:2604.13016.

Reviewer_vLNsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper views OPD as RL with the teacher log-probability as implicit reward. 1) With JustRL/DeepScaleR-1.5B teachers and an R1-Distill-Qwen-1.5B student, OPD raises pass@1 but the gap closes at large k, and an audited AIME coverage analysis finds no OPD-only problem: gains come from sampling efficiency, not new capability. 2) In a collapsed Qwen3-4B → Qwen3-1.7B-Base setting, training converges normally while outputs grow long and repetitive, framed as reward hacking since the top-advantage responses are degenerate even before training. 3) Masking length-capped rollouts or SFT initialization mitigates collapse.

Critical strengths
  1. OPD is widely used in post-training, yet its failures remain unclear. The paper is clearly written, and its implicit-reward lens intuitively explains both gains and collapse, showing that a teacher's generation quality need not reflect how reliably it scores student rollouts.
  2. Beyond raw pass@k, the authors extend sampling for candidate problems, manually audit derivations, and report per-problem results. I appreciate that the one-sided audit and the limits of finite-sample claims are stated explicitly .
Critical weaknesses
  1. Finding 1 is expected and under-attributed. Its question, protocol (pass@k over budgets, coverage, per-problem rates) and conclusion follow Yue et al. [1] with RLVR replaced by OPD, yet [1] is cited only in passing (Sec. 4.1 & 6), not in the Intro or Sec. 3. Also, both successful teachers are RL-trained from the student's own initialization (Sec. 2.3); by [1], their large-k coverage need not exceed the student's, so "no expansion" cannot be separated from "nothing to transfer". Teacher pass@k is not reported and stronger teacher runs (e.g., Qwen3-4B → 1.7B-SFT) are not tested for this, so the abstract's general claim is overstated.

  2. Neither the RL view nor the reward-hacking diagnosis is new. Eq. (2) is the standard reverse-KL expansion; Sec. 2.2 itself credits the reward view to Lu & Thinking Machines Lab (2025). MiniLLM [2], cited here only as an OPD method, already derived OPD as policy gradient with per-step reward log(p/qθ)log(p/q_θ) and found that, while reverse KL still decreases, students "quickly learn to generate repeated, short, or meaningless strings that have high probabilities in the teacher distribution", i.e., reward hacking. This anticipates the core of Sec. 4–5. Cited works report OPD as KL-constrained RL [3], unreliable guidance on drifted prefixes [4], length collapse [5, 6] and no ceiling gain [6].

  3. The mechanism is asserted, not identified. Fig. 7 shows high-advantage responses are degenerate, not why a teacher that rarely writes such text "rewards" it. The expected advantage at a prefix equals −KL(πS∥πT)-\mathrm{KL}(\pi_S\Vert\pi_T), so a high mean advantage mainly reflects teacher–student agreement; repetitive loops are low-entropy, copyable states where both models agree, and can rank highest without any teacher preference. No token-level decomposition (logπTlog π_T vs. logπSlog π_S around the onset of repetition) rules this out or locates where the signal misleads.

  4. The interventions are heuristic and do not test the mechanism. Masking length-capped rollouts is DAPO's overlong filtering [7]; the warmup reuses the SFT checkpoint of Li et al. [8]. Both would curb length collapse under almost any explanation. The mask is coarse: only ~10% of rollouts keep non-zero loss (Sec. 5.2). App. F masks "truncated or failed" responses, Sec. 5.2 only capped ones; if "failed" covers wrong answers, outcome supervision leaks in. Masking only tokens with anomalous advantage, versus response-level masking at a matched token budget and teacher-mixed sampling [2], would be a sharper, more interesting test.

  • [1] Yue et al. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
  • [2] Gu et al. MiniLLM: Knowledge Distillation of Large Language Models.
  • [3] Yang et al. Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation.
  • [4] Fu et al. Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes.
  • [5] Luo et al. Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models.
  • [6] Wang et al. Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations.
  • [7] Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale.
  • [8] Li et al. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe.
Questions for authors

See weakness above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.