- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper approaches the technique of On-Policy Distillation through a KL-Regularized RL perspective. It start from the OPD objective and introduces this idea of token-level reward fitting and learns an optimist estimate of the reward to encourage exploration. By then employing the analytical solution of KL-RL with this optimistic reward, it arrives at an objective that distills the teacher into the student while controlling the entropy of the student (i.e., “imitate this teacher but keeping your options open”). The paper provides empirical evidence in math reasoning benchmarks, claiming improvements over other OPD baselines, and demonstrating better entropy control and off-policy learning.
Critical strengths
-
The paper is well written and very clear. The experimental design is clear, as well as the mathematical derivations and proofs.
-
The proposed method is well developed, assumptions are clear, and the connections between OPD and KL-control justified.
-
The empirical evidence is supportive to the claims: they not only show the improvements over prior methods, but also shows the main two strengths of the method: the capacity of controlling the entropy (preventing collapse and encouraging exploration) and enabling off-policy learning.
-
The paper also provides theoretical analysis on the Regret, which complements well with the empirical evidence.
Critical weaknesses
-
My main concern is the statistical significance of the results. I did not find any references to how many experimental seeds were used, so I am assuming a single one. The paper brings results in benchmarks which usually contain few datapoints (like AIME) and do not report confidence intervals. Given how close the methods’ performance are (sometimes reporting < 1pp gains), it is unclear if these gaps are due to other factors like observation noise, checkpoint selection, etc. I would strongly recommend reporting at least confidence intervals (on a few training/eval seeds) to assess the statistical significance of the results.
-
The paper does not discuss any limitations of the method, and I think this is important. Particularly, I am curious to know whether implicitly learning this optimistic reward function might lead to reward hacking if you train for long enough.
-
I believe it is important to clarify that the connection between OPD and KL-RL/MaxEnt-RL is not a novel contribution of this work (section 4.1) - in fact, it was already presented in prior works (e.g., see Appendix C.1 in [1]), which also points to the direction of defining of OPD defining an implicit reward function defined by the teacher. The text, particularly in the beginning of sec 4.2, and partially the paper title, gives a misleading impression that this paper is introducing this as a novel idea.
-
The related work does a good job on listing related papers, but needs to better contextualize and contrast this work. It now works as a list of what previous papers introduced, but does not clarify what they share in common and what they differ from the current work.
[1] Hubotter et. al. Reinforcement Learning via Self-Distillation, 2026.
Questions for authors
See my question about limitations and reward hacking in the weaknesses.
5 comments
W4. The related work does a good job on listing related papers, but needs to better contextualize and contrast this work. It now works as a list of what previous papers introduced, but does not clarify what they share in common and what they differ from the current work.
Response. We thank the reviewer for pointing this out. We have expanded the related work discussion to clarify the similarities and differences between these studies and our work.
[1] Hubotter et. al. Reinforcement Learning via Self-Distillation, 2026.
[2] Yang et. al. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation, 2026.
Dear Reviewer zrnf,
Thank you again for your detailed and constructive review and your support for our work. As we are about halfway through the discussion period, we would like to kindly follow up.
We would like to briefly summarize our main clarifications and additional analyses:
-
Multi-seed evaluation (W1). We additionally evaluate our methods and all baselines using three random seeds across six benchmarks for Qwen3-8B → Qwen3-4B-Base distillation. LSPD-RB achieves an average Avg@16 of 43.23 ± 0.99, and LSPD achieves 42.89 ± 0.67, compared with 41.91 ± 0.75 for the strongest baseline, demonstrating improved mean performance.
-
Limitations and reward hacking (W2). We add an explicit limitations discussion addressing the exploration–fidelity tradeoff, fixed entropy coefficient, and teacher–student support mismatch. We also extend LSPD training to 200 steps without observing significant performance degradation. The explicit teacher log-probability matching term further constrains deviations from teacher supervision.
-
Contributions beyond MaxEnt-RL (W3). We clarify that our main contribution is not the existing equivalence between OPD and KL-regularized RL, but the integration of off-policy trajectory reuse and exploration into a sample-efficient distillation framework, supported by the theoretical guarantees in Theorem 6.2.
-
Related work (W3 & W4). We add relevant references to Section 4.1 and expand the related work discussion to better distinguish our contributions from existing language model distillation and RL methods.
We would greatly appreciate it if you could let us know whether our responses address your concerns, and we would be happy to clarify any remaining questions.
We thank the reviewer for the constructive feedback and support for acceptance. We address the reviewer's concerns below:
W1. Can you provide the confidence intervals and more seeds for LSPD?
Response. We evaluated the checkpoints from our proposed methods and all baselines for the Qwen3-8B → Qwen3-4B-Base distillation experiment across all six benchmarks using three random seeds. The table below reports the mean performance and standard deviation across the three seeds. The results demonstrate that our proposed methods achieve improved overall performance, with both LSPD-RB (43.23 ± 0.99) and LSPD (42.89 ± 0.67) outperforming the strongest baseline (41.91 ± 0.75) in terms of Avg@16 across 6 benchmarks, which demonstrates statistically significant improvement.
W2. The paper does not discuss any limitations of the method, and I think this is important. Particularly, I am curious to know whether implicitly learning this optimistic reward function might lead to reward hacking if you train for long enough.
Response. We thank the reviewer for highlighting this point and have added a discussion of the limitations. The entropy bonus promotes diversity without directly accounting for response correctness or uncertainty in the implicit reward. An excessively strong bonus may encourage diverse but incorrect continuations, creating a tradeoff between exploration and fidelity to teacher supervision. Our practical algorithm uses a fixed entropy coefficient; adapting this coefficient to the student’s learning progress or the coverage of collected data remains a direction for improvement. Moreover, LSPD inherits the support and coverage limitations of standard OPD: the teacher must assign nonzero probability to tokens sampled by the student. Even when this support condition holds, a substantial mismatch between the teacher and student distributions can limit the effectiveness of distillation.
Regarding reward hacking, we trained LSPD for a full 200 training steps and did not observe significant performance drops, suggesting that reward hacking is not a significant issue within the evaluated training horizon. The practical objective also retains an explicit log-probability matching term (Eq. 4.6), which continually anchors the student to teacher supervision on sampled trajectories and penalizes deviations from these targets. This provides a constraint on the implicit reward throughout optimization.
We have added a concise version of this discussion in our revised manuscripts.
W3. Can you clarify the contribution of this paper beyound MaxEnt-RL?
Response. We agree that the connection between OPD and KL-regularized/maximum-entropy RL presented in Section 4.1 has been established in prior work. We cited [2] in the original manuscript and have now added [1] to Section 4.1 alongside it.
Our contribution goes beyond the objective reformulation in Eq. (4.1). We build on this connection to incorporate two key ideas from reinforcement learning for improving sample efficiency: off-policy trajectory reuse through a replay buffer and exploration. Specifically, our framework shows how reverse-KL-based, OPD-style distillation can benefit from these ideas to reuse historical trajectories and conduct distributional exploration to improve sample efficiency. This integration is the key novelty of our work. We further provide a theoretical analysis of the optimistic formulation in Theorem 6.2, establishing its sample-efficiency guarantees.