- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper systematically investigates the effects of rollout policy, KL direction, and learning rate on strong-to-weak knowledge distillation. Through controlled experiments across different models and reasoning tasks, the authors find that on-policy distillation does not consistently outperform off-policy distillation in terms of task accuracy, catastrophic forgetting, or parameter-update sparsity. Instead, KL direction has a greater influence on task performance, while learning rate largely governs forgetting and sparsity. Theoretical gradient analysis and experiments further show that forward KL is more robust to rollout-policy changes, whereas reverse KL is more sensitive and generally favors student-generated rollouts. The authors also find that on-policy data can improve generalization to harder tasks, although this advantage does not reliably persist after subsequent RLVR.
Critical strengths
1. Well-designed controlled experiments and comprehensive ablation studies.: The paper systematically disentangles the effects of rollout policy, KL direction, and learning rate through well-controlled experiments. Extensive ablation studies across different model families, training configurations, and evaluation metrics further strengthen the reliability of the empirical findings.
2. Insightful theoretical analysis supported by empirical validation.: The paper provides a clear theoretical analysis of the different sensitivities of forward and reverse KL to rollout-policy changes through gradient derivations and stability analysis. The theoretical insights are further supported by experiments along a continuous rollout-policy spectrum, offering a deeper understanding of distillation dynamics.
Critical weaknesses
1. Insufficient justification for the relative importance of the selected factors.: The paper systematically investigates rollout policy, KL direction, and learning rate, and demonstrates that KL direction and learning rate have a greater influence on several training outcomes than rollout policy in the evaluated settings. While the motivation for studying these factors is reasonable, it remains unclear to what extent they can be regarded as the main factors governing on-policy and off-policy distillation dynamics. Other potentially important factors are not systematically examined. A clearer justification or discussion of the scope of these conclusions would strengthen the paper.
2. Limited generalizability beyond strong-to-weak distillation.: Although the paper is motivated by broader discussions of on-policy versus off-policy learning in LLM post-training, its empirical investigation is primarily conducted within a strong-to-weak distillation framework. This setting differs from reward-based reinforcement learning, where reward signals, advantage estimation, and policy-gradient objectives may substantially affect the role of rollout policy. While the authors clearly position their study as distillation-focused, the implications of these differences for generalizing the findings to broader on-policy and off-policy RL settings are not sufficiently discussed.
Questions for authors
Question1: Could the authors further clarify why rollout policy, KL direction, and learning rate are considered the primary factors in their analysis? By the way, how might other factors, such as the teacher–student capability gap, affect their relative importance?
Question2: Could the authors further clarify to what extent their findings on rollout-policy effects in strong-to-weak distillation can generalize to reward-based on-policy and off-policy RL, given the differences in optimization objectives and supervision signals?
2 comments
Official Comment by Reviewer K3Cd
Thank you for the detailed response and additional experiments. The authors' clarifications and additional results have adequately addressed my main concerns regarding the selection of factors and the generalizability of the findings. I appreciate the authors' efforts in strengthening the paper and will maintain my original rating.
We thank the reviewer for their feedback. We appreciate the recognition of the informativeness of our findings, and the breadth of our controlled experiments. Below we address your questions and concerns.
Weakness 1 and Question 1
First, let us clarify why our analysis focuses primarily on rollout policy, KL direction and the learning rate. We focus on rollout policy, as previous works have identified it as a crucial characteristic of post-training methods, affecting catastrophic forgetting, parameter-update sparsity and generalisation. However, previous distillation studies often confounded rollout policy with the KL direction, and we believe that removing this confounding is important to fully isolate the effect of the rollout policy. Finally, we focus on the learning rate as it has been shown to strongly affect the trade-off between learning and forgetting in the SFT setting (Catalan-Tatjer and Geiping, 2026; Rofin et al., 2026) and thus controlling it was necessary to determine to what extent rollout policy also affects this trade-off in the distillation setting.
Additionally, we would like to kindly point out that throughout our experiments we are also considering other factors, such as: (i) the use of gradient clipping (Appendix A.1), (ii) the use of sampled (rather than full-vocabulary) KL divergence (Appendix A.2), (iii) temperature of the sampling policy (ablation in Appendix A.15). We did not find that these factors materially affect the results and conclusions drawn, and hence we discuss them primarily in the Appendix.
That being said, we acknowledge that other factors might also affect distillation dynamics. One such factor, which we have kept fixed so far, is the student-teacher capability gap, which could depend on model size, the type and amount of training, access to privileged information (as per self-distillation), or the response style. Given this broad design space, we prioritised a detailed analysis of the factors above and left a systematic study of capability gaps to future work. Following your suggestion, we have now added a paragraph to the revised Section 7 explicitly discussing these choices.
Nevertheless, to test whether our conclusions generalise beyond our original student-teacher setup, we have now reproduced our main results in two additional settings:
The results in these new settings support our existing conclusions: both on-policy and off-policy rollouts reach similar performance across our three benchmarks, with more stable performance under forward KL, and learning rate largely drives catastrophic forgetting and update sparsity. For both 3B models, on-policy rollouts improve generalisation to harder variants of Countdown. This demonstrate the broader applicability of our findings to other student-teacher capability gaps.
Weakness 2 and Question 2
Yes, indeed, our study is motivated by the broader discussion of the differences between on-policy and off-policy post-training methods, but focuses on distillation specifically. As distillation becomes an important part of the standard post-training pipeline (as seen in the model cards of DeepSeek-v4, Qwen3, GLM-5, Nemotron 3 Ultra), with on-policy distillation gaining a lot of attention from the community, we found it important to evaluate how the rollout policy affects performance in this concrete setting.
We acknowledge that our findings do not directly generalise to reward-based RL. In RLVR, the supervision signal is different, relying on sparse rewards and policy gradients. This change in the training objective might interact differently with the on- and off-policy rollouts and we do not claim that the conclusions drawn in the distillation setting will directly translate there. In fact, our broader methodological contribution is to stress the importance of controlling for the training objective (KL direction) and optimisation regime (e.g. learning rate) before drawing conclusions about the differences between on- and off-policy learning. In particular, our finding that the learning rate plays a crucial role in allowing to navigate the trade-off between learning and forgetting emphasises that any practitioners intending to study the differences between on- and off-policy RLVR should carefully control for this hyperparameter.
Following your suggestion, we have now added a paragraph discussing this explicitly in Section 7.
We thank the reviewer again and are happy to answer further questions.