acceptodds
Under review as a conference paper at ICLR 2027

On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics

Abstract

On-policy learning has been argued to reduce catastrophic forgetting, produce sparser parameter updates, and improve generalisation. However, existing comparisons between supervised fine-tuning and reinforcement learning vary many factors simultaneously, making the contribution of rollout policy difficult to isolate. We study the effect of rollout policy in a controlled strong-to-weak distillation setting, by independently varying rollout policy, token-level KL direction, and learning rate across the Llama 3 and Qwen2.5 model families and reasoning tasks spanning scientific, medical, and arithmetic domains. Our analysis reveals a nuanced picture of distillation dynamics in which rollout policy does not necessarily play a central role. Instead, KL direction more clearly shapes task performance and output coverage, while learning rate governs forgetting and sparsity. Token-level gradient analysis and experiments along a continuous student–teacher rollout-policy spectrum explain this pattern: forward KL is remarkably robust to rollout policy, whereas reverse KL is substantially more sensitive and favours student-generated rollouts. On-policy data nevertheless improves generalisation to harder Countdown variants under both KL directions, although this advantage does not reliably persist after subsequent RLVR. Our broader conclusions remain robust to removing gradient clipping, using sampled KL estimators, and training on tasks requiring longer reasoning chains. Overall, our results challenge the view that on-policy rollouts are inherently preferable and show that their value depends critically on the objective, evaluation setting, and optimisation hyperparameters.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Post-training, on-policy learning, knowledge distillation, on-policy distillation, off-policy learning, fine-tuning LLMs, strong-to-weak distillation, reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_K3CdOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper systematically investigates the effects of rollout policy, KL direction, and learning rate on strong-to-weak knowledge distillation. Through controlled experiments across different models and reasoning tasks, the authors find that on-policy distillation does not consistently outperform off-policy distillation in terms of task accuracy, catastrophic forgetting, or parameter-update sparsity. Instead, KL direction has a greater influence on task performance, while learning rate largely governs forgetting and sparsity. Theoretical gradient analysis and experiments further show that forward KL is more robust to rollout-policy changes, whereas reverse KL is more sensitive and generally favors student-generated rollouts. The authors also find that on-policy data can improve generalization to harder tasks, although this advantage does not reliably persist after subsequent RLVR.

Critical strengths

1. Well-designed controlled experiments and comprehensive ablation studies.: The paper systematically disentangles the effects of rollout policy, KL direction, and learning rate through well-controlled experiments. Extensive ablation studies across different model families, training configurations, and evaluation metrics further strengthen the reliability of the empirical findings.

2. Insightful theoretical analysis supported by empirical validation.: The paper provides a clear theoretical analysis of the different sensitivities of forward and reverse KL to rollout-policy changes through gradient derivations and stability analysis. The theoretical insights are further supported by experiments along a continuous rollout-policy spectrum, offering a deeper understanding of distillation dynamics.

Critical weaknesses

1. Insufficient justification for the relative importance of the selected factors.: The paper systematically investigates rollout policy, KL direction, and learning rate, and demonstrates that KL direction and learning rate have a greater influence on several training outcomes than rollout policy in the evaluated settings. While the motivation for studying these factors is reasonable, it remains unclear to what extent they can be regarded as the main factors governing on-policy and off-policy distillation dynamics. Other potentially important factors are not systematically examined. A clearer justification or discussion of the scope of these conclusions would strengthen the paper.

2. Limited generalizability beyond strong-to-weak distillation.: Although the paper is motivated by broader discussions of on-policy versus off-policy learning in LLM post-training, its empirical investigation is primarily conducted within a strong-to-weak distillation framework. This setting differs from reward-based reinforcement learning, where reward signals, advantage estimation, and policy-gradient objectives may substantially affect the role of rollout policy. While the authors clearly position their study as distillation-focused, the implications of these differences for generalizing the findings to broader on-policy and off-policy RL settings are not sufficiently discussed.

Questions for authors

Question1: Could the authors further clarify why rollout policy, KL direction, and learning rate are considered the primary factors in their analysis? By the way, how might other factors, such as the teacher–student capability gap, affect their relative importance?

Question2: Could the authors further clarify to what extent their findings on rollout-policy effects in strong-to-weak distillation can generalize to reward-based on-policy and off-policy RL, given the differences in optimization objectives and supervision signals?

2 comments

We thank the reviewer for their feedback. We appreciate the recognition of the informativeness of our findings, and the breadth of our controlled experiments. Below we address your questions and concerns.

Weakness 1 and Question 1

First, let us clarify why our analysis focuses primarily on rollout policy, KL direction and the learning rate. We focus on rollout policy, as previous works have identified it as a crucial characteristic of post-training methods, affecting catastrophic forgetting, parameter-update sparsity and generalisation. However, previous distillation studies often confounded rollout policy with the KL direction, and we believe that removing this confounding is important to fully isolate the effect of the rollout policy. Finally, we focus on the learning rate as it has been shown to strongly affect the trade-off between learning and forgetting in the SFT setting (Catalan-Tatjer and Geiping, 2026; Rofin et al., 2026) and thus controlling it was necessary to determine to what extent rollout policy also affects this trade-off in the distillation setting.

Additionally, we would like to kindly point out that throughout our experiments we are also considering other factors, such as: (i) the use of gradient clipping (Appendix A.1), (ii) the use of sampled (rather than full-vocabulary) KL divergence (Appendix A.2), (iii) temperature of the sampling policy (ablation in Appendix A.15). We did not find that these factors materially affect the results and conclusions drawn, and hence we discuss them primarily in the Appendix.

That being said, we acknowledge that other factors might also affect distillation dynamics. One such factor, which we have kept fixed so far, is the student-teacher capability gap, which could depend on model size, the type and amount of training, access to privileged information (as per self-distillation), or the response style. Given this broad design space, we prioritised a detailed analysis of the factors above and left a systematic study of capability gaps to future work. Following your suggestion, we have now added a paragraph to the revised Section 7 explicitly discussing these choices.

Nevertheless, to test whether our conclusions generalise beyond our original student-teacher setup, we have now reproduced our main results in two additional settings:

  1. Using larger 3B student models (in combination with the trained 7B/8B teacher models). The results (reproducing the main comparisons from Figure 1, the learning rate sweep, as well as the analysis of the rollout-policy spectrum across both Llama and Qwen models) are in the revised Appendix A.7.
  2. Using smaller 1B trained teachers (in combination with the non-trained 1B student model). The results (reproducing the main comparisons from Figure 1 using Llama models) are in the revised Appendix A.8.

The results in these new settings support our existing conclusions: both on-policy and off-policy rollouts reach similar performance across our three benchmarks, with more stable performance under forward KL, and learning rate largely drives catastrophic forgetting and update sparsity. For both 3B models, on-policy rollouts improve generalisation to harder variants of Countdown. This demonstrate the broader applicability of our findings to other student-teacher capability gaps.

Weakness 2 and Question 2

Yes, indeed, our study is motivated by the broader discussion of the differences between on-policy and off-policy post-training methods, but focuses on distillation specifically. As distillation becomes an important part of the standard post-training pipeline (as seen in the model cards of DeepSeek-v4, Qwen3, GLM-5, Nemotron 3 Ultra), with on-policy distillation gaining a lot of attention from the community, we found it important to evaluate how the rollout policy affects performance in this concrete setting.

We acknowledge that our findings do not directly generalise to reward-based RL. In RLVR, the supervision signal is different, relying on sparse rewards and policy gradients. This change in the training objective might interact differently with the on- and off-policy rollouts and we do not claim that the conclusions drawn in the distillation setting will directly translate there. In fact, our broader methodological contribution is to stress the importance of controlling for the training objective (KL direction) and optimisation regime (e.g. learning rate) before drawing conclusions about the differences between on- and off-policy learning. In particular, our finding that the learning rate plays a crucial role in allowing to navigate the trade-off between learning and forgetting emphasises that any practitioners intending to study the differences between on- and off-policy RLVR should carefully control for this hyperparameter.

Following your suggestion, we have now added a paragraph discussing this explicitly in Section 7.


We thank the reviewer again and are happy to answer further questions.

Reviewer_K3CdOpenReview ↗

Official Comment by Reviewer K3Cd

Thank you for the detailed response and additional experiments. The authors' clarifications and additional results have adequately addressed my main concerns regarding the selection of factors and the generalizability of the findings. I appreciate the authors' efforts in strengthening the paper and will maintain my original rating.

Reviewer_GrJoOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether student-generated on-policy data is inherently preferable to teacher-generated off-policy data for language-model distillation. Across the Llama 3 and Qwen2.5 families and medical, scientific, and arithmetic reasoning tasks, it separately varies rollout policy, token-level KL direction, and learning rate, measuring task accuracy, capability retention, and parameter-update sparsity. Within the tested configurations, KL direction and learning rate explain more consistent differences than rollout policy: forward KL is comparatively robust to trajectory source, whereas reverse KL is more sensitive. Gradient analysis, a continuous rollout-policy spectrum, and additional experiments on sampled KL, clipping, temperature, and longer reasoning complement the main comparisons. The contribution is a systematic investigation of distillation dynamics.

Critical strengths
  1. The paper separately varies rollout policy, KL direction, and learning rate, uses three seeds for the main comparisons, and evaluates two model families and three tasks. The additional Countdown learning-rate sweep reduces dependence on two arbitrarily selected rates. In Figure 1, forward-KL mean accuracy remains within 71–73% across the compared policy/rate combinations, while forgetting and update sparsity change more substantially with learning rate. Removing clipping, using sampled KL, changing generation temperature, and reporting individual OOD benchmarks provide meaningful robustness checks.

  2. The study goes beyond the two on-/off-policy endpoints by varying sampling along a student–teacher policy spectrum. Under an explicit bounded-Jacobian assumption, Appendix C establishes a forward-KL semi-gradient stability bound, constructs a counterexample to a distribution-free reverse-KL counterpart, and recovers a conditional reverse-KL bound when log-likelihood ratios are controlled. Figure 3 and the Qwen experiments provide complementary empirical evidence of different sensitivities. The theory offers a useful local mechanism with appropriately stated limitations.

Critical weaknesses
  1. The subsequent RLVR experiment does not directly establish that the initial generalization advantage fails to persist. Section 5.3 first evaluates generalization on Countdown-4E, with four operands drawn from 1–10, and then performs RLVR on Countdown-4, whose operand pool additionally includes 12, 15, 20, 25, 50, and 75. Appendix D.6 explicitly states that Figures 6 and 26 report mean correctness rewards on training rollouts rather than held-out accuracy. These curves therefore establish differences in training dynamics under the specified RLVR configuration, but do not directly determine whether the previously measured held-out generalization advantage disappears or reverses: both the task distribution and the evaluation metric have changed. Moreover, the three trajectories vary distillation seeds while fixing the RLVR seed at 42, covering initialization variation rather than independently varied RLVR randomness.
Questions for authors
  1. Are held-out Countdown-4E or Countdown-4 evaluations available for the checkpoints before and after RLVR? With generation temperature, completion limits, and sample counts held fixed, does the ordering of training rewards in Figures 6/26 correspond to the ordering of held-out performance? This would affect assessment of the RLVR conclusion in the abstract.

  2. Appendix D.4 defines sparsity as the fraction of parameter changes below an absolute threshold, and Appendix A.7 checks a threshold of (10−810^{-8}). These results support learning-rate dependence of the specified metric, but two absolute thresholds do not independently distinguish smaller overall displacement from greater concentration of updates in a small subset of coordinates. Have you examined concentration after normalizing the update vector, or policy differences at matched displacement norms? Otherwise, the interpretation should remain explicitly about thresholded absolute parameter changes.

1 comment

We thank the reviewer for their feedback. We appreciate the recognition of the informativeness of our findings, and the breadth and robustness of our experimental results. Below we answer your questions.

Question 1 (and W1)

Thank you for this suggestion. With the results in Figure 6 we wanted to emphasise the training dynamics of the different distillation checkpoints, but we agree that held-out evaluation across both Countdown settings (4 and 4E) would better support our assessment of downstream RLVR performance.

New experiments. We now report in the main text the pass@1 and pass@10 performance on a held-out Countdown-4E test set before and after RLVR, allowing for a direct and fair comparison. Results with the higher learning rate and Countdown-4, as well as the RLVR training dynamics, are shown in Appendix A.4 instead. Additionally, to further improve the robustness of the RLVR analysis, we now extend these experiments to the Qwen2.5-1.5B model, also shown in Appendix A.4.

Results. The new results reveal a more nuanced picture. For Llama, the best off-policy checkpoint still reaches comparable performance after RLVR to the best on-policy checkpoint (both at distillation learning rate 1e-5, with forward KL for OffPD and reverse KL for OnPD), despite the advantage of OnPD training before RLVR. Meanwhile, for Qwen2.5-1.5B, on-policy checkpoints remain stronger than off-policy even after RLVR, particularly at distillation learning rate 5e-5. We have revised the abstract, introduction, and RLVR discussion in Section 5.3 to reflect this distinction: the initial on-policy advantage can persist after RLVR, but whether it does so depends on the model and distillation configuration.

Question 2

Our definition of parameter update sparsity is directly borrowed from previous work (Mukherjee et al. 2025), which uses this exact threshold-based definition to claim that RL natively finetunes smaller subnetworks in LLMs than SFT. Mukherjee et al. specifically attribute this difference between RL and SFT to the use of on-policy data in RL. Thus, in order to directly validate their claim in the distillation setting, we have adopted the same metric throughout our experiments. However, we agree that this single metric might not paint the full picture about the more broader notion of “sparsity” in parameter updates. For this reason, following your suggestion, we now complement our sparsity analysis with two additional metrics.

First, we compute the effective rank of parameter updates, as also reported by Mukherjee et al. This metric is computed by calculating the effective rank of each individual update matrix in the model rieffr^{eff}_i, dividing by the maximum rank of this matrix (rimax=min⁡(mi,ni)r^{max}_i = \min(m_i, n_i) for a matrix of dimension mi×nim_i \times n_i), and averaging these ratios across the model’s parameter matrices 1M∑i=1Mrieffrimax\frac{1}{M} \sum_{i=1}^M \frac{r_i^{eff}}{r_i^{max}}. To find the effective rank, we find the minimal number of singular directions of the matrix which explain 99% of the matrix’s spectral energy, as per the definition of Mukherjee et al. Overall, this metric captures not only the magnitude of the changes to the individual parameters, but also their relative importance.

Secondly, following the reviewer’s suggestion, we also compute the scale-normalised sparsity, where we find the norm of the parameter update vector, ∣∣Δθ∣∣2=∑i=1N(Δθi)2||\Delta\theta||_2 = \sqrt{\sum_{i=1}^N(\Delta\theta_i)^2}, normalise every update element to be Δθ~i=Δθi∣∣Δθ∣∣2\widetilde{\Delta\theta}_i = \frac{\Delta\theta_i}{||\Delta\theta||_2} and then compute the scale-normalised sparsity as 1 - \frac{\\#\{i: |\widetilde{\Delta\theta}_i| > \tau\}}{N}. With this metric, we aim to measure the concentration of updates after controlling for the overall displacement.

We report the results of our learning-rate dependent evaluation under these two new metrics in Appendix A.10 (Figure 32) in the updated manuscript. The results support our previous conclusions, showing that sparsity heavily depends on the learning rate (obtaining R2R^2 values of 0.987 and 0.900 for the two metrics, respectively), but does not differ significantly between the on- and off-policy runs.

In particular, we note that the scale-normalised sparsity has correlation at the level of 0.999994 with the original sparsity, mostly because the norm of the parameter update vector ∣∣Δθ∣∣2||\Delta\theta||_2 ranges between 1.64 and 15.47 across the runs considered. Given that ∣Δθ~∣>τ  ⟺  ∣Δθ∣>τ ∣∣Δθ∣∣2,|\widetilde{\Delta\theta}| > \tau \iff |\Delta\theta| > \tau\ ||\Delta\theta||_2, the scale normalisation only marginally adapts the threshold we consider for the parameter updates, affecting on average only about 0.1% parameters.


We thank the reviewer again and are happy to answer further questions.

Reviewer_GLeROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the role of the rollout policy (on-policy vs. off-policy trajectory generation) in language model post-training by framing the problem within a strong-to-weak knowledge distillation setup. Using Llama-3 (8B →\to 1B) and Qwen2.5 (7B →\to 1.5B) models across medical, scientific, and arithmetic tasks, the authors vary the rollout policy (OnPD vs. OffPD), token-level KL direction (forward vs. reverse KL over the full vocabulary), and learning rate (1×10−51\times 10^{-5} vs. 5×10−55\times 10^{-5}). The paper argues that KL direction primarily dictates in-distribution accuracy, while learning rate governs catastrophic forgetting and parameter-update sparsity, challenging the prevailing view that on-policy learning inherently mitigates forgetting or induces sparse updates.

Critical strengths
  1. Theoretical Characterization of Token-Level KL Semi-Gradients: The mathematical derivations in Section 5.1 and Appendices B–C clearly formalize the structural asymmetry between forward-KL and reverse-KL logit gradients. Showing that the forward-KL semi-gradient satisfies a uniform Lipschitz bound with respect to the total-variation distance between rollout distributions (Theorem C.1), whereas reverse KL lacks a distribution-free bound due to unbounded log-likelihood ratios (Theorem C.2 & Corollary C.3), provides a clean theoretical lens on objective sensitivity.

  2. Continuous Rollout-Policy Spectrum Design: Parameterizing a continuous spectrum πλ\pi_\lambda between teacher and student policies via likelihood-ratio interpolation and contrastive extrapolation (with plausibility masking and log-ratio clipping) is a neat methodological tool for probing intermediate trajectory distributions beyond binary on/off-policy settings.

  3. Multi-Faceted Evaluation Beyond In-Distribution Pass@1: Extending the analysis to examine output diversity (pass@1 vs. pass@10 coverage) and incidental teacher-style adoption (e.g., the Spanish reasoning trace experiment in Appendix A.11) adds interesting perspectives on how mode-covering and mode-seeking objectives behave qualitatively.

Critical weaknesses
  1. Gap Between the Broad Framing and the Specific Distillation Setup (Omission of Standard SFT): While the title and introduction motivate the study through the broader On-Policy vs. Off-Policy post-training debate (e.g., SFT vs. RLVR), the empirical evaluation focuses exclusively on knowledge distillation (OnPD vs. OffPD). In practical offline post-training, SFT is the standard distillation paradigm, whereas full-vocabulary OffPD (particularly OffPD + R-KL) is much less commonly used in practice. Including standard SFT as a primary baseline, or narrowing the paper's framing to specifically reflect distillation, would make the practical takeaways more convincing.

  2. Potential Confounders from the Strong-to-Weak Setup: To cleanly isolate the effect of the rollout policy (on- vs. off-policy), it is desirable to minimize other sources of variation. Using a strong-to-weak setup (8B/7B →\to 1B/1.5B) introduces capacity mismatch and capability gaps as additional variables. It would be helpful to clarify why a same-sized or self-distillation setup (e.g., 1.5B →\to 1.5B) was not used to fully decouple model capacity from rollout dynamics.

  3. Choice of High Learning Rate (5×10−55\times 10^{-5}) and Potential Optimization Instability: As summarized in Appendix E (Table 6), standard learning rates for full-parameter post-training typically fall between 5×10−75\times 10^{-7} and 2×10−52\times 10^{-5}. In comparison, 5×10−55\times 10^{-5} is relatively high and appears to induce optimization instability—where even in-distribution accuracy degrades noticeably under Reverse KL (e.g., Figure 1 left and Figure 8). Since performance degradation under an overly large learning rate is an expected optimization artifact, relying heavily on the 5×10−55\times 10^{-5} regime to draw conclusions about catastrophic forgetting and distillation dynamics may not fully reflect realistic training conditions.

  4. Misleading Interpretation of Figure 1 (Left): Excluding the collapsed 5×10−55\times 10^{-5} runs and looking at the normal LR=1×10−5\text{LR}=1\times 10^{-5} regime in Figure 1 (left), OnPD + R-KL (∼72\sim 72%) matches F-KL (∼71–73\sim 71\text{--}73%), while OffPD + R-KL fails badly (∼35\sim 35%). This clearly shows that the difference lies in On-Policy vs. Off-Policy rollouts when using Reverse KL. Concluding instead that "rollout policy offers no advantage and KL direction is what matters" (Section 4.2) misinterprets the paper's own plots by conflating OffPD + R-KL and the broken 5×10−55\times 10^{-5} runs.

  5. Insufficient Evidence for Subsequent RLVR Claims: The claim that on-policy generalization advantages vanish after subsequent RLVR (Section 5.3, Figure 6) relies on a single synthetic toy task (Countdown-3 →\to Countdown-4) with a 1B model. More extensive experiments on realistic benchmarks (e.g., math or science reasoning) are needed to make this conclusion solid.

Questions for authors

Please see weaknesses above.

8 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

Thank you for your constructive feedback. We address each of your comments below.

Weakness 1

As you correctly note in the summary of our paper, the aim of this work is to study the importance of on-policy rollouts in the strong-to-weak distillation setting. We focus on this setting since it allows to target the effect of the rollout policy specifically, whereas SFT and RLVR differ in many other aspects like the source of training signal, density of supervision and optimisation method. Moreover, distillation has become an important part of recent post-training pipelines, including both on- and off-policy distillation (as stated in the technical reports of models such as DeepSeek-V4, Qwen3, GLM-5, Nemotron 3 Ultra), so we believe that it is important to better understand the role of rollout policy in this specific domain. We do not claim that our findings generalise directly to the difference between on-policy and off-policy learning in RLVR, and we have now made that explicit in the discussion of scope in Section 7.

In our main experiment, we use the full vocabulary KL as we consider it to be the more principled setting (top-k KL approximation are a noisy estimate of the true KL objective; for example, the DeepSeek-v4 technical report advocates for the use of full-vocabulary KL, in order to decrease the variance in the gradient estimation). Nevertheless, we already consider the impact of the KL estimation method in Section 6 and Appendix A.2. There, we present the results of an ablation study in which we use a sampled (top-1) KL estimator instead of the full-vocabulary one. As we show mathematically in Appendix A.2, OffPD with forward KL using the top-1 KL estimator amounts to training with SFT loss on the teacher generated traces (both methods give the same gradients). Hence, the results in Appendix A.2 already implicitly compare an SFT baseline to different distillation variants.

The results show that our findings largely persist in this setting: we observe that forward-KL leads to stable performance increase across both OnPD and OffPD training, with OnPD F-KL attaining highest performance. Meanwhile, the reverse KL runs collapse, across both OnPD and OffPD and at both learning rates, showing that the reverse KL objective is less stable. Further, in this regime catastrophic forgetting and parameter-update sparsity remain governed by the learning rate, rather than by the rollout policy. Finally, we note that the SFT setting (OffPD F-KL in Figure 9, highest performance of 85%) underperforms the full-vocabulary distillation in the same setting (OffPD F-KL in Figure 34, highest average performance of 89%).

Weakness 2

Thank you for this suggestion. We agree that the teacher–student capability gap is an important factor that could influence distillation dynamics. However, it is important to distinguish a confounder from a factor affecting generalisability. Within each of our controlled comparisons, the teacher and initial student are identical across rollout-policy conditions. Thus, the capability gap does not confound our within-setting comparisons.

New experiments in 7/8B→3B and 1B→1B settings. When it comes to generalisability, to investigate whether our conclusions depend on the particular capability gap, we have extended our experiments to two additional configurations:

  • Using larger 3B student models (in combination with the trained 7B/8B teacher models). The results (reproducing the main comparisons from Figure 1, the learning rate sweep, as well as the analysis of the rollout-policy spectrum across both Llama and Qwen models) are in the revised Appendix A.7.
  • Using smaller 1B trained teachers (in combination with the non-trained 1B student model). The results (reproducing the main comparisons from Figure 1 using Llama models) are in the revised Appendix A.8.

The additional experiments support the objective-dependent behaviour observed in the original setting, providing evidence that our main findings are not restricted to the original 7B/8B-to-1B/1.5B configuration.

AuthorsOpenReview ↗

Rebuttal (2/3)

How we decided on the student-teacher setup. We initially experimented with self-distillation, but found that at the 1/1.5B scale, the teacher equipped with privileged information was not able to provide sufficient training signal for the student model to improve on our benchmarks (privileged information provided an increase of 17.0→20.0%17.0 \to 20.0\% for Science, 29.5→31.5%29.5 \to 31.5\% for Medreason and 7.0→35.5%7.0 \to 35.5\% for Countdown-3, respectively). Similar ineptitude of small models was also observed by Shenfeld et al. in their self-distillation paper. Furthermore, a self-distillation setting introduces additional variability with respect to the rollout policy: the teacher policy (instantiated as the student with privileged information) would evolve over the course of training, and hence also vary between the different training runs, making it more difficult to isolate the effect of the rollout policy. Therefore, we opted to use fixed 7/8B models as teachers, obtained with prior training on each benchmark to ensure enough capability gap from the student, such that there is enough room for improvement to allow us to clearly distinguish between the effectiveness of different methods.

Weakness 3

We agree that the higher learning rate of 5e-5 is prone to optimisation instability in our experiments. Nevertheless, in certain settings it achieves the best performance (see, for example, Figure 3 left). Furthermore, we would like to emphasise that our main conclusions regarding the impact of the learning rate rely on a sweep over {1,2,3,4,5,6}e-5, and show both the catastrophic forgetting and sparsity gradually increase over this range. Thus, we disagree that the observed performance is purely an artefact of training instability at the higher learning rate. Moreover, our main results regarding the effect of rollout policy and KL divergence are observed with the more stable 1e-5 learning rate.

We initially experimented with the lower learning rate of 5e-6, but found in preliminary experiments that it did not improve performance or mitigate forgetting over 1e-5, and therefore focus on the latter as the lower LR in our study. Furthermore, the fact that 1e-5 reaches similar performance as 5e-5 while mitigating most of the forgetting across our settings (e.g., Figure 1, Figure 23) suggest that there would be little to gain from considering lower learning rates.

Weakness 4

The reviewer points out an important observation: under forward KL, OnPD matches the performance of OffPD, while under reverse KL OnPD outperforms OffPD. Indeed, this observation underlies our analysis in Section 5. There, we explicitly show that logit gradients of the forward and reverse KL have different sensitivity to the rollout policy (Section 5.1). We experimentally validate this claim using the rollout-policy spectrum, showing that the performance of reverse KL is much stronger in the OnPD regime than in the OffPD regime when it comes to final task performance (Figure 3). In turn, catastrophic forgetting and parameter-update sparsity are not affected by the rollout policy, under either training objective.

Thus, the results in Section 5 indeed reveal that the importance of rollout policy is conditional: rollout policy does matter when using reverse KL as an objective, and when considering the final task performance (and generalisation to harder tasks) as metrics.

Given these observations, we agree that some of the interpretation of Figure 1 we provided in Section 4 could have been misleading and not sufficiently precise. Our main objective in Section 4 was to emphasise that comparable performance can be achieved under both OnPD and OffPD, when choosing training hyperparameters (learning rate and KL direction) carefully. Your comment has inspired us to make the following changes to the manuscript:

  • Updated the “Final task performance” interpretation in Section 4.2 to explicitly state that “When considering the best average performance achieved by OnPD and OffPD under our sweep of KL divergence and learning rate, on-policy rollouts offer no consistent advantage in target-task accuracy: the best mean accuracies across the three datasets are nearly identical, reaching 72% for OnPD and 73% for OffPD (Figure 1, left).”
  • Update the Takeaway box at the end of Section 4.2 to state that: “Across our controlled comparisons, on-policy rollouts can achieve the same average task performance as off-policy rollouts, under suitable choices of hyperparameters. Further, on-policy rollouts offer no consistent advantage in forgetting or update sparsity, which are largely governed by the learning rate.”
  • Updated the opening paragraph of Section 5 to explicitly state that “the effect of rollout policy on task performance depends strongly on the direction of the KL divergence”.
AuthorsOpenReview ↗

Rebuttal (3/3)

Weakness 5

We agree that the original RLVR experiments provided limited evidence about whether the initial generalisation advantage of on-policy distillation remains after further training. We have therefore added held-out evaluations on Countdown-4E, both before and after RLVR (updated Figure 6), and extended the experiments to Qwen2.5-1.5B (Appendix A.4). The new results reveal a more nuanced picture. For Llama, the best off-policy checkpoint reaches comparable performance after RLVR to the best on-policy checkpoint (both at distillation learning rate 1e-5, with forward KL for OffPD and reverse KL for OnPD). Meanwhile, for Qwen2.5-1.5B, on-policy checkpoints remain stronger than off-policy, particularly at distillation learning rate 5e-5. We have revised the abstract, introduction, and RLVR discussion in Section 5.3 to reflect this distinction: the initial on-policy advantage can persist after RLVR, but whether it does so depends on the model and distillation configuration. We also acknowledge that these results are specific to the evaluated Countdown setting and do not establish a universal property of downstream reinforcement learning. We have marked these changes in blue in the updated manuscript.


We thank the reviewer again and are happy to answer further questions.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We are grateful for the detailed feedback. Reviewers found the experiments on the continuous rollout policy spectrum neat and insightful (GLeR, GrJo), our theoretical characterisation of the token-level semi-gradients helpful in formalising the difference between forward and reverse KL (GLeR, K3Cd), and the experiments well-designed and comprehensive (K3Cd, GrJo).

Below, we summarise the main updates, which are highlighted in blue in the revised PDF.

  • Generalisation to different student-teacher capability gaps (K3Cd, GrJo). We have extended our analysis to consider a 3B→3B distillation setup across Llama and Qwen, as well as 1B→1B Llama setup (Appendix A.7 and A.8).
  • More extensive learning rate sweeps (GLeR). We have extended the main learning rate sweeps for the Llama3.2-1B student to Science and Medreason datasets, and additionally added the learning rate of 5e-6 to the sweep (Figure 2, Appendix A.11).
  • More extensive analysis of training with sampled KL (GLeR). We have extended the analysis of training with Sampled KL to Qwen2.5-1B, and also considered a broader range of learning rates (5e-6, 1e-5, 3e-5, 5e-5).
  • Extended evaluation of RLVR training (GrJo, GLeR). We added matched held-out evaluations before and after RLVR on Countdown-4E and Countdown-4, and extended the experiments to Qwen2.5-1.5B.
  • Additional metrics to evaluate update sparsity (K3Cd). We now show that our findings persist when using alternative metrics for sparsity (Appendix A.10).
  • Discussion of the scope and generalisability of the findings (K3Cd). We added discussion points explaining that our results do not directly generalise to RLVR (Section 7).

These revisions strengthen the empirical basis of the paper while making its conclusions more precise: the benefits of on-policy rollouts depend on the objective, optimisation regime, model, and evaluation setting, rather than constituting a universal advantage.

Reviewers GrJo and K3Cd acknowledged that our responses and additional experiments addressed their concerns and maintained their positive ratings of 3 (Weak acceptance). GrJo noted that the revisions addressed “most of my concerns” and made the presentation “clearer and more complete,” while K3Cd stated that they “adequately addressed my main concerns regarding the selection of factors and the generalizability of the findings.” Reviewer GLeR raised further concerns about the experimental setting and the distinction between forgetting and training collapse, which motivated additional experiments and clarifications in our final revision.

Thank you once again for your time and consideration.

Best, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.