acceptodds
Under review as a conference paper at ICLR 2027

Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?

21 views

Abstract

Since the advent of knowledge distillation, KL divergence has been the standard loss in distillation. Recently, on-policy distillation (OPD) has emerged as an efficient post-training paradigm for LLMs. As a distillation method, OPD naturally inherits KL divergence as its standard loss. However, in this work, we find that KL divergence may not be necessary for OPD. We show that simply preserving the update direction is sufficient for effective OPD. As long as the update direction is toward the teacher, OPD works. More precisely, it is not the direction of every token, but the direction of a small subset of tokens where the teacher and student disagree strongly. We first show that simply assigning a reward of \(+1\) to tokens where the teacher probability is higher than the student probability and \(-1\) where it is lower, which merely encourages updates toward the teacher, reproduces almost the same training mode as OPD with reverse KL. We further show that only the direction of a small subset of tokens with large teacher-student disagreement is critical, and training works as long as their update direction is toward the teacher, even if other tokens are pulled away from the teacher. And as an application of these findings, we introduce Consensus Multi-Teacher On-Policy Distillation (C-MOPD) to improve Multi-Teacher On-Policy Distillation (MOPD). Unlike MOPD, which routes each sample to a single teacher and may cause capability conflicts across domains, C-MOPD lets every sample be supervised by all teachers. Experiments show that C-MOPD consistently outperforms MOPD on both math and code benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model. Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_UraaOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

On-policy distillation (OPD) is an increasingly used algorithm for post-training large language models. It relies on using reverse KL divergence as a way to measure the discrepancy between the student's next token distribution and that of the teacher. This is akin to doing RL on a reward function that is the difference between the log probabilities of the student and the teacher on the sampled text. The authors show that replacing this difference with just its sign is enough to reach the same performance as full OPD, giving us insights on the geometry of the OPD optimization objective. Further analysis reveals that only the tokens with the largest disagreement between the student and the teacher matter. The authors also show that the binary reward idea can successfully be extended to multi teacher OPD. Overall, it provides insights into the minimal components needed to post train LLMs to be good reasoning models.

Critical strengths

The main finding is a cute result and is well demonstrated through extensive experiments. The refinements--only certain tokens matter, extension to the multi-teacher setting--make sense and enrich the results of the paper.

Critical weaknesses

While the findings are interesting, the scope of the results remains relatively limited. I expect the paper to be a useful data point, helping shape how the field understands on policy distillation, but not more. That said, I believe the paper should be accepted, hence my score, but I don't think it should be highlighted as a spotlight or an oral.

Minor: on the writing side, the intro reads as a repetition of the abstract; it could be cut down.

Questions for authors

Would there be some practical interest when running this algorithm, for example reduced memory (only need to store 1 bit, instead of 8 bit floating point probabilities)?

OPD is usually run for a limited number of steps; what would happen if we were to run it for longer? Would we expect differences between OPD and BinaryOPD to emerge when training for much longer, as the more fine-grained information in this signal would start to matter more?

3 comments

We sincerely thank the reviewer for the positive assessment. It encourages us a lot. We hope our responses below address your concerns and questions.

All revisions made during the rebuttal are highlighted in blue in the revised PDF.

Q1: Potential memory savings.

Yes, binary rewards could in principle reduce storage through bit-packing, but the absolute saving would be very small. The compressible quantity is only one scalar reward per sampled token, rather than a full-vocabulary distribution. With our batch size of 256 and maximum response length of 8,192 (Appendix E, Table 10), an ideal 1-bit encoding would reduce the reward buffer from 2 MiB to 0.25 MiB compared with 16-bit storage, saving just 3.75 MiB across the entire batch. Even compared with FP32, the saving is only 7.75 MiB. This does not reduce model parameters, optimizer states, or activations, and teacher forward passes are still required. However, regarding practical applications, our finding that only a small fraction of high-disagreement tokens is sufficient for effective OPD suggests potential memory and computational savings during student updates. Specifically, we could compute the output logits and loss only for selected tokens, and potentially further reduce backward computation by avoiding unnecessary gradient calculations for unselected tokens where possible, while retaining the computations needed for attention dependencies. However, realizing these optimizations would require substantial modifications to training infrastructure, which were not feasible for us within the limited rebuttal period. We leave this direction for future work. Thus, our contribution is primarily understanding what really OPD needs.

Q2: Long-term training behavior of BinaryOPD.

To our knowledge, the reason OPD is typically trained for a relatively small number of steps is that it converges rapidly, making substantially longer training unnecessary. Nevertheless, to address the concern that reward magnitude may become important over longer training, we conducted 1,000-step experiments on the Qwen3-4B-Non-Thinking student and Qwen3-4B-Non-Thinking-RL-Math teacher pair, corresponding to 256,000 training samples.

In addition to OPD and BinaryOPD, we further evaluated two variants over the same 1,000 steps to investigate whether only the reward direction matters, or whether the fine-grained magnitude information also becomes important during longer training. Let xt=log⁡(pT/pθ)x_t=\log(p_T/p_\theta) denote the teacher–student log-ratio for the sampled token:

  • RandomOPD: rt=sign⁡(xt)⋅utr_t=\operatorname{sign}(x_t)\cdot u_t, where ut∼U(0,1)u_t\sim\mathcal{U}(0,1). This assigns a random reward magnitude while preserving the teacher-directed update.
  • SinusoidalOPD: rt=sign⁡(xt)(12+12sin⁡(10∣xt∣))r_t=\operatorname{sign}(x_t)\left(\frac{1}{2}+\frac{1}{2}\sin(10|x_t|)\right). This introduces an oscillating, non-monotonic reward magnitude while preserving the update direction.

The results after 1,000 training steps are:

BenchmarkStudentTeacherOPDBinaryOPDRandomOPDSinusoidalOPD
AIME2425.061.261.762.563.361.7
AIME2516.756.354.654.256.755.8
AMC58.488.085.786.687.486.8
MATH50081.393.293.592.993.693.2
Minerva27.736.736.236.336.036.2
OlympiadBench42.960.560.159.860.760.4

All four methods remain comparable after 1,000 steps, with BinaryOPD showing no degradation relative to OPD. Moreover, RandomOPD and SinusoidalOPD also maintain competitive performance despite their arbitrary, non-monotonic reward magnitudes. Furthermore, as shown in Appendix E, Figure 9, all four methods exhibit nearly identical training dynamics in terms of training accuracy, entropy, and response length, despite using different reward magnitudes. These results further support our central finding that the direction of teacher–student updates, rather than the precise reward magnitude, is the critical ingredient for effective OPD, even under substantially extended training.

We have cut down the repetition in introduction.

We thank you again for your positive feedback. We hope that our additional experiments and clarifications have addressed your questions. Please let us know if you have any further questions.

I have read the authors' rebuttal and the other reviewers' reviews and keep my (positive) score as it is.

Dear Reviewer,

Thank you for taking the time to review our rebuttal and for your continued support. We sincerely appreciate your time and effort throughout the review process.

Best regards, The Authors

Reviewer_pEvGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper replaces the sampled-token reverse-KL reward in on-policy distillation with the sign of the teacher-student probability difference. BinaryOPD performs close to OPD across five mathematics and two code student-teacher pairs. Masking experiments show that selected high-disagreement tokens can provide effective sparse supervision. C-MOPD extends the approach to multiple teachers and suppresses strongly conflicting updates. It improves on routed MOPD in the reported math/code setting.

Critical strengths
  1. Tables 2 to 3 compare Qwen-derived and Llama models at different scales under reported matched training settings. BinaryOPD stays close to OPD across these pairs. The Qwen3-4B math average is 65.1 versus 64.1; the two code averages are 55.2 versus 55.4 and 50.6 versus 51.0. These comparisons support the use of a substantially simpler reward without a large performance loss.

  2. ALL+1(A) retains fewer than 1.5% of tokens in the Qwen3-4B math setting and obtains 64.5 versus OPD's 64.1 (Table 4). Appendix D examines additional thresholds and a second model pair. C-MOPD provides an application of these findings: LiveCodeBench v6 improves from 27.9 to 32.9, with gains also reported on the math benchmarks (Table 5).

Critical weaknesses
  1. The connection between the objective and the implemented estimator needs to be explicit. At a fixed prefix, with a fixed teacher, detached rewards, on-policy samples, and no action-dependent weighting, define g(r)=Ea∼pθ[r(a)∇θlog⁡pθ(a)]g(r)=\mathbb E_{a\sim p_\theta}[r(a)\nabla_\theta\log p_\theta(a)]. Then

    =-\nabla_\theta\|p_\theta-p_T\|_1.$$ At equality points this has the usual subgradient interpretation. BinaryOPD retains a local distribution-matching interpretation through L1 distance, or twice total variation. Its success supports replacing KL; it does not by itself show that distribution matching is not the key ingredient. This identity applies to a fixed-prefix update and does not establish equivalence of full sequence-level objectives. Section 3's fixed-prefix term has the opposite sign to the reverse-KL minimization gradient. Differentiating the full on-policy objective would also require state-occupancy terms. A detached-state surrogate may be appropriate and should be identified explicitly if used. Appendix C leaves length normalization, masking denominators, clipping, and advantage processing insufficiently specified; each could affect the analysis.
  2. The interventions support sufficiency more directly than necessity. Under the same estimator, g(1A∪B)=g(−1C)g(1_{A\cup B})=g(-1_C) and g(1)=g(−1)=0g(1)=g(-1)=0. The complementary masks are related by a constant baseline, and ALL±1 removes teacher-dependent reward information. Its failure therefore does not isolate the claimed causal importance of reversing a small token subset. The B-only controls also use uniform signs rather than teacher-directed signs within B (Section 5). Successful training on the sparse subset is a useful finding, but does not establish that low-disagreement information is intrinsically unnecessary. C-MOPD changes both reward shaping and teacher access, so Table 5 supports the complete method more directly than its conflict-filtering explanation.

Questions for authors
  1. What exact loss is differentiated, including detachments, post-mask normalization, advantage processing, and clipping? A loss definition or relevant code would help reconcile Sections 3 to 4 with the estimator above and establish whether the mathematical concerns require only a presentation correction.

  2. Can the conclusions distinguish the demonstrated sufficiency of sparse teacher-directed supervision from necessity, and C-MOPD's overall benefit from its mechanism attribution? Please identify any existing evidence that resolves these distinctions. Clarifying the claims and limiting them to what the evidence supports would address much of this concern without an extensive new experimental suite.

2 comments

We sincerely thank the reviewer for the positive assessment. It encourages us a lot. We hope our responses below address your concerns and questions.

All revisions made during the rebuttal are highlighted in blue in the revised PDF.

Q1: Exact optimization objective and implementation details.

Thank you for catching this. We indeed omitted a minus sign in Section 3 and have corrected it: the expression represents the fixed-prefix reverse-KL descent direction rather than the complete gradient of the sequence-level on-policy objective.

Specifically, we optimize a PPO-style surrogate with sampled prefixes held fixed and rewards detached, without differentiating through the state-occupancy distribution. Token-level rewards are used directly as advantages, without group normalization, baseline subtraction, or reward-to-go computation. Under our setting, the importance ratio equals one, so neither PPO clipping nor dual clipping changes the gradient. The loss is normalized by the number of valid response tokens within each microbatch, including tokens with zero rewards, rather than only the selected tokens.

The exact derivation, loss, detachments, clipping, and normalization are specified in Appendix D. Our code is also provided in the supplementary material.

Q2&W2: Sufficiency vs. necessity and C-MOPD's mechanism.

(1) Sufficiency vs. necessity. To address this concern, we added a B-only Reward Experiment that retains only low-disagreement tokens (group B) while preserving their original OPD or BinaryOPD rewards. We vary ϵ∈{0.02,0.05,0.1,0.2}\epsilon\in\{0.02,0.05,0.1,0.2\} to progressively include tokens with larger teacher–student disagreement. Both methods fail to train effectively when ϵ\epsilon is small, but performance gradually improves as ϵ\epsilon becomes larger, when higher-disagreement tokens are included. We believe this provides additional evidence that only high-disagreement tokens are important for effective OPD, beyond merely demonstrating the sufficiency of sparse supervision. See Section 5.1–5.2, Figure 4.

(2) C-MOPD's mechanism. To distinguish the benefits of multi-teacher access from those of our reward design, we added two baselines, AvgBinary and AvgLogRatio, which also provide supervision from all teachers but simply average their binary rewards or log-ratio rewards. We additionally expanded the experiment from two to three teachers. Both aggregation baselines underperform MOPD, whereas C-MOPD with ϵ∈{0.5,0.8,1.1}\epsilon\in\{0.5,0.8,1.1\} outperforms MOPD across ten benchmarks. We believe this can provide further evidence for the benefit of our conflict-aware reward design rather than accessing to all teachers. See Section 6.1–6.2, Table 4 and Figure 5.

W1: Connection between BinaryOPD and L1 distance.

We agree with the reviewer that the expected BinaryOPD update corresponds to minimizing the L1 distance between teacher and student distributions. However, our motivation is not to propose L1 minimization as an alternative objective, but to use the simplest possible reward to demonstrate that the update direction matters more than the precise reward magnitude.

To further support this point, we introduce two additional variants that preserve the update direction while substantially altering the reward magnitude. Let xt=log⁡(pT/pθ)x_t=\log(p_T/p_\theta) denote the teacher–student log-ratio for the sampled token:

  • RandomOPD: rt=sign⁡(xt)⋅utr_t=\operatorname{sign}(x_t)\cdot u_t, where ut∼U(0,1)u_t\sim\mathcal{U}(0,1).
  • SinusoidalOPD: rt=sign⁡(xt)(12+12sin⁡(10∣xt∣))r_t=\operatorname{sign}(x_t)\left(\frac{1}{2}+\frac{1}{2}\sin(10|x_t|)\right).

We evaluate both variants alongside OPD and BinaryOPD for 1,000 training steps, using Qwen3-4B-Non-Thinking as the student and Qwen3-4B-Non-Thinking-RL-Math as the teacher.

BenchmarkStudentTeacherOPDBinaryOPDRandomOPDSinusoidalOPD
AIME2425.061.261.762.563.361.7
AIME2516.756.354.654.256.755.8
AMC58.488.085.786.687.486.8
MATH50081.393.293.592.993.693.2
Minerva27.736.736.236.336.036.2
OlympiadBench42.960.560.159.860.760.4

All four methods achieve comparable performance. Moreover, their training dynamics are very similar. These results provide further evidence that preserving the teacher-directed update is sufficient for effective OPD, even when the reward magnitude is random or non-monotonic, rather than attributing BinaryOPD's success specifically to L1 minimization. The full experimental setup, benchmark results, and training dynamics are provided in Appendix E, Table 9 and Figure 9.

We thank you again for your constructive feedback. We hope that our additional experiments and clarifications have addressed your concerns, and we would be very grateful if you could adjust your rating accordingly.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer,

As the rebuttal discussion period is coming to an end, we'd really appreciate it if you could take a look at our response when you have time. Please let us know if you have any further questions or concerns.

Thank you very much for your time and consideration!

Best regards, The Authors

Reviewer_qevAOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper asks whether KL divergence is needed for on-policy distillation (OPD). Its proposed BinaryOPD replaces the per-token log-ratio with its sign: +1 where the teacher assigns the sampled token higher probability than the student, −1 where lower. Across seven student–teacher pairs (5 math, 2 code), BinaryOPD matches standard OPD.

A token-level dissection shows that the signal lives in a small set of high-disagreement tokens. Rewarding only those (<1.5% of tokens) in the teacher’s direction reproduces OPD, while pushing them away from the teacher breaks training. The authors explain this with state redundancy during OPD (App. A). They then apply the insight to multi-teacher OPD: C-MOPD supervises every token with all teachers and improves over routed MOPD on both math and code.

Critical strengths

The reviewer truly appreciates the paper as the formulation is very clean, and experimental results are honest and easy to interpret. Although there exist several points to improve, the reviewer believes this paper provides interesting analyses for the question that has long been standing within distillation research -- the need for KL divergence. Here are the strengths:

  1. A clear question with a simple, well-supported answer. BinaryOPD stays within about 1 point of OPD on all seven pairs (Tables 2–3). The pairs span model families (Qwen3, Llama-3.2, DeepSeek-Distill), scales from 1.5B to 30B-A3B MoE, and base, instruct and non-thinking students. The training curves also overlap closely (Figs. 1, 8). This is a useful result for practitioners.

  2. The token-level analysis is insightful. Rewarding only Group A, under 1.5% of tokens, scores 64.5 against 64.1 for full OPD (Table 4). The symmetric negative-reward experiment supports the same conclusion, and the pattern holds across three thresholds and two pairs (Figs. 9–10). It gives a concrete picture of where OPD’s learning signal actually comes from.

  3. There is a plausible mechanism with supporting evidence. The repeated-feedback view (Sec. 4.3) is backed by the state-redundancy analysis in App. A, which explains why magnitude information is dispensable.

  4. The insight leads to a practical method. C-MOPD improves over MOPD on all nine benchmarks in Table 5, notably LiveCodeBench v6 (27.9 →\rightarrow 32.9). On several math benchmarks it essentially matches the math teacher (AMC 88.0, Olympiad 60.4 vs. 60.5). The ε=0\varepsilon=0 vs. ε=0.8\varepsilon=0.8 comparison ties the result back to the Sec. 5 finding.

  5. The method is simple and cheap to adopt. It needs no new hyperparameters beyond ε\varepsilon for C-MOPD, and hyperparameters and setup are clearly reported.

Critical weaknesses
  1. Prior work on non-KL distillation objectives is missing. The paper frames KL as the unquestioned default, but earlier work has already moved away from it.
  • The ICLR 2026 residual-learning KD paper (openreview.net/forum?id=Dh6KxUxG20) argues that white-box methods rely almost exclusively on divergence matching, which assumes the teacher is always correct. It instead trains the student on a residual hidden state, formed by subtracting the projected teacher representation only where the teacher’s top-1 prediction is wrong, and outperforms divergence-based baselines.
  • Classic examples are logit regression with an L2 loss (Ba & Caruana, NeurIPS 2014) and Kim et al. (IJCAI 2021), who compare MSE on logits with KL. Citing these would frame the contribution as extending this line to on-policy LLM distillation with a sign-only signal.
  1. The failing control in Sec. 5 could be sharper. ALL+1(A+B+C) gives +1 to every token, so it carries no teacher information at all, and its failure is expected. A control that excludes a random 5% of tokens instead of Group C would show that it is specifically the high-disagreement tokens that matter.

  2. A main-text claim rests on the appendix. The conclusion states that the verifier signal has little effect, but that evidence is only in App. B. Either move a summary into the main text or soften the claim.

Questions for authors
  1. How sensitive is BinaryOPD to the learning rate, given that it assigns magnitude-1 rewards to the many low-disagreement tokens?

  2. Does C-MOPD scale to more than two teachers? How often do teachers conflict as K grows?

3 comments

We sincerely thank the reviewer for the positive assessment. It encourages us a lot. We hope our responses below address your concerns and questions.

All revisions made during the rebuttal are highlighted in blue in the revised PDF.

Q1: Sensitivity to the learning rate.

We added a learning-rate ablation on the Qwen3-4B-Non-Thinking math pair, keeping all other settings unchanged. At learning rates of 1×10−61\times10^{-6}, 5×10−65\times10^{-6}, and 1×10−51\times10^{-5}, BinaryOPD achieves average accuracies of 65.1, 65.3, and 65.3, respectively, compared with 64.1, 64.7, and 65.2 for OPD. This suggests little sensitivity within the tested range. Details are provided in Appendix C, Table 9.

Q2: Scaling C-MOPD to more than two teachers.

Thank you for this question. We have extended C-MOPD from two to three teachers, incorporating an instruction-following teacher alongside the math and code teachers. As shown in Section 6.2, Table 4, C-MOPD with ϵ∈{0.5,0.8,1.1}\epsilon\in\{0.5,0.8,1.1\} outperforms MOPD across all ten benchmarks, demonstrating its effectiveness with more teachers.

Regarding teacher conflicts, Figure 5 shows that with ϵ=0.0\epsilon=0.0, the Zero Reward Ratio increases beyond 90%, indicating that most tokens cannot be updated under strict teacher agreement. In contrast, with ϵ∈{0.5,0.8,1.1}\epsilon\in\{0.5,0.8,1.1\}, the Zero Reward Ratio initially increases before stabilizing at much lower levels (approximately 45–50%). This suggests that allowing mild disagreements enables more effective updates with more teachers.

W1: Prior work on non-KL distillation.

Thank you for pointing out these references. We have added a discussion in Section 2, “Beyond KL-based Knowledge Distillation.” Ba & Caruana (2014) and Kim et al. (2021) study logit-level L2/MSE objectives, while On et al. (2026) use residual learning with teacher representations and prediction errors. These works primarily operate on fixed training data. Instead, our work investigates on-policy LLM distillation on student-generated trajectories, showing that preserving the direction of teacher–student disagreement suffices without retaining the precise reward magnitude.

W2: Random token-exclusion control.

Following your valuable suggestion, we added Random-5% Drop, which randomly excludes 5% of tokens and assigns +1 to the remaining tokens. Like ALL+1(A+B+C), this control fails to produce effective training, unlike selectively excluding Group C. The symmetric negative-reward control also fails. The training dynamics are provided in Section 5.2, Figure 3.

We additionally conducted a B-only Reward Experiment, retaining only low-disagreement tokens with their original OPD or BinaryOPD rewards. By gradually increasing the disagreement threshold ϵ\epsilon, we progressively include tokens with larger teacher–student disagreement. Both methods fail to train effectively at small ϵ\epsilon, but performance improves as higher-disagreement tokens are included. This further supports the importance of high-disagreement tokens. See Section 5.2, Figure 4.

W3: Verifier-related claim in the conclusion.

Thank you for pointing out this oversight. We have removed the verifier-related claim from the conclusion (Section 7). The corresponding analysis remains in Appendix B as an auxiliary investigation rather than a main-text conclusion.

We thank you again for your positive feedback. We hope that our additional experiments and clarifications have addressed your concerns, and we would be very grateful if you could adjust your rating accordingly.

Reviewer_qevAOpenReview ↗

Changed to Clear Acceptance

Dear Authors,

The Reviewer appreciate the efforts in resolving the comments. The Reviewer have taken a comprehensive look at the paper again and believe the paper is qualified for ICLR, given the strong results across a wide spectrum of model sizes.

During reviewing this paper, the Reviewer think that the paper is very well-organized with the presentations properly handled, and the concepts are illustrated comprehensibly. Please keep up the good work in this area. The Reviewer also hold a belief that KL Divergence is not a must in distillation, so would be very happy to see this paper published.

Good luck with the final results.

Dear Reviewer,

Thank you very much for your kind words and positive feedback. Your encouraging comments mean a lot to us and truly motivate us to continue pursuing research in this direction. We sincerely appreciate your time and thoughtful consideration of our work.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.