acceptodds
Under review as a conference paper at ICLR 2027

Divergence-Routed On-Policy Distillation for Language Models

Abstract

On-policy distillation (OPD) transfers knowledge from a teacher model to a student along student-generated trajectories, typically by minimizing reverse Kullback–Leibler (RKL). However, RKL can provide weak corrective signals for teacher-supported tokens to which the student assigns very low probability. This raises two questions: which positions are most consequential for the behavioral gap between the student and teacher, and what additional supervision is needed at these positions? Through cross-sampling, we find that divergence-based routing more effectively pinpoints positions contributing to the teacher–student performance gap than entropy-based. Further analysis reveals teacher-mode under-coverage at high-JS positions, with standard OPD increasing the large teacher–student probability gaps. Motivated by this findings, we propose Divergence-Routed On-Policy Distillation (DrOPD). DrOPD retains RKL supervision at all positions, and selectively adds forward Kullback–Leibler (FKL) at positions with the high Jensen–Shannon (JS) divergence. Experiments across six math benchmarks show that DrOPD consistently improves over standard OPD and entropy-based routing. DrOPD also reduces the severe under-coverage cases by 37.9% relative to OPD, supporting targeted FKL as an effective complement to OPD.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On Policy Distillation, Reinforcement learning, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WUG1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DrOPD retains reverse-KL supervision at all response positions and adds forward KL at the 20% of positions with the highest student-teacher JS divergence. Cross-sampling experiments motivate the routing rule, and a local gradient analysis explains the added corrective signal. The method is evaluated on six math benchmarks for two Qwen3 student-teacher pairs, with entropy-routing controls and ablations of the loss allocation and candidate set.

Critical strengths
  1. The gradient analysis gives a clear motivation for adding forward KL when the student assigns little probability to a teacher candidate. The identities are valid for the candidate-normalized distributions used in the derivation (Section 3 and supplementary Section 2.2). The cross-sampling study also compares divergence with entropy and random intervention.

  2. The entropy-routing baselines use the same all-position reverse KL and top-20% forward-KL allocation, making them useful comparisons. Mean accuracy rises from 31.57 to 33.41 for the smaller pair and from 44.64 to 45.23 for the larger pair (Table 1). The supplement reports training cost as well as accuracy; the smaller-pair run takes about 8% longer than OPD.

Critical weaknesses
  1. The supplied implementation appears to optimize a different surrogate from the one derived in the paper. Equation 8 and supplementary Section 2.3 use candidate-normalized log probabilities. In dp_actor.py, _forward_micro_batch instead gathers full-vocabulary log-softmax values, kl_val uses the raw student-teacher log-probability difference, and these gathered values enter the policy loss without candidate renormalization. For forward KL, this gives a different local logit gradient and can affect logits outside the candidate set. The code comments describe this behavior. This need not be a bad update, but the theory and implementation need to agree about which method is being evaluated. This is the main reason for my current weak-reject recommendation.

  2. The advantage specifically due to JS routing is less clear than the advantage over pure reverse KL. The entropy controls help, but a training control that applies forward KL to randomly chosen 20% of positions is missing; random intervention during decoding tests a different question. Such a control would be useful given the relatively small margins over the strongest entropy baseline and over OPD for the larger pair.

Questions for authors
  1. Which probability convention and code version were used for Table 1? Please reconcile the released reward/policy-loss path with Equation 8 and supplementary Section 2.3. A small fixed-context gradient check would help resolve whether the difference is intentional.

  2. Do you have a random-20% routing training result under the same loss coefficient and budget? Please also clarify the checkpoint-selection rule and report any existing repeat runs for the larger-pair result.

Reviewer_m2PfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies where on-policy distillation (OPD), which minimizes the RKL between the student and teacher next-token distributions along student-generated trajectories, fails to transfer teacher knowledge. This paper utilize observes that the JSD between teacher and student exhibits the token positions of teacher-mode under-coverage that standard OPD does not sufficiently repair by the cross-sampling method. To provide appropriate supervision at high-disagreement positions, this paper proposes DrOPD, which retains RKL at all positions and adds FKL at the top-ranked positions by JSD of renormalized teacher and student distributions on the union of top-K candidates. Under the mathematical reasoning tasks, DrOPD shows the performance improvements.

Critical strengths
  1. This manuscript is well written and easy to follow along with their reasoning. The analysis-then-method structure makes the motivation of each design decision clear.

  2. The proposed method has a simple and intuitive structure, which makes it broadly applicable. DrOPD adds a gated FKL term to the standard OPD objective and requires no auxiliary models or additional several hyperparameters beyond the routing fraction and the candidate count.

Critical weaknesses
  1. There is insufficient justification for why the proposed method improves performance, and the design reads as heuristic. The gradient analysis on the logit space motivates adding FKL. However, this work does not verify why the top fraction of positions ranked by JSD is the core candidate set or why supervision there delivers into downstream performance improvement. The motivating analysis in Section 2 is also narrow in ways that matter: it uses a single model pair, a single benchmark dataset, and some fixed hyper-parameters (N=320, top-16). Also, I am not sure that these experimental results are general case. Furthermore, computing both divergences on a renormalized union of top-k candidates may not consistent with the motivation, since the teacher-supported tokens with very low student probability lie outside the student’s top-k and the renormalized teacher distribution discards exactly the mass the method is meant to recover. Finally, while the truncated objective is not a divergence between the true distributions, its bias is not clearly analyzed.

  2. The proposed objective, which utilizes the RKL as a default and adds FKL at a selected subset of positions, is structurally close to EOPD. The main methodological difference lies in the routing criterion and the position-selection rule. In my opinion, these are incremental and limited improvement. Furthermore, the current manuscript does not provide the in-depth theoretical and empirical comparison with EOPD: there are only performance comparisons and several detailed experiments are only compared with vanilla OPD. Furthermore, although various prior studies were explained in the related work section, the current experimental baseline is limited to EOPD alone, making it difficult to fully demonstrate the effectiveness of the proposed method.

Questions for authors
  1. I understand that the high-disagreement position (Eq. 7) is calculated at the batch level. I'm curious to see how performance compares when this is extended to calculate the high-disagreement position for each individual instance.
Reviewer_UhpMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In on-policy distillation (OPD), student LLM is trained on its own rollouts to match teacher via reverse KL (RKL). Authors argue RKL fails to teach tokens where teacher probability is high but student probability is near zero ("mode under-coverage"). They propose DrOPD, keeping RKL on all tokens and adding forward KL (FKL) on top 20% tokens ranked by teacher-student JS divergence (over union of both models' top-16 tokens). On two Qwen3 pairs (1.7B ←\leftarrow 4B, 4B ←\leftarrow 8B) and 6 math benchmarks, DrOPD improves unweighted mean accuracy over standard OPD and entropy-routed baselines.

Critical strengths
  1. Simple, practical method: DrOPD only needs top-kk logits that existing OPD pipelines already compute, so it is simple drop-in change.
  2. Clear gain over OPD baseline on 1.7B: On 1.7B pair, DrOPD stays above paper's OPD baseline across all checkpoints (Fig. 4c) and improves all 6 benchmarks.
  3. Clean ablation grid: 2D ablation of loss direction (RKL vs FKL) by token selection (Tab. 2) is clean and informative.
Critical weaknesses

While DrOPD beats OPD baseline (this paper's own baseline) on 1.7B, experiments do not support main claim that JS-divergence routing beats entropy routing by fixing mode under-coverage.

W1. Main claim does not hold on large benchmarks (Tab. 1).

  • Paper claims DrOPD consistently beats entropy routing. Yet on 1.7B, DrOPD loses to student-entropy routing (Stu-entropy) on 4 of 6 benchmarks, including all 3 large ones: MATH500 (-0.28), OlympiadBench (-0.67), and Minerva (-0.56). It also loses to teacher-entropy (Tch-entropy) on two largest sets: OlympiadBench (-0.89) and MATH500 (-0.85).
  • DrOPD's higher unweighted mean comes purely from two smallest sets, AIME24 and AIME25 (only 30 problems each, where 1 pt ≈\approx 2.4 correct samples). Weighted by problem count across all 1,547 test problems, DrOPD (43.39) is actually lower than both Stu-entropy (43.75) and Tch-entropy (43.84). On 4B, all methods are within 0.4 pts.
  • Also, ablations (Sec. 4.5-4.6) drop Minerva and OlympiadBench, two of the three large benchmarks where DrOPD loses to Stu-entropy.

W2. Checkpoints and hyperparameters seem selected on test data from single runs.

  • In Tab. 1, 6-benchmark means for OPD (31.57) and DrOPD (33.41) match single-checkpoint peaks on test curves in Fig. 4c (e.g., OPD spikes to ~31.6 at step 180 while step 160 and 200 are ~28-29; DrOPD peaks at step 220 at ~33.4 vs ~32.4 at end). Picking best test checkpoint per method biases comparison.
  • Key hyperparameters (ρ=20\rho=20%, k=16k=16, union support) are also tuned directly on 4 test benchmarks (Fig. 5).
  • All numbers are single runs without seeds or error bars, even though test curves jump by 1-3.5 pts between adjacent steps (Fig. 4c), wider than margins between methods.

W3. Missing controls to show JS routing or FKL actually drives gain over OPD.

  • No random-routing baseline: No test on adding FKL at random 20% token positions during training, which is basic null baseline for any routing claim.
  • Upweighting high-JS tokens vs adding FKL: In Tab. 2, Sparse RKL (RKL only on top-20% JS tokens, no FKL at all) already gets ~62% of DrOPD's gain over Full RKL (34.94 vs 33.80, vs DrOPD's 35.64), while Sparse FKL drops below OPD (33.29). This suggests gain mostly comes from focusing gradients on high-JS tokens (or dropping noisy low-JS tokens), not from needing FKL there.

W4. Overstated novelty and circular diagnostics.

  • Prior work: Loss setup (full RKL + gated FKL) is same as EOPD (Jin et al., ICML 2026) [R1], just swapping teacher-entropy gate for JS gate. Also, finding that divergence finds critical tokens better than entropy in token replacement was already shown in Huang et al. (ICLR 2026) [R2], and token-level FKL/RKL gradient mixing was studied in ToDi (EMNLP 2025) [R3] and AKL (COLING 2025) [R4], none of which (except EOPD) are cited.
  • Circular diagnostics: In inference token replacement (Fig. 1a), swapping student distribution with teacher's only changes output when they disagree (pt≠qtp_t \neq q_t), so divergence beats entropy on "accuracy per intervention" almost by definition. Likewise, showing DrOPD reduces prob gaps at high-JS positions (Sec. 4.4) only shows added FKL loss is optimized there, and paper does not check if Stu-entropy or Tch-entropy also reduce this gap.

References

[R1] W. Jin, T. Min, Y. Yang, et al. Entropy-Aware On-Policy Distillation of Language Models. ICML 2026. arXiv:2603.07079.

[R2] K.-X. Huang, H. Meng, J. Wu, et al. Beyond Magnitude: Leveraging Direction of RLVR Updates for LLM Reasoning. ICLR 2026. arXiv:2603.22117.

[R3] S. Jung, S. Yoon, D. Kim, H. Lee. ToDi: Token-wise Distillation via Fine-Grained Divergence Control. EMNLP 2025. arXiv:2505.16297.

[R4] T. Wu, C. Tao, J. Wang, et al. Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models. COLING 2025. arXiv:2404.02657.

Questions for authors
  1. Test-set selection & multi-seed variance (W1, W2): Are Tab. 1 scores taken from peak checkpoints on test curves (Fig. 4c)? Can you report last-checkpoint (or val-selected) accuracy over 3 seeds, especially on large benchmarks (MATH500, OlympiadBench, Minerva)?
  2. Isolating routing vs loss effect (W3): How does DrOPD compare with (a) adding FKL at random 20% positions, and (b) upweighting RKL (2x) on top-20% JS positions w/o FKL?
  3. Mechanism on baselines (W4): What is severe-gap ratio (Fig. 4b) for Stu-entropy, Tch-entropy, and EOPD?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.