acceptodds
Under review as a conference paper at ICLR 2027

Tail-Aware Top- On-Policy Distillation

Abstract

On-policy distillation (OPD) has emerged as an effective paradigm for transferring knowledge between language models, where a student is trained to align its next-token distribution with the teacher's along its own trajectories. To provide dense supervision at tractable cost, many works minimize the reverse Kullback–Leibler (KL) divergence between the student and teacher's normalized distributions over the teacher's top- tokens. However, this normalized objective discards the information about tail probability: the total probability outside the teacher's top- tokens. As a result, the optimization can steadily increase the student's tail probability and entropy, empirically degrading downstream accuracy. To address this issue, we propose Tail-Aware Top- OPD (), a novel distillation method that restores the missing tail probability signal. In particular, TA-OPD minimizes the reverse KL divergence over the top- tokens plus a tail token that carries the tail probability. In effect, TA-OPD better aligns the student's next-token distribution with the teacher's, preventing the increase in tail probability and entropy caused by top- normalization. Extensive experiments demonstrate the superiority of TA-OPD, improving Avg@8 by up to 8.05 points on common benchmarks.

Primary area:
reinforcement learning
Keywords:
On-policy distillation, Knowledge distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VHBsOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates a failure mode of normalized top-k OPD, where the student model is trained to match the teacher's next-token distribution over the teacher's top-k vocabulary, with both distributions normalized over the selected vocab set StkS_t^k, which can potentially cause issues when the student and teacher are mis-aligned. The authors observe that the tail probabilities (the probability mass outside the teacher's top-k vocabulary) of both the student and teacher steadily increase during distillation with normalized top-k OPD, as this strategy discards tail information. To address this issue, they propose Tail-Aware Top-k OPD (TA-OPD), which additionally takes the aggregated tail probability into account by introducing a single tail token representing this probability mass and minimizing reverse KL divergence over the augmented distribution. However, introducing tail token is not a new idea, as it was previously introduced in Qi et al. (2026).

Experiments across multiple language-model families trained on math reasoning tasks demonstrate that TA-OPD mitigates tail-probability inflation and yields modest improvements in downstream task performance over baselines, while showing a comparably larger performance gap on OOD benchmarks (or less forgetting of OOD knowledge), especially when the student and teacher are mis-aligned.

[References]

Qi, P., Zhou, X., Liu, Z., Pang, T., Du, C., Lin, M. and Lee, W.S., 2026. Rethinking the trust region in llm reinforcement learning. arXiv preprint arXiv:2602.04879.

Critical strengths
  • The observation that tail probability inflation occurs for both the student and teacher is helpful.
  • The OOD performance gap is comparatively large relative to in-distribution performance, supporting that TA-OPD excels at preserving prior knowledge.
Critical weaknesses
  • The idea itself is not novel. It was already introduced in another paper, although plugging it into OPD is appreciated. The propositions are mostly trivial claims.
  • There is no strong evidence that tail probability inflation leads to performance degradation. Figure 2 strongly indicates that high tail probability leads to low accuracy with normalized top-k OPD. However, the results do not align with those in Table 1, where the MATH500 accuracy of the Norm. top-k baseline is 68.78, which does not align with the claim in Figure 2. I would expect its accuracy to be below 25, with a tail probability of ~0.5, but it isn't. This really confuses me about the overall results. Relatedly, the weak overall improvements in in-distribution performance make the claim weaker (an average 0–2% improvement is just noise under T=0.7). Considering that the OOD performance gap looks better, I would consider reframing the paper.
Questions for authors

Additional experiments required

  1. Is k=16k=16 a practical setup? The authors mention that considering the full vocabulary could make training costs high, but setting k=32k=32 or 6464 seems to mitigate the issue. I am curious whether increasing kk to 32 or 64 substantially increases the training cost (i.e., peak memory or training time). If not, there is no reason to use a larger kk setting, as k=64k=64 seems to properly address the tail probability inflation issue.

  2. Missing baseline results that must be compared with. In Table 1, you must report the teacher's and student's initial performance before distillation.

  3. It is unclear whether tail inflation induces performance degradation. Following the point raised in the weakness section, I would suggest reporting the performance of normalized top-k OPD and TA-OPD with different values of kk (2, 4, 8, 16, 32, 64). Currently, you only show tail probability changes for these settings in Figure 8 without reporting any performance changes.

Questions that do not require experiments

  1. Inconsistency between the claim and results. Why is the comparison in Table 2 different from the setting in Table 1?

  2. Why did you choose the reverse KL formulation instead of the forward KL formulation over student-sampled trajectories? Reverse KL is mode-seeking, so it could be vulnerable to the teacher–student mis-alignment issue.

  3. Please demonstrate the reason for distilling models of the same size. There is no motivation to distill in this case, as we can simply deploy the teacher model. If the goal was to claim transferability, you should have claimed that OOD knowledge degradation is not as severe as with other methods.

  4. Temperature inconsistency. What are the rollout and evaluation temperature settings? I am curious why you used different temperatures for Figure 2 (1.0) and Table 1 (0.7). The temperature descriptions in Section 5.1 and Table 4 are also inconsistent.

  5. Also, why is the student–teacher setting in Table 2 different from that in Table 1?

Slightly annoying things that might not change the score but are worth addressing

  • Figure 1 is too far from line43.
  • I don't understand why sample-corrected TA-OPD is introduced after the main results section. I recommend putting this content in the Method section rather than in a separate Discussion section.
5 comments
AuthorsOpenReview ↗

Response to Reviewer VHBs - part 1

We sincerely appreciate the reviewer's constructive feedback. Please find our response below:

1. Novelty relative to Qi et al. (2026) [W1]

Our contribution goes beyond applying an existing construction to OPD. As recognized by Reviewer AM8K (“The mathematical analysis is another strength”) and Reviewer vVo8 (“solid theoretical analysis”), our propositions are not trivial. We further clarify the differences from DPPO [1] below.

First, DPPO assumes access to the sampled token and constructs its support as

\cup \{\hat{y}_t\},$$ where $\mu$ is the behavior policy and $\hat y_t$ is the sampled token, and computes the KL over $S_t^{\mathrm{DPPO}}\cup\{v_{\mathrm{tail}}\}$. The KL estimator remains biased. When sampled token is available, we use SC-TA-OPD, which is unbiased to full vocabulary KL in both value and gradient. Second, DPPO uses the approximate KL between the behavior and current policies to determine whether to mask a policy-gradient update, replacing PPO’s ratio-based clipping. It does not directly optimize this divergence through its gradient. In our setting, the approximate KL itself serves as the distillation loss, making its optimization properties central. We identify tail-probability inflation in normalized top-$k$ OPD, introduce an aggregate-tail token to address it, and analyze the resulting TA-OPD loss and its gradient properties. **2. Temperature Inconsistency [W2, Q4, Q7]** The temperature setup in Section 5.1 and Table 4 is consistent: we use temperature 1.0 and top-$p$ 1.0 for training rollouts, and temperature 0.7 and top-$p$ 0.95 for evaluation, following prior work [1]. The only exception is Figure 2, which uses temperature 1.0 and top-$p$ 1.0 for evaluation. This is because any other temperature would change the probability of sampling tail tokens and make it differ from the measured tail probability. A low temperature and top-$p$ sampling mitigate the effect of tail probability increase by sampling fewer tail tokens, which explains why normalized top-$k$ OPD with Qwen2.5-7B-Instruct achieves 68.78% on MATH500 in Table 1, much higher than its accuracy in Figure 2. We clarify both the temperature setup and its effect on the Table 1 vs. Figure 2 accuracy gap in the revised manuscript. Specifically, we add the sentence "the validation sampling temperature is $1$, which ensures that the probability of sampling tail tokens is the same as the measured tail probability" at line 199 and "low evaluation temperatures and top-$p$ can mitigate the accuracy degradation by sampling fewer tail tokens" at line 209. **3. Relationship between tail probability increase and performance degradation [W2, Q3]** As suggested, we report the MATH500 accuracy of normalized top-$k$ OPD and TA-OPD with different $k$ (evaluation temperature 0.7, top-$p$ 0.95), using Qwen2.5-Math-1.5B as the student and DeepSeek-R1-Distill-Qwen-7B as the teacher. |$k$|8|16|32|64| |-|-:|-:|-:|-:| |Norm. top-$k$ OPD (%)|31.0|48.6|52.4|54.0| |TA-OPD (%)|58.2|61.0|–|–| When $k$ increases from 8 to 16, the tail probability increase issue is substantially mitigated and the validation accuracy improves from 31.0\% to 48.6\%. Combined with Figure 8, these results show that tail probability increase leads to performance degradation. Moreover, TA-OPD with $k=16$ (61.0%) outperforms normalized top-$k$ OPD with $k=64$ (54.0%). Experiments of TA-OPD with $k=32$ and $k=64$ are still running, and we will include them in the final version. **4. Significance of in-distribution improvements [W2]** The evaluation results are reported as Avg@8 (except for MMLU-Pro), i.e., the accuracy averaged over 8 sampled responses per question, which already reduces the variance of evaluation. Thus, the improvement is not noisy. In Appendix G.2 of the revised manuscript, we report the mean $\pm$ standard deviation of in-distribution macro-averaged accuracy across eight evaluation rounds, together with p-values from two-sided Welch's t-tests comparing TA-OPD with the best-performing baseline. The results show significant improvements for Qwen2.5-7B-Instruct and Llama-3.1-8B ($p<0.05$). **5. Practicality of $k=16$ and training cost of larger $k$ [Q1]** $k=16$ is a practical setup and is adopted by prior work [2, 3]. Our Figure 5 shows that the performance of top-$k$ OPD saturates quickly as $k$ increases and $k=16$ already achieves strong performance. However, increasing $k$ substantially increases the training cost. With Qwen2.5-Math-1.5B as the student and DeepSeek-R1-Distill-Qwen-7B as the teacher, training with $k=64$ increases the peak memory from 53 GB to 66 GB and the time per step from 130 s to 440 s compared with $k=16$.
AuthorsOpenReview ↗

Response to Reviewer VHBs - part 2

6. Initial performance of teachers and students [Q2]

We thank the reviewer for the suggestion. We add the initial performance of teachers and students in Table 1 of the revised manuscript.

7. Settings of Table 2 [Q4, Q8]

Table 2 is an ablation study of the bias-correction term, rather than a comparison with baselines. Its goal is to examine whether correcting the bias of TA-OPD improves performance, so we only compare TA-OPD with SC-TA-OPD. For the ablation study, we use smaller LLMs to reduce computational cost. But the evaluation setup is the same as that in Table 1.

8. Choice of reverse KL [Q5]

We use reverse KL for two reasons. First, reverse KL is the standard objective in OPD, adopted by most OPD works [4] and industrial LLMs [5, 6]. Moreover, [7] shows that reverse KL achieves higher Avg@k than forward KL in OPD. Second, off-policy distillation typically minimizes forward KL [8] over teacher-sampled trajectories. When switching to student-sampled trajectories, a natural idea is to use reverse KL to avoid using importance sampling.

9. Motivation for same-size distillation [Q6]

One important application of OPD is model merging, where domain-specialized teachers are trained from the same base model and then merged into a single student of the same size. For example, DeepSeek-V4 [5] and MiMo-V2-Flash [6] first train several domain experts independently, and then consolidate them into a unified model via OPD with the reverse KL loss. In this setting, the teacher cannot be deployed, since each teacher only excels in its own domain.

10. Presentation issues

We remove "see Figure 1" from the Introduction. Regarding sample-corrected TA-OPD, we position it as an extension of TA-OPD, since it additionally uses the sampled token. We introduce it after the main results to show that TA-OPD alone, without the sampled-token information, already addresses the tail probability increase issue.

References

[1] Qi et al. Rethinking the Trust Region in LLM Reinforcement Learning. ICML, 2026.

[2] Li et al. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. TMLR, 2026.

[3] Jin et al. Entropy-Aware On-Policy Distillation of Language Models. ICML, 2026.

[4] Lu and Thinking Machines Lab. On-Policy Distillation. Thinking Machines Lab: Connectionism, 2025.

[5] DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. Technical Report, 2026.

[6] Xiaomi LLM-Core Team. MiMo-V2-Flash Technical Report. arXiv:2601.02780, 2026.

[7] Zhao et al. Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillation. arXiv:2605.16826, 2026.

[8] Kim, Y., and Rush, A. M. Sequence-Level Knowledge Distillation. EMNLP, 2016.

Thank you to the authors for the detailed rebuttal, the additional experiments, and the clarifications in the revised manuscript. I appreciate the clear explanations regarding the training rollout versus evaluation temperature settings (which indeed follow standard conventions), the motivation for the smaller model pairs in Table 2, and the addition of the initial student/teacher baselines in Table 1.

However, after reviewing the updated results and responses, several methodological and empirical concerns remain that prevent me from raising my score at this stage:

  1. Practical significance under standard decoding and comparison to Sampled-token OPD

While the temperature explanation resolves the numerical discrepancy between Figure 2 (T=1.0, p=1.0) and Table 1 (T=0.7, p=0.95), it also weakens the practical severity of the claimed failure mode. Under standard inference decoding, Normalized top-k OPD recovers from 20.60% to 68.78% on MATH500, indicating that standard nucleus/temperature sampling already mitigates the vast majority of the tail-inflation degradation.

More importantly, when comparing TA-OPD against Sampled-token OPD, which requires only a single token log-probability, has no tail-inflation issue, and is ~12% faster in wall-clock runtime. The in-distribution gains of TA-OPD against sampled-token OPD on instruction-tuned models is still marginal.

  1. Confounded evaluation in the Llama-3.1-8B -> DeepSeek-R1-Distill-Llama-8B experiment

Adding the pre-distillation baselines in Table 1 reveals a striking anomaly in the Llama-3.1-8B setup: the initial student achieves only 1.37% on ARC-c and 2.43% on MMLU-Pro, the teacher achieves 25.10% on ARC-c, yet the TA-OPD student trained solely on math data jumps to 65.70% on ARC-c (maybe format/parser failure?). The large gains on this pair appear confounded by template/format alignment rather than clean knowledge distillation or OOD preservation. Could the authors clarify why the teacher scores only 25.10% on ARC-c while the math-distilled student reaches 65.70%?

  1. Implausible computational overhead reported for k=64 vs. k=16

The authors state that increasing k from 16 to 64 increases peak GPU memory from 53 GB to 66 GB (+13 GB) and step time from 130 s to 440 s (3.38x). However, the full-vocabulary forward/backward pass of the student and forward pass of the teacher are already computed regardless of k, and as shown in Listing 1, the top-k loss tensors have shape (batch, seq_len, k). Increasing k by 48 elements would add less than 500MB of tensor memory per batch (nowhere near 13 GB).

Could the authors clarify what caused this 3.4x slowdown and +13 GB memory spike? Specifically, did the average rollout response length change substantially between k=16 and k=64, or is this overhead an artifact of serializing larger top-k Python dictionaries over vLLM's IPC interface?

Reviewer_vVo8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Overall, this is a well-written if somewhat niche paper, which proposes a simple but elegant method of using a special tail token to capture the tail distribution in on-policy tail-aware top-k distribution. Over a range of experiments for KD on mathematical reasoning tasks, the paper demonstrates the superiority of the proposed method to vanilla normalised top-k on-policy distillation and MOPD (a recently-proposed competitor method focusing on multi-teacher top-k on-policy distillation).

Critical strengths
  • a novel method for on-policy top-k knowledge distillation, focusing specifically on how to better capture long-tail distributions through the use of a special tail token
  • a simple formulation with solid theoretical analysis and underpinnings
  • solid experiments with clear empirical findings
Critical weaknesses
  • a major concern with any on-policy knowledge distillation work is the overhead associated with rollouts justifies the use of on-policy vs. off-policy methods (such as Dasgupta et al. 2026 mentioned in Related Work); the paper presents no analysis of this computational cost and the tradeoff in performance vs. cost for the method to have any sense of whether this cost is justified, and all of the experiments focus exclusively on on-policy methods without any empirical comparison with Dasgupta et al. 2026 or other off-policy methods
  • there is also no comparison (covering both additional overhead and performance gains) with full-vocab on-policy knowledge distillation to better contextualise the work and what the tradeoffs in using top-k distillation are
Questions for authors

in addition to the questions implicit in the two weaknesses above:

  • for the comparison with MOPD, is the top-k formulation used (I assume yes given the label in Table 1)? given that MOPD is primarily concerned with multi-teacher distillation and orthogonal to the method proposed in this paper, when MOPD is combined with the proposed top-k distillation method, do the results improve further over the proposed TA-OPD method, or are the effects of multi-teacher distillation annulled in explicitly modelling the tail distribution?
  • for the sample-corrected TA-OPD variant, are the results in Table 2 based on k=64, in which case do they differ for lower values of k (ie does the sample correction matter more for smaller values)?
  • mathematical reasoning entails a relatively controlled language compared to more general tasks, generally with lower next-token perplexity because of the structure and predictability of mathematical reasoning. for more general language domains (eg tasks such as machine translation), are higher values of k generally required, in which case is the utility of the proposed method is curtailed? I guess implicit in this: is mathematical reasoning a biased selection of evaluation setting to test the method, and the relative improvements are less for other truly-long-tail distillation scenarios?
4 comments
AuthorsOpenReview ↗

Response to Reviewer vVo8 - part 1

We thank the reviewer for the insightful comments. Please find our response below:

1. Justification of using on-policy distillation [W1]

Before presenting the empirical results, we clarify that off-policy distillation and on-policy distillation (OPD) are complementary stages rather than competing methods. Off-policy distillation efficiently transfers capability to a weak student, whose own rollouts are too poor to provide useful signal. OPD then refines the student on its own distribution, removing the train–inference mismatch of off-policy training [1] with less forgetting [2]. In practice, off-policy and on-policy distillation are often applied sequentially. For example, Qwen3-1.7B-Base is first distilled off-policy from Qwen3-235B-A22B (or Qwen3-32B) and then trained with OPD to obtain Qwen3-1.7B. Moreover, DeepSeek-V4 [3] and MiMo-V2-Flash [4] use OPD to merge multiple domain experts into one model, a setting where off-policy distillation is ill-suited because it causes more forgetting of the other experts’ knowledge.

In Appendix F.4 of the revised manuscript, we conduct a new comparison between on-policy and off-policy distillation at comparable cost. We use Qwen3-1.7B-Base as the student and Qwen3-8B as the teacher. For off-policy distillation, we minimize full-vocabulary forward KL. The results in the Table below show that our method outperforms off-policy distillation.

MethodMATH500MinervaOlympiadAMCAIME 24AIME 25Avg.
TA-OPD69.9325.4134.4135.949.177.0830.32
Off-policy63.0023.9026.2228.9210.006.6726.45

2. Comparison with full-vocabulary OPD [W2]

Full-vocabulary OPD is computationally prohibitive in practice, as it requires the teacher's probabilities over the entire vocabulary (151,936 tokens for Qwen2.5) at every position, about 9,500× more than k=16k=16. Increasing kk from 16 to 64 already increases the peak memory from 53 GB to 66 GB and the time per step from 130 s to 440 s (with Qwen2.5-Math-1.5B as the student and DeepSeek-R1-Distill-Qwen-7B as the teacher). Due to the high cost, we cannot provide results for full-vocabulary OPD.

However, we show that TA-OPD closely approximates full-vocabulary OPD with k=16k=16: its objective recovers about 95% of the full-vocabulary reverse KL (Appendix F.2), and it performs comparably to sample-corrected TA-OPD (Table 2), which is an unbiased estimate of full-vocabulary OPD (Proposition 5).

3. Clarification of MOPD top-kk loss [Q1]

Is the MOPD top-kk formulation used in the comparison?

Yes, “MOPD top-kk” in Table 1 refers to the top-kk loss proposed in [2], evaluated in our single-teacher setting: ℓtMOPD=∑v∈Stk(pt(v)log⁡pt(v)qt(v)−pt(v)+qt(v)).\ell_t^{\mathrm{MOPD}}=\sum_{v\in S_t^k}\Big(p_t(v)\log\frac{p_t(v)}{q_t(v)}-p_t(v)+q_t(v)\Big).

Does combining MOPD with ℓtMOPD\ell_t^{\mathrm{MOPD}} improve further over TA-OPD alone?

We have not evaluated this combination. We clarify that ℓtMOPD\ell_t^{\mathrm{MOPD}} is not specifically designed for multi-teacher distillation: in [2]’s multi-teacher experiments, it underperforms sampled-token OPD. Similarly, in our single-teacher experiments, ℓtMOPD\ell_t^{\mathrm{MOPD}} generally underperforms sampled-token OPD, while TA-OPD achieves the highest ID and OOD average scores across all model pairs. Thus, the available results do not suggest a particular advantage of ℓtMOPD\ell_t^{\mathrm{MOPD}} in the multi-teacher setting; whether ℓtMOPD\ell_t^{\mathrm{MOPD}} outperforms TA-OPD in the multi-teacher setting remains an open question.

Are the effects of multi-teacher distillation annulled in explicitly modelling the tail distribution?

TA-OPD's strong OOD performance in Table 1 suggests that explicitly modeling the tail may help preserve complementary capabilities rather than eliminate the benefits of multiple teachers.

AuthorsOpenReview ↗

Response to Reviewer vVo8 - part 2

4. Effect of kk on sample-corrected TA-OPD [Q2]

The results in Table 2 are based on k=16k=16. Yes, the sample correction matters more for smaller kk, since the bias of TA-OPD relative to full-vocabulary OPD grows with the tail probability. In the extreme case of k=0k=0, TA-OPD provides no training signal, while SC-TA-OPD reduces to sampled-token OPD, which still achieves strong performance (Table 1).

5. Generalization beyond mathematical reasoning [Q3]

We clarify that the main application of OPD is to transfer the capabilities of domain-specialized teachers into a student model, as adopted in recent industrial LLMs [3, 4], rather than general language tasks such as machine translation. Following this setting, recent OPD works [5-7] mainly evaluate on mathematical reasoning, and we adopt the same setting.

Regarding whether a larger kk is required in general (or long-tailed) domains, prior work shows that OPD remains effective with very sparse teacher supervision on general tasks. For example, on the Tulu3 prompts, sampled-token OPD with Qwen3-8B as the teacher improves the IFEval score of a mid-trained Qwen3-8B student from 79% to 83% [8]. Moreover, sample-corrected TA-OPD provides an unbiased estimate of the full-vocabulary reverse KL for any kk (Proposition 5), and thus does not rely on a large kk to cover the teacher distribution.

References

[1] Agarwal et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR, 2024.

[2] Ma et al. MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training. arXiv:2606.30406, 2026.

[3] DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. Technical Report, 2026.

[4] Xiaomi LLM-Core Team. MiMo-V2-Flash Technical Report. arXiv:2601.02780, 2026.

[5] Fu et al. Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes. COLM, 2026.

[6] Li et al. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. TMLR, 2026.

[7] Xing et al. Trust Region On-Policy Distillation. arXiv:2606.01249, 2026.

[8] Lu and Thinking Machines Lab. On-Policy Distillation. Thinking Machines Lab: Connectionism, 2025.

I thank the authors for the detailed response, which addresses many (but not all) of my concerns. Two major things that were not addressed in the response are: (a) the computational cost (yes, I agree that off- and on-policy distillation are complementary and often applied sequentially, but I want to understand the computational cost-performance tradeoff, which was not addressed in the response or updated paper); and (b) KD for tasks other than mathematical reasoning (yes, I get that most authors focus exclusively on mathematical reasoning, but for methods which specifically target the long tail, its shape and behaviour are very different for mathematical reasoning vs. lower-entropy language tasks such as machine translation, and I want to understand whether the method behaves similarly for such tasks). If some explicit analysis can be added for these, I will consider increasing my score.

We appreciate the reviewer's clarification and agree that both the cost–performance tradeoff and applicability beyond mathematical reasoning warrant explicit empirical evaluation. We address both concerns below.

Cost–performance tradeoff of on-policy and off-policy distillation

We add an explicit accuracy–runtime comparison in Appendix F.4, using Qwen3-1.7B-Base and Qwen3-8B. The results show that, excluding teacher-trajectory sampling cost, off-policy distillation provides faster initial gains but achieves lower final accuracy than TA-OPD.

Performance of TA-OPD on machine translation

We added English-to-Chinese translation experiments in Appendix F.12. We use the Qwen3-1.7B-Base and Qwen3-8B model pair. Both top-kk methods use k=16k=16. The training set is the English-to-Chinese training split of ALMA-Human-Parallel, containing 15,406 English--Chinese parallel examples, and the test set is the English-to-Chinese subset of WMT22-Test. The results are shown in the table below.

MethodBLEU ↑COMET ↑
Sampled-token OPD39.570.8667
Normalized top-kk OPD1.500.5536
TA-OPD39.770.8644

TA-OPD performs comparably to sampled-token OPD, while normalized top-kk OPD collapses, with all test outputs reaching the 512-token generation limit. These results show that TA-OPD remains effective beyond mathematical reasoning with k=16k=16.

Reviewer_AM8KOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a failure mode of normalized top-kk on-policy distillation (OPD), where the student assigns increasing probability mass to tokens outside the teacher’s top-kk support. The authors analyze the conditions for this behavior through gradient dynamics and propose Tail-Aware Top-kk OPD (TA-OPD), which minimizes reverse KL over the top-kk tokens and an aggregate tail category. A sampled-token correction is also introduced to obtain unbiased per-prefix estimates of the full-vocabulary reverse KL. Experiments across multiple student–teacher pairs report improvements in mathematical reasoning and out-of-distribution performance over normalized top-kk OPD.

Critical strengths
  • The paper addresses a practically relevant problem in distillation with limited teacher information. It shows that normalized top-kk OPD can exhibit increasing tail probability and declining performance across several student–teacher settings.
  • The mathematical analysis is another strength. The paper connects gradient dynamics with covariance and identifies conditions under which tail probability increases.
  • The method shows substantial improvements in some settings. TA-OPD improves the average score across the math benchmarks over normalized top-kk OPD for all three model pairs, with an 8.05-point gain in the Llama setting.
Critical weaknesses
  • The novelty of the objective and the additional contribution over prior work are not fully clear. The top-kk + aggregate-tail KL in Eq. (4) is very similar to the KL approximation used in DPPO. For a fixed support, it also has the same form as the distillation objective in SDPO [1]. The paper explains its difference from DPPO by using the approximation directly as an optimization objective, but SDPO also uses it directly for distillation. It would help to discuss the differences from SDPO in support selection and teacher construction, and to distinguish the contribution of the tail-inflation analysis on the teacher’s top-kk support from the construction of the objective itself.

  • Fu et al. show that top-pp rollout sampling can improve the stability and performance of normalized teacher top-kk OPD. The present paper generates training rollouts without top-pp truncation (p=1.0p=1.0). It is therefore unclear whether TA-OPD’s gains remain when this stabilization is applied. A comparison using the same stabilization for both methods under matched training budgets and evaluation conditions, or a top-pp sensitivity analysis for both methods, would help address this concern.

  • The mechanism behind the additional gains over MOPD needs further explanation. MOPD also has an optimum at pt(v)=qt(v)p_t(v)=q_t(v) for v∈Stkv\in S_t^k in distribution space, so the aggregate tail mass matches at the oracle optimum as well. TA-OPD differs from MOPD by an additional tail-mass penalty. Thus, the performance difference cannot be explained solely by whether the tail mass matches at the optimum. Beyond the reported performance comparison, it would be useful to examine how this penalty affects tail mass during training and whether it improves training stability.

  • The paper does not report results across training seeds or confidence intervals. This makes it difficult to assess how consistent the improvements are. Reporting these results would strengthen the empirical evidence.

References

  • [1] Hübotter, J., Lübeck, F., Behric, L. D., Baumann, A., Bagatella, M., Marta, D., Hakimi, I., Shenfeld, I., Kleine Buening, T., Guestrin, C., & Krause, A. (2026). Reinforcement learning via self-distillation. In T. Zhang, M. Dudik, M. Jaggi, A. Agarwal, S. Li, D. Schuurmans, J. Zhu, F. Berkenkamp, H. Dong, & A. Bietti (Eds.), Proceedings of the 43rd International Conference on Machine Learning (Vol. 306, pp. 48677–48721). PMLR.
Questions for authors
  • [Proposition 2] The claim that tail probability increases whenever the student and teacher are misaligned is stronger than the stated result. Proposition 2 establishes a first-order increase under a specific covariance condition. A difference between the two distributions alone does not guarantee tail growth. The corresponding statements should reflect this condition.

  • [Proposition 3] The gradient formula itself is valid. However, the student assigning too much probability to the tail does not, by itself, imply that the sum of the head gradients is negative. The preceding explanation uses the condition that the tail log-ratio exceeds the average head log-ratio, which is different from simply requiring pttail>qttailp_t^{\mathrm{tail}}>q_t^{\mathrm{tail}}. Could the authors keep this distinction explicit in the interpretation?

  • [Distribution alignment] TA-OPD matches the top-kk probabilities and the aggregate tail mass, but it does not directly match the conditional distribution within the tail. Therefore, the claim that it “faithfully aligns” the full next-token distribution seems too broad. The wording should reflect the scope of the coarse-grained approximation.

  • [Full-KL gap] The exact gap is pttailDKL(p~t∣q~t)p_t^{\mathrm{tail}}D_{\mathrm{KL}}(\tilde p_t|\tilde q_t), so a small tail mass alone does not guarantee a small gap. The conditional tail KL also needs to be considered. The small gap observed in Appendix F.2 should be presented as an empirical finding, with a clear distinction from a general theoretical guarantee based on tail mass alone.

  • [Figure 15] Normalizing the teacher distribution as defined in the text gives two peaks of approximately 0.1060.106 and 0.0930.093, which do not match the teacher curve in Figure 15. Could the authors clarify whether this comes from a typo in the formula or a difference in the actual experimental setup? The distribution definition and the figure should be consistent.

  • [Figure readability] The axis labels, tick labels, and legends in some figures are difficult to read at a normal PDF viewing scale. Please increase their font sizes.

3 comments
AuthorsOpenReview ↗

Response to Reviewer AM8K - part 1

We thank the reviewer for the insightful comments. Please find our response below:

1. Novelty and relationship to DPPO and SDPO [W1]

Thank you for bringing SDPO [1] to our attention. We clarify three distinctions below.

Our work explains when and why tail supervision matters in external-teacher OPD. SDPO’s student and teacher derive from the same LLM conditioned on different prefixes, so their capability gap is expected to be relatively small. Our analyses show that normalized top-kk can inflate tail probability, especially with large student–teacher capability gaps; with small capability gaps, this inflation need not occur, and normalized top-kk and TA-OPD can perform comparably. This explains the comparable performance of normalized top-kk and top-kk + tail in self-distillation reported by [2]. In other words, tail supervision may offer limited benefits in self-distillation, but our work establishes its role in addressing tail-probability inflation and improving performance in external-teacher OPD, especially when the student-teacher capability gap is large.

Different support selection. SDPO uses the student’s top-kk support, whereas our method and tail-inflation analysis focus on the teacher’s top-kk support. In Appendix C, we find that using the student's top-kk support, normalized top-kk OPD causes response-length degeneration: rollout lengths rapidly increase and saturate at the maximum generation length.

Our sample-corrected extension provides an additional theoretical contribution. DPPO [3] assumes access to the sampled token and constructs its support as St=TopK(μ(⋅∣st))∪{yt}S_t = \mathrm{TopK}(\mu(\cdot \mid s_t)) \cup \{y_t\}, where μ\mu is the behavior policy and y^t\hat y_t is the sampled token, and computes the KL over StDPPO∪{vtail}S_t^{\mathrm{DPPO}}\cup\{v_{\mathrm{tail}}\}. The KL estimator remains biased. When sampled token is available, we use SC-TA-OPD, which is unbiased to full vocabulary KL in both value and gradient.

We cite SDPO and add these distinctions in Appendix F.9 of the revised manuscript.

2. Comparison under top-pp rollout stabilization [W2]

TA-OPD’s gains remain when top-pp sampling is applied. In the table below, we compare TA-OPD and normalized top-kk OPD under top-pp sampling with Qwen2.5-Math-1.5B as the student and DeepSeek-R1-Distill-Qwen-7B as the teacher.

MethodRollout top-ppMATH-500 Accuracy
Norm. top-kk1.048.6%
TA-OPD1.061.0%
Norm. top-kk0.956.8%
TA-OPD0.964.4%

Top-pp rollout sampling improves both methods, but TA-OPD still outperforms normalized top-kk OPD by 7.6 percentage points at top-pp=0.9 (64.4% vs. 56.8%).

3. Mechanism behind the gains over MOPD top-kk loss [W3]

We conduct a new comparison in Appendix F.3 of the revised manuscript to explain the gain from two perspectives.

  1. Better tail-mass alignment. Appendix F.3 shows that across the three student–teacher pairs in Table 1, MOPD top-kk loss yields substantially larger tail-mass residual, measured as ∣pstudenttail−pteachertail∣|p_{\mathrm{student}}^{\mathrm{tail}}-p_{\mathrm{teacher}}^{\mathrm{tail}}|, than TA-OPD. Furthermore, for Qwen3-1.7B, training with MOPD top-kk loss steadily increases the tail probabilities and entropy. The larger tail-mass discrepancy may contribute to MOPD’s weaker downstream performance.
  2. Better approximation of the full-vocabulary objective. TA-OPD is empirically a tight lower bound of the full-vocabulary reverse KL, and Figure 13 shows that training with TA-OPD empirically reduces the full-vocabulary loss. In contrast, reducing MOPD top-kk loss may not reduce the full-vocabulary loss. This closer alignment with the full-vocabulary objective may help explain TA-OPD’s stronger downstream performance.

4. Results across training seeds or confidence intervals [W4]

The evaluation results are reported as Avg@8 (except for MMLU-Pro), which averages over 8 sampled responses per question to reduce evaluation variance. In Appendix G.2 of the revised manuscript, we report the mean ±\pm standard deviation of in-distribution macro-averaged accuracy across eight evaluation rounds, together with p-values from two-sided Welch's t-tests comparing TA-OPD with the best-performing baseline for each model pair. The results show significant improvements for Qwen2.5-7B-Instruct and Llama-3.1-8B (p<0.05p<0.05). Below, we present results for the Qwen2.5-7B-Instruct and OpenThinker3-7B model pair.

MethodMean ± Stdpp-value vs. best baseline
Sampled-token37.23 ± 0.31—
Norm. top-kk33.42 ± 0.32—
MOPD top-kk37.37 ± 0.34—
TA-OPD38.68 ± 0.152.20×10−62.20 \times 10^{-6}

Regarding training seeds, each experiment requires substantial computational resources, making it difficult to run multiple trials with different random seeds. Given the computational constraint, it is common practice to perform only a single run per setting as adopted in reference [3] mentioned by the reviewer.

AuthorsOpenReview ↗

Response to Reviewer AM8K - part 2

5. Presentation Issue [Q1–Q4]

We thank the reviewer for identifying these problems and providing practical suggestions. The following corrections have been made in the revised manuscript:

  1. Proposition 2. We change the unconditional claim to: “Our theoretical analysis suggests that the optimization tends to increase the student's tail probability when the student's distribution is misaligned with the teacher's, a common occurrence in distillation.”
  2. Proposition 3. We conjecture that the concern may refer to the sentence: “It can therefore explicitly raise the top-k logits relative to the tail whenever the student overweights the tail relative to the teacher.” We have revised the sentence to avoid misunderstanding.
  3. Distribution alignment. We change "faithfully align" to "better align the next-token distribution compared to normalized top-kk OPD".
  4. Full-KL gap. We remove “tight” from the theoretical analysis and describe the gap as “empirically small,” as measured in Appendix F.2.

6. Teacher distribution in synthetic experiment [Q5]

We thank the reviewer for the careful check. This is a typo in the formula. The correct teacher distribution is given by: q=softmax(z)q = \mathrm{softmax}(z) with zv=5exp⁡(−(v−10)22⋅22)+4.5exp⁡(−(v−20)22⋅1.22)+εv,z_v = 5\exp(-\frac{(v-10)^2}{2\cdot 2^2}) + 4.5\exp(-\frac{(v-20)^2}{2\cdot 1.2^2}) + \varepsilon_v, where εv∼N(0,0.122)\varepsilon_v \sim \mathcal{N}(0, 0.12^2). We correct the formula in the revised manuscript.

7. Figure readability [Q6]

Thank you for the feedback. We increase the font size for Figures 1, 3, 4, 12, 13, 18.

References

[1] Hübotter et al. Reinforcement Learning via Self-Distillation. ICML, 2026.

[2] Zucchet, N., & Linderman, S. W. (2026). Divergence controls entropy in distillation. arXiv preprint arXiv:2610.03529.

[3] Qi et al. Rethinking the Trust Region in LLM Reinforcement Learning. ICML, 2026.

[4] Fu et al. Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes. COLM, 2026.

Thank you for the detailed rebuttal. The additional experiments and clarifications address several of my concerns.

However, some concerns remain regarding methodological novelty, the fairness of baseline comparisons, and the lack of independent training-seed evaluations.

Nevertheless, given the additional evidence provided, I will raise my rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We appreciate the reviewers’ thoughtful feedback and valuable comments on our work. In the following responses, we have addressed the reviewers' comments and concerns point by point. We have also revised the manuscript according to the reviewers' suggestions, and we believe this makes our paper much stronger. The main changes$^1$ are summarized below:

  • Qualified claims about distribution alignment, tail-probability increase, and the gap to full-vocabulary KL in the Abstract, Introduction, and Section 4. [AM8K]
  • Clarified the evaluation sampling settings in Sections 3.2 and 5.1. [VHBs]
  • Added the initial performance of student and teacher models in Table 1. [VHBs]
  • Added comparisons with MOPD top-kk loss in Appendix F.3. [AM8K]
  • Added comparisons with off-policy distillation in Appendix F.4. [vVo8]
  • Added discussion of DPPO and SDPO in Related Work and in Appendix F.9. [AM8K, VHBs]
  • Added evaluation variability and statistical significance results in Appendix G.2. [AM8K, VHBs]

1^1 For clarity, we highlight the revised parts of the manuscript in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.