- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates a failure mode of normalized top-k OPD, where the student model is trained to match the teacher's next-token distribution over the teacher's top-k vocabulary, with both distributions normalized over the selected vocab set , which can potentially cause issues when the student and teacher are mis-aligned. The authors observe that the tail probabilities (the probability mass outside the teacher's top-k vocabulary) of both the student and teacher steadily increase during distillation with normalized top-k OPD, as this strategy discards tail information. To address this issue, they propose Tail-Aware Top-k OPD (TA-OPD), which additionally takes the aggregated tail probability into account by introducing a single tail token representing this probability mass and minimizing reverse KL divergence over the augmented distribution. However, introducing tail token is not a new idea, as it was previously introduced in Qi et al. (2026).
Experiments across multiple language-model families trained on math reasoning tasks demonstrate that TA-OPD mitigates tail-probability inflation and yields modest improvements in downstream task performance over baselines, while showing a comparably larger performance gap on OOD benchmarks (or less forgetting of OOD knowledge), especially when the student and teacher are mis-aligned.
[References]
Qi, P., Zhou, X., Liu, Z., Pang, T., Du, C., Lin, M. and Lee, W.S., 2026. Rethinking the trust region in llm reinforcement learning. arXiv preprint arXiv:2602.04879.
Critical strengths
- The observation that tail probability inflation occurs for both the student and teacher is helpful.
- The OOD performance gap is comparatively large relative to in-distribution performance, supporting that TA-OPD excels at preserving prior knowledge.
Critical weaknesses
- The idea itself is not novel. It was already introduced in another paper, although plugging it into OPD is appreciated. The propositions are mostly trivial claims.
- There is no strong evidence that tail probability inflation leads to performance degradation. Figure 2 strongly indicates that high tail probability leads to low accuracy with normalized top-k OPD. However, the results do not align with those in Table 1, where the MATH500 accuracy of the Norm. top-k baseline is 68.78, which does not align with the claim in Figure 2. I would expect its accuracy to be below 25, with a tail probability of ~0.5, but it isn't. This really confuses me about the overall results. Relatedly, the weak overall improvements in in-distribution performance make the claim weaker (an average 0–2% improvement is just noise under T=0.7). Considering that the OOD performance gap looks better, I would consider reframing the paper.
Questions for authors
Additional experiments required
-
Is a practical setup? The authors mention that considering the full vocabulary could make training costs high, but setting or seems to mitigate the issue. I am curious whether increasing to 32 or 64 substantially increases the training cost (i.e., peak memory or training time). If not, there is no reason to use a larger setting, as seems to properly address the tail probability inflation issue.
-
Missing baseline results that must be compared with. In Table 1, you must report the teacher's and student's initial performance before distillation.
-
It is unclear whether tail inflation induces performance degradation. Following the point raised in the weakness section, I would suggest reporting the performance of normalized top-k OPD and TA-OPD with different values of (2, 4, 8, 16, 32, 64). Currently, you only show tail probability changes for these settings in Figure 8 without reporting any performance changes.
Questions that do not require experiments
-
Inconsistency between the claim and results. Why is the comparison in Table 2 different from the setting in Table 1?
-
Why did you choose the reverse KL formulation instead of the forward KL formulation over student-sampled trajectories? Reverse KL is mode-seeking, so it could be vulnerable to the teacher–student mis-alignment issue.
-
Please demonstrate the reason for distilling models of the same size. There is no motivation to distill in this case, as we can simply deploy the teacher model. If the goal was to claim transferability, you should have claimed that OOD knowledge degradation is not as severe as with other methods.
-
Temperature inconsistency. What are the rollout and evaluation temperature settings? I am curious why you used different temperatures for Figure 2 (1.0) and Table 1 (0.7). The temperature descriptions in Section 5.1 and Table 4 are also inconsistent.
-
Also, why is the student–teacher setting in Table 2 different from that in Table 1?
Slightly annoying things that might not change the score but are worth addressing
- Figure 1 is too far from line43.
- I don't understand why sample-corrected TA-OPD is introduced after the main results section. I recommend putting this content in the Method section rather than in a separate Discussion section.
5 comments
Response to Reviewer VHBs - part 2
6. Initial performance of teachers and students [Q2]
We thank the reviewer for the suggestion. We add the initial performance of teachers and students in Table 1 of the revised manuscript.
7. Settings of Table 2 [Q4, Q8]
Table 2 is an ablation study of the bias-correction term, rather than a comparison with baselines. Its goal is to examine whether correcting the bias of TA-OPD improves performance, so we only compare TA-OPD with SC-TA-OPD. For the ablation study, we use smaller LLMs to reduce computational cost. But the evaluation setup is the same as that in Table 1.
8. Choice of reverse KL [Q5]
We use reverse KL for two reasons. First, reverse KL is the standard objective in OPD, adopted by most OPD works [4] and industrial LLMs [5, 6]. Moreover, [7] shows that reverse KL achieves higher Avg@k than forward KL in OPD. Second, off-policy distillation typically minimizes forward KL [8] over teacher-sampled trajectories. When switching to student-sampled trajectories, a natural idea is to use reverse KL to avoid using importance sampling.
9. Motivation for same-size distillation [Q6]
One important application of OPD is model merging, where domain-specialized teachers are trained from the same base model and then merged into a single student of the same size. For example, DeepSeek-V4 [5] and MiMo-V2-Flash [6] first train several domain experts independently, and then consolidate them into a unified model via OPD with the reverse KL loss. In this setting, the teacher cannot be deployed, since each teacher only excels in its own domain.
10. Presentation issues
We remove "see Figure 1" from the Introduction. Regarding sample-corrected TA-OPD, we position it as an extension of TA-OPD, since it additionally uses the sampled token. We introduce it after the main results to show that TA-OPD alone, without the sampled-token information, already addresses the tail probability increase issue.
References
[1] Qi et al. Rethinking the Trust Region in LLM Reinforcement Learning. ICML, 2026.
[2] Li et al. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. TMLR, 2026.
[3] Jin et al. Entropy-Aware On-Policy Distillation of Language Models. ICML, 2026.
[4] Lu and Thinking Machines Lab. On-Policy Distillation. Thinking Machines Lab: Connectionism, 2025.
[5] DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. Technical Report, 2026.
[6] Xiaomi LLM-Core Team. MiMo-V2-Flash Technical Report. arXiv:2601.02780, 2026.
[7] Zhao et al. Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillation. arXiv:2605.16826, 2026.
[8] Kim, Y., and Rush, A. M. Sequence-Level Knowledge Distillation. EMNLP, 2016.
Thank you to the authors for the detailed rebuttal, the additional experiments, and the clarifications in the revised manuscript. I appreciate the clear explanations regarding the training rollout versus evaluation temperature settings (which indeed follow standard conventions), the motivation for the smaller model pairs in Table 2, and the addition of the initial student/teacher baselines in Table 1.
However, after reviewing the updated results and responses, several methodological and empirical concerns remain that prevent me from raising my score at this stage:
- Practical significance under standard decoding and comparison to Sampled-token OPD
While the temperature explanation resolves the numerical discrepancy between Figure 2 (T=1.0, p=1.0) and Table 1 (T=0.7, p=0.95), it also weakens the practical severity of the claimed failure mode. Under standard inference decoding, Normalized top-k OPD recovers from 20.60% to 68.78% on MATH500, indicating that standard nucleus/temperature sampling already mitigates the vast majority of the tail-inflation degradation.
More importantly, when comparing TA-OPD against Sampled-token OPD, which requires only a single token log-probability, has no tail-inflation issue, and is ~12% faster in wall-clock runtime. The in-distribution gains of TA-OPD against sampled-token OPD on instruction-tuned models is still marginal.
- Confounded evaluation in the Llama-3.1-8B -> DeepSeek-R1-Distill-Llama-8B experiment
Adding the pre-distillation baselines in Table 1 reveals a striking anomaly in the Llama-3.1-8B setup: the initial student achieves only 1.37% on ARC-c and 2.43% on MMLU-Pro, the teacher achieves 25.10% on ARC-c, yet the TA-OPD student trained solely on math data jumps to 65.70% on ARC-c (maybe format/parser failure?). The large gains on this pair appear confounded by template/format alignment rather than clean knowledge distillation or OOD preservation. Could the authors clarify why the teacher scores only 25.10% on ARC-c while the math-distilled student reaches 65.70%?
- Implausible computational overhead reported for k=64 vs. k=16
The authors state that increasing k from 16 to 64 increases peak GPU memory from 53 GB to 66 GB (+13 GB) and step time from 130 s to 440 s (3.38x). However, the full-vocabulary forward/backward pass of the student and forward pass of the teacher are already computed regardless of k, and as shown in Listing 1, the top-k loss tensors have shape (batch, seq_len, k). Increasing k by 48 elements would add less than 500MB of tensor memory per batch (nowhere near 13 GB).
Could the authors clarify what caused this 3.4x slowdown and +13 GB memory spike? Specifically, did the average rollout response length change substantially between k=16 and k=64, or is this overhead an artifact of serializing larger top-k Python dictionaries over vLLM's IPC interface?
Response to Reviewer VHBs - part 1
We sincerely appreciate the reviewer's constructive feedback. Please find our response below:
1. Novelty relative to Qi et al. (2026) [W1]
Our contribution goes beyond applying an existing construction to OPD. As recognized by Reviewer AM8K (“The mathematical analysis is another strength”) and Reviewer vVo8 (“solid theoretical analysis”), our propositions are not trivial. We further clarify the differences from DPPO [1] below.
First, DPPO assumes access to the sampled token and constructs its support as
\cup \{\hat{y}_t\},$$ where $\mu$ is the behavior policy and $\hat y_t$ is the sampled token, and computes the KL over $S_t^{\mathrm{DPPO}}\cup\{v_{\mathrm{tail}}\}$. The KL estimator remains biased. When sampled token is available, we use SC-TA-OPD, which is unbiased to full vocabulary KL in both value and gradient. Second, DPPO uses the approximate KL between the behavior and current policies to determine whether to mask a policy-gradient update, replacing PPO’s ratio-based clipping. It does not directly optimize this divergence through its gradient. In our setting, the approximate KL itself serves as the distillation loss, making its optimization properties central. We identify tail-probability inflation in normalized top-$k$ OPD, introduce an aggregate-tail token to address it, and analyze the resulting TA-OPD loss and its gradient properties. **2. Temperature Inconsistency [W2, Q4, Q7]** The temperature setup in Section 5.1 and Table 4 is consistent: we use temperature 1.0 and top-$p$ 1.0 for training rollouts, and temperature 0.7 and top-$p$ 0.95 for evaluation, following prior work [1]. The only exception is Figure 2, which uses temperature 1.0 and top-$p$ 1.0 for evaluation. This is because any other temperature would change the probability of sampling tail tokens and make it differ from the measured tail probability. A low temperature and top-$p$ sampling mitigate the effect of tail probability increase by sampling fewer tail tokens, which explains why normalized top-$k$ OPD with Qwen2.5-7B-Instruct achieves 68.78% on MATH500 in Table 1, much higher than its accuracy in Figure 2. We clarify both the temperature setup and its effect on the Table 1 vs. Figure 2 accuracy gap in the revised manuscript. Specifically, we add the sentence "the validation sampling temperature is $1$, which ensures that the probability of sampling tail tokens is the same as the measured tail probability" at line 199 and "low evaluation temperatures and top-$p$ can mitigate the accuracy degradation by sampling fewer tail tokens" at line 209. **3. Relationship between tail probability increase and performance degradation [W2, Q3]** As suggested, we report the MATH500 accuracy of normalized top-$k$ OPD and TA-OPD with different $k$ (evaluation temperature 0.7, top-$p$ 0.95), using Qwen2.5-Math-1.5B as the student and DeepSeek-R1-Distill-Qwen-7B as the teacher. |$k$|8|16|32|64| |-|-:|-:|-:|-:| |Norm. top-$k$ OPD (%)|31.0|48.6|52.4|54.0| |TA-OPD (%)|58.2|61.0|–|–| When $k$ increases from 8 to 16, the tail probability increase issue is substantially mitigated and the validation accuracy improves from 31.0\% to 48.6\%. Combined with Figure 8, these results show that tail probability increase leads to performance degradation. Moreover, TA-OPD with $k=16$ (61.0%) outperforms normalized top-$k$ OPD with $k=64$ (54.0%). Experiments of TA-OPD with $k=32$ and $k=64$ are still running, and we will include them in the final version. **4. Significance of in-distribution improvements [W2]** The evaluation results are reported as Avg@8 (except for MMLU-Pro), i.e., the accuracy averaged over 8 sampled responses per question, which already reduces the variance of evaluation. Thus, the improvement is not noisy. In Appendix G.2 of the revised manuscript, we report the mean $\pm$ standard deviation of in-distribution macro-averaged accuracy across eight evaluation rounds, together with p-values from two-sided Welch's t-tests comparing TA-OPD with the best-performing baseline. The results show significant improvements for Qwen2.5-7B-Instruct and Llama-3.1-8B ($p<0.05$). **5. Practicality of $k=16$ and training cost of larger $k$ [Q1]** $k=16$ is a practical setup and is adopted by prior work [2, 3]. Our Figure 5 shows that the performance of top-$k$ OPD saturates quickly as $k$ increases and $k=16$ already achieves strong performance. However, increasing $k$ substantially increases the training cost. With Qwen2.5-Math-1.5B as the student and DeepSeek-R1-Distill-Qwen-7B as the teacher, training with $k=64$ increases the peak memory from 53 GB to 66 GB and the time per step from 130 s to 440 s compared with $k=16$.