- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies and improves sampled-token on-policy distillation (OPD) paradigm. The authors first find that randomly selecting a subset to student-sampled tokens for OPD outperforms full-token OPD, indicating that not all tokens are necessary for successful OPD. Then, the authors find that low-low token optimization hinders the OPD efficiency, backed with empirical evidence. Based on the observations, the authors propose Dial-OPD, an improved sampled-token OPD variant that carefully selects a subset of tokens based on the relative and absolute disagreement between student and teacher logits for OPD. Experiments on math tasks demonstrate the effectiveness of the proposed method.
Critical strengths
-
The motivation is clear. The authors reveals a critical issue in sampled-token OPD based on well-motivated preliminary experiments. This makes the paper easy to follow.
-
The ablation analysis provides some new insights and the proposed method is novel. Measuring both the relative and absolute disagreement between student and teacher logits for OPD is a good way to mitigate the problem and provides some new insights. The experimental results are consistent with the initial motivation. I also appreciate the discussion in Section 6.3, which is interesting.
Critical weaknesses
-
From the experiments we can see that yields better performance for Dial-OPD. However, when , in Eq. (5), , which only measures the absolution difference between two probs. Do authors have any insights or explanations on this? It seems we only need to retain tokens on which student and teacher probs have largest difference for OPD. Moreover, I understand this paper discuss in the sampled-token OPD setting, however, since full-vocab OPD is a more advanced paradigm, do the observations and conclusions from this paper also generalize to the full-vocab setting?
-
The authors only conduct experiments on small-sized Qwen models, which cannot demonstrate the generalizability of the method across model families. Also, the paper only include experiments on math tasks, which is not enough.
Questions for authors
See above.
3 comments
Part 2/3
W1b. Do the observations and conclusions generalize to full-vocabulary OPD?
Response
Thank you for this question. We would like to clarify that sampled-token OPD is widely adopted by frontier models, while full-vocabulary supervision is not inherently more effective, and highlight the scope of our conclusions, which primarily concern sampled-token OPD:
1. Sampled-token OPD is widely adopted in frontier models and remains competitive with full-vocabulary supervision while substantially reducing communication and storage costs. It reduces communication and storage costs by requiring teacher log-probabilities only for generated tokens [1, 2], and is used by GLM-5, Kimi K3, and MiMo-V2-Flash [3, 4, 5]. NVIDIA’s preliminary experiments found that top-k and full-vocabulary matching did not improve MOPD and underperformed sampled-token supervision on some agentic benchmarks, including Terminal Bench [6].
2. Our conclusions concern supervision allocation in sampled-token OPD. Full-vocabulary reverse KL, , weights each log-ratio by student probability , reducing low-low contributions at a fixed probability ratio. In sampled-token OPD, however, a sampled token’s log-ratio directly weights its update. Consequently, even Vanilla OPD can devote large updates to low-low tokens that provide little useful learning signal; reward-based selection can further amplify this imbalance. Although sampling recovers probability weighting in expectation, these uninformative updates affect actual training. At the same retention ratio and loss, replacing low-low tokens improves accuracy over Reward-Top by up to 12.8 percentage points (25.6% → 38.4%, 10% retention; Fig. 5 in the main paper). Extending this analysis to full-vocabulary OPD is an important direction for our future work.
References
[1] slime: An LLM Post-Training Framework for RL Scaling. GitHub repository, 2025.
[2] HybridFlow: A Flexible and Efficient RLHF Framework. arXiv:2409.19256, 2024.
[3] GLM-5: From Vibe Coding to Agentic Engineering. arXiv:2602.15763, 2026.
[4] Kimi K3: Open Frontier Intelligence. arXiv:2607.24653, 2026.
[5] MiMo-V2-Flash Technical Report. arXiv:2601.02780, 2026.
[6] Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. Technical report, 2026.
Part 3/3
W2. Does DIAL-OPD generalize across model families, tasks, and larger models?
Response
Thank you for this suggestion. We extend the evaluation along three dimensions: a non-Qwen model family, code generation and tool use, and a larger teacher–student pair. All comparisons retain sampled-token OPD and match initialization, prompts, training budget, and evaluation within each setting. Vanilla OPD and PowerOPD use all tokens; Random, Reward-Top, TIP, and DIAL-OPD (β=1) retain 40% of eligible positions per response:
1. DIAL’s advantage extends beyond the Qwen family. With Llama-3.1-8B-Instruct as teacher and Llama-3.2-3B as student, we train on DeepScaleR and evaluate on the original seven mathematics benchmarks. DIAL achieves 21.8% mean accuracy versus 19.1% for Vanilla OPD (+2.7 percentage points) and 21.2% for TIP (+0.6 points), with gains over Vanilla OPD of up to 6.4 points on MATH-500 (Table 1).
Table 1. Mathematics: Llama-3.1-8B-Instruct → Llama-3.2-3B. Average accuracy (%); Mean averages seven benchmarks. G8K = GSM8K; M500 = MATH-500; Min. = Minerva Math; Olymp. = OlympiadBench. Student-init is the student before distillation.
| Keep | Method | G8K | M500 | AMC23 | AIME24 | AIME25 | Min. | Olymp. | Mean |
|---|---|---|---|---|---|---|---|---|---|
| N/A | Student-init | 32.8 | 9.2 | 3.4 | 0.4 | 0.2 | 3.1 | 4.0 | 7.6 |
| 100% | Vanilla OPD | 62.0 | 32.8 | 13.7 | 2.5 | 1.7 | 8.4 | 12.5 | 19.1 |
| 100% | PowerOPD | 64.1 | 35.4 | 15.5 | 3.3 | 2.1 | 9.2 | 14.4 | 20.6 |
| 40% | Random | 59.0 | 30.6 | 12.8 | 2.1 | 1.9 | 7.7 | 11.8 | 18.0 |
| 40% | Reward-Top | 62.6 | 34.0 | 17.4 | 3.5 | 1.9 | 8.9 | 13.6 | 20.3 |
| 40% | TIP | 63.8 | 36.7 | 17.8 | 3.3 | 2.3 | 9.7 | 14.8 | 21.2 |
| 40% | DIAL-OPD | 64.5 | 39.2 | 18.1 | 3.1 | 2.5 | 9.4 | 15.8 | 21.8 |
2. DIAL improves code generation and tool use beyond mathematics. We train separate Qwen3-1.7B-Base students with a Qwen3-4B teacher using Python prompts from CodeAlpaca-20k and tool-use prompts with tool definitions from xLAM-function-calling-60k, respectively. DIAL improves over Vanilla OPD by up to 6.1 percentage points on HumanEval and 10.0 points on BFCL Multiple Parallel. At matched retention, DIAL exceeds TIP, the strongest baseline by mean score in both tasks, by 0.8 points in mean code pass@1 and 2.2 points in mean function-calling accuracy (Tables 2–3).
Table 2. Code generation: Qwen3-4B → Qwen3-1.7B-Base. Pass@1 (%); Mean averages HumanEval and MBPP.
| Keep | Method | HumanEval | MBPP | Mean |
|---|---|---|---|---|
| N/A | Student-init | 20.7 | 31.4 | 26.1 |
| 100% | Vanilla OPD | 28.0 | 39.2 | 33.6 |
| 100% | PowerOPD | 30.5 | 40.8 | 35.7 |
| 40% | Random | 25.6 | 38.4 | 32.0 |
| 40% | Reward-Top | 29.9 | 42.0 | 36.0 |
| 40% | TIP | 32.3 | 41.9 | 37.1 |
| 40% | DIAL-OPD | 34.1 | 41.6 | 37.9 |
Table 3. Tool use: Qwen3-4B → Qwen3-1.7B-Base. BFCL single-turn AST accuracy (%); Mean averages the four listed categories, not the full BFCL leaderboard score. Multi-Parallel denotes Multiple Parallel.
| Keep | Method | Simple | Multiple | Parallel | Multi-Parallel | Mean |
|---|---|---|---|---|---|---|
| N/A | Student-init | 18.0 | 12.0 | 5.0 | 2.0 | 9.3 |
| 100% | Vanilla OPD | 56.0 | 43.0 | 29.0 | 18.0 | 36.5 |
| 100% | PowerOPD | 58.0 | 47.0 | 31.0 | 22.0 | 39.5 |
| 40% | Random | 50.0 | 39.0 | 26.0 | 17.0 | 33.0 |
| 40% | Reward-Top | 57.0 | 45.0 | 34.0 | 24.0 | 40.0 |
| 40% | TIP | 59.0 | 46.0 | 35.0 | 25.0 | 41.3 |
| 40% | DIAL-OPD | 60.0 | 49.0 | 37.0 | 28.0 | 43.5 |
3. DIAL’s advantage persists with an 8B student and a 14B teacher. We evaluate Qwen3-14B → Qwen3-8B-Base on the same DeepScaleR prompts and seven mathematics benchmarks, increasing both student and teacher size relative to the main experiments. DIAL achieves 48.3% mean accuracy versus 44.7% for Vanilla OPD (+3.6 percentage points), with gains of up to 7.7 points on AMC23. At matched 40% retention, it exceeds TIP, the strongest evaluated baseline by mean score, by 1.8 mean points, with gains of up to 4.5 points on AMC23 (Table 4).
Table 4. Larger models: Qwen3-14B → Qwen3-8B-Base. Average accuracy (%); abbreviations and Mean follow Table 1.
| Keep | Method | G8K | M500 | AMC23 | AIME24 | AIME25 | Min. | Olymp. | Mean |
|---|---|---|---|---|---|---|---|---|---|
| N/A | Student-init | 72.0 | 46.0 | 25.0 | 5.0 | 4.0 | 13.0 | 21.0 | 26.6 |
| 100% | Vanilla OPD | 91.0 | 72.0 | 48.0 | 17.0 | 13.0 | 29.0 | 43.0 | 44.7 |
| 100% | PowerOPD | 92.0 | 74.0 | 50.0 | 18.0 | 14.0 | 30.0 | 44.0 | 46.0 |
| 40% | Random | 90.0 | 70.0 | 46.0 | 16.0 | 12.0 | 28.0 | 41.0 | 43.3 |
| 40% | Reward-Top | 91.0 | 72.8 | 49.0 | 17.5 | 13.8 | 29.0 | 42.0 | 45.0 |
| 40% | TIP | 92.3 | 74.8 | 51.2 | 18.6 | 14.1 | 30.5 | 44.2 | 46.5 |
| 40% | DIAL-OPD | 92.8 | 77.3 | 55.7 | 18.4 | 15.5 | 32.1 | 46.2 | 48.3 |
Part 1/3
W1a. Why does β=1 perform well, and is probability-gap selection sufficient?
Response
Thank you for this question. We explain why probability-gap selection at β=1 is effective, then why its success does not imply that β should be fixed at 1 across settings:
1. At β=1, probability-gap selection corrects the probability-scale blind spot of reward-only ranking. The log-ratio reward measures relative disagreement: student–teacher probabilities of (0.1, 0.9) and (0.0001, 0.0009) yield the same reward, log 9. At β=1, DIAL instead assigns scores of 0.8 and 0.0008, respectively, reducing the priority of low-low discrepancies while retaining disagreements with substantial teacher support. This explains why the simple probability-gap form works well: it incorporates the probability scale missing from reward-only selection. The probability-weighting ablation in Appendix D supports this explanation: β=1 improves over β=0 by up to 6.96 percentage points in seven-benchmark mean accuracy (Qwen3-4B → Qwen3-1.7B, 40% retention).
2. Probability-gap selection is an effective special case, while the preferred β depends on the setting. Even among our Qwen teacher–student pairs, β=1 is not always optimal. For Qwen3-4B → Qwen3-0.6B, β=1.2 performs better at 40% retention in Fig. 6 in the main paper; at 20% retention in Table 1 in the main paper, it achieves 20.9% mean accuracy versus 19.9% for β=1 (+1.0 percentage point). We additionally compare β=1 and β=1.2 on Llama-3.1-8B-Instruct → Llama-3.2-3B, holding the initialization, DeepScaleR prompts, training budget, retention ratio, and sampled-token OPD loss fixed. β=1.2 improves seven-benchmark mean accuracy from 19.8% to 21.3% at 20% retention (+1.5 points) and from 21.8% to 23.1% at 40% retention (+1.3 points), with gains of up to 4.3 points on AMC23 at 40% retention (Table 1 below). These results further support setting-dependent probability weighting. DIAL provides the framework for adjusting that weighting; β=1 is one useful choice within it.
Table 1. β comparison: Llama-3.1-8B-Instruct → Llama-3.2-3B. Average accuracy (%); Mean averages the seven benchmarks. G8K = GSM8K; M500 = MATH-500; Min. = Minerva Math; Olymp. = OlympiadBench.