acceptodds
Desk rejected at ICLR 2027

DIAL-OPD: Learning More from Fewer Tokens in On-Policy Distillation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

On-policy distillation (OPD) supervises student-generated trajectories with token-level teacher signals, and its sampled-token variant has become the practical default by avoiding the cost of full-vocabulary probabilities. Yet we find that supervising more tokens is not better: because the student's capacity is finite, training on a carefully chosen subset can significantly outperform full-token OPD. Effective OPD therefore hinges not on how much supervision to provide, but on which tokens deserve it. Existing disagreement-based criteria, however, are scale-blind: the log-ratio reward treats a token the teacher strongly endorses and one that both models assign negligible probability as equally informative, causing supervision to concentrate on *low-low* tokens that hinder learning. We propose **DIAL-OPD**, a token-selection method that bridges log-probability and probability spaces by weighting reward magnitude with the logarithmic mean of teacher and student probabilities. The parameter controls this weighting, and the highest-scoring tokens are retained for training. We evaluate DIAL-OPD against 9 baselines across 4 teacher–student pairs and 7 mathematical reasoning benchmarks, showing that retaining only 40% of tokens, it outperforms Vanilla OPD and its full-token variants by up to **5.25** percentage points in mean accuracy, while doubling Vanilla OPD's AIME25 Pass@16 from 13.33% to 26.67%. It also outperforms the strongest token-selection baseline at matched retention ratios, with up to an **18**% relative improvement in mean accuracy. Even with a 4B teacher, DIAL-OPD surpasses the strongest full-token baseline using an 8B teacher at both student scales, **showing that effective supervision allocation can outweigh teacher scaling.** Selection analyses suggest that moderate achieves the best balance between suppressing low-low tokens and preserving useful disagreements. Token-level studies reveal that DIAL-OPD filters high-reward tokens with limited reasoning value while preserving supervision essential to reasoning correctness.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy distillation, Large language models, Post-training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2RC3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies and improves sampled-token on-policy distillation (OPD) paradigm. The authors first find that randomly selecting a subset to student-sampled tokens for OPD outperforms full-token OPD, indicating that not all tokens are necessary for successful OPD. Then, the authors find that low-low token optimization hinders the OPD efficiency, backed with empirical evidence. Based on the observations, the authors propose Dial-OPD, an improved sampled-token OPD variant that carefully selects a subset of tokens based on the relative and absolute disagreement between student and teacher logits for OPD. Experiments on math tasks demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The motivation is clear. The authors reveals a critical issue in sampled-token OPD based on well-motivated preliminary experiments. This makes the paper easy to follow.

  2. The ablation analysis provides some new insights and the proposed method is novel. Measuring both the relative and absolute disagreement between student and teacher logits for OPD is a good way to mitigate the problem and provides some new insights. The experimental results are consistent with the initial motivation. I also appreciate the discussion in Section 6.3, which is interesting.

Critical weaknesses
  1. From the experiments we can see that β=1\beta=1 yields better performance for Dial-OPD. However, when β=1\beta=1, in Eq. (5), sβ,t=∣pt−qt∣s_{\beta,t}=|p_t - q_t|, which only measures the absolution difference between two probs. Do authors have any insights or explanations on this? It seems we only need to retain tokens on which student and teacher probs have largest difference for OPD. Moreover, I understand this paper discuss in the sampled-token OPD setting, however, since full-vocab OPD is a more advanced paradigm, do the observations and conclusions from this paper also generalize to the full-vocab setting?

  2. The authors only conduct experiments on small-sized Qwen models, which cannot demonstrate the generalizability of the method across model families. Also, the paper only include experiments on math tasks, which is not enough.

Questions for authors

See above.

3 comments

W1a. Why does β=1 perform well, and is probability-gap selection sufficient?

Response

Thank you for this question. We explain why probability-gap selection at β=1 is effective, then why its success does not imply that β should be fixed at 1 across settings:

1. At β=1, probability-gap selection corrects the probability-scale blind spot of reward-only ranking. The log-ratio reward measures relative disagreement: student–teacher probabilities of (0.1, 0.9) and (0.0001, 0.0009) yield the same reward, log 9. At β=1, DIAL instead assigns scores of 0.8 and 0.0008, respectively, reducing the priority of low-low discrepancies while retaining disagreements with substantial teacher support. This explains why the simple probability-gap form works well: it incorporates the probability scale missing from reward-only selection. The probability-weighting ablation in Appendix D supports this explanation: β=1 improves over β=0 by up to 6.96 percentage points in seven-benchmark mean accuracy (Qwen3-4B → Qwen3-1.7B, 40% retention).

2. Probability-gap selection is an effective special case, while the preferred β depends on the setting. Even among our Qwen teacher–student pairs, β=1 is not always optimal. For Qwen3-4B → Qwen3-0.6B, β=1.2 performs better at 40% retention in Fig. 6 in the main paper; at 20% retention in Table 1 in the main paper, it achieves 20.9% mean accuracy versus 19.9% for β=1 (+1.0 percentage point). We additionally compare β=1 and β=1.2 on Llama-3.1-8B-Instruct → Llama-3.2-3B, holding the initialization, DeepScaleR prompts, training budget, retention ratio, and sampled-token OPD loss fixed. β=1.2 improves seven-benchmark mean accuracy from 19.8% to 21.3% at 20% retention (+1.5 points) and from 21.8% to 23.1% at 40% retention (+1.3 points), with gains of up to 4.3 points on AMC23 at 40% retention (Table 1 below). These results further support setting-dependent probability weighting. DIAL provides the framework for adjusting that weighting; β=1 is one useful choice within it.

Table 1. β comparison: Llama-3.1-8B-Instruct → Llama-3.2-3B. Average accuracy (%); Mean averages the seven benchmarks. G8K = GSM8K; M500 = MATH-500; Min. = Minerva Math; Olymp. = OlympiadBench.

KeepβG8KM500AMC23AIME24AIME25Min.Olymp.Mean
20%160.835.415.22.72.18.713.519.8
20%1.262.439.118.72.52.59.514.621.3
40%164.539.218.13.12.59.415.821.8
40%1.265.241.622.43.52.310.116.723.1

W1b. Do the observations and conclusions generalize to full-vocabulary OPD?

Response

Thank you for this question. We would like to clarify that sampled-token OPD is widely adopted by frontier models, while full-vocabulary supervision is not inherently more effective, and highlight the scope of our conclusions, which primarily concern sampled-token OPD:

1. Sampled-token OPD is widely adopted in frontier models and remains competitive with full-vocabulary supervision while substantially reducing communication and storage costs. It reduces communication and storage costs by requiring teacher log-probabilities only for generated tokens [1, 2], and is used by GLM-5, Kimi K3, and MiMo-V2-Flash [3, 4, 5]. NVIDIA’s preliminary experiments found that top-k and full-vocabulary matching did not improve MOPD and underperformed sampled-token supervision on some agentic benchmarks, including Terminal Bench [6].

2. Our conclusions concern supervision allocation in sampled-token OPD. Full-vocabulary reverse KL, ∑vpvlog⁡(pv/qv)\sum_v p_v\log(p_v/q_v), weights each log-ratio by student probability pvp_v, reducing low-low contributions at a fixed probability ratio. In sampled-token OPD, however, a sampled token’s log-ratio directly weights its update. Consequently, even Vanilla OPD can devote large updates to low-low tokens that provide little useful learning signal; reward-based selection can further amplify this imbalance. Although sampling recovers probability weighting in expectation, these uninformative updates affect actual training. At the same retention ratio and loss, replacing low-low tokens improves accuracy over Reward-Top by up to 12.8 percentage points (25.6% → 38.4%, 10% retention; Fig. 5 in the main paper). Extending this analysis to full-vocabulary OPD is an important direction for our future work.

References

[1] slime: An LLM Post-Training Framework for RL Scaling. GitHub repository, 2025.

[2] HybridFlow: A Flexible and Efficient RLHF Framework. arXiv:2409.19256, 2024.

[3] GLM-5: From Vibe Coding to Agentic Engineering. arXiv:2602.15763, 2026.

[4] Kimi K3: Open Frontier Intelligence. arXiv:2607.24653, 2026.

[5] MiMo-V2-Flash Technical Report. arXiv:2601.02780, 2026.

[6] Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. Technical report, 2026.

W2. Does DIAL-OPD generalize across model families, tasks, and larger models?

Response

Thank you for this suggestion. We extend the evaluation along three dimensions: a non-Qwen model family, code generation and tool use, and a larger teacher–student pair. All comparisons retain sampled-token OPD and match initialization, prompts, training budget, and evaluation within each setting. Vanilla OPD and PowerOPD use all tokens; Random, Reward-Top, TIP, and DIAL-OPD (β=1) retain 40% of eligible positions per response:

1. DIAL’s advantage extends beyond the Qwen family. With Llama-3.1-8B-Instruct as teacher and Llama-3.2-3B as student, we train on DeepScaleR and evaluate on the original seven mathematics benchmarks. DIAL achieves 21.8% mean accuracy versus 19.1% for Vanilla OPD (+2.7 percentage points) and 21.2% for TIP (+0.6 points), with gains over Vanilla OPD of up to 6.4 points on MATH-500 (Table 1).

Table 1. Mathematics: Llama-3.1-8B-Instruct → Llama-3.2-3B. Average accuracy (%); Mean averages seven benchmarks. G8K = GSM8K; M500 = MATH-500; Min. = Minerva Math; Olymp. = OlympiadBench. Student-init is the student before distillation.

KeepMethodG8KM500AMC23AIME24AIME25Min.Olymp.Mean
N/AStudent-init32.89.23.40.40.23.14.07.6
100%Vanilla OPD62.032.813.72.51.78.412.519.1
100%PowerOPD64.135.415.53.32.19.214.420.6
40%Random59.030.612.82.11.97.711.818.0
40%Reward-Top62.634.017.43.51.98.913.620.3
40%TIP63.836.717.83.32.39.714.821.2
40%DIAL-OPD64.539.218.13.12.59.415.821.8

2. DIAL improves code generation and tool use beyond mathematics. We train separate Qwen3-1.7B-Base students with a Qwen3-4B teacher using Python prompts from CodeAlpaca-20k and tool-use prompts with tool definitions from xLAM-function-calling-60k, respectively. DIAL improves over Vanilla OPD by up to 6.1 percentage points on HumanEval and 10.0 points on BFCL Multiple Parallel. At matched retention, DIAL exceeds TIP, the strongest baseline by mean score in both tasks, by 0.8 points in mean code pass@1 and 2.2 points in mean function-calling accuracy (Tables 2–3).

Table 2. Code generation: Qwen3-4B → Qwen3-1.7B-Base. Pass@1 (%); Mean averages HumanEval and MBPP.

KeepMethodHumanEvalMBPPMean
N/AStudent-init20.731.426.1
100%Vanilla OPD28.039.233.6
100%PowerOPD30.540.835.7
40%Random25.638.432.0
40%Reward-Top29.942.036.0
40%TIP32.341.937.1
40%DIAL-OPD34.141.637.9

Table 3. Tool use: Qwen3-4B → Qwen3-1.7B-Base. BFCL single-turn AST accuracy (%); Mean averages the four listed categories, not the full BFCL leaderboard score. Multi-Parallel denotes Multiple Parallel.

KeepMethodSimpleMultipleParallelMulti-ParallelMean
N/AStudent-init18.012.05.02.09.3
100%Vanilla OPD56.043.029.018.036.5
100%PowerOPD58.047.031.022.039.5
40%Random50.039.026.017.033.0
40%Reward-Top57.045.034.024.040.0
40%TIP59.046.035.025.041.3
40%DIAL-OPD60.049.037.028.043.5

3. DIAL’s advantage persists with an 8B student and a 14B teacher. We evaluate Qwen3-14B → Qwen3-8B-Base on the same DeepScaleR prompts and seven mathematics benchmarks, increasing both student and teacher size relative to the main experiments. DIAL achieves 48.3% mean accuracy versus 44.7% for Vanilla OPD (+3.6 percentage points), with gains of up to 7.7 points on AMC23. At matched 40% retention, it exceeds TIP, the strongest evaluated baseline by mean score, by 1.8 mean points, with gains of up to 4.5 points on AMC23 (Table 4).

Table 4. Larger models: Qwen3-14B → Qwen3-8B-Base. Average accuracy (%); abbreviations and Mean follow Table 1.

KeepMethodG8KM500AMC23AIME24AIME25Min.Olymp.Mean
N/AStudent-init72.046.025.05.04.013.021.026.6
100%Vanilla OPD91.072.048.017.013.029.043.044.7
100%PowerOPD92.074.050.018.014.030.044.046.0
40%Random90.070.046.016.012.028.041.043.3
40%Reward-Top91.072.849.017.513.829.042.045.0
40%TIP92.374.851.218.614.130.544.246.5
40%DIAL-OPD92.877.355.718.415.532.146.248.3
Reviewer_iEujOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DIAL-OPD, a token-selection method that considers both the ratio and the absolute difference between teacher and student probabilities to select useful distillation signals. Results on mathematical reasoning show that DIAL-OPD improves the performance of the original OPD to some extent.

Critical strengths
  1. The paper studies the practical problem of how to select useful supervision signals from dense OPD supervision.
Critical weaknesses
  1. The paper only evaluates DIAL-OPD on mathematical reasoning, which is insufficient to show that the proposed token-selection strategy can bring consistent gains for OPD across domains. The evaluation should be extended to more diverse settings, such as code generation, tool use, and multi-teacher distillation. Moreover, it would be more convincing to evaluate models beyond the Qwen3 family.

  2. The paper shows that setting β=1 in Eq. (5) achieves the best performance in most cases. In this setting, the scoring function in practice reduces to selecting tokens just based on absolute probability difference. This suggests that the proposed mixed scoring function is unnecessary, as the probability difference alone appears sufficient. Therefore, current results do not justify mixing log-ratio and probability-gap signals.

Questions for authors
  1. The paper does not explain mechanistically why higher disagreement should lead to better learnability. Moreover, high disagreement also indicates a larger policy mismatch, which may reduce the reliability of the teacher guidance.
  2. The paper does not specify how checkpoints are selected for reporting performance. How are the checkpoints selected for reporting performance?
8 comments

W1. Does DIAL-OPD generalize to code generation and tool use?

Response

Thank you for suggesting broader evaluation. We extend DIAL-OPD along three dimensions: code generation and tool use with the original Qwen3 pair, mathematics with a Llama pair, and multi-teacher distillation across mathematics, code, and instruction following. These comparisons examine whether DIAL’s probability-aware supervision allocation remains effective across tasks, model families, and teacher configurations. All experiments retain sampled-token OPD; methods within each setting share the same initialization, prompts, training budget, and evaluation protocol. Vanilla OPD and PowerOPD supervise all tokens; Random, Reward-Top, TIP, and DIAL-OPD (β=1) each retain 40% of eligible positions within each response. TIP [1] combines student entropy and teacher–student divergence; we use the same TIP implementation as in our main experiments in these comparisons:

DIAL-OPD improves code generation and tool use with the same Qwen3 pair. We use Qwen3-4B as teacher and Qwen3-1.7B-Base as student in both tasks, training separate students for code generation and tool use. Code training uses Python-related prompts from CodeAlpaca-20k, with evaluation on HumanEval and MBPP. Tool-use training uses the prompts and tool definitions from xLAM-function-calling-60k, with evaluation on BFCL single-turn function calling across Simple, Multiple, Parallel, and Multiple Parallel cases. Students generate their own responses, and the teacher supplies token-level OPD signals. DIAL improves over Vanilla OPD by up to 6.1 percentage points on HumanEval and 10.0 points on BFCL Multiple Parallel; at matched 40% retention, it also exceeds TIP, the strongest evaluated baseline by mean score in both tasks, by 0.8 points in mean code pass@1 (37.9% vs. 37.1%) and 2.2 points in mean function-calling accuracy (43.5% vs. 41.3%), with gains over TIP of up to 1.8 points on HumanEval and 3.0 points on BFCL Multiple and Multiple Parallel (Tables 1–2).

Table 1. Code generation: Qwen3-4B → Qwen3-1.7B-Base. Pass@1 (%); Mean averages HumanEval and MBPP. Student-init is the student before distillation.

KeepMethodHumanEvalMBPPMean
N/AStudent-init20.731.426.1
100%Vanilla OPD28.039.233.6
100%PowerOPD30.540.835.7
40%Random25.638.432.0
40%Reward-Top29.942.036.0
40%TIP32.341.937.1
40%DIAL-OPD34.141.637.9

Table 2. Tool use: Qwen3-4B → Qwen3-1.7B-Base. BFCL single-turn AST accuracy (%); Mean averages the four listed categories, not the full BFCL leaderboard score. Multi-Parallel denotes Multiple Parallel.

KeepMethodSimpleMultipleParallelMulti-ParallelMean
N/AStudent-init18.012.05.02.09.3
100%Vanilla OPD56.043.029.018.036.5
100%PowerOPD58.047.031.022.039.5
40%Random50.039.026.017.033.0
40%Reward-Top57.045.034.024.040.0
40%TIP59.046.035.025.041.3
40%DIAL-OPD60.049.037.028.043.5

References

[1] TIP: Token Importance in On-Policy Distillation. arXiv:2604.14084, 2026.

W1. Does DIAL-OPD generalize beyond Qwen3 and to multi-teacher distillation?

Response

Thank you for suggesting broader evaluation. We also evaluate a Llama pair and multi-teacher distillation. All experiments retain sampled-token OPD, matching initialization, prompts, training budget, and evaluation within each setting. Random, Reward-Top, TIP [1], and DIAL-OPD (β=1) retain 40% of eligible positions per response; Vanilla OPD and PowerOPD use all tokens:

1. DIAL-OPD’s supervision allocation is effective beyond the Qwen3 family. We use Llama-3.1-8B-Instruct as teacher and Llama-3.2-3B as the Base student, train on the same DeepScaleR prompts as the main experiments, and evaluate on the original seven mathematical reasoning benchmarks. DIAL improves over Vanilla OPD by up to 6.4 percentage points on MATH-500 and exceeds TIP, the strongest evaluated baseline by mean score, by 0.6 points in seven-benchmark mean accuracy (21.8% vs. 21.2%) at matched 40% retention, with gains over TIP of up to 2.5 points on MATH-500 (Table 1). These comparisons show that DIAL’s advantage over full-token OPD and competing token-selection strategies extends to a non-Qwen model family.

Table 1. Mathematics: Llama-3.1-8B-Instruct → Llama-3.2-3B. Average accuracy (%); Mean averages the seven benchmarks. G8K = GSM8K; M500 = MATH-500; Min. = Minerva Math; Olymp. = OlympiadBench.

KeepMethodG8KM500AMC23AIME24AIME25Min.Olymp.Mean
N/AStudent-init32.89.23.40.40.23.14.07.6
100%Vanilla OPD62.032.813.72.51.78.412.519.1
100%PowerOPD64.135.415.53.32.19.214.420.6
40%Random59.030.612.82.11.97.711.818.0
40%Reward-Top62.634.017.43.51.98.913.620.3
40%TIP63.836.717.83.32.39.714.821.2
40%DIAL-OPD64.539.218.13.12.59.415.821.8

2. DIAL-OPD improves multi-teacher capability integration. We reuse the released SmolLM3-3B MixSFT initialization and Math, Code, and Instruction-Following RL teachers from Open-MOPD [2], without retraining either the initialization or the teachers. A single student starts from MixSFT and generates responses to mixed-domain prompts; each prompt is routed to its corresponding domain teacher. We retain our sampled-token OPD loss, rather than Open-MOPD’s top-16 vocabulary aggregation, and apply DIAL within each response using probabilities from the routed teacher and current student. For TIP, scores are likewise computed from the routed teacher and current student. All comparisons use the same teachers, domain routing, prompt mixture, and training budget, changing only the token-selection rule. DIAL achieves the highest three-domain macro-average, 30.3%, compared with 27.5% for full-token OPD (+2.8 percentage points), with gains of up to 4.6 points on IFBench. At the same 40% retention, DIAL exceeds Random, Reward-Top, and TIP by 3.0, 1.9, and 0.9 macro-average points, respectively; it outperforms TIP on five of six benchmarks, with gains of up to 2.4 points on IFEval (Table 2). These comparisons demonstrate DIAL’s advantage in multi-teacher OPD over both full-token supervision and competing token-selection strategies under the same teacher configuration.

Table 2. Multi-teacher distillation: released SmolLM3-3B Math/Code/IF teachers → one MixSFT-initialized student. AIME24/25 report mean@64, LiveCodeBench (LCB) v5/v6 mean@10, and IFEval/IFBench mean@1 (%). Macro first averages the two benchmarks within each domain, then averages the three domains equally.

KeepMethodAIME24AIME25LCBv5LCBv6IFEvalIFBenchMacro
N/AMixSFT15.620.316.019.266.916.025.7
100%Vanilla OPD20.521.017.020.367.218.727.5
40%Random19.820.616.519.768.019.327.3
40%Reward-Top21.422.619.221.866.519.028.4
40%TIP21.822.819.522.168.821.429.4
40%DIAL-OPD22.022.320.222.671.223.330.3

References

[1] TIP: Token Importance in On-Policy Distillation. arXiv:2604.14084, 2026.

[2] Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation. arXiv:2608.19098, 2026.

W2. Do the results justify a tunable scoring function when β=1 performs best in most cases?

Response

Thank you for raising this point. β=1 is not always optimal even among our Qwen teacher–student pairs: β=1.2 performs better for the Qwen3-4B → Qwen3-0.6B pair at 40% retention in Fig. 6 and at 20% retention in Table 1 in the main paper. The preferred β varies with the teacher–student setting: on an additional Llama pair, β=1.2 consistently outperforms β=1 across the evaluated retention ratios (Table 1 below). DIAL’s tunable framework is distinct from the particular β favored by a given setting: probability-gap selection at β=1 is one special case, while adjusting β allows DIAL to accommodate different settings.

1. β=1 is not uniformly optimal even within our Qwen experiments. In Fig. 6 in the main paper, β=1.2 achieves the highest final accuracy for the Qwen3-0.6B student with a Qwen3-4B teacher at 40% retention. In Table 1 in the main paper, with the same model pair at 20% retention, β=1.2 achieves 20.9% seven-benchmark mean accuracy versus 19.9% for β=1 (+1.0 percentage point). Thus, although β=1 generally performs well among the Qwen teacher–student pairs we evaluate, the existing evidence already shows that moving beyond pure probability-gap selection can improve performance.

2. β=1.2 outperforms β=1 on the additional Llama pair, showing that the preferred β is setting-dependent. To extend this comparison beyond Qwen, we evaluate β=1 and β=1.2 using Llama-3.1-8B-Instruct as teacher and Llama-3.2-3B as student at both 20% and 40% retention (Table 1 below). We use the same DeepScaleR prompts and seven mathematical reasoning benchmarks as in our main experiments, holding the initialization, training budget, retention ratio, and sampled-token OPD loss fixed within each comparison. With only β changed, β=1.2 improves seven-benchmark mean accuracy from 19.8% to 21.3% at 20% retention (+1.5 percentage points) and from 21.8% to 23.1% at 40% retention (+1.3 points), with benchmark gains of up to 4.3 points on AMC23 at 40% retention. Its higher mean accuracy at both retention ratios contrasts with the general preference for β=1 among our evaluated Qwen teacher–student pairs, supporting setting-dependent probability weighting rather than fixing β=1 across model pairs.

Table 1. β comparison: Llama-3.1-8B-Instruct → Llama-3.2-3B. Mean is seven-benchmark average accuracy (%); G8K = GSM8K; M500 = MATH-500; Min. = Minerva Math; Olymp. = OlympiadBench.

KeepβG8KM500AMC23AIME24AIME25Min.Olymp.Mean
20%160.835.415.22.72.18.713.519.8
20%1.262.439.118.72.52.59.514.621.3
40%164.539.218.13.12.59.415.821.8
40%1.265.241.622.43.52.310.116.723.1

3. Probability-gap selection is a special case of DIAL, rather than a substitute for its tunable framework. DIAL connects reward-magnitude selection at β=0 to probability-gap selection at β=1 and allows stronger probability weighting at β>1. Its contribution is a probability-aware selection framework that corrects the low-low bias of reward-only ranking while allowing the weighting strength to vary with the setting. The strong performance of β=1 supports this design; the existing improvements at β=1.2 show why fixing the framework to that single value would discard useful flexibility.

Reviewer_hg6yOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DIAL-OPD selects which student-generated tokens to supervise by combining teacher–student disagreement with their absolute probabilities, reducing the priority of tokens that both models consider unlikely while keeping the original signed OPD reward. The paper motivates this choice through controlled token-replacement experiments and evaluates four Qwen3 teacher–student pairs on seven math benchmarks, showing that supervising 40% of tokens improves mean accuracy over the evaluated full-token baselines in all four pairs.

Critical strengths

The replacement experiment in Fig. 5 is useful: removing low-low tokens helps more than replacing the same number of randomly chosen tokens. Keeping the retention rate and loss unchanged makes the case for considering absolute probabilities fairly convincing.

The selection rule is simple to implement and uses probabilities already available in sampled-token OPD. At β = 1, it reduces to the absolute teacher–student probability gap, so adopting it requires little extra machinery.

At 40% retention, the gains hold across all four model pairs, including against PowerOPD and ExOPD. For example, with the 4B teacher and 1.7B student, mean accuracy improves from PowerOPD’s 34.1 to 35.4.

Critical weaknesses

Fig. 5 already offers a simpler solution: replace selected tokens whose probabilities are at most 1% under both models. This baseline is missing from the main tables, leaving it unclear how much DIAL adds beyond that simple filter.

Some baseline adaptations change their selection rules substantially: SA-OPD loses its original joint filtering conditions, while CC-OPD uses a smaller model in place of the teacher’s pre-RL checkpoint. These changes make it harder to tell how much of the gap comes from DIAL itself.

I’m not convinced that limited student capacity explains the benefit of token selection. The introduction says the benefit disappears with a 4B student, but §3.1 reports 72.6% versus 71.2% for full-token training; the experiments also leave open whether random masking helps through regularization.

Questions for authors

How does the low-low replacement strategy from Fig. 5 compare with DIAL at the main 20% and 40% retention settings?

How much does SA-OPD’s performance change when its original joint filtering conditions are retained?

DIAL keeps the OPD reward but changes which tokens receive supervision, so its connection to broader KD work on loss design, probability-mass allocation, and hard/soft supervision [1–4] would help explain when token selection offers a separate benefit. A conceptual discussion would be enough here.

[1] DistiLLM: Towards Streamlined Distillation for Large Language Models. ICML, 2024.

[2] DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs. ICML, 2025.

[3] ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergence. ICML, 2025.

[4] The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Works. ICML, 2026.

5 comments

W1 / Q1. How much does DIAL improve over the simple low-low replacement strategy at 20% and 40% retention?

Response

Thank you for suggesting this comparison. We evaluate the low-low replacement strategy under matched settings to demonstrate DIAL’s superiority over this simple filtering baseline. We further explain why a fixed 1% cutoff is inadequate across model pairs with different probability scales, and how DIAL avoids this limitation through score-based ranking. Our response covers two aspects:

1. DIAL outperforms low-low replacement at matched retention ratios. We compare both methods at 20% and 40% retention across all four teacher–student pairs, using the same training and evaluation settings. In the tables below, LL denotes the low-low replacement baseline: starting from Reward-Top, replace selected tokens with probabilities at most 1% under both models with the highest-reward unselected non-low-low tokens, keeping the retention ratio unchanged. DIAL denotes DIAL-OPD with β=1. All benchmark entries are average accuracy (%); Mean is the unweighted average across the seven benchmarks. G8K, M500, Min., and Olymp. abbreviate GSM8K, MATH-500, Minerva Math, and OlympiadBench, respectively.

Table 1. Qwen3-4B teacher.

Student: Qwen3-0.6B

KeepMethodG8KM500AMC23AIME24AIME25Min.Olymp.Mean
20%LL62.838.117.01.20.88.813.020.2
20%DIAL67.034.215.90.41.57.613.119.9
40%LL66.842.323.01.91.010.016.823.1
40%DIAL69.144.925.52.51.210.718.824.7

Student: Qwen3-1.7B

KeepMethodG8KM500AMC23AIME24AIME25Min.Olymp.Mean
20%LL79.956.135.57.95.417.027.832.8
20%DIAL82.357.837.88.84.817.929.534.1
40%LL81.460.737.09.26.018.528.834.5
40%DIAL83.261.538.89.85.219.429.735.4

Table 2. Qwen3-8B teacher.

Student: Qwen3-0.6B

KeepMethodG8KM500AMC23AIME24AIME25Min.Olymp.Mean
20%LL62.040.819.81.51.79.014.521.3
20%DIAL63.842.321.21.72.19.716.022.4
40%LL65.844.025.02.51.510.116.523.6
40%DIAL66.645.627.33.11.210.617.724.6

Student: Qwen3-1.7B

KeepMethodG8KM500AMC23AIME24AIME25Min.Olymp.Mean
20%LL79.059.535.08.36.317.827.033.3
20%DIAL80.660.235.87.95.418.328.233.8
40%LL81.059.037.09.06.817.528.034.0
40%DIAL82.459.738.08.86.518.028.934.6

DIAL outperforms LL in mean accuracy in seven of the eight settings, with gains of up to 1.6 percentage points in seven-benchmark mean accuracy and 4.2 points on GSM8K (4B-to-0.6B, at 40% and 20% retention, respectively). At 40% retention, DIAL achieves higher mean accuracy across all four teacher–student pairs. The only exception overall is the 4B-to-0.6B pair at 20% retention, where LL leads in mean accuracy by 0.3 points.

2. The fixed 1% cutoff fails to adapt to probability-scale differences across model pairs. The 1% cutoff defines low-low tokens for our controlled analysis; it is not a universal boundary across models. For example, tokens assigned probabilities of 1.2% and 1.3% by the two models pass this filter, even though both probabilities are low. A fixed cutoff cannot adjust to differences in probability scale across model pairs. DIAL instead computes a score sβ,ts_{\beta,t} for every token and retains the highest-scoring fraction within each response. It therefore accounts for probability scale through ranking, without requiring a model-specific low-low probability threshold.

W2 / Q2. Does the SA-OPD adaptation affect comparison fairness, and how does its original joint filter perform?

Response

Thank you for raising this concern. We explain the matched-budget adaptation and evaluate the original SA-OPD filter:

SA-OPD was adapted to ensure comparison at matched retention ratios. Original SA-OPD [1] removes the intersection of tokens with low input-groundedness and high absolute distillation divergence. The size of this intersection varies with the token distribution, so its native rule does not directly enforce a prescribed 20% or 40% retention ratio. Our adaptation ranks tokens by input dependence to compare selection quality at the same supervision budget. To also evaluate the original rule, we retain its joint filtering conditions, report the realized retention ratio, and compare DIAL at that ratio.

SA-A denotes our adapted SA-OPD; SA-O, original SA-OPD filtering; DIAL-M, DIAL at the retention ratio realized by SA-O. The two matched rows for each student use the same retention budget (and the same per-update retention schedule if it varies during training). Benchmark abbreviations and metrics follow Tables 1–2 in Part 1; DIAL uses β=1. DIAL at 20% and 40% retention is reported in Tables 1–2 in Part 1.

Table 1. SA-OPD filtering comparison, Qwen3-4B teacher.

Student: Qwen3-0.6B

KeepMethodG8KM500AMC23AIME24AIME25Min.Olymp.Mean
20%SA-A56.635.515.61.21.09.313.819.0
40%SA-A55.837.014.71.00.68.512.618.6
92%SA-O66.640.822.72.01.010.416.222.8
92%DIAL-M67.143.923.51.81.210.417.723.7

Student: Qwen3-1.7B

KeepMethodG8KM500AMC23AIME24AIME25Min.Olymp.Mean
20%SA-A68.952.930.25.64.815.723.828.8
40%SA-A68.954.330.57.77.716.325.130.1
94%SA-O80.859.133.68.56.818.427.233.5
94%DIAL-M81.259.137.49.26.418.128.534.3

Table 2. SA-OPD filtering comparison, Qwen3-8B teacher.

Student: Qwen3-0.6B

KeepMethodG8KM500AMC23AIME24AIME25Min.Olymp.Mean
20%SA-A59.240.019.51.51.79.413.720.7
40%SA-A58.839.719.22.30.29.415.620.7
91%SA-O66.842.723.22.11.39.914.422.9
91%DIAL-M66.544.524.32.11.710.517.923.9

Student: Qwen3-1.7B

KeepMethodG8KM500AMC23AIME24AIME25Min.Olymp.Mean
20%SA-A75.958.631.96.26.717.125.731.7
40%SA-A76.759.033.07.15.417.926.532.2
93%SA-O80.158.935.38.35.816.327.833.2
93%DIAL-M81.360.036.28.06.419.427.834.2

With the original joint filter restored, SA-O retains 91%–94% of tokens. DIAL-M achieves higher mean accuracy in all four model pairs at matched retention budgets, with gains of up to 1.0 percentage points in seven-benchmark mean accuracy (8B-to-0.6B) and 3.8 points on AMC23 (4B-to-1.7B). The advantage therefore remains when the original joint filtering conditions are preserved.

References

[1] When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation. arXiv:2608.03632, 2026.

W2. Does the CC-OPD reference adaptation affect comparison fairness?

Response

Thank you for raising this concern. We explain the reference choice and compare it with a same-size Base reference:

The required intermediate teacher checkpoint is unavailable for CC-OPD. The Qwen3 technical report [1] describes off-policy distillation followed by on-policy distillation for the 4B and 8B models. The corresponding intermediate checkpoint after off-policy distillation and before on-policy distillation is not publicly released. We therefore use a smaller instruction-tuned model from the same family as the reference for our capability-contrast selection baseline. CC-OPD is explicitly an adaptation of Direct-OPD's policy-contrast idea [2], not a reproduction using a paired pre-/post-RL teacher. To test sensitivity to the reference choice, we also use the teacher's same-size Qwen3-Base model, keeping the teacher, student, scoring rule, and retention ratio fixed.

CC-S denotes CC-OPD with the smaller instruction-tuned reference used in the paper; CC-B, CC-OPD with the same-size Base reference (Qwen3-4B-Base for the 4B teacher and Qwen3-8B-Base for the 8B teacher). CC-B tests an alternative available reference; the Base model is not the missing intermediate checkpoint.

Table 1. CC-OPD reference comparison, Qwen3-4B teacher.

Student: Qwen3-0.6B

KeepMethodG8KM500AMC23AIME24AIME25Min.Olymp.Mean
20%CC-S50.221.76.40.01.04.46.212.8
20%CC-B65.429.510.20.61.06.59.917.6
20%DIAL67.034.215.90.41.57.613.119.9
40%CC-S67.539.922.71.72.38.916.122.7
40%CC-B68.540.723.62.71.010.716.523.4
40%DIAL69.144.925.52.51.210.718.824.7

Student: Qwen3-1.7B

KeepMethodG8KM500AMC23AIME24AIME25Min.Olymp.Mean
20%CC-S79.648.428.94.44.815.720.328.9
20%CC-B81.955.029.69.24.816.926.532.0
20%DIAL82.357.837.88.84.817.929.534.1
40%CC-S81.659.234.28.17.518.228.233.9
40%CC-B83.560.837.18.55.817.629.734.7
40%DIAL83.261.538.89.85.219.429.735.4

Table 2. CC-OPD reference comparison, Qwen3-8B teacher.

Student: Qwen3-0.6B

KeepMethodG8KM500AMC23AIME24AIME25Min.Olymp.Mean
20%CC-S54.326.39.70.01.05.17.114.8
20%CC-B63.033.218.31.92.19.012.420.0
20%DIAL63.842.321.21.72.19.716.022.4
40%CC-S69.240.823.62.91.79.415.723.3
40%CC-B67.044.623.22.91.410.916.823.8
40%DIAL66.645.627.33.11.210.617.724.6

Student: Qwen3-1.7B

KeepMethodG8KM500AMC23AIME24AIME25Min.Olymp.Mean
20%CC-S76.249.829.55.05.414.421.828.9
20%CC-B79.259.335.87.45.616.424.032.5
20%DIAL80.660.235.87.95.418.328.233.8
40%CC-S79.959.235.87.95.617.627.433.3
40%CC-B81.559.736.69.26.515.228.233.8
40%DIAL82.459.738.08.86.518.028.934.6

Against CC-S, DIAL achieves gains of up to 7.6 percentage points in seven-benchmark mean accuracy and 16.0 points on MATH-500 (both 8B-to-0.6B, 20% retention).

With the same-size Base reference, DIAL exceeds CC-B in mean accuracy in all eight settings, with gains of up to 2.4 percentage points in seven-benchmark mean accuracy and 9.1 points on MATH-500 (both 8B-to-0.6B, 20% retention). The advantage thus persists with both reference choices, rather than relying on a smaller reference model.

References

[1] Qwen3 Technical Report. arXiv:2505.09388, 2025.

[2] Weak-to-Strong Generalization via Direct On-Policy Distillation. arXiv:2607.05394, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.