- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes RIVAD, a reasoning distillation framework that combines three training modes: off-policy distillation using teacher-generated responses, on-policy distillation using student-generated responses, and self-distillation using a student-derived teacher conditioned on partial reasoning steps. The method dynamically adjusts the sampling probabilities of these modes based on their validation loss improvements, increasing the training allocation for a mode when its loss decreases more slowly than a predefined threshold. The authors evaluate RIVAD using Qwen2.5 and Gemma2 models on several reasoning benchmarks, including GSM8K, MATH, and MBPP, and compare it against existing knowledge distillation methods.
Critical strengths
None
Critical weaknesses
The proposed method looks like a simple combination of existing distillation techniques, with little motivation for its design choices. In particular, the paper increases the sampling ratio when a certain mode's loss does not decrease fast enough. However, a slow decrease in loss does not mean that increasing the training budget would yield greater benefits. It could also mean that the training is close to saturation or that the supervision signal is noisy. The paper provides little analysis or justification for this strategy. It is also unclear where the performance improvement actually comes from.
The paper does not clearly explain why randomly masking reasoning steps would provide better supervision. While the authors include a limited ablation on different masking strategies, the improvement over using the full reasoning context is very small. It is unclear why this particular design is necessary or whether the improvement simply comes from giving the self-teacher additional information.
The paper does not compare its method with several standard training and distillation approaches, including SFT, rejection fine-tuning, GRPO, and OPSD. These baselines are important for understanding whether the proposed method offers any meaningful improvement over existing approaches, especially OPSD, which is closely related to the proposed self-distillation method.
The experiments use relatively old and small models, such as Qwen2.5 and Gemma2, and the evaluation benchmarks are still GSM8K, MBPP, etc., all of which predate 2025. It is hard to say how relevant the proposed method is to current reasoning models or what value it could bring to the community.
Questions for authors
What is the motivation behind the proposed adaptive routing strategy? Why should allocating more training budget to a mode with slower loss improvement lead to better performance? Can the authors provide experimental evidence supporting this design?
What motivates the random masking of reasoning steps in the self-distillation mode?
5 comments
Why Random Step Masking Outperforms Full Context in `SELF` and Comparisons with OPSD, RFT, GRPO
2. Why Random Step Masking Outperforms Full Context in SELF (Weakness 2 & Question 2)
- Preventing Reference-Copying Shortcuts:
In
SELF, both student and self-teacher score the same canonical trajectory under teacher forcing (Table 1, Eq. 1). Conditioning a self-teacher on the complete untruncated solution (No truncation) exposes the entire reference solution and final answer 100% of the time, inducing reference-copying shortcuts and overly sharp distributions that degrade transfer to a question-only student [1, 2]. Sampling and randomly masking segments (with at least one hidden and one visible step, Eq. 3–4) withholds parts of the derivation and forces the self-teacher to model inter-step conditional relationships. - Accuracy and Efficiency Gains (Table 8):
Random (Ours)achieves the highest average accuracy (54.76%), outperforming bothNo truncation(54.42%) and static truncation (Fixed 25%: 54.32%,Fixed 50%: 54.38%), with a substantial +2.4 point improvement on MBPP (49.2% vs. 46.8%) alongside gains on MATH (53.0% vs. 52.7%) and STEM (52.6% vs. 52.1%), while shortening the self-teacher's prompt length and reducing GPU memory overhead.
3. Comparisons with OPSD, RFT, and GRPO (Weakness 3)
-
Relationship to OPSD: Standard OPSD [3] supervises student-generated rollouts using full reference solutions without an external teacher. This reliance on self-supervision can be particularly problematic for compact reasoning models: on Qwen3-1.7B with thinking enabled, recent work reports an average accuracy drop from 37.2% to 30.8% under long-budget evaluation [2]. The original OPSD study also shows that omitting per-token pointwise clipping can cause performance collapse on this model [3], while Purified OPSD identifies reference-induced shortcuts that undermine transferable reasoning in long-CoT models [1]. These findings suggest that access to correct solutions alone does not ensure that a compact model can effectively supervise itself. Our
SELF-onlyablation (41.54% on Qwen2.5-1.5B and 13.5% on Gemma2-2B) and our full-referenceNo truncationablation (54.42% vs. 54.76% forRandommasking; 46.8% vs. 49.2% on MBPP, Table 8) directly demonstrate why privileged self-distillation is substantially more effective when (i) using random step masking on canonical trajectories and (ii) combined with external-teacher supervision (OFF/ON) inRIVAD(54.76% / 40.0%). -
Why White-Box KD Is Orthogonal to RFT and GRPO: Our work studies white-box token-level knowledge distillation from a larger teacher on general multi-task reasoning data (UltraInteract 78K), matching the exact setting of
MiniLLM,GKD,DistiLLM,Speculative KD,TSD-KD,CSD, andAMiD. RFT and GRPO are outcome-verifier RL/filtering methods that require rule-based scalar verifiers and multi-rollout group sampling with non-trivial initial pass rates. For compact 1.5B–2B students with low base accuracy on hard tasks (16.9% and 16.1% on MATH) and general reasoning datasets without rule verifiers, white-box KD provides dense token-level supervision at single-trajectory cost (70.3 min/epoch, Table 7) as an orthogonal training stage prior to RL.
Standardized ICLR 2026 Benchmark Protocol and Non-Saturated Tasks (Weakness 4)
4. Standardized ICLR 2026 Benchmark Protocol and Non-Saturated Tasks (Weakness 4)
Our experimental setup (Qwen2.5-14B 1.5B and Gemma2-9B 2B on UltraInteract 78K across 8 benchmarks) strictly follows the benchmark protocol established by the latest ICLR 2026 white-box KD papers: TSD-KD [4], CSD [5], and AMiD [6] (Lines 346–388). Beyond GSM8K and MBPP, our 8-benchmark suite includes MATH, GSM-Plus, MMLU-Pro-Math, BBH-CoT, MMLU-STEM, and SciQ, which remain far from saturated at the 1.5B–2B scale:
| Benchmark (Student Model) | Base Student | SFT (Seq-KD) | Best Prior ICLR 2026 Baseline | RIVAD (Ours) | Gain Over Best Baseline |
|---|---|---|---|---|---|
MATH (Qwen2.5-1.5B, Table 2) | 16.9% | 16.5% | 29.3% (CSD) | 53.0% | +23.7 pts (+80.9% rel.) |
GSM-Plus (Qwen2.5-1.5B, Table 2) | 38.8% | 38.2% | 42.1% (AMiD) | 50.2% | +8.1 pts (+19.2% rel.) |
8-Task Avg (Qwen2.5-1.5B, Table 2) | 44.6% | 44.7% | 49.5% (CSD) | 55.5% | +6.0 pts (+12.1% rel.) |
MATH (Gemma2-2B, Table 6) | 16.1% | 17.7% | 20.5% (AMiD) | 24.1% | +3.6 pts (+17.6% rel.) |
5-Task Avg (Gemma2-2B, Table 6) | 34.1% | 34.0% | 36.4% (AMiD) | 40.0% | +3.6 pts (+9.9% rel.) |
References
[1] Shen et al., Purified OPSD: On-policy self-distillation without losing how to think, arXiv:2607.02234, 2026.
[2] Kaur et al., Rethinking on-policy self-distillation for thinking models, arXiv:2607.05184, 2026.
[3] Zhao et al., Self-distilled reasoner: On-policy self-distillation for large language models, arXiv:2601.18734, 2026.
[4] Kim & Baek, Explain in your own words: Improving reasoning via token-selective dual knowledge distillation, ICLR 2026.
[5] Kim et al., Distillation of large language models via concrete score matching, ICLR 2026.
[6] Shin et al., AMiD: Knowledge distillation for LLMs with -mixture assistant distribution, ICLR 2026.
Response to Reviewer hPZ1
We thank Reviewer hPZ1 for the feedback. Below, we address each concern using our theoretical analysis, controlled ablations, and experimental comparisons.
1. Adaptive Routing Motivation, Saturation/Noise Concerns, and Source of Gains (Weakness 1 & Question 1)
Intuition: Strong Teacher Guidance First, Complementary Signals as the Student Improves: At the beginning of training, the student is relatively weak and benefits from guidance from a stronger external teacher. We therefore assign a high initial probability to
OFF, which distills the external teacher's distributions along canonical reference responses (ρOFF(1)≫ρON(1),ρSELF(1), Eq. 5). This provides a foundation before placing greater weight on supervision that depends on the student's evolving capabilities. As the student becomes more capable, the additional benefit of OFF-dominated training may diminish, motivating greater use ofONandSELFas complementary signals.ONsupplies external-teacher feedback on the student's own generated trajectories, whileSELFadds privileged-context supervision from a student-derived teacher on canonical responses. The intuition is to adapt the balance of supervision as learning progresses, while retaining external-teacher guidance throughOFFandON.Connecting This Intuition to the Progress Signal and Theoretical Analysis: Under OFF-dominated training, shared student parameters allow
ONandSELFto benefit indirectly fromOFFupdates while receiving relatively few direct updates (Section 3.3, Lines 284–286). If this indirect transfer slows, allocating more direct updates to these modes may provide useful complementary supervision. Equation (8) uses Im(t)<δ as feedback for increasing their allocation. This is a motivation for the rule, rather than a guarantee that slow progress uniquely identifies a transfer plateau or rules out saturation and noise. Theorem 1 (Eq. 13, proved in Appendix A, Eq. 14–25) provides the following bound:DTm≤D0mexp(−∑t=1T1{m(t)=m}Im(t)+aϵ∑t=1T1{m(t)=m}),ϵ=ηG2.
When Im(t)<δ, increasing ρm (Eq. 8) shifts training budget from potentially interfering cross-mode updates (m(t)=m) toward direct contraction (m(t)=m, Lines 333–341).
Built-In Safeguards Against Noise and Late-Stage Saturation:
ON) and fixed step masks on canonical responses (SELF) prior to refreshing pT~ (Algorithm 2, Lines 11–15), eliminating mini-batch sampling variance.SELFalways scores canonical responses r=y rather than noisy student rollouts (Lines 225–229), andONoptimizes only LKDON against the frozen external teacher pT without LCE (Eq. 11), avoiding direct hard-label imitation of sampled tokens.OFFanchor throughout late training.Empirical Evidence for Multi-Mode Synergy and Adaptive Routing (Tables 3 & 4): Combining privileged
SELFwith external-teacher supervision under our adaptive routing boosts 5-task accuracy over standalone modes (OFF: 40.28%,ON: 41.12%,SELF: 41.54%) to 54.12% (OFF + SELF), 55.58% (ON + SELF), and 54.76% (RIVAD) on Qwen2.5, and from 13.5%–35.1% to 40.0% (RIVAD) on Gemma2. Within the 3-mode setup,RIVADoutperformsUniform (1/3, 1/3, 1/3)on both Qwen2.5 (54.76% vs. 54.38%, including +1.8% on MBPP and +0.8% on STEM) and Gemma2 (40.0% vs. 39.5%) while requiring fewer expensive on-policyONrollouts.