acceptodds
Under review as a conference paper at ICLR 2027

RIVAD: Reasoning with Information-Variant Views for Adaptive Multi-Mode Distillation

10 views

Abstract

Large language models (LLMs) exhibit strong multi-step reasoning capabilities, yet transferring these abilities to compact models remains challenging. Existing methods typically rely on either teacher-generated trajectories, which provide stable supervision but may not reflect the prefixes encountered by the student at inference, or student-generated trajectories, which reduce this mismatch at higher training cost. We introduce RIVAD, a reasoning distillation framework that combines information-variant self-distillation with adaptive multi-mode training. Training integrates three complementary modes: canonical off-policy distillation from an external teacher, privileged-context self-distillation, and on-policy distillation over fresh student-generated responses. In the self-distillation mode, RIVAD constructs information-variant views by exposing a self-teacher to different partial, order-preserving subsets of the canonical reasoning, while the student receives only the original question. An adaptive training strategy dynamically adjusts the contribution of different supervision modes according to the student's learning progress, allowing the training allocation to evolve throughout optimization. Together, these components combine stable canonical targets, information-variant privileged guidance, and feedback on student-visited prefixes in a unified framework for distilling multi-step reasoning into compact language models.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LLM distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_hPZ1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes RIVAD, a reasoning distillation framework that combines three training modes: off-policy distillation using teacher-generated responses, on-policy distillation using student-generated responses, and self-distillation using a student-derived teacher conditioned on partial reasoning steps. The method dynamically adjusts the sampling probabilities of these modes based on their validation loss improvements, increasing the training allocation for a mode when its loss decreases more slowly than a predefined threshold. The authors evaluate RIVAD using Qwen2.5 and Gemma2 models on several reasoning benchmarks, including GSM8K, MATH, and MBPP, and compare it against existing knowledge distillation methods.

Critical strengths

None

Critical weaknesses

The proposed method looks like a simple combination of existing distillation techniques, with little motivation for its design choices. In particular, the paper increases the sampling ratio when a certain mode's loss does not decrease fast enough. However, a slow decrease in loss does not mean that increasing the training budget would yield greater benefits. It could also mean that the training is close to saturation or that the supervision signal is noisy. The paper provides little analysis or justification for this strategy. It is also unclear where the performance improvement actually comes from.

The paper does not clearly explain why randomly masking reasoning steps would provide better supervision. While the authors include a limited ablation on different masking strategies, the improvement over using the full reasoning context is very small. It is unclear why this particular design is necessary or whether the improvement simply comes from giving the self-teacher additional information.

The paper does not compare its method with several standard training and distillation approaches, including SFT, rejection fine-tuning, GRPO, and OPSD. These baselines are important for understanding whether the proposed method offers any meaningful improvement over existing approaches, especially OPSD, which is closely related to the proposed self-distillation method.

The experiments use relatively old and small models, such as Qwen2.5 and Gemma2, and the evaluation benchmarks are still GSM8K, MBPP, etc., all of which predate 2025. It is hard to say how relevant the proposed method is to current reasoning models or what value it could bring to the community.

Questions for authors

What is the motivation behind the proposed adaptive routing strategy? Why should allocating more training budget to a mode with slower loss improvement lead to better performance? Can the authors provide experimental evidence supporting this design?

What motivates the random masking of reasoning steps in the self-distillation mode?

5 comments
AuthorsOpenReview ↗

Response to Reviewer hPZ1

We thank Reviewer hPZ1 for the feedback. Below, we address each concern using our theoretical analysis, controlled ablations, and experimental comparisons.


1. Adaptive Routing Motivation, Saturation/Noise Concerns, and Source of Gains (Weakness 1 & Question 1)

  • Intuition: Strong Teacher Guidance First, Complementary Signals as the Student Improves: At the beginning of training, the student is relatively weak and benefits from guidance from a stronger external teacher. We therefore assign a high initial probability to OFF, which distills the external teacher's distributions along canonical reference responses (ρOFF(1)≫ρON(1),ρSELF(1)\rho_{\text{OFF}}^{(1)} \gg \rho_{\text{ON}}^{(1)}, \rho_{\text{SELF}}^{(1)}, Eq. 5). This provides a foundation before placing greater weight on supervision that depends on the student's evolving capabilities. As the student becomes more capable, the additional benefit of OFF-dominated training may diminish, motivating greater use of ON and SELF as complementary signals. ON supplies external-teacher feedback on the student's own generated trajectories, while SELF adds privileged-context supervision from a student-derived teacher on canonical responses. The intuition is to adapt the balance of supervision as learning progresses, while retaining external-teacher guidance through OFF and ON.

  • Connecting This Intuition to the Progress Signal and Theoretical Analysis: Under OFF-dominated training, shared student parameters allow ON and SELF to benefit indirectly from OFF updates while receiving relatively few direct updates (Section 3.3, Lines 284–286). If this indirect transfer slows, allocating more direct updates to these modes may provide useful complementary supervision. Equation (8) uses Im(t)<δI_m^{(t)} < \delta as feedback for increasing their allocation. This is a motivation for the rule, rather than a guarantee that slow progress uniquely identifies a transfer plateau or rules out saturation and noise. Theorem 1 (Eq. 13, proved in Appendix A, Eq. 14–25) provides the following bound:

    DTm≤D0mexp⁡(−∑t=1T1{m(t)=m}Im(t)+ϵa∑t=1T1{m(t)≠m}),ϵ=ηG2.D_T^m \le D_0^m \exp \left(-\sum_{t=1}^T \mathbf{1}\{m(t)=m\}I_m^{(t)} + \frac{\epsilon}{a}\sum_{t=1}^T \mathbf{1}\{m(t)\neq m\}\right), \quad \epsilon = \eta G^2.

    When Im(t)<δI_m^{(t)} < \delta, increasing ρm\rho_m (Eq. 8) shifts training budget from potentially interfering cross-mode updates (m(t)≠mm(t) \neq m) toward direct contraction (m(t)=mm(t) = m, Lines 333–341).

  • Built-In Safeguards Against Noise and Late-Stage Saturation:

    • Denoised routing evaluation: Dm(t)D_m^{(t)} is evaluated on a fixed held-out set Dh\mathcal{D}_h using deterministic greedy decoding (ON) and fixed step masks on canonical responses (SELF) prior to refreshing pT~p_{\tilde{T}} (Algorithm 2, Lines 11–15), eliminating mini-batch sampling variance.
    • Denoised training targets: SELF always scores canonical responses r=yr=y rather than noisy student rollouts (Lines 225–229), and ON optimizes only LKDON\mathcal{L}_{\text{KD}}^{\text{ON}} against the frozen external teacher pTp_T without LCE\mathcal{L}_{\text{CE}} (Eq. 11), avoiding direct hard-label imitation of sampled tokens.
    • Bounded late-stage caps: Upper bounds ρSELFmax⁡=ρONmax⁡=0.25\rho_{\text{SELF}}^{\max} = \rho_{\text{ON}}^{\max} = 0.25 (Eq. 8–9) stop probability growth once reached, preserving a permanent ≥50%\ge 50\% OFF anchor throughout late training.
  • Empirical Evidence for Multi-Mode Synergy and Adaptive Routing (Tables 3 & 4): Combining privileged SELF with external-teacher supervision under our adaptive routing boosts 5-task accuracy over standalone modes (OFF: 40.28%, ON: 41.12%, SELF: 41.54%) to 54.12% (OFF + SELF), 55.58% (ON + SELF), and 54.76% (RIVAD) on Qwen2.5, and from 13.5%–35.1% to 40.0% (RIVAD) on Gemma2. Within the 3-mode setup, RIVAD outperforms Uniform (1/3, 1/3, 1/3) on both Qwen2.5 (54.76% vs. 54.38%, including +1.8% on MBPP and +0.8% on STEM) and Gemma2 (40.0% vs. 39.5%) while requiring fewer expensive on-policy ON rollouts.

AuthorsOpenReview ↗

Why Random Step Masking Outperforms Full Context in `SELF` and Comparisons with OPSD, RFT, GRPO

2. Why Random Step Masking Outperforms Full Context in SELF (Weakness 2 & Question 2)

  • Preventing Reference-Copying Shortcuts: In SELF, both student pθ(⋅∣x,y<i)p_\theta(\cdot \mid x, y_{<i}) and self-teacher pT~(⋅∣x~,y<i)p_{\tilde{T}}(\cdot \mid \tilde{x}, y_{<i}) score the same canonical trajectory r=y=(z1,…,zL)r=y=(z_1, \dots, z_L) under teacher forcing (Table 1, Eq. 1). Conditioning a self-teacher on the complete untruncated solution (No truncation) exposes the entire reference solution and final answer zLz_L 100% of the time, inducing reference-copying shortcuts and overly sharp distributions that degrade transfer to a question-only student [1, 2]. Sampling d∼U(0,0.5)d \sim \mathcal{U}(0, 0.5) and randomly masking segments hj∼Bernoulli(d)h_j \sim \text{Bernoulli}(d) (with at least one hidden and one visible step, Eq. 3–4) withholds parts of the derivation and forces the self-teacher to model inter-step conditional relationships.
  • Accuracy and Efficiency Gains (Table 8): Random (Ours) achieves the highest average accuracy (54.76%), outperforming both No truncation (54.42%) and static truncation (Fixed 25%: 54.32%, Fixed 50%: 54.38%), with a substantial +2.4 point improvement on MBPP (49.2% vs. 46.8%) alongside gains on MATH (53.0% vs. 52.7%) and STEM (52.6% vs. 52.1%), while shortening the self-teacher's prompt length and reducing GPU memory overhead.

3. Comparisons with OPSD, RFT, and GRPO (Weakness 3)

  • Relationship to OPSD: Standard OPSD [3] supervises student-generated rollouts using full reference solutions without an external teacher. This reliance on self-supervision can be particularly problematic for compact reasoning models: on Qwen3-1.7B with thinking enabled, recent work reports an average accuracy drop from 37.2% to 30.8% under long-budget evaluation [2]. The original OPSD study also shows that omitting per-token pointwise clipping can cause performance collapse on this model [3], while Purified OPSD identifies reference-induced shortcuts that undermine transferable reasoning in long-CoT models [1]. These findings suggest that access to correct solutions alone does not ensure that a compact model can effectively supervise itself. Our SELF-only ablation (41.54% on Qwen2.5-1.5B and 13.5% on Gemma2-2B) and our full-reference No truncation ablation (54.42% vs. 54.76% for Random masking; 46.8% vs. 49.2% on MBPP, Table 8) directly demonstrate why privileged self-distillation is substantially more effective when (i) using random step masking on canonical trajectories and (ii) combined with external-teacher supervision (OFF/ON) in RIVAD (54.76% / 40.0%).

  • Why White-Box KD Is Orthogonal to RFT and GRPO: Our work studies white-box token-level knowledge distillation from a larger teacher on general multi-task reasoning data (UltraInteract 78K), matching the exact setting of MiniLLM, GKD, DistiLLM, Speculative KD, TSD-KD, CSD, and AMiD. RFT and GRPO are outcome-verifier RL/filtering methods that require rule-based scalar verifiers and multi-rollout group sampling with non-trivial initial pass rates. For compact 1.5B–2B students with low base accuracy on hard tasks (16.9% and 16.1% on MATH) and general reasoning datasets without rule verifiers, white-box KD provides dense token-level supervision at single-trajectory cost (70.3 min/epoch, Table 7) as an orthogonal training stage prior to RL.

AuthorsOpenReview ↗

Standardized ICLR 2026 Benchmark Protocol and Non-Saturated Tasks (Weakness 4)

4. Standardized ICLR 2026 Benchmark Protocol and Non-Saturated Tasks (Weakness 4)

Our experimental setup (Qwen2.5-14B →\to 1.5B and Gemma2-9B →\to 2B on UltraInteract 78K across 8 benchmarks) strictly follows the benchmark protocol established by the latest ICLR 2026 white-box KD papers: TSD-KD [4], CSD [5], and AMiD [6] (Lines 346–388). Beyond GSM8K and MBPP, our 8-benchmark suite includes MATH, GSM-Plus, MMLU-Pro-Math, BBH-CoT, MMLU-STEM, and SciQ, which remain far from saturated at the 1.5B–2B scale:

Benchmark (Student Model)Base StudentSFT (Seq-KD)Best Prior ICLR 2026 BaselineRIVAD (Ours)Gain Over Best Baseline
MATH (Qwen2.5-1.5B, Table 2)16.9%16.5%29.3% (CSD)53.0%+23.7 pts (+80.9% rel.)
GSM-Plus (Qwen2.5-1.5B, Table 2)38.8%38.2%42.1% (AMiD)50.2%+8.1 pts (+19.2% rel.)
8-Task Avg (Qwen2.5-1.5B, Table 2)44.6%44.7%49.5% (CSD)55.5%+6.0 pts (+12.1% rel.)
MATH (Gemma2-2B, Table 6)16.1%17.7%20.5% (AMiD)24.1%+3.6 pts (+17.6% rel.)
5-Task Avg (Gemma2-2B, Table 6)34.1%34.0%36.4% (AMiD)40.0%+3.6 pts (+9.9% rel.)

References

[1] Shen et al., Purified OPSD: On-policy self-distillation without losing how to think, arXiv:2607.02234, 2026.
[2] Kaur et al., Rethinking on-policy self-distillation for thinking models, arXiv:2607.05184, 2026.
[3] Zhao et al., Self-distilled reasoner: On-policy self-distillation for large language models, arXiv:2601.18734, 2026.
[4] Kim & Baek, Explain in your own words: Improving reasoning via token-selective dual knowledge distillation, ICLR 2026.
[5] Kim et al., Distillation of large language models via concrete score matching, ICLR 2026.
[6] Shin et al., AMiD: Knowledge distillation for LLMs with α\alpha-mixture assistant distribution, ICLR 2026.

Reviewer_TaWMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a white-box reasoning distillation framework that allocates training updates among off-policy distillation from a fixed teacher (OFF), on-policy distillation on newly generated student trajectories (ON), and self-distillation with privileged context (SELF). SELF uses a lagged student snapshot as its teacher and supplies it with reference reasoning from which some steps have been randomly removed, while the student receives only the original question. A scheduler progressively increases the sampling probabilities of ON and SELF based on relative changes in their held-out distillation losses. Experiments use Qwen2.5 and Gemma2 teacher–student pairs. Table 2 reports an average accuracy of 55.5% across eight tasks for the Qwen student, compared with 49.5% for the strongest listed baseline, with particularly large gains on mathematical tasks.

Critical strengths
  1. The framework addresses meaningful differences among supervision sources in distillation. OFF provides stable reference supervision, ON covers prefixes actually visited by the student, and SELF uses additional context to strengthen supervision derived from the student itself. Implementing all three through output-distribution distillation is relatively straightforward and does not require matching teacher and student hidden dimensions.

  2. The reported mathematical reasoning gains are substantial, and the evaluation includes a second model family. In Table 2, the Qwen student reaches 53.0% on MATH, compared with 29.3% for CSD, with clear improvements on GSM8K and GSM-Plus as well. The Gemma results in Table 6 provide initial evidence that the method is not specific to one model family. These gains warrant careful examination rather than dismissing the framework as a simple combination of supervision modes.

Critical weaknesses
  1. The experiments do not sufficiently establish adaptive multi-mode allocation as the central contribution. In the five-task comparison in Table 3, RIVAD scores 54.76, while the simpler ON + SELF configuration reaches 55.58. Uniform mixing of all three modes scores 54.38, only 0.38 percentage points below the adaptive version. In Appendix Table 8, the random-context strategy also improves over full context by only 0.34 points. These central ablations provide neither uncertainty estimates for the differences nor results across repeated training runs, and there is no comparison with a fixed or predetermined schedule matched for total mode usage. The large gains therefore more clearly support combining SELF with external-teacher supervision than they establish the necessity of adaptive scheduling or information-variant views. Equation (8) only allows the ON and SELF probabilities to increase monotonically; it does not directly estimate the causal training benefit of a mode. The cumulative progress bound in Theorem 1 also does not establish an advantage of this rule over fixed mixing.

  2. There are substantive inconsistencies between the method definition and the reported experiments that affect reproducibility and interpretation. Equation (11) states that ON uses only KD, whereas lines 8–9 of Appendix Algorithm 2 add CE in every mode. Section 3.2 constructs SELF context by randomly hiding steps in the original reference reasoning, but Appendix C describes separately generating supporting facts, hints, and formulas with a teacher. These constructions change both the information supplied and the preparation cost, and the paper does not identify which one produced the main results. Table 4 also reports 45.9 for Gemma ON/STEM, but the arithmetic mean of the five displayed scores is 39.04 rather than the reported 35.1. If the STEM entry should instead be 25.9, it would match the MiniLLM entry in Table 6, but I cannot determine the intended correction. This discrepancy even affects the conclusion that adding SELF consistently improves ON. In addition, the appendices do not complete key settings such as the initial routing probabilities and their caps, the routing interval, and the maximum step-drop rate. The standard errors in Table 2 are not accompanied by a definition of the unit of replication or the number of runs. These issues require consistent configurations and result records rather than being treated as inconsequential presentation errors.

Questions for authors
  1. Please clarify the training objective and SELF context actually used for Tables 2–8. Does ON include CE? Is the privileged context a subset of reference reasoning steps or separately generated hints? Please also explain the inconsistency between ON/STEM and its reported average in Table 4 and provide the complete configurations corresponding to the results.

  2. Under matched training-token, teacher-query, and compute budgets, does RIVAD consistently outperform ON + SELF, uniform three-mode mixing, and a predetermined schedule matched for overall mode usage? Can existing repeated runs provide means and intervals for the key differences? These results would directly affect my assessment of adaptive allocation.

  3. When comparing held-out losses across checkpoints, how are SELF teacher refreshes and changes in ON-generated trajectories handled? Could you provide the actual mode probabilities, losses, and downstream performance over training, and clarify whether the standard errors in Table 2 measure question-level uncertainty or variation across independent training runs?

2 comments
AuthorsOpenReview ↗

Response to Reviewer TaWM

We thank Reviewer TaWM for the thorough review and for recognizing the conceptual clarity of our framework and its substantial reasoning gains across both model families. Below, we clarify the exact implementation, configurations, and efficiency trade-offs.


1. Training Objective, Privileged Context, Table 4 Correction, and Configurations (Weakness 2 & Question 1)

  • Mode-Specific Objective (Eq. 11 vs. Algorithm 2): All reported experiments follow Equation (11): OFF and SELF optimize (1−β)LCE+βLKD(1-\beta)\mathcal{L}_{\text{CE}} + \beta\mathcal{L}_{\text{KD}} on the canonical response yy, whereas ON optimizes only LKDON\mathcal{L}_{\text{KD}}^{\text{ON}} without LCE\mathcal{L}_{\text{CE}}. Because ON trajectories y^∼pθ\hat{y} \sim p_\theta are student-generated, omitting LCE\mathcal{L}_{\text{CE}} prevents reinforcing student rollout errors as ground truth while still transferring token-level teacher distributions pTp_T. Line 9 of Appendix Algorithm 2 omitted the condition m∈{OFF,SELF}m \in \{\text{OFF}, \text{SELF}\} from Eq. (11); we will update Line 9 in Algorithm 2 to match Eq. (11) exactly.

  • Canonical Step-Masking Context (Section 3.2 vs. Appendix C): All experimental results in Tables 2–8 use the canonical step-masking mechanism in Section 3.2 (Lines 209–224): the canonical response yy is split at double newlines (\n\n) into ordered steps z1,…,zLz_1, \dots, z_L, and a subset of steps is randomly masked via Eq. (3)–(4) without generating external hints. The short text in Appendix C (Lines 801–809) and "geometry/pooling" entries in Appendix B were leftover notes from an early draft and were not used in any reported runs. We will remove those obsolete notes in the revision.

  • Table 4 Typo Correction and Controlled In-Pipeline Comparisons: Thank you for catching the transcription typo in Table 4: for Gemma2 ON (MiniLLM), the STEM score is 25.9 (as correctly reported in Table 6, Line 818), rather than 45.9, which yields the reported 5-task average of 35.1 (35.04). Likewise, for Gemma2 RIVAD, the five task scores (54.8, 37.0, 24.1, 38.0, 46.2 in Table 6; printed as 46.1 in Table 4) average to 40.0 (matching Table 6 and Line 431), rather than 40.1. With the correct 25.9 STEM score for ON, adding SELF (ON + SELF: 40.1 on STEM, 37.0 Avg.) consistently improves over ON (25.9 on STEM, 35.1 Avg.). Moreover, alongside the TSD-KD reference rows, we directly executed SELF-only (41.54% Qwen Table 3 / 13.5% Gemma Table 4), CSD (49.5% Qwen Table 2 / 35.6% Gemma Table 6), AMiD (49.4% / 36.4%), OFF + SELF (54.12% / 39.9%), ON + SELF (55.58% / 37.0%), Uniform (54.38% / 39.5%), and RIVAD (54.76% [55.5% 8-task] / 40.0%) within our identical training and evaluation pipeline, confirming that the large multi-mode gains come directly from our framework.

  • Complete Routing and Masking Configurations (Appendix B): Note that ON + SELF and OFF + SELF in Tables 3–4 also use our adaptive routing rule (Eq. 8) restricted to two active modes:

Hyperparameter3-Mode RIVAD2-Mode ON + SELF2-Mode OFF + SELF
Initial ρ(1)=(ρOFF(1),ρSELF(1),ρON(1))\rho^{(1)} = (\rho_{\text{OFF}}^{(1)}, \rho_{\text{SELF}}^{(1)}, \rho_{\text{ON}}^{(1)})(0.85,0.10,0.05)(0.85, 0.10, 0.05)(0.00,0.95,0.05)(0.00, 0.95, 0.05)(0.90,0.10,0.00)(0.90, 0.10, 0.00)
Upper caps (ρSELFmax⁡,ρONmax⁡)(\rho_{\text{SELF}}^{\max}, \rho_{\text{ON}}^{\max})(0.25,0.25)(0.25, 0.25)(−,0.25)(-, 0.25)(0.25,−)(0.25, -)
Step increments (ηSELF,ηON)(\eta_{\text{SELF}}, \eta_{\text{ON}})(0.05,0.025)(0.05, 0.025)(−,0.025)(-, 0.025)(0.05,−)(0.05, -)
Residual probability assignmentρOFF=1−ρSELF−ρON\rho_{\text{OFF}} = 1 - \rho_{\text{SELF}} - \rho_{\text{ON}}ρSELF=1−ρON\rho_{\text{SELF}} = 1 - \rho_{\text{ON}}ρOFF=1−ρSELF\rho_{\text{OFF}} = 1 - \rho_{\text{SELF}}
Routing interval KK / Threshold δ\delta100100 updates / 0.050.05100100 updates / 0.050.05100100 updates / 0.050.05
Max step-drop dmax⁡d_{\max} / KL β\beta / Temp τ\tau / Top-kk0.50.5 / 0.50.5 / 2.02.0 / 50500.50.5 / 0.50.5 / 2.02.0 / 50500.50.5 / 0.50.5 / 2.02.0 / 5050
AuthorsOpenReview ↗

Response to Reviewer TaWM

2. Compute-, Token-, and Rollout-Budget Advantage of Adaptive Allocation (Weakness 1 & Question 2)

  • Why RIVAD Is Strictly More Compute-Efficient Than Uniform (1/3, 1/3, 1/3): All configurations run for the exact same number of optimizer updates and batch size (128 examples/update), with one teacher forward pass and one student forward-backward pass per update. However, ON is by far the most expensive mode because every update additionally requires autoregressive student rollout generation (y^∼pθ\hat{y} \sim p_\theta, up to 512 new tokens), whereas OFF and SELF use single-pass teacher forcing on canonical responses yy.
    1. Uniform (1/3, 1/3, 1/3) executes expensive ON rollouts on 33.3% of all updates throughout training. By contrast, RIVAD initializes ρON(1)=0.05\rho_{\text{ON}}^{(1)} = 0.05 and caps ρONmax⁡=0.25\rho_{\text{ON}}^{\max} = 0.25, executing 25%–85% fewer on-policy rollout generations while outperforming Uniform on both Qwen2.5 (54.76% vs. 54.38%, including +1.8% on MBPP, +0.8% on STEM, +0.6% on MATH) and Gemma2 (40.0% vs. 39.5%, including +1.3% on MATH and +1.4% on STEM).
    2. Cross-Architecture Robustness and Token Efficiency (RIVAD vs. ON + SELF): Both ON + SELF and RIVAD use our adaptive routing mechanism with the same ON rollout bounds (ρON∈[0.05,0.25]\rho_{\text{ON}} \in [0.05, 0.25]). However, ON + SELF allocates the remaining 75%–95% of updates entirely to SELF (which prepends privileged context steps to the teacher prompt, up to 1,536 prompt tokens and 2,560 total tokens vs. 512 prompt and 1,024 total tokens for OFF, Table 5). While ON + SELF specializes well on Qwen2.5 math tasks (where standalone SELF-only already reaches 28.8% on MATH vs. 16.9% base), RIVAD processes shorter OFF prompts on ≥50%\ge 50\% of updates and outperforms ON + SELF on Qwen2.5 MBPP (49.2% vs. 48.2%) and STEM (52.6% vs. 52.2%), as well as by +3.0 points on Gemma2 (40.0% vs. 37.0%), where early canonical OFF grounding is essential because the initial 2B self-teacher is weak (SELF-only: 13.5%).
    3. Random Step Masking Efficiency (Table 8): Random masking outperforms No truncation (54.76% vs. 54.42%, including +2.4% on MBPP: 49.2% vs. 46.8%) while shortening the self-teacher's prompt context by removing up to 50% of reference steps on average, contributing to RIVAD's low runtime (70.3 min/epoch) and reduced peak GPU memory (114.7 GB/GPU vs. 127.9 GB for CSD and 125.5 GB for AMiD, Table 7).

3. Checkpoint Loss Comparability, Routing Progression, and Standard Errors (Question 3)

  • Pre-Synchronization Evaluation Protocol: At each routing checkpoint (s mod K=0s \bmod K = 0), DSELFD_{\text{SELF}} and DOND_{\text{ON}} are evaluated on the fixed held-out set Dh\mathcal{D}_h using fixed per-example masks (SELF) and deterministic greedy decoding (ON) before synchronizing pT~←pθp_{\tilde{T}} \leftarrow p_\theta (Algorithm 2, Lines 11–15). Because pT~p_{\tilde{T}} remains frozen throughout the K=100K=100 updates between checkpoints, ISELF(t)I_{\text{SELF}}^{(t)} measures improvement against a stationary self-teacher target within that interval, normalized relative to each mode's own prior checkpoint loss (Eq. 7).
  • Routing Schedule Progression and Downstream Impact: Starting from (ρOFF(1),ρSELF(1),ρON(1))=(0.85,0.10,0.05)(\rho_{\text{OFF}}^{(1)}, \rho_{\text{SELF}}^{(1)}, \rho_{\text{ON}}^{(1)}) = (0.85, 0.10, 0.05), early OFF-dominated updates reduce both DSELFD_{\text{SELF}} and DOND_{\text{ON}} via positive cross-mode transfer (Im(t)≥δ=0.05I_m^{(t)} \ge \delta = 0.05). Whenever indirect transfer slows below δ=0.05\delta = 0.05 at a K=100K=100-step checkpoint, ρSELF\rho_{\text{SELF}} increments by ηSELF=0.05\eta_{\text{SELF}}=0.05 (reaching its 0.250.25 cap after 3 triggered checkpoints) and ρON\rho_{\text{ON}} increments by ηON=0.025\eta_{\text{ON}}=0.025 (reaching its 0.250.25 cap after 8 triggered checkpoints), transitioning to the bounded regime (ρOFF,ρSELF,ρON)=(0.50,0.25,0.25)(\rho_{\text{OFF}}, \rho_{\text{SELF}}, \rho_{\text{ON}}) = (0.50, 0.25, 0.25). Across supervision stages, Tables 3 and 4 show downstream accuracy progressing from single-mode (OFF: 40.28% / 34.6%, ON: 41.12% / 35.1%, SELF: 41.54% / 13.5%) to two-mode (OFF + SELF: 54.12% / 39.9%, ON + SELF: 55.58% / 37.0%) and three-mode (Uniform: 54.38% / 39.5%, RIVAD: 54.76% / 40.0%).
  • Definition of Standard Errors: The ±\pm values in Tables 2 and 6 denote question-level (sample-level) standard errors across the test instances of each benchmark under our default training seed (seed 10, Table 5). We will state this explicitly in the captions of Tables 2 and 6.
Reviewer_2m9oOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RIVAD to improve the reasoning ability of smaller language models through knowledge distillation. It combines three training modes: off-policy, on-policy, and self-distillation. In the self-distillation mode, a copy of the student receives partial reasoning steps as additional context. The method also adjusts how often each mode is used based on held-out KL loss. Experiments on Qwen2.5 and Gemma2 show improvements over several distillation baselines, especially on mathematical reasoning tasks.

Critical strengths
  • Privileged-context self-distillation: The proposed self-distillation method uses partial reference reasoning to provide additional supervision during training. It allows the student to benefit from privileged context without requiring it at inference time.

  • Strong empirical results: RIVAD achieves higher average accuracy than existing baselines on both Qwen2.5 and Gemma2. The improvement on MATH is particularly substantial (53.0% vs. 29.3% for the strongest baseline), suggesting its potential for improving multi-step reasoning.

Critical weaknesses
  • Limited evidence for the contributions of each component: The ablation results do not clearly show how much each component contributes. In Table 3, adaptive routing only slightly improves over uniform mixing (54.76 vs. 54.38), while ON + SELF performs better (55.58). Random truncation also shows only a small gain over no truncation (54.76 vs. 54.42, Table 8). These results make the benefits of the proposed components difficult to isolate.

  • Reliability of the adaptive routing signal: The KL losses across checkpoints may not be directly comparable. ON uses newly generated student responses at each checkpoint, and the self-teacher also changes during training. Therefore, a lower KL loss does not necessarily mean the student has improved.

Questions for authors
  • How are canonical reasoning steps segmented? The paper does not provide a comprehensive specification for how to identify the boundaries of semantic steps in each type of reasoning text.
  • Is it possible for the privileged context to contain the final answer? Does the Self-teacher provide better supervision because it understands the relationships between some of the reasoning steps, or simply because it sees the final answer in advance?
  • How do the authors ensure that KL losses across routing checkpoints are comparable when ON trajectories and SELF teacher distributions change?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 2m9o

We thank Reviewer 2m9o for the positive assessment and for highlighting both our privileged-context self-distillation design and the substantial reasoning improvements (such as reaching 53.0% on MATH). Below, we address each concern and question directly.


1. Isolating Component Contributions and Rollout Efficiency (Weakness 1)

  • Synergy Between External-Teacher Supervision and Privileged SELF: Within our unified training and evaluation pipeline, standalone SELF-only reaches 41.54% on Qwen2.5 (Table 3) and 13.5% on Gemma2 (Table 4), while strong external-teacher baselines evaluated in the exact same pipeline achieve 49.5% (CSD) and 49.4% (AMiD) across the Qwen2.5 suite (29.3% and 28.3% on MATH, Table 2) and 35.6% / 36.4% on Gemma2 (Table 6). Combining privileged SELF with external-teacher supervision under our framework elevates 5-task accuracy to 54.12%–55.58% on Qwen2.5 (55.5% across all 8 tasks, 53.0% on MATH) and 40.0% on Gemma2.

  • Two-Mode vs. Three-Mode Adaptive Routing and Rollout Efficiency (Tables 3 & 4): In Tables 3 and 4, OFF + SELF and ON + SELF are two-mode instances of RIVAD's adaptive routing (updating mode probabilities via Eq. 8 with the third mode set to 0). Among the three modes, ON is by far the most computationally expensive because each step requires fresh autoregressive student rollout generation (y^∼pθ\hat{y} \sim p_\theta, up to 512 tokens) followed by external-teacher scoring:

Multi-Mode ConfigurationRouting RuleON Rollout ShareQwen2.5 AvgQwen2.5 MBPP / STEMGemma2 AvgGemma2 MATH / STEM
OFF + SELF (2-Mode)Adaptive (Eq. 8)0%54.1247.6 / 52.039.923.7 / 44.7
ON + SELF (2-Mode)Adaptive (Eq. 8)5% -> 25%55.5848.2 / 52.237.021.8 / 40.1
Uniform (1/3, 1/3, 1/3)Static Equal33.3% (Fixed)54.3847.4 / 51.839.522.8 / 44.8
RIVAD (3-Mode, Ours)Adaptive (Eq. 8)5% -> 25%54.7649.2 / 52.640.024.1 / 46.2
  1. Cross-architecture robustness (RIVAD vs. ON + SELF): On Qwen2.5-1.5B, standalone SELF-only already improves substantially over the base student on MATH (28.8% vs. 16.9%), enabling 2-mode ON + SELF to specialize effectively on math tasks while 3-mode RIVAD achieves the best accuracy on MBPP (49.2% vs. 48.2%) and STEM (52.6% vs. 52.2%). On Gemma2-2B, however, the initial student is too weak to self-teach without canonical grounding (SELF-only drops to 13.5% vs. 34.1% base). Consequently, omitting OFF causes ON + SELF to drop to 37.0% on Gemma2, whereas 3-mode RIVAD anchors early training in OFF (ρOFF(1)=0.85\rho_{\text{OFF}}^{(1)}=0.85) and achieves the best overall performance (40.0%).
  2. Compute efficiency over Uniform: While Uniform executes expensive ON rollouts on 33.3% of all updates throughout training, RIVAD starts at ρON(1)=0.05\rho_{\text{ON}}^{(1)}=0.05 and caps ρONmax⁡=0.25\rho_{\text{ON}}^{\max}=0.25—outperforming Uniform on both Qwen2.5 (54.76% vs. 54.38%, +1.8% on MBPP, +0.8% on STEM) and Gemma2 (40.0% vs. 39.5%, +1.3% on MATH, +1.4% on STEM) while saving 25%–85% of on-policy rollout generations.
AuthorsOpenReview ↗

Response to Reviewer 2m9o

2. Canonical Reasoning Segmentation (Question 1)

  • Explicit Step Boundaries: In our implementation, canonical responses yy are segmented at double-newline boundaries (\n\n), removing empty segments and preserving their original order z1,…,zLz_1, \dots, z_L. In multi-step math, science, and code trajectories from UltraInteract, logical derivation blocks and code blocks are naturally separated by paragraph breaks. Using \n\n boundaries provides deterministic step units for order-preserving masking (Eq. 3–4) while keeping the supervised target sequence r=yr=y completely unchanged (Lines 209–224). We will state this \n\n segmentation rule explicitly in Section 3.2.

3. Does SELF Succeed via Inter-Step Relations or Final-Answer Exposure? (Question 2)

  • Direct Evidence from No Truncation vs. Random Masking (Table 8): In our segmented responses z1,…,zLz_1, \dots, z_L, the final answer resides in the concluding segment zLz_L. Under No truncation (54.42% Avg., Table 8), the self-teacher receives the complete canonical solution—meaning the final answer zLz_L is visible 100% of the time. In contrast, under Random (Ours) (54.76% Avg.), every segment zjz_j (including the final-answer segment zLz_L) is independently masked via hj∼Bernoulli(d)h_j \sim \text{Bernoulli}(d) with d∼U(0,0.5)d \sim \mathcal{U}(0, 0.5) and at least one segment guaranteed hidden (Eq. 3–4).
  • Why Partial Masking Outperforms Full Final-Answer Visibility: If the benefit of SELF merely came from seeing the final answer in advance, No truncation (100% final-answer visibility) would strictly outperform Random masking (where zLz_L is frequently masked out). Instead, randomly masking steps outperforms both No truncation (+2.4 points on MBPP [49.2% vs. 46.8%], +0.3 on MATH, +0.5 on STEM) and static truncation (Fixed 25%: 54.32%, Fixed 50%: 54.38%), while shortening the self-teacher's prompt context. This confirms that withholding parts of the reference mitigates answer-copying shortcuts and promotes genuine step-to-step conditional modeling.

4. Comparability and Stability of Held-Out Routing Losses (Weakness 2 & Question 3)

  • Pre-Synchronization Evaluation Protocol: At each routing checkpoint (s mod K=0s \bmod K = 0), held-out losses DSELF(t)D_{\text{SELF}}^{(t)} and DON(t)D_{\text{ON}}^{(t)} are evaluated on the fixed disjoint set Dh\mathcal{D}_h using deterministic greedy decoding (ON) and fixed per-example step masks on canonical responses (SELF), executed prior to refreshing the self-teacher pT~←pθp_{\tilde{T}} \leftarrow p_\theta (Algorithm 2, Lines 11–15).
  • Within-Interval Anchor and Relative Normalization: Because pT~p_{\tilde{T}} is synchronized to pθp_\theta at the start of each K=100K=100-step interval and kept frozen throughout that interval, DSELF(t)D_{\text{SELF}}^{(t)} directly measures how effectively the student closes the predictive gap to its privileged-context counterpart over those 100 steps. Furthermore, Eq. (7) computes the dimensionless relative improvement Im(t)=(Dm(t−1)−Dm(t))/(Dm(t−1)+ϵa)I_m^{(t)} = (D_m^{(t-1)} - D_m^{(t)}) / (D_m^{(t-1)} + \epsilon_a) within each mode independently, rather than comparing raw loss magnitudes across modes.

Thank you for the detailed response. The clarifications on reasoning step segmentation and adaptive routing are helpful. I will keep my original rating of 3.

AuthorsOpenReview ↗

Response to Reviewer 2m9o

Thank you for taking the time to read our rebuttal. We appreciate your thoughtful reassessment and feedback.

Reviewer_2z28OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes to a new distillation formwork that combines offline, online, and self distillation and adaptively select these objectives  during training. For self distillation, the author also proposed a new variant that partially masks reasoning steps to prevent information leakage from gold trajectories.

Critical strengths
  1. The work combines popular distillation techniques and conducted a system-level study on combining them and their relative contribution. This challenges the conventional perception that these methods are mutually exclusive, which could inspire future works on multi-objective distillation
  2. Empirical results show that proposed method achieves stronger performance than single-objective baselines, highlighting the necessity of multi-objective training
  3. The author conducted studies on two model families to show the generalizability of the findings
Critical weaknesses
  1. The author claim its proposed variants of OPSD as a key contribution, so I would expect more ablation and analysis on the effectiveness of “SELF” setup compared with vanilla OPSD. However this comparison is missing and existing results does not establish “SELF” is a better approach than vanilla OPSD in single-objective setting, or as a component of proposed RIVAD framework
  2. While the author explored two model families, the baselines Qwen2.5 and Gemma are quite outdated. More over, ablation studies show conflicting results with no clear winner. Specifically, on Qwen2.5 results, ON + SELF setup outperforms 4/6 benchmarks tested than RIVAD, on Gemma experiments, OFF + SELF seemed to be better than  ON + SELF setup. These differences are not analyzed or carefully discussed. The author suggest RIVAD “provides a more robust distillation strategy than relying on a fixed or single mode.” , however for any specific applications it seems alternative methods may work better. Moreover, this paper provide no insight on when adaptive combination is useful and when they are not.
Questions for authors

See weakness section

Additional questions 1.Could you explore more model families like Qwen3 or Gemma 3 and further discuss the different behavior across model families. Specifically, discussions on when should RIVAD is more preferable and when alternatives is more preferable will strength the contribution of this work 2. Could you provide analysis on if SELF is truly better than conventional OPSD?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 2z28

We thank Reviewer 2z28 for recognizing the value of our system-level multi-objective study and strong empirical gains across two model families. Below, we address the comparison with OPSD, the behavior of pairwise vs. three-mode routing across model families, and our benchmark protocol.


1. Conceptual and Empirical Distinction Between SELF and Vanilla OPSD (Weakness 1 & Question 2)

  • Decoupling Privileged Context from Rollout Noise and Full-Reference Shortcuts: Conventional OPSD [1] uses a privileged self-teacher conditioned on the full reference solution to supervise student-generated on-policy rollouts (r=y^∼pθr = \hat{y} \sim p_\theta). Recent studies [2, 3] show that on compact models, this design faces two key limitations: (i) supervising noisy student rollouts with a self-teacher can reinforce student errors when the base model is weak, and (ii) conditioning on the complete untruncated solution can induce reference-copying shortcuts. In contrast, our SELF mode fixes the scored trajectory to the canonical response r=yr=y and conditions the self-teacher on randomly masked, order-preserving subsets of reasoning steps (Table 1, Eq. 1–4), requiring only single-pass teacher forcing instead of expensive on-policy rollout generation.

  • Empirical Evidence from Our Ablations (Tables 3, 4, and 8):

    1. Why standalone self-distillation is insufficient on compact models: Standalone SELF-only reaches 41.54% on Qwen2.5-1.5B and drops to 13.5% on Gemma2-2B, where the 2B student is initially too weak to self-teach without external grounding (scoring 5.23% on GSM8K and 6.38% on MATH, compared with 53.7% and 16.1% for the base student). Pairing SELF with external-teacher supervision in our framework lifts performance to 54.12%–55.58% on Qwen2.5 and 39.9%–40.0% on Gemma2.
    2. Why partial step masking outperforms full-solution conditioning: In Table 8, conditioning the self-teacher on the complete reference solution (No truncation, which mirrors OPSD's full-solution context) achieves 54.42% (46.8% on MBPP), whereas our random step masking (Random: 54.76%) improves MBPP by +2.4 points (49.2% vs. 46.8%), MATH by +0.3 (53.0% vs. 52.7%), and STEM by +0.5 (52.6% vs. 52.1%) while reducing context length and GPU memory.

2. When Pairwise Subsets vs. Three-Mode RIVAD Are Preferable (Weakness 2 & Question 1)

In Tables 3 and 4, ON + SELF and OFF + SELF are two-mode instances of RIVAD's adaptive routing (updating probabilities via Eq. 8 with the third mode set to 0). Comparing the two model families reveals a clear mechanistic principle governing when a specialized 2-mode subset vs. 3-mode RIVAD excels:

Model Family (5-Task Suite)Base StudentStandalone SELF-onlyOFF + SELF (No ON)ON + SELF (No OFF)Uniform (1/3,1/3,1/3)RIVAD (Adaptive 3-Mode)
Qwen2.5-1.5B (Table 3)40.14 (MATH: 16.9)41.54 (MATH: 28.8)54.1255.58 (MBPP: 48.2)54.38 (MBPP: 47.4)54.76 (MBPP: 49.2, STEM: 52.6)
Gemma2-2B (Table 4)34.10 (MATH: 16.1)13.50 (MATH: 6.38)39.9037.00 (MATH: 21.8)39.50 (MATH: 22.8)40.00 (MATH: 24.1, STEM: 46.2)
  • Regime 1 — Strong Initial Self-Teacher (Qwen2.5-1.5B): On Qwen2.5-1.5B, standalone SELF-only already improves substantially over the base student on MATH (28.8% vs. 16.9%), MBPP (48.8% vs. 38.4%), and STEM (51.1% vs. 49.5%). Because the student's initial self-teacher is already capable on math, devoting the entire budget to ON + SELF (ρOFF=0\rho_{\text{OFF}}=0) maximizes math performance (55.58% 5-task average). Conversely, on tasks where the 14B external teacher holds a massive advantage—MBPP (78.9%) and STEM (70.5%)—retaining canonical OFF grounding in 3-mode RIVAD achieves the best accuracy on both MBPP (49.2% vs. 48.2%) and STEM (52.6% vs. 52.2%).
  • Regime 2 — Weak Initial Self-Teacher (Gemma2-2B): On Gemma2-2B, standalone SELF-only collapses to 13.5% (well below the base student's 34.1%). Consequently, omitting canonical OFF supervision in ON + SELF degrades accuracy to 37.0%, whereas OFF + SELF (39.9%) and 3-mode RIVAD (40.0%) remain strong by anchoring early training in canonical teacher responses (ρOFF(1)=0.85\rho_{\text{OFF}}^{(1)}=0.85).
  • Practical Guideline: When the student already has strong initial self-teaching competence on a target domain, 2-mode ON + SELF is an effective specialization; across general tasks and diverse student capacities, 3-mode RIVAD serves as a robust default that avoids mode-pruning failure modes and consistently outperforms static Uniform (1/3, 1/3, 1/3) on both Qwen2.5 (54.76% vs. 54.38%) and Gemma2 (40.0% vs. 39.5%) while using 25%–85% fewer expensive ON rollouts. We will add this discussion to Section 4.3.
AuthorsOpenReview ↗

3. Model Families and Standardized ICLR 2026 Benchmark Protocol

3. Model Families and Standardized ICLR 2026 Benchmark Protocol (Weakness 2 & Question 1)

  • Alignment with Latest ICLR 2026 White-Box KD Works: Our experimental setup (Qwen2.5-14B →\to 1.5B and Gemma2-9B →\to 2B trained on UltraInteract 78K across the 8-benchmark suite: MATH, GSM-Plus, MMLU-Pro-Math, BBH-CoT, MMLU-STEM, MBPP, GSM8K, and SciQ) strictly follows the standardized protocol established by the latest ICLR 2026 white-box KD papers (TSD-KD [4], CSD [5], and AMiD [6]) (Lines 346–388). This ensures direct comparability against all nine baselines while demonstrating large gains on challenging benchmarks such as MATH (53.0% vs. 29.3% for CSD on Qwen2.5; 24.1% vs. 20.5% for AMiD on Gemma2).

References

[1] Zhao et al., Self-distilled reasoner: On-policy self-distillation for large language models, arXiv:2601.18734, 2026.
[2] Shen et al., Purified OPSD: On-policy self-distillation without losing how to think, arXiv:2607.02234, 2026.
[3] Kaur et al., Rethinking on-policy self-distillation for thinking models, arXiv:2607.05184, 2026.
[4] Kim & Baek, Explain in your own words: Improving reasoning via token-selective dual knowledge distillation, ICLR 2026.
[5] Kim et al., Distillation of large language models via concrete score matching, ICLR 2026.
[6] Shin et al., AMiD: Knowledge distillation for LLMs with α\alpha-mixture assistant distribution, ICLR 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.