acceptodds
Under review as a conference paper at ICLR 2027

When Mixing Beats Either Policy: Rethinking Rollout Sampling in Large Language Model Distillation

Abstract

On-policy distillation (OPD) trains a student language model on its own rollouts while using a teacher's logits as supervision. However, sampling entirely from the student has two shortcomings: the teacher's guidance becomes unreliable on student-generated trajectories that deviate from the teacher's distribution, and on hard problems the student rarely produces correct paths for the teacher to reinforce. Sampling from the teacher instead incurs high variance in the gradient correction and risks catastrophic forgetting. In this paper, we prove that neither is optimal: the gradient variance of OPD is strictly convex in the student–teacher mixing strength, and its minimum lies strictly between pure student and pure teacher sampling. Building on this theoretical result, we propose MixOPD, which replaces the on-policy sampling distribution with a geometric student–teacher mixture. This method includes two variants: a fixed mixing strength version and MixOPD-Ada, which adapts the mixing strength at each token using the entropy gap between the two models. We implement both variants natively inside the vLLM inference engine with only 39% wall-clock overhead over vanilla OPD. Experiments on agentic, mathematical reasoning, and code generation tasks across two student sizes show that both variants consistently outperform standard OPD by an average of 4 points, indicating that the choice of rollout sampling distribution is a substantial and underexplored lever for improving distillation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VmjAOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes MixOPD, a distillation method that generates training rollouts by mixing the student's and teacher's predictions at each token. It uses a geometric mixture and corrects for the shifted sampling distribution with importance weighting. The authors aim to show that, under certain conditions, mixing can reduce gradient variance compared to sampling from either the student or the teacher alone. Two variants are presented: one uses a fixed mixing ratio, and the other adjusts the ratio according to the model's entropy. The authors claim that both variants outperform standard OPD on agentic, math, and coding tasks, with only a modest increase in compute.

Critical strengths
  • Practical implementation of mixed sampling. The paper addresses how to combine the student's and teacher's predictions when generating training rollouts. Because the vLLM implementation reuses the teacher logits, token-level mixing is possible with only a modest increase in training time.
  • Strong results on ALFWorld. Fixed-ratio MixOPD outperforms OPD at both student sizes. The gain is larger than on the math and coding benchmarks, which suggests that mixed sampling is useful for multi-turn tasks.
Critical weaknesses
  1. Unclear contribution relative to prior work. Existing methods already use teacher–student mixed sampling or token-level teacher intervention [1, 2, 3], and comparisons against these methods are needed to establish the contribution of MixOPD. Although the paper discusses TRB as related work [4], it should state clearly that TRB also derives a per-prefix geometric mixture from the same KL-constrained formulation. As it stands, the distinction from TRB is not apparent beyond the use of token-level geometric mixing.
  2. The theory and its link to the gains are not clearly validated. The interior optimum of the variance depends on specific conditions, but the paper does not verify that these conditions hold throughout training. It also does not show that the chosen mixing ratio actually reduces variance, or that the variance reduction explains the observed performance gains.
  3. Insufficient experimental details. Importance weighting, clipping, hyperparameter selection, evaluation-time sampling, and run-to-run variance are not adequately reported. In addition, the Qwen3-4B baseline scores differ from those in the technical report [5], and neither the thinking mode nor the generation settings are specified. These omissions make the results difficult to interpret.

[1] Gu et al. MiniLLM: Knowledge Distillation of Large Language Models. ICLR, 2024.
[2] Xu et al. Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling. ICLR, 2025.
[3] Koo et al. SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models. Findings of NAACL, 2025.
[4] Plyusov et al. Trust-Region Behavior Blending for On-Policy Distillation. arXiv:2605.31159, 2026.
[5] Yang et al. Qwen3 Technical Report. arXiv:2505.09388, 2025.

Questions for authors
  1. How are the importance weights computed and clipped in MixOPD and IW-OPD? If the weights are token-level, how is unbiasedness guaranteed when the prefix distribution shifts?
  2. Is thinking mode used during training and during evaluation, respectively? What is the performance of the student before training under the same thinking mode and token budget?
  3. Each math benchmark contains only 30 problems. How many responses per problem were used to estimate pass@1? Are the gains over OPD consistent across training seeds? The authors should report confidence intervals for these gains and specify which validation split was used to select the mixing hyperparameters.
  4. Does the chosen mixing ratio actually reduce gradient variance? Comparing variance and task performance across mixing ratios on the same model and dataset is needed to attribute the gains to variance reduction.
5 comments
AuthorsOpenReview ↗

Response to Reviewer VmjA(1/4)

Response to Reviewer VmjA

Thank you for recognizing the practicality of our mixed-sampling implementation and the empirical results on ALFWorld, and for your constructive suggestions on our contribution, theoretical validation, and experimental reproducibility. We address your concerns point by point below.


Q1. Contribution and comparison with TRB (Weakness 1)

Unclear contribution relative to prior work. Existing methods already use teacher–student mixed sampling or token-level teacher intervention [2, 3, 4], and comparisons against these methods are needed to establish the contribution of MixOPD. Although the paper discusses TRB as related work [1], it should state clearly that TRB also derives a per-prefix geometric mixture from the same KL-constrained formulation. As it stands, the distinction from TRB is not apparent beyond the use of token-level geometric mixing.

A1. Compared with prior OPD methods that sample from mixed distributions, MixOPD contributes a conditional gradient-variance analysis of geometric mixture sampling, together with practical strategies for sustained mixing and entropy-gap-driven adaptation. To evaluate these strategies, we conducted additional comparisons with TRB and MiniLLM-style arithmetic mixed sampling under our experimental settings. The results show that MixOPD retains an advantage over these methods. We elaborate on the specific distinctions between MixOPD and these methods below.

1. Technical novelty. Among works on teacher–student mixed sampling, TRB and MiniLLM are most closely related to ours; our main innovations concern the theoretical basis for the mixing distribution, gradient-variance analysis, and control of the mixing strength. TRB [1], like MixOPD, uses a sampling distribution based on a geometric mixture. Our additional contribution is a conditional gradient-variance analysis: Theorem 2 gives conditions under which an interior mixture has lower variance than both pure-student and pure-teacher sampling for the ideal importance-weighted estimator at a fixed prefix. TRB does not provide this analysis. We also extend mixing beyond TRB's early warmup: fixed MixOPD maintains mixing throughout training, and MixOPD-Ada uses the token-level entropy gap to control its strength, instead of annealing a per-prefix KL budget to zero during early warmup. MiniLLM [2] uses arithmetic mixing, �0��0�. Our Theorem 1 instead derives the geometric mixture as the distribution closest to the teacher under a KL budget around the student, providing an optimization-based justification for this choice. We further analyze variance along the geometric path and introduce entropy-gap adaptation, whereas MiniLLM's main experiments use a fixed mixing strength throughout training. In addition to the methods above, SWITCH [3] switches between student and teacher policies, while Speculative KD [4] accepts student proposals or replaces them with teacher samples. In both cases, each token is generated by either the teacher or the student. Compared with these binary decisions, geometric mixing continuously reweights candidate tokens using their relative probabilities under the student and teacher, allowing teacher guidance to refine the student's sampling preferences. Theorem 1 further shows that it yields the sampling distribution closest to the teacher under a given KL budget around the student. We have revised Section 2 to make these distinctions explicit and added the baseline comparisons to Appendix D.1 (Table 6).

2. Empirical comparison. We implemented TRB and the OPD component of MiniLLM with a Qwen3-4B student (omitting MiniLLM's SFT stage for a fair comparison), using Qwen3-30B-A3B as the teacher for math/code and Qwen3-14B for ALFWorld. TRB starts with a KL budget of 0.01, which decays to zero within the first 15 updates. MiniLLM uses the hyperparameters reported in its paper, with all remaining settings matched to our main experiments. Their results are shown below. The OPD, MixOPD, and MixOPD-Ada entries use the original Table 1 runs; A6 reports the additional three-seed results.

MethodALF SeenALF UnseenAIME24AIME25HMMT-NovHumanEvalMBPPMean
OPD32.937.355.349.335.776.259.849.5
TRB42.940.357.047.739.976.859.852.1
MiniLLM40.038.858.250.436.377.458.551.4
MixOPD44.344.857.050.937.279.460.053.4
MixOPD-Ada42.144.059.551.539.278.060.253.5
AuthorsOpenReview ↗

Response to Reviewer VmjA(2/4)

On ALFWorld, MixOPD improves seen/unseen success over TRB by 1.4/4.5 percentage points, and unseen success over MiniLLM by 6.0 points. On mathematics, MixOPD-Ada improves the three-benchmark mean over TRB and MiniLLM by 1.9 and 1.8 points, respectively. Across all seven benchmarks, OPD, TRB, and MiniLLM achieve average scores of 49.5, 52.1, and 51.4, respectively, compared with 53.4/53.5 for MixOPD/MixOPD-Ada. These results support the effectiveness of our mixing strategies.


Q2. Validation of the theory and the source of gains (Weakness 2)

The theory and its link to the gains are not clearly validated. The interior optimum of the variance depends on specific conditions, but the paper does not verify that these conditions hold throughout training. It also does not show that the chosen mixing ratio actually reduces variance, or that the variance reduction explains the observed performance gains.

A2. We added variance comparisons for MixOPD and MixOPD-Ada at different training stages in the Qwen3-4B mathematics setting (Appendix D.4, Table 9). We generated responses from each method's checkpoints at steps 0, 25, and 50. Holding the model parameters and contexts (the questions and previously generated text) fixed, we compare the total variance of single-token output-logit gradients under each method's mixed sampling, pure-student sampling, and pure-teacher sampling. We compute the gradient variance at each measured context by taking the exact expectation over the full vocabulary, then report the mean of these variances. The table reports the mean gradient variance under pure-student sampling (α=0), each method's mixed sampling, and pure-teacher sampling (α=1). Lower values indicate lower variance.

Training methodStepα=0Respective mixed sampling strategyα=1
MixOPD01.151.978.32
MixOPD251.061.285.25
MixOPD500.830.722.56
MixOPD-Ada01.151.487.32
MixOPD-Ada250.300.271.00
MixOPD-Ada500.190.160.57

MixOPD-Ada has lower variance than both endpoints by step 25; at step 50, both MixOPD and MixOPD-Ada have lower variance than both endpoints, with reductions of approximately 13% and 16% relative to pure-student sampling, respectively. Mixed-sampling variance decreases across the three checkpoints for both methods.

In addition, under the settings above, we computed the endpoint derivatives corresponding to Theorem 2, as reported below. The results are consistent with the variance changes in the table above: at checkpoints satisfying M′(0)<0 and M′(1)>0, the mixed sampling strategies we actually used also yielded lower variance than both pure-student and pure-teacher sampling.

Training methodTraining stepM′(0)M′(1)
MixOPD00.7310.10
MixOPD250.428.36
MixOPD50-0.723.25
MixOPD-Ada00.399.54
MixOPD-Ada25-0.902.98
MixOPD-Ada50-0.611.97

Regarding your concern about the causal link between reducing gradient variance and improving performance, prior work provides an optimization-based explanation for this connection. The stochastic-gradient analysis of Bottou et al. [5] shows that, under its smoothness, moment, and step-size assumptions, a smaller gradient-variance bound reduces the noise term in the convergence bound when other constants are held fixed (Theorem 4.8). In the more closely related setting of language-model distillation, MiniLLM [2] also reports that its variance-reducing single-step decomposition improves validation and test performance in an ablation study (Section 3.4, Table 4). These studies provide theoretical and empirical support for the links between gradient variance and both optimization efficiency and distillation performance.


Q3. Experimental details and reproducibility (Weakness 3)

Insufficient experimental details. Importance weighting, clipping, hyperparameter selection, evaluation-time sampling, and run-to-run variance are not adequately reported. In addition, the Qwen3-4B baseline scores differ from those in the technical report [6], and neither the thinking mode nor the generation settings are specified. These omissions make the results difficult to interpret.

A3. We provide these implementation and evaluation details in Appendix C (Tables 4–5) and summarize them below.

1. Importance weighting and clipping. Following Eq. (5) and Section 3.4, we compute the token-level IS weight as �0��0�: the student's probability of the sampled token divided by its probability under the normalized geometric mixture used to sample it, both evaluated at the same prefix �1��1�. We cap the weight at 1.5, �2��2�, then detach it and apply it to that token's loss without batch normalization.

AuthorsOpenReview ↗

Response to Reviewer VmjA(3/4)

2. Hyperparameter settings. Fixed MixOPD uses �0��0�; Ada uses a global mixing cap of 0.3 and entropy-gap threshold $(\tau=0.4)$. These three values were selected by grid search using the full training set. The IS cap is specified in point 1.

For math, Section 4.1 reports a learning rate of �1��1� and a training batch size of 1,024 prompts. The core training settings for the 4B ALFWorld MixOPD/MixOPD-Ada main-table runs are:

ParameterALFWorld setting
OptimizerAdamW; �2��2�, �3��3�; weight decay 0.01
Learning rate�4��4�, constant; no warmup
Training batch size128 episodes
Minibatch / microbatch size128 / 1 per GPU

Training and evaluation generation settings are given in point 3 below.

3. Generation and evaluation settings, and baseline comparison. All evaluations use the student alone.

Math uses non-thinking mode during training and evaluation. Evaluation uses temperature/top-p = 1, a 16,384-token limit, and 32 responses per problem, following IW-OPD [7]; mean response correctness estimates pass@1. Code also uses non-thinking mode during training and evaluation, follows its EvalPlus greedy single-sample evaluation protocol, and uses a uniform 512-token output limit across methods.

For ALFWorld, thinking is enabled during both training and evaluation, allowing reasoning before each action. This choice was informed by preliminary experiments in which the non-thinking OPD configuration did not train stably. We use Qwen3-14B as a thinking-capable teacher. Training rollouts use temperature 1.0, while evaluation uses temperature 0.4 following the TCOD reference configuration [8]. Both use top-p 1.0, one response per turn, at most 30 action turns, and 2,048 output tokens per turn, retaining the two most recent interaction rounds. Evaluation reports task success on 140 seen and 134 unseen games.

Our AIME24/AIME25 baseline scores of 22.3/21.8 are broadly consistent with the non-thinking results of 25.0/19.1 in the Qwen3 technical report [6]. The numerical differences can be explained by the different inference settings: our temperature/top-p are 1/1, whereas the report uses 0.7/0.8.

4. Run-to-run variance. Please refer to A6 for the mean performance and standard deviation across three training seeds.


Q4. Importance weights, clipping, and shifted prefixes (Original question 1)

How are the importance weights computed and clipped in MixOPD and IW-OPD? If the weights are token-level, how is unbiasedness guaranteed when the prefix distribution shifts?

A4. MixOPD computes the token-level ratio �5��5�, caps it at 1.5, and applies it with stop-gradient and no batch normalization. IW-OPD instead uses prefix-dependent surrogate weights in [1,1.5][1,1.5] [7].

The unbiasedness statement in our analysis concerns the token gradient at a fixed prefix. With exact importance weights, changing the token-sampling distribution preserves this conditional gradient mean. Correcting a trajectory-level expectation would additionally require products of likelihood ratios for the preceding tokens. Our implementation uses token-level correction and clips large weights, following the standard bias–variance trade-off used in RL to limit high-variance updates and stabilize training [10]. MiniLLM [2] similarly approximates cumulative importance weights with token-level ratios, and REOPOLD [9] clips distillation credits to stabilize training.


Q5. Thinking mode and the untrained student (Original question 2)

Is thinking mode used during training and during evaluation, respectively? What is the performance of the student before training under the same thinking mode and token budget?

A5. Math uses non-thinking mode during training and evaluation. Evaluation follows IW-OPD [7]: temperature/top-p = 1, a 16,384-token limit, and 32 responses per problem. Code follows its EvalPlus greedy single-sample protocol, with non-thinking generation and a uniform 512-token limit across methods.

For ALFWorld, thinking is enabled during both training and evaluation, allowing reasoning before each action. This choice was informed by preliminary experiments in which the non-thinking OPD configuration did not train stably. We use Qwen3-14B as a thinking-capable teacher. Training rollouts use temperature 1.0, while evaluation uses temperature 0.4 following the TCOD reference configuration [8]. Evaluation uses top-p 1.0, one response per turn, at most 30 action turns, and 2,048 output tokens per turn, retaining the two most recent interaction rounds.

The untrained Qwen3-4B scores 22.3/21.8 on AIME24/AIME25, broadly consistent with the non-thinking results of 25.0/19.1 in the Qwen3 technical report [6]. The numerical differences can be explained by the different inference settings.


Reviewer_MfEeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presented MIXOPD, using a mixed distribution between student and teacher as the sampling policy. The theoretic analysis shows that the geometric mixture achieves lower variance without changing the optimization objective. Empirical results show that MIXOPD improves the overall performance against sampling from student or teach policies.

Critical strengths
  • The motivation is reasonable, with a good presentation.
  • The theoretic analysis makes the method more solid.
  • The evaluation is substantial.
Critical weaknesses
  • The main concern is the non-trivial overhead. Though the authors claim that 39% wall-clock overhead is small, actually this is very large. Additionally, recording log⁡πT(yt)\log \pi_T(y_t) for sampled token only have other limitations, for example, backward through full vocabulary KL instead of MC KL would be infeasible.
  • The theorectic analysis can only guarantee there exists an α\alpha which has the smallest variance, but it cannot guarantee the chosen α\alpha has a smaller variance than α=0\alpha=0 and α=1\alpha=1. How to find the optimal alphaalpha remains open.
Questions for authors
  • Equation 5 is based on a naive reward. Typically we use advantage by subtracting a baseline for variance reduction. If we use advantage in this formulation, does the theory still achieve the same conclusion?
  • The minimum lies strictly between pure student and pure teacher sampling is a bit over-claiming, due to the existence of condition (i) and (ii). It should be at most empirically, not theorectically strictly. And to be honest, I don't see why these two conditions are true even intuitively. They require better justifications.
  • I don't see a consistent advantage for MIXOPD-ADA compared to MIXOPD. So do we really need MIXOPD-ADA?
  • The empirical improvement in math and coding (figure 1) seems marginal. Any insights on the reason?
4 comments
AuthorsOpenReview ↗

Response to Reviewer MfEe(1/3)

Response to Reviewer MfEe

Thank you for your positive assessment of our motivation, presentation, theoretical analysis, and empirical evaluation, and for your constructive suggestions on computational overhead, the conditions under which our theory applies, and the value of the adaptive strategy. We address your concerns point by point below.


Q1. Overhead and full-vocabulary KL (Weakness 1)

The main concern is the non-trivial overhead. Though the authors claim that 39% wall-clock overhead is small, actually this is very large. Additionally, recording log⁡πT(yt)\log \pi_T(y_t) for sampled token only have other limitations, for example, backward through full vocabulary KL instead of MC KL would be infeasible.

A1.

1. Performance under the same training budget

MixOPD retains its performance advantage at comparable training time. We extended OPD from 50 to 70 steps, using the training time of 50-step MixOPD as the budget reference (Appendix D.3, Table 8). This comparison uses the Qwen3-4B student and Qwen3-30B-A3B teacher on 32 NVIDIA H200 GPUs. Reported times include rollout generation, teacher inference, and student parameter updates, excluding model initialization and task evaluation.

Task / timing experimentMethodTraining stepsTime (minutes)
Mathematics & codeOPD50178.82
Mathematics & codeOPD70246.10
Mathematics & codeMixOPD50244.19

MixOPD outperforms extended OPD on all five mathematics and code benchmarks, with an average of 57.1 versus 54.3, using 244.19 versus 246.10 minutes. Extending OPD even lowers its five-benchmark mean from 55.2 to 54.3; additional training time does not close the gap. The 50-step entries below reproduce the three-seed means, and Avg is the unweighted mean of the five displayed benchmarks.

MethodAIME24AIME25HMMT-NovHumanEvalMBPPAvg
OPD55.449.235.576.759.455.2
OPD (extended)55.749.035.173.857.954.3
MixOPD57.251.037.979.360.157.1
MixOPD-Ada59.651.639.577.260.457.7

2. Feasibility of full-vocabulary KL

MixOPD supports both sampled-token and full-vocabulary KL losses. Its geometric sampling rule already computes the teacher's full next-token distribution at each step. The original implementation retains the sampled token's log-probability for the distillation update. This is a standard implementation option: verl explicitly implements single-sample KL estimators from student and teacher log-probabilities at the sampled token [1].

Our full-vocabulary KL experiment below, included in Appendix D.5 (Table 10), recomputes the teacher logits during the student update. It achieves comparable ALFWorld performance and training time, directly demonstrating compatibility with full-vocabulary supervision.

MethodTraining stepsALF Seen (%)ALF Unseen (%)Training time (minutes)
MixOPD (sampled-token)5043.644.3495.68
MixOPD (full-vocabulary KL, teacher recomputation)5045.342.4507.96

Q2. Whether the chosen coefficient beats both endpoints (Weakness 2)

The theorectic analysis can only guarantee there exists an α\alpha which has the smallest variance, but it cannot guarantee the chosen α\alpha has a smaller variance than α=0\alpha=0 and α=1\alpha=1. How to find the optimal alphaalpha remains open.

A2. We added variance comparisons in Appendix D.4 (Table 9) for MixOPD and MixOPD-Ada during training to examine whether the mixing strategies used in practice achieve lower variance than pure-student and pure-teacher sampling.

1. Does the chosen mixing strategy outperform both endpoints? In the Qwen3-4B mathematics setting, we generated responses from the step 0, 25, and 50 checkpoints of MixOPD (fixed α=0.3\alpha=0.3) and MixOPD-Ada. Holding model parameters and contexts fixed, we compare the total variance of single-token output-logit gradients under each method's mixed sampling, pure-student sampling, and pure-teacher sampling. We take the exact expectation over the full vocabulary and then average the variances over the measured contexts. The table directly compares the mean gradient variances under pure-student sampling (α=0\alpha=0), each method's mixed sampling, and pure-teacher sampling (α=1\alpha=1). Lower values indicate lower variance.

Training methodStepα=0\alpha=0Respective mixed sampling strategyα=1\alpha=1
MixOPD01.151.978.32
MixOPD251.061.285.25
MixOPD500.830.722.56
MixOPD-Ada01.151.487.32
MixOPD-Ada250.300.271.00
MixOPD-Ada500.190.160.57
AuthorsOpenReview ↗

Response to Reviewer MfEe(2/3)

2. How can the mixing coefficient be selected adaptively? MixOPD-Ada adaptively selects the mixing coefficient token by token using the student–teacher entropy gap at the current context: it increases teacher influence when the student is more uncertain relative to the teacher and decreases it when the entropy gap is smaller. It uses the distributions already available during sampling, without evaluating gradient variance for each candidate coefficient. At step 25, Ada's mixed-sampling variance is 0.27, below both pure-student sampling (0.30) and pure-teacher sampling (1.00). These results support entropy-based adaptation as an effective practical coefficient-selection strategy; this strategy also outperforms fixed MixOPD on the mathematics benchmarks and MBPP.


Q3. Distillation reward and the role of an advantage baseline (Original question 1)

Equation 5 is based on a naive reward. Typically we use advantage by subtracting a baseline for variance reduction. If we use advantage in this formulation, does the theory still achieve the same conclusion?

A3. The reward in Eq. (5) follows directly from the reverse-KL distillation loss. At a fixed prefix hh, let Lh=KL(πS(⋅∣h)∣πT(⋅∣h))L_h=\mathrm{KL}(\pi_S(\cdot\mid h)\\|\pi_T(\cdot\mid h)). Its negative gradient is

−∇θLh=Ey∼πS(⋅∣h)[log⁡πT(y∣h)πS(y∣h)⏟R(y∣h)∇θlog⁡πS(y∣h)].-\nabla_\theta L_h =\mathbb{E}_{y\sim\pi_S(\cdot\mid h)}\left[ \underbrace{\log\frac{\pi_T(y\mid h)}{\pi_S(y\mid h)}}_{R(y\mid h)} \nabla_\theta\log\pi_S(y\mid h) \right].

The additional score-function term vanishes because EπS[∇θlog⁡πS]=0\mathbb{E}_{\pi_S}[\nabla_\theta\log\pi_S]=0. Sampling from qαq_\alpha and multiplying by πS/qα\pi_S/q_\alpha gives the gradient estimator whose second moment is analyzed in Eq. (5). Thus, RR is the negative log-probability ratio from the distillation objective. Our implementation uses one rollout per query and this direct token-level reward, with a zero baseline and no advantage whitening. A state-dependent baseline can provide complementary variance reduction; our analysis focuses on the estimator obtained directly from the distillation loss.


Q4. Conditions for a strictly interior minimum (Original question 2)

The minimum lies strictly between pure student and pure teacher sampling is a bit over-claiming, due to the existence of condition (i) and (ii). It should be at most empirically, not theorectically strictly. And to be honest, I don't see why these two conditions are true even intuitively. They require better justifications.

A4. The strictly interior optimum is a theoretical conclusion under the two explicit endpoint conditions in Theorem 2. Strict convexity together with M′(0)<0M'(0)<0 and M′(1)>0M'(1)>0 gives a unique minimizer in (0,1)(0,1). The revised abstract summarizes when mixed sampling improves variance; Section 3.3 explains the endpoint conditions, with checkpoint measurements in Appendix D.4 (Table 9).

The two conditions describe complementary effects of mixed sampling. Condition (i), M′(0)=−Cov⁡πS(c,R)<0M'(0)=-\operatorname{Cov}_{\pi_S}(c,R)<0, means that teacher-preferred tokens carry more gradient energy under the student distribution. This is natural when the student still underweights informative teacher-preferred tokens that require substantial updates. Condition (ii), M′(1)>0M'(1)>0, reflects the effect of large student-to-teacher importance ratios: adding student influence improves coverage of student-favored tokens and reduces their contribution to the second moment. These effects are particularly relevant in mid-to-late training, when the student and teacher are closer overall but informative token-level disagreements remain.

We measured these derivatives at the Qwen3-4B mathematics checkpoints used in A2, holding model parameters and contexts fixed and taking exact expectations over the vocabulary. The table reports derivatives of the mean output-logit gradient second moment; its derivatives equal those of the corresponding variance because the gradient mean is unchanged by importance-weighted resampling.

Training methodTraining stepM′(0)M'(0)M′(1)M'(1)
MixOPD00.7310.10
MixOPD250.428.36
MixOPD50-0.723.25
MixOPD-Ada00.399.54
MixOPD-Ada25-0.902.98
MixOPD-Ada50-0.611.97

Both endpoint conditions hold for MixOPD at step 50 and for MixOPD-Ada at steps 25 and 50. These are also the checkpoints at which the actual mixing rules in A2 achieve lower variance than both endpoints. The measured endpoint signs and the variance at the chosen coefficients therefore jointly support the theoretical account of mixed sampling.


AuthorsOpenReview ↗

Response to Reviewer MfEe(3/3)

Q5. Whether MixOPD-Ada is necessary (Original question 3)

I don't see a consistent advantage for MIXOPD-ADA compared to MIXOPD. So do we really need MIXOPD-ADA?

A5. MixOPD-Ada provides context-dependent control of teacher influence, complementing fixed mixing. In Table 1, Ada outperforms fixed MixOPD on all three mathematics benchmarks, while fixed MixOPD performs better on ALFWorld. Ada therefore offers an adaptive option with demonstrated empirical benefits, whose additional gains depend on the task.

Ada assigns greater teacher influence where the student is uncertain and the teacher is more certain. Across the three training seeds in Appendix D.2 (Table 7), it improves the mathematics mean from 48.7 for fixed MixOPD to 50.2, and MBPP from 60.1 to 60.4. The variance measurements in A2 also show that Ada reduces variance below both endpoints by step 25. These results support context-dependent mixing as a useful alternative to the fixed strategy.


Q6. Why gains differ across domains (Original question 4)

The empirical improvement in math and coding (figure 1) seems marginal. Any insights on the reason?

A6. Math and code also show clear gains, although their magnitude is smaller than on ALFWorld. Following the suggestion of Reviewer 1 (VmjA), we conducted three training runs using the Qwen3-4B settings of Table 1 and report their mean and standard deviation in Appendix D.2 (Table 7). Using the displayed benchmark means and equal weighting within each domain, MixOPD improves over OPD by 2.0 and 1.7 percentage points on math and code, respectively; MixOPD-Ada improves by 3.5 and 0.8 percentage points. Combining the three math and two code benchmarks, the two methods improve over OPD by an average of 1.9 and 2.4 percentage points, respectively. Thus, the results averaged over three training runs show that mixed sampling also provides practically meaningful gains over OPD on math and code; its benefits are not confined to ALFWorld.

We believe the larger gains on ALFWorld may reflect the sensitivity of multi-turn interaction to early decisions. Early actions change subsequent environment states, so mixed sampling may improve not only the current action but also the trajectories available for subsequent learning. Figure 1(a) shows that the teacher's task completion rate decreases as the student-generated prefix grows, providing evidence for this trajectory sensitivity.

References

[1] verl contributors. On-Policy Distillation: sampled-token KL loss modes.


Thanks again for your detailed comments. Please let us know if there is more to clarify.

Thanks for the rebuttal. Although I respect the efforts for such a long response, I don't think it substantially addresses my concerns. I will keep my score.

non-trivial overhead

Thanks for the extra experiments. But it's hard to be convinced to accept such a complicated rollout system and large overhead by a simple experiment with incremental improvement.

Feasibility of full-vocabulary KL

Recomputes the teacher logits during the student update will incur extra non-trivial overhead, otherwise it costs huge memory to store full-vocabulary probabilities from rollout engine to training engine. The authors claim that this forward pass can be skipped (no recompute), that's why I ask this question.

Does the chosen mixing strategy outperform both endpoints?

From the table, I don't see the mixed sampling strategy consistently has a lower variance even empirically. Theorectical justification is also missing.

How to find the optimal α\alpha remains open.

The response basically describes their heuristic method again, which is not what I asked. I'm asking how to find the theorectical optimal solution. At least, the authors should prove their solution won't behave worse than α=0\alpha = 0 and α=1\alpha=1.

If we use advantage in this formulation, does the theory still achieve the same conclusion?

A comon practice is to use KL itself as a baseline to reduce variance, which can make a difference on the theorectical analysis. The authors said they use a zero baseline, and did not answer my question directly.

The minimum lies strictly between pure student and pure teacher sampling is a bit over-claiming, due to the existence of condition (i) and (ii). It should be at most empirically, not theorectically strictly. And to be honest, I don't see why these two conditions are true even intuitively. They require better justifications.

The real question is: this claim is only valid when these two conditions are true, so making this strong claim without mentioning the conditions is a bit over-claiming. And the evidences for these two conditions are from empirical results, instead of theoretical analysis (so they can be false in some cases).

I don't see a consistent advantage for MIXOPD-ADA compared to MIXOPD. So do we really need MIXOPD-ADA?

In Figure 1 and Table 1, MIXOPD-ADA performs worse than MIXOPD in both agentic and code tasks. Even in Math, the difference is not significant. That's the reason I asked this question. Obviously, the response doesn't help. You can always find some cases that MIXOPD-ADA performs a bit better, but if the conclusion is not consistent, it means nothing.

The empirical improvement in math and coding (figure 1) seems marginal.

Two or one percentage points are just marginal from my view.

Reviewer_q8yMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MixOPD, which improves on-policy distillation by generating training trajectories from a token-level geometric mixture of student and teacher policies. The method introduces teacher guidance during rollout generation and uses importance weighting to correct for the modified sampling distribution. It includes fixed and entropy-based adaptive mixing variants. Experiments across agentic, mathematical reasoning, and code generation benchmarks show improved student performance over standard OPD under the same training-step budget.

Critical strengths

The method improves learning effectiveness per training step by incorporating teacher logits directly into rollout generation. This allows teacher guidance to influence the trajectories the student learns from, potentially providing more informative training examples than purely student-generated rollouts. The reported improvements across two student sizes and multiple task domains support the usefulness of this approach under a fixed number of training updates.

Critical weaknesses

The experiments do not establish whether these improvements translate into better performance under an equal computational budget. MixOPD requires teacher inference at every decoding step, and the reported per-step wall-clock time increases from 207 seconds for standard OPD to 288 seconds for MixOPD, approximately 39% overhead. Comparisons at equal training steps therefore allocate more computation to MixOPD. Evaluations under matched wall-clock time, GPU-hours, or FLOPs—or comparisons of the cost required to reach a given performance level—are needed to determine whether the method offers a practical efficiency advantage.

Questions for authors

see weakness

1 comment

Response to Reviewer q8yM

Thank you for recognizing the improvements in learning effectiveness per training step through teacher guidance and the performance gains across student sizes and task domains under a fixed number of training steps, and for your constructive suggestions on fair comparisons under equal computational budgets. We address your concern below.


Q1. Performance under an equal computational budget (Weakness and question)

The experiments do not establish whether these improvements translate into better performance under an equal computational budget. MixOPD requires teacher inference at every decoding step, and the reported per-step wall-clock time increases from 207 seconds for standard OPD to 288 seconds for MixOPD, approximately 39% overhead. Comparisons at equal training steps therefore allocate more computation to MixOPD. Evaluations under matched wall-clock time, GPU-hours, or FLOPs—or comparisons of the cost required to reach a given performance level—are needed to determine whether the method offers a practical efficiency advantage.

see weakness

A1. MixOPD retains its performance advantage at comparable training time. We extended OPD from 50 to 70 steps, using the training time of 50-step MixOPD as the budget reference (Appendix D.3, Table 8). This comparison uses the Qwen3-4B student and Qwen3-30B-A3B teacher on 32 NVIDIA H200 GPUs. Reported times include rollout generation, teacher inference, and student parameter updates, excluding model initialization and task evaluation.

Task / timing experimentMethodTraining stepsTime (minutes)
Mathematics & codeOPD50178.82
Mathematics & codeOPD70246.10
Mathematics & codeMixOPD50244.19

MixOPD outperforms extended OPD on all five mathematics and code benchmarks, with an average of 57.1 versus 54.3, using 244.19 versus 246.10 minutes. Extending OPD lowers its five-benchmark mean from 55.2 to 54.3; additional training time does not close the gap. The 50-step entries below reproduce the three-seed means, and Avg is the unweighted mean of the five displayed benchmarks.

MethodAIME24AIME25HMMT-NovHumanEvalMBPPAvg
OPD55.449.235.576.759.455.2
OPD (extended)55.749.035.173.857.954.3
MixOPD57.251.037.979.360.157.1
MixOPD-Ada59.651.639.577.260.457.7

Thanks again for your constructive suggestions. Hope our explanations can ease your concerns. Please let us know if there is more to clarify.

Comments on the paper

1 comment

We sincerely thank all reviewers for their detailed reading and valuable comments. We have addressed their concerns and incorporated the corresponding revisions into the updated manuscript, with two additional pages. The revisions are marked in blue. The main updates are:

  • Sec. 2: Clarify the relationship to TRB and MiniLLM, including the distinctions in variance analysis, sustained mixing, and entropy-based adaptation.
  • Abstract and Sec. 3.3: Clarify the variance result and explain the endpoint conditions in the context of mid-to-late distillation.
  • Appendix C (Tables 4–5): Add importance-weighting and clipping details, hyperparameter selection using the full training set, optimizer settings, thinking modes, and evaluation configurations.
  • Appendix D.1 (new, Table 6): Add comparisons with TRB and MiniLLM under the Qwen3-4B experimental settings.
  • Appendix D.2 (new, Table 7): Report mean performance and standard deviation across three training seeds for OPD, MixOPD, and MixOPD-Ada with Qwen3-4B.
  • Sec. 4.5 and Appendix D.3 (new, Table 8): Add a matched-training-time comparison of 50-step MixOPD and 70-step OPD on mathematics and code benchmarks.
  • Appendix D.4 (new, Table 9): Report measured conditional gradient variances and endpoint derivatives at steps 0, 25, and 50, connecting the endpoint signs with variance reductions at the chosen coefficients.
  • Appendix D.5 (new, Table 10): Add an ALFWorld comparison of sampled-token and full-vocabulary KL supervision, including performance and training time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.