- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes MixOPD, a distillation method that generates training rollouts by mixing the student's and teacher's predictions at each token. It uses a geometric mixture and corrects for the shifted sampling distribution with importance weighting. The authors aim to show that, under certain conditions, mixing can reduce gradient variance compared to sampling from either the student or the teacher alone. Two variants are presented: one uses a fixed mixing ratio, and the other adjusts the ratio according to the model's entropy. The authors claim that both variants outperform standard OPD on agentic, math, and coding tasks, with only a modest increase in compute.
Critical strengths
- Practical implementation of mixed sampling. The paper addresses how to combine the student's and teacher's predictions when generating training rollouts. Because the vLLM implementation reuses the teacher logits, token-level mixing is possible with only a modest increase in training time.
- Strong results on ALFWorld. Fixed-ratio MixOPD outperforms OPD at both student sizes. The gain is larger than on the math and coding benchmarks, which suggests that mixed sampling is useful for multi-turn tasks.
Critical weaknesses
- Unclear contribution relative to prior work. Existing methods already use teacher–student mixed sampling or token-level teacher intervention [1, 2, 3], and comparisons against these methods are needed to establish the contribution of MixOPD. Although the paper discusses TRB as related work [4], it should state clearly that TRB also derives a per-prefix geometric mixture from the same KL-constrained formulation. As it stands, the distinction from TRB is not apparent beyond the use of token-level geometric mixing.
- The theory and its link to the gains are not clearly validated. The interior optimum of the variance depends on specific conditions, but the paper does not verify that these conditions hold throughout training. It also does not show that the chosen mixing ratio actually reduces variance, or that the variance reduction explains the observed performance gains.
- Insufficient experimental details. Importance weighting, clipping, hyperparameter selection, evaluation-time sampling, and run-to-run variance are not adequately reported. In addition, the Qwen3-4B baseline scores differ from those in the technical report [5], and neither the thinking mode nor the generation settings are specified. These omissions make the results difficult to interpret.
[1] Gu et al. MiniLLM: Knowledge Distillation of Large Language Models. ICLR, 2024.
[2] Xu et al. Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling. ICLR, 2025.
[3] Koo et al. SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models. Findings of NAACL, 2025.
[4] Plyusov et al. Trust-Region Behavior Blending for On-Policy Distillation. arXiv:2605.31159, 2026.
[5] Yang et al. Qwen3 Technical Report. arXiv:2505.09388, 2025.
Questions for authors
- How are the importance weights computed and clipped in MixOPD and IW-OPD? If the weights are token-level, how is unbiasedness guaranteed when the prefix distribution shifts?
- Is thinking mode used during training and during evaluation, respectively? What is the performance of the student before training under the same thinking mode and token budget?
- Each math benchmark contains only 30 problems. How many responses per problem were used to estimate pass@1? Are the gains over OPD consistent across training seeds? The authors should report confidence intervals for these gains and specify which validation split was used to select the mixing hyperparameters.
- Does the chosen mixing ratio actually reduce gradient variance? Comparing variance and task performance across mixing ratios on the same model and dataset is needed to attribute the gains to variance reduction.
5 comments
Response to Reviewer VmjA(2/4)
On ALFWorld, MixOPD improves seen/unseen success over TRB by 1.4/4.5 percentage points, and unseen success over MiniLLM by 6.0 points. On mathematics, MixOPD-Ada improves the three-benchmark mean over TRB and MiniLLM by 1.9 and 1.8 points, respectively. Across all seven benchmarks, OPD, TRB, and MiniLLM achieve average scores of 49.5, 52.1, and 51.4, respectively, compared with 53.4/53.5 for MixOPD/MixOPD-Ada. These results support the effectiveness of our mixing strategies.
Q2. Validation of the theory and the source of gains (Weakness 2)
The theory and its link to the gains are not clearly validated. The interior optimum of the variance depends on specific conditions, but the paper does not verify that these conditions hold throughout training. It also does not show that the chosen mixing ratio actually reduces variance, or that the variance reduction explains the observed performance gains.
A2. We added variance comparisons for MixOPD and MixOPD-Ada at different training stages in the Qwen3-4B mathematics setting (Appendix D.4, Table 9). We generated responses from each method's checkpoints at steps 0, 25, and 50. Holding the model parameters and contexts (the questions and previously generated text) fixed, we compare the total variance of single-token output-logit gradients under each method's mixed sampling, pure-student sampling, and pure-teacher sampling. We compute the gradient variance at each measured context by taking the exact expectation over the full vocabulary, then report the mean of these variances. The table reports the mean gradient variance under pure-student sampling (α=0), each method's mixed sampling, and pure-teacher sampling (α=1). Lower values indicate lower variance.
| Training method | Step | α=0 | Respective mixed sampling strategy | α=1 |
|---|---|---|---|---|
| MixOPD | 0 | 1.15 | 1.97 | 8.32 |
| MixOPD | 25 | 1.06 | 1.28 | 5.25 |
| MixOPD | 50 | 0.83 | 0.72 | 2.56 |
| MixOPD-Ada | 0 | 1.15 | 1.48 | 7.32 |
| MixOPD-Ada | 25 | 0.30 | 0.27 | 1.00 |
| MixOPD-Ada | 50 | 0.19 | 0.16 | 0.57 |
MixOPD-Ada has lower variance than both endpoints by step 25; at step 50, both MixOPD and MixOPD-Ada have lower variance than both endpoints, with reductions of approximately 13% and 16% relative to pure-student sampling, respectively. Mixed-sampling variance decreases across the three checkpoints for both methods.
In addition, under the settings above, we computed the endpoint derivatives corresponding to Theorem 2, as reported below. The results are consistent with the variance changes in the table above: at checkpoints satisfying M′(0)<0 and M′(1)>0, the mixed sampling strategies we actually used also yielded lower variance than both pure-student and pure-teacher sampling.
| Training method | Training step | M′(0) | M′(1) |
|---|---|---|---|
| MixOPD | 0 | 0.73 | 10.10 |
| MixOPD | 25 | 0.42 | 8.36 |
| MixOPD | 50 | -0.72 | 3.25 |
| MixOPD-Ada | 0 | 0.39 | 9.54 |
| MixOPD-Ada | 25 | -0.90 | 2.98 |
| MixOPD-Ada | 50 | -0.61 | 1.97 |
Regarding your concern about the causal link between reducing gradient variance and improving performance, prior work provides an optimization-based explanation for this connection. The stochastic-gradient analysis of Bottou et al. [5] shows that, under its smoothness, moment, and step-size assumptions, a smaller gradient-variance bound reduces the noise term in the convergence bound when other constants are held fixed (Theorem 4.8). In the more closely related setting of language-model distillation, MiniLLM [2] also reports that its variance-reducing single-step decomposition improves validation and test performance in an ablation study (Section 3.4, Table 4). These studies provide theoretical and empirical support for the links between gradient variance and both optimization efficiency and distillation performance.
Q3. Experimental details and reproducibility (Weakness 3)
Insufficient experimental details. Importance weighting, clipping, hyperparameter selection, evaluation-time sampling, and run-to-run variance are not adequately reported. In addition, the Qwen3-4B baseline scores differ from those in the technical report [6], and neither the thinking mode nor the generation settings are specified. These omissions make the results difficult to interpret.
A3. We provide these implementation and evaluation details in Appendix C (Tables 4–5) and summarize them below.
1. Importance weighting and clipping. Following Eq. (5) and Section 3.4, we compute the token-level IS weight as : the student's probability of the sampled token divided by its probability under the normalized geometric mixture used to sample it, both evaluated at the same prefix . We cap the weight at 1.5, , then detach it and apply it to that token's loss without batch normalization.
Response to Reviewer VmjA(3/4)
2. Hyperparameter settings. Fixed MixOPD uses ; Ada uses a global mixing cap of 0.3 and entropy-gap threshold $(\tau=0.4)$. These three values were selected by grid search using the full training set. The IS cap is specified in point 1.
For math, Section 4.1 reports a learning rate of and a training batch size of 1,024 prompts. The core training settings for the 4B ALFWorld MixOPD/MixOPD-Ada main-table runs are:
| Parameter | ALFWorld setting |
|---|---|
| Optimizer | AdamW; , ; weight decay 0.01 |
| Learning rate | , constant; no warmup |
| Training batch size | 128 episodes |
| Minibatch / microbatch size | 128 / 1 per GPU |
Training and evaluation generation settings are given in point 3 below.
3. Generation and evaluation settings, and baseline comparison. All evaluations use the student alone.
Math uses non-thinking mode during training and evaluation. Evaluation uses temperature/top-p = 1, a 16,384-token limit, and 32 responses per problem, following IW-OPD [7]; mean response correctness estimates pass@1. Code also uses non-thinking mode during training and evaluation, follows its EvalPlus greedy single-sample evaluation protocol, and uses a uniform 512-token output limit across methods.
For ALFWorld, thinking is enabled during both training and evaluation, allowing reasoning before each action. This choice was informed by preliminary experiments in which the non-thinking OPD configuration did not train stably. We use Qwen3-14B as a thinking-capable teacher. Training rollouts use temperature 1.0, while evaluation uses temperature 0.4 following the TCOD reference configuration [8]. Both use top-p 1.0, one response per turn, at most 30 action turns, and 2,048 output tokens per turn, retaining the two most recent interaction rounds. Evaluation reports task success on 140 seen and 134 unseen games.
Our AIME24/AIME25 baseline scores of 22.3/21.8 are broadly consistent with the non-thinking results of 25.0/19.1 in the Qwen3 technical report [6]. The numerical differences can be explained by the different inference settings: our temperature/top-p are 1/1, whereas the report uses 0.7/0.8.
4. Run-to-run variance. Please refer to A6 for the mean performance and standard deviation across three training seeds.
Q4. Importance weights, clipping, and shifted prefixes (Original question 1)
How are the importance weights computed and clipped in MixOPD and IW-OPD? If the weights are token-level, how is unbiasedness guaranteed when the prefix distribution shifts?
A4. MixOPD computes the token-level ratio , caps it at 1.5, and applies it with stop-gradient and no batch normalization. IW-OPD instead uses prefix-dependent surrogate weights in [7].
The unbiasedness statement in our analysis concerns the token gradient at a fixed prefix. With exact importance weights, changing the token-sampling distribution preserves this conditional gradient mean. Correcting a trajectory-level expectation would additionally require products of likelihood ratios for the preceding tokens. Our implementation uses token-level correction and clips large weights, following the standard bias–variance trade-off used in RL to limit high-variance updates and stabilize training [10]. MiniLLM [2] similarly approximates cumulative importance weights with token-level ratios, and REOPOLD [9] clips distillation credits to stabilize training.
Q5. Thinking mode and the untrained student (Original question 2)
Is thinking mode used during training and during evaluation, respectively? What is the performance of the student before training under the same thinking mode and token budget?
A5. Math uses non-thinking mode during training and evaluation. Evaluation follows IW-OPD [7]: temperature/top-p = 1, a 16,384-token limit, and 32 responses per problem. Code follows its EvalPlus greedy single-sample protocol, with non-thinking generation and a uniform 512-token limit across methods.
For ALFWorld, thinking is enabled during both training and evaluation, allowing reasoning before each action. This choice was informed by preliminary experiments in which the non-thinking OPD configuration did not train stably. We use Qwen3-14B as a thinking-capable teacher. Training rollouts use temperature 1.0, while evaluation uses temperature 0.4 following the TCOD reference configuration [8]. Evaluation uses top-p 1.0, one response per turn, at most 30 action turns, and 2,048 output tokens per turn, retaining the two most recent interaction rounds.
The untrained Qwen3-4B scores 22.3/21.8 on AIME24/AIME25, broadly consistent with the non-thinking results of 25.0/19.1 in the Qwen3 technical report [6]. The numerical differences can be explained by the different inference settings.
Response to Reviewer VmjA(1/4)
Response to Reviewer VmjA
Thank you for recognizing the practicality of our mixed-sampling implementation and the empirical results on ALFWorld, and for your constructive suggestions on our contribution, theoretical validation, and experimental reproducibility. We address your concerns point by point below.
Q1. Contribution and comparison with TRB (Weakness 1)
A1. Compared with prior OPD methods that sample from mixed distributions, MixOPD contributes a conditional gradient-variance analysis of geometric mixture sampling, together with practical strategies for sustained mixing and entropy-gap-driven adaptation. To evaluate these strategies, we conducted additional comparisons with TRB and MiniLLM-style arithmetic mixed sampling under our experimental settings. The results show that MixOPD retains an advantage over these methods. We elaborate on the specific distinctions between MixOPD and these methods below.
1. Technical novelty. Among works on teacher–student mixed sampling, TRB and MiniLLM are most closely related to ours; our main innovations concern the theoretical basis for the mixing distribution, gradient-variance analysis, and control of the mixing strength. TRB [1], like MixOPD, uses a sampling distribution based on a geometric mixture. Our additional contribution is a conditional gradient-variance analysis: Theorem 2 gives conditions under which an interior mixture has lower variance than both pure-student and pure-teacher sampling for the ideal importance-weighted estimator at a fixed prefix. TRB does not provide this analysis. We also extend mixing beyond TRB's early warmup: fixed MixOPD maintains mixing throughout training, and MixOPD-Ada uses the token-level entropy gap to control its strength, instead of annealing a per-prefix KL budget to zero during early warmup. MiniLLM [2] uses arithmetic mixing, �0�. Our Theorem 1 instead derives the geometric mixture as the distribution closest to the teacher under a KL budget around the student, providing an optimization-based justification for this choice. We further analyze variance along the geometric path and introduce entropy-gap adaptation, whereas MiniLLM's main experiments use a fixed mixing strength throughout training. In addition to the methods above, SWITCH [3] switches between student and teacher policies, while Speculative KD [4] accepts student proposals or replaces them with teacher samples. In both cases, each token is generated by either the teacher or the student. Compared with these binary decisions, geometric mixing continuously reweights candidate tokens using their relative probabilities under the student and teacher, allowing teacher guidance to refine the student's sampling preferences. Theorem 1 further shows that it yields the sampling distribution closest to the teacher under a given KL budget around the student. We have revised Section 2 to make these distinctions explicit and added the baseline comparisons to Appendix D.1 (Table 6).
2. Empirical comparison. We implemented TRB and the OPD component of MiniLLM with a Qwen3-4B student (omitting MiniLLM's SFT stage for a fair comparison), using Qwen3-30B-A3B as the teacher for math/code and Qwen3-14B for ALFWorld. TRB starts with a KL budget of 0.01, which decays to zero within the first 15 updates. MiniLLM uses the hyperparameters reported in its paper, with all remaining settings matched to our main experiments. Their results are shown below. The OPD, MixOPD, and MixOPD-Ada entries use the original Table 1 runs; A6 reports the additional three-seed results.