- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper provides a unified framework for OPD, organizing existing methods along four design spaces: rollout source, trajectory weighting, token position weighting, and local supervision loss. Based on the framework, the authors evaluate 30+ OPD methods on math rasoning benchmarks. Results show that student-generated rollouts are more effective than teacher-generated ones (I think it's the very initial motivation of OPD), failure-only training can match full trajectory supervision, and selective token weighting and adaptive vocabulary allocation can improve distillation performance. By combining several design choices, they also obtain accuracy improvements of ~1% and reduce training time ~15%.
Critical strengths
- The framework is clear and unified.
- Empirical comparisons are extensive and comprehensive, which include over 30 configurations.
- The paper identifies several potentially useful findings, such as the effectiveness of failure-only supervision and the random token windows, which may help guide further OPD design.
Critical weaknesses
- Limit of novelty. The central contribution is a taxonomy of existing OPD methods, rather than a fundamentally new algorithm or theoretical insight. Many previous works (A Formula-Driven Survey and Research Agenda for On-Policy Distillation, Uni-OPD) have already explored unified formulations and design dimensions of OPD.
- Some empirical conclusions are less informative than they appear. For example, "the student-generated rollouts are more effective" is the very initial motivation in the Thinking Machine blog of OPD. Failure-only training retains approximately 93% of trajectories, making a comparable performance to vanilla unsurprising. Similarly, SelecTKD assigns full weight to approximately 98% of tokens, so its results provide limited evidence for the effectiveness of selective supervision.
- Limit improvements. The final combined recipe achieves only 0.18–1.29 percentage points of improvement over Vanilla OPD. The paper also lacks a comparison to stronger existing OPD baselines, which makes it difficult to assess whether the proposed combination is useful.
Questions for authors
See weakness.
8 comments
Response to Reviewer J81V (2/5)
(A1, continued)
We add additional analysis of why random placement works. Consider a batch of responses, where response has tokens and of its positions are supervised (). Let indicate whether position of response is supervised, and let be the local loss at that position. Our default token-mean aggregation (Eq. (69)) gives the batch loss
Random scatter draws positions uniformly from each response, so and the denominator equals . Taking the expectation of the resulting loss over the randomly drawn positions gives
which is the full-token objective with sequence-mean aggregation (Eq. (70)). The positions are drawn independently of the student parameters and the denominators are fixed during each update (Appendix B.1), so the same identity holds for the gradient. Sparsity therefore adds sampling noise without changing the target, which is why random scatter matches full-token supervision with the same sequence-mean aggregation. For the random window, which supervises consecutive positions at a uniformly random offset, and for ,
which equals for . The window therefore weights interior positions by instead of and gives less weight to the first and last positions, so it approximates when . A fixed prefix or suffix sets for every position outside its range, so its expected objective differs from full supervision, which matches its lower accuracy. These results support Finding 3: where supervision is applied matters, and a small budget is enough when the supervised positions are chosen at random.
We will revise the paper to include these contributions and the comparison with prior work.
W2: Some empirical conclusions are less informative than they appear. For example, “the student-generated rollouts are more effective” is the very initial motivation in the Thinking Machine blog of OPD. Failure-only training retains approximately 93% of trajectories, making a comparable performance to vanilla unsurprising. Similarly, SelecTKD assigns full weight to approximately 98% of tokens, so its results provide limited evidence for the effectiveness of selective supervision.
A2: We thank the reviewer for these comments on our findings.
Student-generated rollouts. Finding 1 does not restate the motivation of OPD. Finding 1 shows that mixing teacher-generated content into student rollouts does not improve performance over student-only rollouts. We test three mixing schemes from prior work: fixed teacher mixing (GKD [5]), DAgger-style annealing [6, 7], and AggreVaTe-style handoff [7, 8]. None outperforms student-only rollouts. This differs from the Thinking Machines blog [9], which compares on-policy distillation with off-policy distillation and RL but does not examine teacher–student rollout mixing. We will revise Finding 1 to make this distinction clear.
Failure-only training. We agree with the reviewer that failure-only training leaves out only a small share of trajectories: it retains 92.3% of them. While it reduces the time spent on student updates, generation time remains unchanged because all trajectories must still be generated and verified. Therefore, failure-only selection barely changes the training time. We then revise Finding 2 to state that trajectory selection is not needed, and no longer apply failure-only selection: all trajectories are used, as in Vanilla. Following the revised Finding 2, we retrained the combined configuration in Section 6 without trajectory selection, as in Vanilla, and with three seeds. This improves its accuracy and leaves its training time unchanged; the updated Table 4 is shown in A3.
SelecTKD. SelecTKD [10] improves average accuracy from 32.82% to 34.19% by reducing only 2.34% of token positions, while keeping full weight on the remaining 97.66% (Appendix B.3). These positions are where the student's top choice falls outside the teacher's top five. This shows that even a small number of disagreement positions can affect training performance, and that selective weighting can help.
Response to Reviewer J81V (3/5)
W3: Limit improvements. The final combined recipe achieves only 0.18–1.29 percentage points of improvement over Vanilla OPD. The paper also lacks a comparison to stronger existing OPD baselines, which makes it difficult to assess whether the proposed combination is useful.
A3: We thank the reviewer for the comments. We add the standard deviation across seeds to every table in Sections 5 and 6, as shown below for Table 4:
| Teacher → student | Vanilla acc. (%) | Ours acc. (%) | Δ | Vanilla time (h) | Ours time (h) | Speedup |
|---|---|---|---|---|---|---|
| 8B → 4B | 42.25 ± 0.43 | 42.97 ± 0.36 | +0.72 | 12.40 | 11.05 | 1.12× |
| 32B → 4B | 40.71 ± 0.31 | 41.23 ± 0.33 | +0.52 | 9.30 | 7.42 | 1.25× |
| 32B → 8B | 43.15 ± 0.12 | 44.65 ± 0.25 | +1.50 | 12.38 | 10.33 | 1.20× |
Following the revised Finding 2, we retrained the combined configuration without trajectory selection, as in Vanilla, which improves its accuracy while leaving its training time unchanged.
Each evaluation in Table 4 averages 32 samples per problem on AIME24, AIME25, and AMC23 and 3 samples on Minerva and MATH500, so the reported accuracy does not depend on a single sampled answer. Our method improves accuracy over Vanilla on all three teacher-student pairs, with an average gain of 0.91 percentage points (). It also achieves 1.12×-1.25× training speedups, with an overall geometric mean speedup of 1.19× (). Both -values are from one-sided paired -tests across the three configurations.
Section 5 already compares the representative existing methods for each component, and we select the best of them to construct the final combined configuration. The comparisons in our unified framework are reliable because we use a commonly evaluated Qwen student–teacher pair [2, 3, 7, 9, 11–19]. All methods share the same training configuration, and results are averaged over multiple seeds. We compare the combined configuration only with Vanilla for two reasons. First, Vanilla is a strong baseline that already outperforms many existing methods. Second, every method in Section 5 is compared with Vanilla under the same protocol, so Vanilla is the common reference for all of them.
We further run instruction-following experiments to test the findings beyond mathematical reasoning. We keep the main pair (Qwen3-4B-Instruct-2507 teacher, Qwen3-1.7B-Base student) and the main training settings. We use Nemotron-RL-instruction_following [20] as the training set and evaluate on IFEval [21] and IFBench [22], reporting strict prompt-level accuracy averaged over four samples per prompt. We train Vanilla and four single designs (SelecTKD, quantile budgeting, Skew KL, and hard clipping) with three seeds each. All four designs match or exceed Vanilla on both benchmarks, with quantile budgeting again the strongest.
| Method | IFEval | IFBench |
|---|---|---|
| Vanilla | 54.57 ±1.10 | 19.75 ±0.72 |
| SelecTKD | 55.63 ±1.18 | 20.47 ±0.92 |
| Quantile budgeting | 56.38 ±0.97 | 21.42 ±0.84 |
| Skew KL | 55.32 ±1.20 | 19.99 ±1.32 |
| Hard clipping | 54.62 ±0.60 | 21.42 ±0.74 |
Table 2 with the standard deviation across seeds is shown below.
Response to Reviewer J81V (1/5)
A1: We thank the reviewer for the useful comments.
Our paper makes two novel contributions. First, we build a unified framework that isolates the effect of each design choice. Second, motivated by the analysis of this unified framework, we further design new methods, such as quantile budgeting for vocabulary allocation and random token windows for supervision placement.
The unified framework defines the OPD design space using four factors of a single training objective (Eq. (2)): rollout source, trajectory weight, token-position weight, and local loss. By changing these factors, we construct different OPD methods (Appendix A, Eqs. (7)–(68)) and evaluate each against the same Vanilla baseline. This lets us compare over 30 configurations under one protocol, from more than ten published methods and from designs we propose. We measure the effect of each design choice and analyze why a choice helps or hurts, which yields Findings 1 to 4. Existing works do not support this analysis. For example, A Formula-Driven Survey and Research Agenda for On-Policy Distillation [1] classifies OPD methods by the variables in their objectives and estimators. It also organizes the results reported in the literature. It records which variables each method changes but does not measure their effects, because it trains no models. The results it collects also come from different models, data, and training budgets, so they cannot be compared directly. Uni-OPD [2] is a unified OPD framework for training both LLMs and MLLMs rather than a study of how each component affects OPD.
We also propose several new methods for OPD based on our unified framework. For example, quantile budgeting addresses vocabulary allocation. Existing support methods such as teacher top-K [3] keep the same number of candidates at every position, although the teacher's probability mass is concentrated at some positions and spread out at others. Quantile budgeting scores each candidate token v by max(pt(v),qt(v)), where pt and qt are the student and teacher distributions at position t. It then applies one threshold pooled over the micro-batch, so positions receive different numbers of entries while the average stays at eight. The teacher's top token is always kept (Eqs. (47) and (48)). It therefore uses the vocabulary budget more efficiently: at the same average budget it retains 97.7% of the teacher probability mass, versus 54.7% for a fixed budget, and it is the best single design in Table 2. It is also part of the combined configuration in Section 6.
Another example concerns token placement. Inspired by existing token-selection methods, in particular ESR [4], which supervises only a prefix of each response, we asked whether the placement of a small supervision budget matters. Keeping full generation and the prefix baseline's 100-token budget, we first place the budget as a uniformly random contiguous window (random window, Eq. (28)) and then as positions drawn uniformly from each response (random scatter). They both keep the accuracy of full-token supervision. ESR [4] and a fixed suffix with the same budget fall behind.