acceptodds
Under review as a conference paper at ICLR 2027

The Design Space of On-Policy Distillation: A Unified Framework and Empirical Analysis

Abstract

On-policy distillation (OPD) transfers reasoning capabilities from a stronger teacher to a student by training on student-generated responses. Yet it remains unclear which parts of OPD are essential for effective learning and which can be simplified. We introduce a unified training objective that organizes representative OPD methods along four design axes: rollout source, trajectory weighting, token-position weighting, and local supervision loss. Guided by this framework, we conduct a systematic empirical study of over 30 OPD configurations on mathematical reasoning tasks. We find that teacher feedback remains effective on student-generated trajectories. In our setting, training only on failed trajectories matches the accuracy of using all trajectories. At the token level, a random 100-token window remains competitive with full-token supervision, while SelecTKD, which emphasizes teacher–student agreement, achieves the highest accuracy among the tested token-weighting methods. Preserving teacher probability information remains important: adaptive vocabulary allocation retains more teacher probability mass at the same target average budget, whereas replacing probabilities with ranks or top- membership lowers accuracy. Skew KL improves accuracy using a mixture of teacher and student probabilities, while hard clipping limits extreme learning signals. Finally, we combine these choices into a single OPD recipe and evaluate it on three larger teacher–student model pairs, improving accuracy by up to points and reducing total training time by . Together, our framework and findings clarify where OPD can be simplified and where richer teacher feedback remains important.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy distillation, LLM post-training, Mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_J81VOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides a unified framework for OPD, organizing existing methods along four design spaces: rollout source, trajectory weighting, token position weighting, and local supervision loss. Based on the framework, the authors evaluate 30+ OPD methods on math rasoning benchmarks. Results show that student-generated rollouts are more effective than teacher-generated ones (I think it's the very initial motivation of OPD), failure-only training can match full trajectory supervision, and selective token weighting and adaptive vocabulary allocation can improve distillation performance. By combining several design choices, they also obtain accuracy improvements of ~1% and reduce training time ~15%.

Critical strengths
  • The framework is clear and unified.
  • Empirical comparisons are extensive and comprehensive, which include over 30 configurations.
  • The paper identifies several potentially useful findings, such as the effectiveness of failure-only supervision and the random token windows, which may help guide further OPD design.
Critical weaknesses
  • Limit of novelty. The central contribution is a taxonomy of existing OPD methods, rather than a fundamentally new algorithm or theoretical insight. Many previous works (A Formula-Driven Survey and Research Agenda for On-Policy Distillation, Uni-OPD) have already explored unified formulations and design dimensions of OPD.
  • Some empirical conclusions are less informative than they appear. For example, "the student-generated rollouts are more effective" is the very initial motivation in the Thinking Machine blog of OPD. Failure-only training retains approximately 93% of trajectories, making a comparable performance to vanilla unsurprising. Similarly, SelecTKD assigns full weight to approximately 98% of tokens, so its results provide limited evidence for the effectiveness of selective supervision.
  • Limit improvements. The final combined recipe achieves only 0.18–1.29 percentage points of improvement over Vanilla OPD. The paper also lacks a comparison to stronger existing OPD baselines, which makes it difficult to assess whether the proposed combination is useful.
Questions for authors

See weakness.

8 comments
AuthorsOpenReview ↗

Response to Reviewer J81V (1/5)

W1: Limit of novelty. The central contribution is a taxonomy of existing OPD methods, rather than a fundamentally new algorithm or theoretical insight. Many previous works (A Formula-Driven Survey and Research Agenda for On-Policy Distillation, Uni-OPD) have already explored unified formulations and design dimensions of OPD.

A1: We thank the reviewer for the useful comments.

Our paper makes two novel contributions. First, we build a unified framework that isolates the effect of each design choice. Second, motivated by the analysis of this unified framework, we further design new methods, such as quantile budgeting for vocabulary allocation and random token windows for supervision placement.

The unified framework defines the OPD design space using four factors of a single training objective (Eq. (2)): rollout source, trajectory weight, token-position weight, and local loss. By changing these factors, we construct different OPD methods (Appendix A, Eqs. (7)–(68)) and evaluate each against the same Vanilla baseline. This lets us compare over 30 configurations under one protocol, from more than ten published methods and from designs we propose. We measure the effect of each design choice and analyze why a choice helps or hurts, which yields Findings 1 to 4. Existing works do not support this analysis. For example, A Formula-Driven Survey and Research Agenda for On-Policy Distillation [1] classifies OPD methods by the variables in their objectives and estimators. It also organizes the results reported in the literature. It records which variables each method changes but does not measure their effects, because it trains no models. The results it collects also come from different models, data, and training budgets, so they cannot be compared directly. Uni-OPD [2] is a unified OPD framework for training both LLMs and MLLMs rather than a study of how each component affects OPD.

We also propose several new methods for OPD based on our unified framework. For example, quantile budgeting addresses vocabulary allocation. Existing support methods such as teacher top-K [3] keep the same number of candidates at every position, although the teacher's probability mass is concentrated at some positions and spread out at others. Quantile budgeting scores each candidate token vv by max⁡(pt(v),qt(v))\max(p_t(v), q_t(v)), where ptp_t and qtq_t are the student and teacher distributions at position tt. It then applies one threshold pooled over the micro-batch, so positions receive different numbers of entries while the average stays at eight. The teacher's top token is always kept (Eqs. (47) and (48)). It therefore uses the vocabulary budget more efficiently: at the same average budget it retains 97.7% of the teacher probability mass, versus 54.7% for a fixed budget, and it is the best single design in Table 2. It is also part of the combined configuration in Section 6.

Another example concerns token placement. Inspired by existing token-selection methods, in particular ESR [4], which supervises only a prefix of each response, we asked whether the placement of a small supervision budget matters. Keeping full generation and the prefix baseline's 100-token budget, we first place the budget as a uniformly random contiguous window (random window, Eq. (28)) and then as positions drawn uniformly from each response (random scatter). They both keep the accuracy of full-token supervision. ESR [4] and a fixed suffix with the same budget fall behind.

AuthorsOpenReview ↗

Response to Reviewer J81V (2/5)

(A1, continued)

We add additional analysis of why random placement works. Consider a batch of NN responses, where response ii has Ti≥KT_i \ge K tokens and KK of its positions are supervised (K=100K = 100). Let mit∈0,1m_{it} \in \\{0,1\\} indicate whether position tt of response ii is supervised, and let ℓit\ell_{it} be the local loss at that position. Our default token-mean aggregation (Eq. (69)) gives the batch loss

L^=∑i,tmit,ℓit∑j,umju,∑tmit=K.\hat{L} = \frac{\sum_{i,t} m_{it}\\,\ell_{it}}{\sum_{j,u} m_{ju}}, \qquad \sum_{t} m_{it} = K.

Random scatter draws KK positions uniformly from each response, so Pr⁡(mit=1)=K/Ti\Pr(m_{it}=1) = K/T_i and the denominator equals NKNK. Taking the expectation EU\mathbb{E}_U of the resulting loss L^scatter\hat{L}_{\text{scatter}} over the randomly drawn positions gives

EU[L^scatter]=1NK∑i∑t=1TiKTi,ℓit=1N∑i1Ti∑t=1Tiℓit=Lseq,\mathbb{E}_{U}\big[\hat{L}_{\text{scatter}}\big] = \frac{1}{NK}\sum_{i}\sum_{t=1}^{T_i}\frac{K}{T_i}\\,\ell_{it} = \frac{1}{N}\sum_{i}\frac{1}{T_i}\sum_{t=1}^{T_i}\ell_{it} = L^{\text{seq}},

which is the full-token objective with sequence-mean aggregation (Eq. (70)). The positions are drawn independently of the student parameters θ\theta and the denominators are fixed during each update (Appendix B.1), so the same identity holds for the gradient. Sparsity therefore adds sampling noise without changing the target, which is why random scatter matches full-token supervision with the same sequence-mean aggregation. For the random window, which supervises KK consecutive positions at a uniformly random offset, and for Ti≥2KT_i \ge 2K,

Pr⁡(mit=1)=min⁡(t,,K,,Ti−t+1)Ti−K+1,\Pr(m_{it}=1) = \frac{\min(t,\\,K,\\,T_i-t+1)}{T_i-K+1},

which equals K/(Ti−K+1)K/(T_i-K+1) for K≤t≤Ti−K+1K \le t \le T_i-K+1. The window therefore weights interior positions by 1/(Ti−K+1)1/(T_i-K+1) instead of 1/Ti1/T_i and gives less weight to the first and last K−1K-1 positions, so it approximates LseqL^{\text{seq}} when Ti≫KT_i \gg K. A fixed prefix or suffix sets mit=0m_{it}=0 for every position outside its range, so its expected objective differs from full supervision, which matches its lower accuracy. These results support Finding 3: where supervision is applied matters, and a small budget is enough when the supervised positions are chosen at random.

We will revise the paper to include these contributions and the comparison with prior work.

W2: Some empirical conclusions are less informative than they appear. For example, “the student-generated rollouts are more effective” is the very initial motivation in the Thinking Machine blog of OPD. Failure-only training retains approximately 93% of trajectories, making a comparable performance to vanilla unsurprising. Similarly, SelecTKD assigns full weight to approximately 98% of tokens, so its results provide limited evidence for the effectiveness of selective supervision.

A2: We thank the reviewer for these comments on our findings.

Student-generated rollouts. Finding 1 does not restate the motivation of OPD. Finding 1 shows that mixing teacher-generated content into student rollouts does not improve performance over student-only rollouts. We test three mixing schemes from prior work: fixed teacher mixing (GKD [5]), DAgger-style annealing [6, 7], and AggreVaTe-style handoff [7, 8]. None outperforms student-only rollouts. This differs from the Thinking Machines blog [9], which compares on-policy distillation with off-policy distillation and RL but does not examine teacher–student rollout mixing. We will revise Finding 1 to make this distinction clear.

Failure-only training. We agree with the reviewer that failure-only training leaves out only a small share of trajectories: it retains 92.3% of them. While it reduces the time spent on student updates, generation time remains unchanged because all trajectories must still be generated and verified. Therefore, failure-only selection barely changes the training time. We then revise Finding 2 to state that trajectory selection is not needed, and no longer apply failure-only selection: all trajectories are used, as in Vanilla. Following the revised Finding 2, we retrained the combined configuration in Section 6 without trajectory selection, as in Vanilla, and with three seeds. This improves its accuracy and leaves its training time unchanged; the updated Table 4 is shown in A3.

SelecTKD. SelecTKD [10] improves average accuracy from 32.82% to 34.19% by reducing only 2.34% of token positions, while keeping full weight on the remaining 97.66% (Appendix B.3). These positions are where the student's top choice falls outside the teacher's top five. This shows that even a small number of disagreement positions can affect training performance, and that selective weighting can help.

AuthorsOpenReview ↗

Response to Reviewer J81V (3/5)

W3: Limit improvements. The final combined recipe achieves only 0.18–1.29 percentage points of improvement over Vanilla OPD. The paper also lacks a comparison to stronger existing OPD baselines, which makes it difficult to assess whether the proposed combination is useful.

A3: We thank the reviewer for the comments. We add the standard deviation across seeds to every table in Sections 5 and 6, as shown below for Table 4:

Teacher → studentVanilla acc. (%)Ours acc. (%)ΔVanilla time (h)Ours time (h)Speedup
8B → 4B42.25 ± 0.4342.97 ± 0.36+0.7212.4011.051.12×
32B → 4B40.71 ± 0.3141.23 ± 0.33+0.529.307.421.25×
32B → 8B43.15 ± 0.1244.65 ± 0.25+1.5012.3810.331.20×

Following the revised Finding 2, we retrained the combined configuration without trajectory selection, as in Vanilla, which improves its accuracy while leaving its training time unchanged.

Each evaluation in Table 4 averages 32 samples per problem on AIME24, AIME25, and AMC23 and 3 samples on Minerva and MATH500, so the reported accuracy does not depend on a single sampled answer. Our method improves accuracy over Vanilla on all three teacher-student pairs, with an average gain of 0.91 percentage points (p=0.046p=0.046). It also achieves 1.12×-1.25× training speedups, with an overall geometric mean speedup of 1.19× (p=0.016p=0.016). Both pp-values are from one-sided paired tt-tests across the three configurations.

Section 5 already compares the representative existing methods for each component, and we select the best of them to construct the final combined configuration. The comparisons in our unified framework are reliable because we use a commonly evaluated Qwen student–teacher pair [2, 3, 7, 9, 11–19]. All methods share the same training configuration, and results are averaged over multiple seeds. We compare the combined configuration only with Vanilla for two reasons. First, Vanilla is a strong baseline that already outperforms many existing methods. Second, every method in Section 5 is compared with Vanilla under the same protocol, so Vanilla is the common reference for all of them.

We further run instruction-following experiments to test the findings beyond mathematical reasoning. We keep the main pair (Qwen3-4B-Instruct-2507 teacher, Qwen3-1.7B-Base student) and the main training settings. We use Nemotron-RL-instruction_following [20] as the training set and evaluate on IFEval [21] and IFBench [22], reporting strict prompt-level accuracy averaged over four samples per prompt. We train Vanilla and four single designs (SelecTKD, quantile budgeting, Skew KL, and hard clipping) with three seeds each. All four designs match or exceed Vanilla on both benchmarks, with quantile budgeting again the strongest.

MethodIFEvalIFBench
Vanilla54.57 ±1.1019.75 ±0.72
SelecTKD55.63 ±1.1820.47 ±0.92
Quantile budgeting56.38 ±0.9721.42 ±0.84
Skew KL55.32 ±1.2019.99 ±1.32
Hard clipping54.62 ±0.6021.42 ±0.74

Table 2 with the standard deviation across seeds is shown below.

Reviewer_nth3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a unified framework for analyzing the design space of on-policy distillation (OPD), organizing existing methods into four components: rollout source, trajectory weighting, token-position weighting, and local supervision loss. Based on this formulation, the authors conduct an empirical study of over 30 OPD configurations on mathematical reasoning benchmarks to investigate the effects of different training choices on model accuracy and efficiency. The analysis covers student- and teacher-generated trajectories, selective supervision, vocabulary allocation, teacher information preservation, and distillation objectives. Based on the empirical findings, the authors combine several design choices into a training configuration and evaluate it on three larger teacher–student model pairs.

Critical strengths
  1. The paper provides a unified perspective on different OPD design choices. By decomposing existing approaches into four components, the proposed framework offers a structured way to compare different training strategies and identify their respective contributions to student learning.
  2. The paper includes a relatively comprehensive empirical comparison. The evaluation covers over 30 configurations, multiple mathematical reasoning benchmarks, and additional teacher–student model pairs. The analyses of trajectory selection, token weighting, vocabulary allocation, and training efficiency provide empirical observations about the behavior of different OPD strategies.
Critical weaknesses
  1. The proposed unified framework mainly reorganizes existing OPD methods into four design dimensions, without introducing a fundamentally new algorithm or optimization principle. The final training recipe also combines previously proposed techniques, such as SelecTKD, Skew KL, and selective supervision. The paper should better clarify its novel contributions beyond empirical comparisons and the integration of existing methods.
  2. The experiments focus exclusively on mathematical reasoning tasks using Qwen3 teacher–student pairs. It remains unclear whether the identified design principles generalize to other model families or capabilities such as coding, instruction following, and general knowledge. Broader evaluations are needed before drawing general conclusions about OPD design choices.
  3. Although the paper systematically evaluates individual OPD components, most comparisons vary one design choice at a time. The final configuration combines several choices, but their interactions are not sufficiently investigated. A more comprehensive factorial analysis or progressive combination ablation would help determine whether the selected components provide complementary benefits or introduce conflicting optimization effects.
  4. For example, the paper suggests that failure-only training can achieve comparable accuracy while reducing supervision, but this setting retains 92.3% of trajectories, making the practical benefit relatively limited. Similarly, the superiority of several design choices may depend on the specific training configuration. Additional experiments with controlled supervision budgets and different training conditions would strengthen these conclusions.
  5. Although the individual components are compared against representative existing approaches, the final combined configuration is primarily evaluated against Vanilla OPD. Comparisons with complete, competitive OPD training recipes under matched computational budgets would be necessary to establish whether the proposed configuration provides meaningful advantages over current approaches.
  6. In Table 3, the combined recipe improves accuracy by only 0.18–1.29 percentage points across the three larger model pairs. Although training-time reductions are reported, the paper does not sufficiently establish the statistical reliability of these final improvements. Additional repeated experiments and a more detailed accuracy–efficiency trade-off analysis would strengthen the practical significance of the proposed recipe.
Questions for authors

please see the above comments.

8 comments
AuthorsOpenReview ↗

Response to Reviewer nth3 (1/7)

W1: The proposed unified framework mainly reorganizes existing OPD methods into four design dimensions, without introducing a fundamentally new algorithm or optimization principle. The final training recipe also combines previously proposed techniques, such as SelecTKD, Skew KL, and selective supervision. The paper should better clarify its novel contributions beyond empirical comparisons and the integration of existing methods.

A1: We thank the reviewer for this comment.

Our paper makes two novel contributions. First, we build a unified framework that isolates the effect of each design choice. Second, motivated by the analysis of this unified framework, we further design new methods, such as quantile budgeting for vocabulary allocation and random token windows for supervision placement.

The unified framework defines the OPD design space using four factors of a single training objective (Eq. (2)): rollout source, trajectory weight, token-position weight, and local loss. By changing these factors, we construct different OPD methods (Appendix A, Eqs. (7)–(68)) and evaluate each against the same Vanilla baseline. This lets us compare over 30 configurations under one protocol, from more than ten published methods and from designs we propose. We measure the effect of each design choice and analyze why a choice helps or hurts, which yields Findings 1 to 4. Existing works do not support this analysis. For example, A Formula-Driven Survey and Research Agenda for On-Policy Distillation [1] classifies OPD methods by the variables in their objectives and estimators. It also organizes the results reported in the literature. It records which variables each method changes but does not measure their effects, because it trains no models. The results it collects also come from different models, data, and training budgets, so they cannot be compared directly. Uni-OPD [2] is a unified OPD framework for training both LLMs and MLLMs rather than a study of how each component affects OPD.

We also propose several new methods for OPD based on our unified framework. For example, quantile budgeting addresses vocabulary allocation. Existing support methods such as teacher top-K [3] keep the same number of candidates at every position, although the teacher's probability mass is concentrated at some positions and spread out at others. Quantile budgeting scores each candidate token vv by max⁡(pt(v),qt(v))\max(p_t(v), q_t(v)), where ptp_t and qtq_t are the student and teacher distributions at position tt. It then applies one threshold pooled over the micro-batch, so positions receive different numbers of entries while the average stays at eight. The teacher's top token is always kept (Eqs. (47) and (48)). It therefore uses the vocabulary budget more efficiently: at the same average budget it retains 97.7% of the teacher probability mass, versus 54.7% for a fixed budget, and it is the best single design in Table 2. It is also part of the combined configuration in Section 6.

Another example concerns token placement. Inspired by existing token-selection methods, in particular ESR [4], which supervises only a prefix of each response, we asked whether the placement of a small supervision budget matters. Keeping full generation and the prefix baseline's 100-token budget, we first place the budget as a uniformly random contiguous window (random window, Eq. (28)) and then as positions drawn uniformly from each response (random scatter). They both keep the accuracy of full-token supervision. ESR [4] and a fixed suffix with the same budget fall behind.

AuthorsOpenReview ↗

Response to Reviewer nth3 (2/7)

(A1, continued)

We add additional analysis of why random placement works. Consider a batch of NN responses, where response ii has Ti≥KT_i \ge K tokens and KK of its positions are supervised (K=100K = 100). Let mit∈0,1m_{it} \in \\{0,1\\} indicate whether position tt of response ii is supervised, and let ℓit\ell_{it} be the local loss at that position. Our default token-mean aggregation (Eq. (69)) gives the batch loss

L^=∑i,tmit,ℓit∑j,umju,∑tmit=K.\hat{L} = \frac{\sum_{i,t} m_{it}\\,\ell_{it}}{\sum_{j,u} m_{ju}}, \qquad \sum_{t} m_{it} = K.

Random scatter draws KK positions uniformly from each response, so Pr⁡(mit=1)=K/Ti\Pr(m_{it}=1) = K/T_i and the denominator equals NKNK. Taking the expectation EU\mathbb{E}_U of the resulting loss L^scatter\hat{L}_{\text{scatter}} over the randomly drawn positions gives

EU[L^scatter]=1NK∑i∑t=1TiKTi,ℓit=1N∑i1Ti∑t=1Tiℓit=Lseq,\mathbb{E}_{U}\big[\hat{L}_{\text{scatter}}\big] = \frac{1}{NK}\sum_{i}\sum_{t=1}^{T_i}\frac{K}{T_i}\\,\ell_{it} = \frac{1}{N}\sum_{i}\frac{1}{T_i}\sum_{t=1}^{T_i}\ell_{it} = L^{\text{seq}},

which is the full-token objective with sequence-mean aggregation (Eq. (70)). The positions are drawn independently of the student parameters θ\theta and the denominators are fixed during each update (Appendix B.1), so the same identity holds for the gradient. Sparsity therefore adds sampling noise without changing the target, which is why random scatter matches full-token supervision with the same sequence-mean aggregation. For the random window, which supervises KK consecutive positions at a uniformly random offset, and for Ti≥2KT_i \ge 2K,

Pr⁡(mit=1)=min⁡(t,,K,,Ti−t+1)Ti−K+1,\Pr(m_{it}=1) = \frac{\min(t,\\,K,\\,T_i-t+1)}{T_i-K+1},

which equals K/(Ti−K+1)K/(T_i-K+1) for K≤t≤Ti−K+1K \le t \le T_i-K+1. The window therefore weights interior positions by 1/(Ti−K+1)1/(T_i-K+1) instead of 1/Ti1/T_i and gives less weight to the first and last K−1K-1 positions, so it approximates LseqL^{\text{seq}} when Ti≫KT_i \gg K. A fixed prefix or suffix sets mit=0m_{it}=0 for every position outside its range, so its expected objective differs from full supervision, which matches its lower accuracy. These results support Finding 3: where supervision is applied matters, and a small budget is enough when the supervised positions are chosen at random.

We will revise the paper to include these contributions and the comparison with prior work.

W2: The experiments focus exclusively on mathematical reasoning tasks using Qwen3 teacher–student pairs. It remains unclear whether the identified design principles generalize to other model families or capabilities such as coding, instruction following, and general knowledge. Broader evaluations are needed before drawing general conclusions about OPD design choices.

A2: We thank the reviewer for this comment.

We follow the standard practice in OPD research, where Qwen-family teacher-student pairs are the default choice: of the 16 OPD studies we cite, 13 use only Qwen-based pairs [2, 3, 5–15]. Qwen has become the default for two reasons. First, token-level OPD compares teacher and student distributions over a shared vocabulary, and Qwen3 provides base and post-trained models at many sizes with one tokenizer, so teacher and student sizes can be varied without re-tokenization. Second, many widely used open reasoning models, such as DeepSeek-R1-Distill-Qwen and OpenMath-Nemotron, are built on Qwen, so strong teachers and trainable students come from the same family. Mathematical reasoning is likewise the standard domain: several recent OPD studies train only on math [8, 12, 14–16]. Our study already used more than 15,000 GPU hours of training and evaluation. We also add the standard deviation across seeds to the main results in Table 2, shown below.

AuthorsOpenReview ↗

Response to Reviewer nth3 (3/7)

(A2, continued)

MethodAIME24AIME25AMC23MinervaMATH500Average
Vanilla5.69 ± 0.697.60 ± 0.3142.79 ± 0.5215.28 ± 0.2666.56 ± 1.1432.82 ± 0.11
(a) Rollout source
Fixed mixing3.80 ± 0.566.20 ± 0.5034.70 ± 0.4714.50 ± 0.7864.10 ± 0.6929.55 ± 0.20
Teacher-only0.00 ± 0.001.20 ± 0.1625.60 ± 0.4712.00 ± 0.4548.90 ± 0.8021.80 ± 0.11
DAgger5.40 ± 0.836.70 ± 0.6237.50 ± 1.0415.70 ± 0.9065.50 ± 0.6731.19 ± 0.26
AggreVaTe5.00 ± 0.587.10 ± 0.5340.90 ± 0.6713.60 ± 0.7264.20 ± 0.2731.20 ± 0.27
(b) Trajectory weighting
Correct-only5.00 ± 1.425.00 ± 1.0839.40 ± 1.3715.10 ± 1.3062.60 ± 1.9830.51 ± 1.12
Failure-only5.56 ± 0.897.64 ± 0.3343.23 ± 0.3915.36 ± 0.3966.09 ± 0.4932.82 ± 0.36
FiRe5.00 ± 0.217.53 ± 0.5742.40 ± 1.1815.40 ± 0.9467.16 ± 0.6032.80 ± 0.44
RG-OPD5.45 ± 0.577.43 ± 0.6943.31 ± 0.7615.07 ± 0.7566.38 ± 0.3232.80 ± 0.53
(c) Token-position weighting
Prefix3.80 ± 0.616.70 ± 0.5844.10 ± 0.7312.70 ± 0.7761.50 ± 0.8030.87 ± 0.31
Suffix0.40 ± 0.322.10 ± 0.4723.40 ± 1.0411.50 ± 0.8449.50 ± 1.2021.44 ± 0.40
Random window6.20 ± 0.299.20 ± 0.5242.50 ± 0.8717.20 ± 0.3867.10 ± 0.4333.62 ± 0.22
Random scatter7.10 ± 0.597.50 ± 0.3339.10 ± 0.5115.90 ± 0.6865.80 ± 0.6732.02 ± 0.15
TIP5.56 ± 0.517.67 ± 0.7743.02 ± 0.6615.36 ± 0.6266.47 ± 0.1832.87 ± 0.23
SelecTKD6.70 ± 1.165.40 ± 0.6643.10 ± 0.8416.90 ± 0.7870.70 ± 0.5134.19 ± 0.26
TA-OPD6.70 ± 0.395.40 ± 0.3040.30 ± 0.6516.40 ± 0.4462.90 ± 0.4731.41 ± 0.14
(d) Local supervision loss
Vocabulary support and budget allocation
Teacher top-K1.15 ± 0.541.98 ± 0.3828.20 ± 0.9512.70 ± 0.1449.64 ± 1.2523.03 ± 0.51
Intersection7.10 ± 0.487.10 ± 0.4743.10 ± 0.7115.40 ± 0.7268.20 ± 0.8633.46 ± 0.17
Union (FKL)6.20 ± 0.557.90 ± 0.5841.60 ± 0.7715.70 ± 0.8167.10 ± 0.6132.85 ± 0.16
Union (RKL)4.20 ± 0.332.90 ± 0.3431.20 ± 0.7011.60 ± 0.8346.90 ± 0.7623.33 ± 0.31
Student-tail7.10 ± 0.455.80 ± 0.3741.90 ± 0.7017.60 ± 0.3769.70 ± 1.4533.93 ± 0.21
Fixed budget5.40 ± 0.427.10 ± 0.4843.80 ± 0.8315.20 ± 1.1867.60 ± 0.8033.20 ± 0.53
Quantile budgeting7.90 ± 0.6610.00 ± 0.3245.00 ± 0.7618.10 ± 0.6369.90 ± 0.5635.49 ± 0.14
Teacher information
Standardized values3.23 ± 0.315.97 ± 0.7837.97 ± 0.5914.09 ± 0.7763.84 ± 0.3330.13 ± 0.06
Rank only1.01 ± 0.261.46 ± 0.4821.72 ± 0.6911.23 ± 1.2345.84 ± 0.7520.01 ± 0.47
Membership only3.80 ± 0.573.30 ± 0.3931.60 ± 0.8910.90 ± 0.9556.30 ± 0.5025.57 ± 0.46
Distillation loss
Skew KL7.90 ± 0.648.30 ± 0.4543.10 ± 1.4017.00 ± 0.5069.90 ± 0.4834.54 ± 0.34
Forward KL2.90 ± 0.427.90 ± 0.3544.40 ± 0.7416.50 ± 0.5668.60 ± 0.6933.73 ± 0.53
JSD6.98 ± 0.428.23 ± 0.2842.97 ± 0.6715.44 ± 0.5669.31 ± 0.1433.83 ± 0.26
EOPD (adapted)5.97 ± 0.428.33 ± 0.4242.58 ± 2.1913.93 ± 0.9464.87 ± 2.2932.13 ± 1.35
Squared log-ratio5.45 ± 0.327.05 ± 0.3742.06 ± 1.1314.91 ± 1.0467.31 ± 0.4732.63 ± 0.26
OPD + stop-prob. distillation6.20 ± 0.568.80 ± 0.5445.00 ± 0.6616.30 ± 0.6769.00 ± 0.4534.45 ± 0.13
Update strength
Hard clipping8.30 ± 0.608.30 ± 0.4941.90 ± 1.1015.90 ± 0.7869.60 ± 0.7133.93 ± 0.38
Positive-side clipping2.95 ± 0.423.99 ± 0.6134.58 ± 1.2713.52 ± 0.7457.60 ± 0.4027.29 ± 0.61
Soft-log compression6.35 ± 0.837.36 ± 0.3243.85 ± 0.3315.97 ± 0.7168.98 ± 0.9533.92 ± 0.55
Power credit6.20 ± 0.759.60 ± 0.5443.10 ± 0.5615.00 ± 0.6469.30 ± 0.5833.83 ± 0.16
Tanh transform4.13 ± 0.613.85 ± 0.2833.70 ± 0.5913.28 ± 0.9558.33 ± 0.4227.33 ± 0.21
Reviewer_oLfjOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides a unified framework spanning supervision data generation, trajectory- or token-level reweighting, and the choice of distillation loss for on-policy distillation. Key findings include the importance of teacher feedback on student-generated trajectories and agreement between teacher and student probabilities.

Critical strengths
  • The findings are informative and supported by technical analysis and experimental evidence.
  • The paper provides a good-quality survey of existing methods.
  • The paper is well structured.
Critical weaknesses
  • Table 1: The rollout sources should also include methods that interleave generation between student and teacher models, such as Speculative Knowledge Distillation. Such methods would also be relevant to the experimental comparisons in Table 2.
  • The conclusions are mainly based on one student–teacher model configuration. I think the key findings still need to be validated across different student–teacher model families.
  • For each method evaluated in Table 2, an explicit instantiation of its objective, or at least its key components, should be provided in the main paper to clarify what changes across methods. For instance, the concrete definition of ℓSquant\ell^{\mathrm{Squant}}. The layout of Table 1 is good, but its framework is not explicitly mapped to each method in Table 2.
Questions for authors
  • What is the performance gap between the initial student and teacher models? I suggest reporting teacher performance as a reference point in Table 2. This would help contextualize the gains from self versus teacher-provided supervision.
  • Another important question is whether the gap between student and teacher models affects the best distillation strategy. For example, distillation between models from the same family may be easier than between models from more divergent families. This aspect was not investigated in this work.
6 comments
AuthorsOpenReview ↗

Response to Reviewer oLfj (1/4)

W1: Table 1: The rollout sources should also include methods that interleave generation between student and teacher models, such as Speculative Knowledge Distillation. Such methods would also be relevant to the experimental comparisons in Table 2.

A1: We thank the reviewer for pointing us to Speculative Knowledge Distillation (SKD) [1]. SKD fits our framework as a rollout source. The student generates each response. When it proposes a token outside the teacher's top-KK tokens, the teacher replaces it with a sampled token, and the student continues from the corrected prefix. Teacher and student tokens are thus mixed within a response, and the amount of teacher assistance adapts to the student. We follow the SKD paper for the sampling rule (K=25K=25, with the teacher sampling at temperature 0.2) and keep all other settings as in our main experiments. We train this SKD entry with three seeds. Compared with the mixing methods in Table 2(a), SKD achieves the highest accuracy but still underperforms Vanilla. Therefore, our main findings still hold. We will discuss SKD in the introduction, related work, method, and experiment sections of the revision.

MethodAIME24AIME25AMC23MinervaMATH500Average
Vanilla5.69 ± 0.697.60 ± 0.3142.79 ± 0.5215.28 ± 0.2666.56 ± 1.1432.82 ± 0.11
SKD5.34 ± 0.517.12 ± 0.4741.00 ± 0.9214.20 ± 0.6665.80 ± 0.5431.81 ± 0.31

W2: The conclusions are mainly based on one student–teacher model configuration. I think the key findings still need to be validated across different student–teacher model families.

A2: We thank the reviewer for this comment.

We follow the standard practice in OPD research, where Qwen-family teacher-student pairs are the default choice: of the 16 OPD studies we cite, 13 use only Qwen-based pairs [2–14]. Qwen has become the default for two reasons. First, token-level OPD compares teacher and student distributions over a shared vocabulary (Section 3.1), and Qwen3 provides base and post-trained models at many sizes with one tokenizer, so teacher and student sizes can be varied without re-tokenization. Second, many widely used open reasoning models, such as DeepSeek-R1-Distill-Qwen and OpenMath-Nemotron, are built on Qwen, so strong teachers and trainable students come from the same family. Mathematical reasoning is likewise the standard domain: several recent OPD studies train only on math [6, 10, 12, 14, 15]. Our study already used more than 15,000 GPU hours of training and evaluation.

We also test whether the findings hold at larger model scales. Section 6 applies the combined configuration to three larger Qwen3 pairs (8B → 4B, 32B → 4B, and 32B → 8B), so that, with the main pair, students range from 1.7B to 8B parameters and teachers from 4B to 32B. Failure-only selection retains 92.3% of trajectories, and because all trajectories must still be generated and verified, it barely changes the training time. We therefore revise Finding 2 to state that trajectory selection is not needed. Following the revised Finding 2, we retrained the combined configuration in Section 6 without trajectory selection, as in Vanilla, and with three seeds. This improves its accuracy and leaves its training time unchanged. On all three pairs, the configuration assembled from Findings 1 to 4 improves accuracy over Vanilla while reducing training time (Table 4), validating our findings across model scales.

Teacher → studentVanilla acc. (%)Ours acc. (%)ΔVanilla time (h)Ours time (h)Speedup
8B → 4B42.25 ± 0.4342.97 ± 0.36+0.7212.4011.051.12×
32B → 4B40.71 ± 0.3141.23 ± 0.33+0.529.307.421.25×
32B → 8B43.15 ± 0.1244.65 ± 0.25+1.5012.3810.331.20×

We further run instruction-following experiments to test the findings beyond mathematical reasoning. We keep the main pair (Qwen3-4B-Instruct-2507 teacher, Qwen3-1.7B-Base student) and the main training settings. We use Nemotron-RL-instruction_following [16] as the training set and evaluate on IFEval [17] and IFBench [18], reporting strict prompt-level accuracy averaged over four samples per prompt. We train Vanilla and four single designs (SelecTKD, quantile budgeting, Skew KL, and hard clipping) with three seeds each. All four designs match or exceed Vanilla on both benchmarks, with quantile budgeting again the strongest.

MethodIFEvalIFBench
Vanilla54.57 ±1.1019.75 ±0.72
SelecTKD55.63 ±1.1820.47 ±0.92
Quantile budgeting56.38 ±0.9721.42 ±0.84
Skew KL55.32 ±1.2019.99 ±1.32
Hard clipping54.62 ±0.6021.42 ±0.74

We also add the standard deviation across seeds to the main results in Table 2, shown below.

AuthorsOpenReview ↗

Response to Reviewer oLfj (2/4)

(A2, continued)

MethodAIME24AIME25AMC23MinervaMATH500Average
Vanilla5.69 ± 0.697.60 ± 0.3142.79 ± 0.5215.28 ± 0.2666.56 ± 1.1432.82 ± 0.11
(a) Rollout source
Fixed mixing3.80 ± 0.566.20 ± 0.5034.70 ± 0.4714.50 ± 0.7864.10 ± 0.6929.55 ± 0.20
Teacher-only0.00 ± 0.001.20 ± 0.1625.60 ± 0.4712.00 ± 0.4548.90 ± 0.8021.80 ± 0.11
DAgger5.40 ± 0.836.70 ± 0.6237.50 ± 1.0415.70 ± 0.9065.50 ± 0.6731.19 ± 0.26
AggreVaTe5.00 ± 0.587.10 ± 0.5340.90 ± 0.6713.60 ± 0.7264.20 ± 0.2731.20 ± 0.27
(b) Trajectory weighting
Correct-only5.00 ± 1.425.00 ± 1.0839.40 ± 1.3715.10 ± 1.3062.60 ± 1.9830.51 ± 1.12
Failure-only5.56 ± 0.897.64 ± 0.3343.23 ± 0.3915.36 ± 0.3966.09 ± 0.4932.82 ± 0.36
FiRe5.00 ± 0.217.53 ± 0.5742.40 ± 1.1815.40 ± 0.9467.16 ± 0.6032.80 ± 0.44
RG-OPD5.45 ± 0.577.43 ± 0.6943.31 ± 0.7615.07 ± 0.7566.38 ± 0.3232.80 ± 0.53
(c) Token-position weighting
Prefix3.80 ± 0.616.70 ± 0.5844.10 ± 0.7312.70 ± 0.7761.50 ± 0.8030.87 ± 0.31
Suffix0.40 ± 0.322.10 ± 0.4723.40 ± 1.0411.50 ± 0.8449.50 ± 1.2021.44 ± 0.40
Random window6.20 ± 0.299.20 ± 0.5242.50 ± 0.8717.20 ± 0.3867.10 ± 0.4333.62 ± 0.22
Random scatter7.10 ± 0.597.50 ± 0.3339.10 ± 0.5115.90 ± 0.6865.80 ± 0.6732.02 ± 0.15
TIP5.56 ± 0.517.67 ± 0.7743.02 ± 0.6615.36 ± 0.6266.47 ± 0.1832.87 ± 0.23
SelecTKD6.70 ± 1.165.40 ± 0.6643.10 ± 0.8416.90 ± 0.7870.70 ± 0.5134.19 ± 0.26
TA-OPD6.70 ± 0.395.40 ± 0.3040.30 ± 0.6516.40 ± 0.4462.90 ± 0.4731.41 ± 0.14
(d) Local supervision loss
Vocabulary support and budget allocation
Teacher top-K1.15 ± 0.541.98 ± 0.3828.20 ± 0.9512.70 ± 0.1449.64 ± 1.2523.03 ± 0.51
Intersection7.10 ± 0.487.10 ± 0.4743.10 ± 0.7115.40 ± 0.7268.20 ± 0.8633.46 ± 0.17
Union (FKL)6.20 ± 0.557.90 ± 0.5841.60 ± 0.7715.70 ± 0.8167.10 ± 0.6132.85 ± 0.16
Union (RKL)4.20 ± 0.332.90 ± 0.3431.20 ± 0.7011.60 ± 0.8346.90 ± 0.7623.33 ± 0.31
Student-tail7.10 ± 0.455.80 ± 0.3741.90 ± 0.7017.60 ± 0.3769.70 ± 1.4533.93 ± 0.21
Fixed budget5.40 ± 0.427.10 ± 0.4843.80 ± 0.8315.20 ± 1.1867.60 ± 0.8033.20 ± 0.53
Quantile budgeting7.90 ± 0.6610.00 ± 0.3245.00 ± 0.7618.10 ± 0.6369.90 ± 0.5635.49 ± 0.14
Teacher information
Standardized values3.23 ± 0.315.97 ± 0.7837.97 ± 0.5914.09 ± 0.7763.84 ± 0.3330.13 ± 0.06
Rank only1.01 ± 0.261.46 ± 0.4821.72 ± 0.6911.23 ± 1.2345.84 ± 0.7520.01 ± 0.47
Membership only3.80 ± 0.573.30 ± 0.3931.60 ± 0.8910.90 ± 0.9556.30 ± 0.5025.57 ± 0.46
Distillation loss
Skew KL7.90 ± 0.648.30 ± 0.4543.10 ± 1.4017.00 ± 0.5069.90 ± 0.4834.54 ± 0.34
Forward KL2.90 ± 0.427.90 ± 0.3544.40 ± 0.7416.50 ± 0.5668.60 ± 0.6933.73 ± 0.53
JSD6.98 ± 0.428.23 ± 0.2842.97 ± 0.6715.44 ± 0.5669.31 ± 0.1433.83 ± 0.26
EOPD (adapted)5.97 ± 0.428.33 ± 0.4242.58 ± 2.1913.93 ± 0.9464.87 ± 2.2932.13 ± 1.35
Squared log-ratio5.45 ± 0.327.05 ± 0.3742.06 ± 1.1314.91 ± 1.0467.31 ± 0.4732.63 ± 0.26
OPD + stop-prob. distillation6.20 ± 0.568.80 ± 0.5445.00 ± 0.6616.30 ± 0.6769.00 ± 0.4534.45 ± 0.13
Update strength
Hard clipping8.30 ± 0.608.30 ± 0.4941.90 ± 1.1015.90 ± 0.7869.60 ± 0.7133.93 ± 0.38
Positive-side clipping2.95 ± 0.423.99 ± 0.6134.58 ± 1.2713.52 ± 0.7457.60 ± 0.4027.29 ± 0.61
Soft-log compression6.35 ± 0.837.36 ± 0.3243.85 ± 0.3315.97 ± 0.7168.98 ± 0.9533.92 ± 0.55
Power credit6.20 ± 0.759.60 ± 0.5443.10 ± 0.5615.00 ± 0.6469.30 ± 0.5833.83 ± 0.16
Tanh transform4.13 ± 0.613.85 ± 0.2833.70 ± 0.5913.28 ± 0.9558.33 ± 0.4227.33 ± 0.21
AuthorsOpenReview ↗

Response to Reviewer oLfj (3/4)

W3: For each method evaluated in Table 2, an explicit instantiation of its objective, or at least its key components, should be provided in the main paper to clarify what changes across methods. For instance, the concrete definition of ℓSquant\ell^{S_{\mathrm{quant}}}. The layout of Table 1 is good, but its framework is not explicitly mapped to each method in Table 2.

A3: We thank the reviewer for this suggestion. Each method in Table 2 changes one of the four factors (μk,wk,mk,t,ℓk,t)(\mu_k, w_k, m_{k,t}, \ell_{k,t}) in Eq. (2) relative to Vanilla, (πk,1,1,ℓk,tsamp)(\pi_k, 1, 1, \ell^{\mathrm{samp}}_{k,t}), and the blocks of Table 2 group the methods by the factor they change. Appendix A gives the complete objective of every method, indexed by design family in Table 5. In the revision, we will make this mapping explicit in the main paper: each row of Table 2 will cite its objective in Appendix A, and Section 5 will give the key definitions, such as ℓSquant\ell^{S_{\mathrm{quant}}}.

Q1: What is the performance gap between the initial student and teacher models? I suggest reporting teacher performance as a reference point in Table 2. This would help contextualize the gains from self versus teacher-provided supervision.

A-Q1: We thank the reviewer for this suggestion. Under our evaluation protocol (Section 4), the initial student, Qwen3-1.7B-Base, scores 14.28 and the teacher, Qwen3-4B-Instruct-2507, scores 64.37, a gap of 50.1 points:

AIME24AIME25AMC23MinervaMATH500Average
Initial student (Qwen3-1.7B-Base)1.51.119.56.330.114.28
Teacher (Qwen3-4B-Instruct-2507)42.447.593.128.990.564.37

We will add the initial student's and the teacher's scores to Table 2 as reference points.

Q2: Another important question is whether the gap between student and teacher models affects the best distillation strategy. For example, distillation between models from the same family may be easier than between models from more divergent families. This aspect was not investigated in this work.

A-Q2: We thank the reviewer for this question. Within a model family, the strategy recommended by our findings remains effective as the teacher-student gap changes. Section 6 applies the configuration selected on the main pair, whose teacher-to-student size ratio is 2.4×, to three pairs with ratios of 2× (8B → 4B), 4× (32B → 8B), and 8× (32B → 4B). With the same configuration, it improves accuracy over Vanilla on all three pairs while reducing training time (Table 4).

We agree with the reviewer that distillation within a model family is easier than across families. The main reason is tokenizer mismatch: token-level OPD compares teacher and student distributions over a shared vocabulary (Section 3.1), but models from different families usually use different tokenizers. Distilling across families therefore requires aligning the two tokenizers first, which typically requires additional training. This is beyond the scope of this paper, and we will discuss it in the Conclusion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of the Revision

We thank all reviewers for their comments. We have uploaded a revised paper, with all changes marked in purple. The main changes are listed below.

1. Speculative Knowledge Distillation (Reviewer oLfj). SKD is now a rollout source in Table 1 and Table 2(a), with its objective in Appendix A (Eq. (10a)). It is the most accurate of the mixing methods (31.81) but remains below Vanilla (32.82). The introduction, related work, and Section 5.1 discuss it.

2. Mapping between the framework and Table 2 (Reviewer oLfj). Table 2 has a new Eq. column that cites each method's objective in Appendix A, and Section 5.4 states the quantile-budgeting rule. Table 2 also reports the initial student (14.28) and the teacher (64.37) as reference points.

3. Statistical reliability (Reviewer nth3). Tables 2–4 and the result tables in the appendix now report the standard deviation over three seeds. Section 6 adds one-sided paired tt-tests across the three larger pairs: the average accuracy gain is 0.91 points (p=0.046p=0.046) and the geometric-mean speedup is 1.19× (p=0.016p=0.016).

4. Findings 1 and 2 and the combined configuration (Reviewers nth3 and J81V). Finding 1 now states that mixing teacher-generated content into student rollouts does not improve over student-only rollouts. Finding 2 now states that trajectory selection is not needed. The combined configuration therefore uses all trajectories, and we retrained it with three seeds on the three larger pairs. It gains 0.72, 0.52, and 1.50 points over Vanilla and reduces total training time by 15.5% (Table 4). The abstract, introduction, and conclusion use the updated numbers.

5. Beyond mathematical reasoning (Reviewers nth3 and oLfj). The new Section 5.5 and Table 3 report instruction-following results on IFEval and IFBench for Vanilla and four single designs. All four designs match or exceed Vanilla on both benchmarks, and quantile budgeting is again the strongest.

6. Why random placement works (Reviewers nth3 and J81V). Section 5.3 and the new Appendix D show that, in expectation, random scatter preserves the full-token objective with sequence-mean aggregation and a random window approximates it, whereas a fixed prefix or suffix changes the objective.

7. Contributions and prior work (Reviewers nth3 and J81V). The introduction now states that the framework isolates the effect of each design choice and lists the new designs it leads to, such as quantile budgeting and random token windows. Related work compares our study with the formula-driven survey of Zhang (2026) and with Uni-OPD.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.