acceptodds
Under review as a conference paper at ICLR 2027

Bootstrapped On-Policy Distillation

11 views

Abstract

On-policy distillation (OPD) improves language models by distilling teacher predictions on student-generated trajectories, while on-policy self-distillation (OPSD) removes the need for an external teacher by using a reference-conditioned copy of the model as the teacher. Despite its great early improvements, we find that OPSD does not remain consistently effective under continued training and can eventually suffer substantial performance degradation. We study this training behavior and link the degradation to a growing mismatch between the evolving reference-free student and the frozen privileged teacher, together with increasing student uncertainty on its evolving on-policy states. This growing mismatch motivates us to first learn from privileged reference supervision and then transfer the acquired knowledge to a fresh student. We introduce Bootstrapped On-Policy Distillation (BOPD), a two-stage framework that first uses OPSD to internalize task-relevant information from reference-conditioned supervision into an intermediate model, and then distills this reference-free model into a fresh copy of the initial model. Experiments across 4B and 8B models show that BOPD improves mathematical reasoning beyond both the initial models and the constructed OPSD teachers, while also outperforming the constructed teachers on code generation. These results suggest that privileged reference supervision can be more effectively exploited by first transferring its task-relevant signal into a task-adapted model and then distilling from that model without reference conditioning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On policy distillation, self-improvement

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4fQmOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

BOPD first builds a teacher through reference-conditioned self-distillation, then transfers its behavior without references to a fresh copy of the original model. Qwen3-4B/8B experiments examine reasoning, coding, general-capability retention, and training stability.

Critical strengths
  1. Average mathematics performance exceeds both the starting model and the intermediate teacher at both scales, supporting the usefulness of the two-stage procedure.
  2. The approach requires no architectural changes or additional inference overhead. Ablations also document when reference conditioning and repeated distillation become counterproductive.
Critical weaknesses
  1. The causal explanation remains unresolved. Divergence and entropy trends correlate with deterioration but do not isolate privileged-context mismatch as its source. Changing rollout distributions and insufficient controls for teacher configuration and loss clipping leave optimization-related explanations open.
  2. The empirical advantage lacks convincing robustness. BOPD exceeds the best reported 4B OPSD checkpoint by only 0.93 percentage points on Avg@6, while its 8B coding improvement over initialization is 0.09 points. Neither uncertainty estimates nor validation-selected baseline comparisons are provided. Appendix F’s inconsistent data provenance and evaluation batches further limit conclusions about repeated distillation.
Questions for authors
  1. Which Table 3 comparisons share identical training samples and evaluation settings, given the discrepancies disclosed in Appendix F?
  2. Evidence request: Does the improvement persist across training seeds against tuned, validation-selected OPSD under equal compute, including a non-thinking student paired with a thinking teacher?
  3. What is the exact clipped training loss, and what evidence separates its optimization effects from the proposed privileged-context explanation?
5 comments

We thank the reviewer for the constructive feedback and for recognizing the value of our two-stage distillation framework. We conducted additional experiments to address the concerns regarding experimental consistency, robustness, fair comparison, and mechanistic interpretation.

1. Training Data and Evaluation Consistency

We apologize for the inaccurate description in Appendix F. All distillation runs in Table 3 use the same set of 3,200 training examples and the same main training hyperparameters. All configurations follow the same evaluation protocol, including benchmarks, decoding settings, and metrics. We will correct and clarify these descriptions in Appendix F.

2. Multi-seed Robustness

We repeated the mathematical reasoning and code-generation experiments on Qwen3-4B and Qwen3-8B using three training seeds. The following tables report results averaged across seeds with n=6.

Mathematical Reasoning

ModelMethodAIME24AIME25HMMT25Mean
Qwen3-4BOPSD24.63 / 44.4420.56 / 43.3310.93 / 22.2218.70 / 36.67
BOPD30.37 / 56.6725.37 / 54.4412.96 / 32.2222.90 / 47.78
Qwen3-8BOPSD25.93 / 48.8921.11 / 42.2211.30 / 25.5619.45 / 38.89
BOPD28.89 / 53.4422.59 / 45.5512.59 / 30.0021.36 / 43.00

Averaged across seeds, BOPD improves both Avg@6 and Pass@6 on all three mathematical benchmarks at both model sizes.

Code Generation

ModelMethodHumanEval+MBPP+LCB v5-newLCB v6-newMean
Qwen3-4BOPSD75.8159.8820.5622.1044.59
BOPD77.2461.9920.7624.7646.19
Qwen3-8BOPSD80.0863.3220.9624.9547.33
BOPD81.9166.0523.5525.5249.26

BOPD improves mean coding performance by 1.60 points at 4B and 1.93 points at 8B, with positive average gains across all four coding benchmarks.

3. Validation-selected OPSD

For the OPSD baseline, we follow the empirically supported configuration from the original OPSD paper. Its ablations investigate different distillation objectives, including forward KL, reverse KL, and JSD, and as full-vocabulary and sampled-token distillation, etc. Based on these findings and our own preliminary checks, we adopt forward KL with full-vocabulary distillation and the corresponding recommended training settings. Thus, our OPSD baseline follows an established and empirically validated training recipe.

We further evaluated checkpoints from steps 0 to 200 at 20-step intervals and selected the best checkpoint using held-out validation data. The step-120 checkpoint achieved the highest validation accuracy.

MethodAIME24AIME25HMMT25Mean
OPSD (120, validation-selected)27.22 / 53.3321.11 / 50.0010.56 / 20.0019.63 / 41.11
BOPD (100+100)32.78 / 60.0025.56 / 50.0011.11 / 23.3323.15 / 44.44
BOPD (120+100)34.44 / 63.3321.11 / 46.6716.11 / 40.0023.89 / 50.00

The validation-selected OPSD checkpoint achieves 19.63% mean Avg@6, compared with 23.15% for the original BOPD. Furthermore, using the selected step-120 checkpoint as the Stage I teacher yields 23.89% Avg@6 and 50.00% Pass@6 after an additional 100-step OPD stage. These results suggest that improvements in OPSD teacher construction can also benefit subsequent BOPD training.

4. Computational Cost and Matched-Budget Comparison

Our original submission compared BOPD (100+100 steps) with continued OPSD (200 steps). We further evaluated BOPD with a shorter 50+50-step schedule against 100-step OPSD and measured training costs using four NVIDIA A100 GPUs with n=6.

Total StepsMethodAIME24AIME25HMMT25MeanGPU-hours
100BOPD (50+50)26.11 / 46.6723.33 / 43.3316.67 / 26.6722.04 / 38.893.25
OPSD25.00 / 50.0019.44 / 40.0013.89 / 26.6719.44 / 38.893.27
200BOPD (100+100)32.78 / 60.0025.56 / 50.0011.11 / 23.3323.15 / 44.446.40
OPSD12.22 / 33.337.78 / 20.007.22 / 13.339.07 / 22.226.48

Under matched total training steps, BOPD improves mean Avg@6 by 2.60 points at 100 steps and 14.08 points at 200 steps, with nearly identical measured GPU-hour costs. These results show that BOPD's gains are not solely attributable to additional training steps.

5. Non-thinking Student with Thinking Teacher

We further evaluated BOPD with a non-thinking student and a thinking teacher on Qwen3-4B and Qwen3-8B with n=6.

ModelMethodAIME24AIME25HMMT25Mean
Qwen3-4BInitial71.67 / 83.3366.67 / 86.6745.56 / 63.3361.30 / 77.78
OPSD73.89 / 80.0065.00 / 80.0037.78 / 56.6758.89 / 72.22
BOPD75.56 / 86.6768.33 / 80.0046.11 / 56.6763.33 / 74.45
Qwen3-8BInitial76.67 / 83.3368.89 / 83.3347.22 / 63.3364.26 / 76.66
OPSD77.22 / 83.3367.22 / 80.0046.67 / 63.3363.70 / 75.55
BOPD79.44 / 86.6772.22 / 80.0047.22 / 56.6766.29 / 74.45

At 100 total steps, BOPD improves mean Avg@6 from 19.44% to 22.04%, with nearly identical training costs (3.25 vs. 3.27 GPU-hours). At 200 steps, BOPD achieves 23.15%, compared with 9.07% for continued OPSD, again at comparable costs (6.40 vs. 6.48 GPU-hours). These results demonstrate BOPD's effectiveness under approximately matched computational budgets.

6. Loss Clipping and Optimization Effects

Exact Clipped Loss

We follow the official OPSD implementation for pointwise loss clipping. Let ptp_t and qtq_t denote the student and frozen teacher distributions at each student-generated prefix. The training objective is Lc=1∣T∣∑t∈T∑v∈Vmin⁡(qt(v)[log⁡qt(v)−log⁡pt(v)],c)\mathcal{L}_{c}=\frac{1}{|\mathcal{T}|}\sum_{t\in\mathcal{T}}\sum_{v\in\mathcal{V}}\min(q_t(v)[\log q_t(v)-\log p_t(v)],c)

Here, T\mathcal{T} denotes the set of valid completion positions. The cap c=0.05c=0.05 is applied to each signed vocabulary-level contribution before summation. Both stages of the main BOPD experiments use c=0.05c=0.05.

Loss Clipping Ablation

To investigate the effect of loss clipping on prolonged OPSD training, we evaluated five configurations on Qwen3-4B: (1) 100 steps with clipping, (2) 200 steps with clipping, (3) 100 steps with clipping followed by 100 steps without clipping, (4) 100 steps without clipping, and (5) 200 steps without clipping. The entirely unclipped configurations were trained from the initial model using the same training seed (42).

ConfigurationAIME24AIME25HMMT25Mean
OPSD (Clip 100)25.00 / 50.0019.44 / 40.0013.89 / 26.6719.44 / 38.89
OPSD (Clip 200)12.22 / 33.337.78 / 20.007.22 / 13.339.07 / 22.22
OPSD (Clip 100 → No Clip 100)23.33 / 33.3320.56 / 36.678.33 / 16.6717.41 / 28.89
OPSD (No Clip 100)23.89 / 43.3321.11 / 33.3310.56 / 26.6718.52 / 34.44
OPSD (No Clip 200)22.78 / 43.3318.33 / 36.6711.67 / 20.0017.59 / 33.33
BOPD (100+100)32.78 / 60.0025.56 / 50.0011.11 / 23.3323.15 / 44.44

Removing loss clipping substantially mitigates the performance degradation of prolonged OPSD training. After 100 steps of clipped OPSD, disabling clipping for the subsequent 100 steps improves mean Avg@6 from 9.07% (continued clipped OPSD) to 17.41%, although it remains below the step-100 performance of 19.44%. Similarly, when OPSD is trained entirely without clipping, its performance still decreases from 18.52% at step 100 to 17.59% at step 200. Neither configuration surpasses BOPD (23.15%).

These results suggest that removing loss clipping can alleviate OPSD's performance degradation but does not yield further improvements. In contrast, BOPD achieves additional performance gains through its second-stage distillation.

7. Mechanism analysis

Why OPSD Fail

Our original analysis shows that prolonged OPSD training is accompanied by increasing teacher–student divergence and on-policy entropy. To better understand this mismatch, we further examined how the privileged teacher supervises correct student-generated solutions that differ from the reference. We found that the reference-conditioned teacher sometimes assigns lower probabilities to key steps in valid alternative solutions. When we replayed trajectories from training, we found that the reference-conditioned teacher sometimes assigned negative token-level advantage to key steps in correct student solutions that differed from the reference. For example, on an intersecting-chords problem, the student directly applied the intersecting-chords theorem, whereas the reference used similar triangles. Replaying the student's correct trajectory yielded a mean token-level advantage of −0.196 for the key formula, where advantage is defined as log⁡qt(yt)−log⁡pt(yt)\log q_t(y_t)-\log p_t(y_t). Similarly, on a chocolate-bar problem, the student used the invariant that each break increases the number of pieces by one, while the reference used a different counting approach. The corresponding mean advantage was −1.284.

In our stratified sample of correct training trajectories, approximately 19.4% followed valid alternative solution paths. These observations suggest that reference-conditioned supervision can conflict with valid student reasoning, potentially reducing the probability of correct continuations that differ from the reference. Such conflicts provide a possible explanation for the growing teacher–student mismatch during OPSD training.

Why does BOPD outperform its OPSD teacher?

We further analyzed the original Qwen3-4B results across 90 problems from AIME24, AIME25, and HMMT25. A problem is considered solved if at least one of six sampled responses is correct.

Outcome for M0M_0 and M1M_1ProblemsSolved by M2M_2
Both solved2625/26
Only M0M_0 solved42/4
Only M1M_1 solved94/9
Neither solved519/51

Here, M0M_0 is the initial model, M1M_1 is the OPSD teacher, and M2M_2 is the BOPD student.

OPSD expanded problem-solving coverage, while performance declined on some problems the base model had previously solved. OPSD solved nine problems that M0M_0 missed and lost coverage on four that M0M_0 solved. BOPD recovered two of those four, retained four of the nine OPSD gains, and solved nine problems that both models missed. We further investigated how BOPD solved problems that neither the base model nor the OPSD teacher had solved.

To understand these gains, we examined the models’ reasoning traces. We found M0M_0 may start with a correct approach and make a mistake later, while M1M_1 may fail on the same problem because it takes the wrong reasoning path from the outset. In a seating-arrangement problem, M0M_0 correctly interpreted the constraint as forbidding three consecutive occupied seats but made a counting error. In contrast, M1M_1 initially misinterpreted the constraint as forbidding any two adjacent occupied seats.However, when provided with M0M_0's correct intermediate prefix, M1M_1 successfully completed the solution by counting blocks of occupied seats.

This example illustrates an important property of on-policy distillation: even when the teacher fails to generate a correct solution from scratch, it may still provide useful guidance on intermediate states reached by the student. Since BOPD initializes Stage II from M0M_0 and queries M1M_1 on student-generated prefixes, the student can potentially exploit useful teacher predictions that are not reflected in the teacher's own sampled trajectories.

BOPD can also mitigate local conflicts between reference-conditioned supervision in OPSD and valid alternative solution paths. Holding the student and evaluated prefixes fixed, we replaced the reference-conditioned OPSD teacher with the reference-free bootstrapped teacher M1M_1. The mean token-level advantage increased from −1.284 to −0.536 for the chocolate-bar invariant and from −0.072 to −0.032 for the counting step in a doubles-handshake problem. These results suggest that the bootstrapped teacher can provide less conflicting supervision on certain valid alternative reasoning steps. Together with the problem-level analysis, they offer a possible explanation for why a fresh student trained through on-policy distillation can outperform its teacher.

Reviewer_TqnUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates why on-policy self-distillation can become less effective when training continues for too long. The authors suggest this happens because the teacher uses reference information unavailable to the student, creating an increasing mismatch between them. To address this problem, they introduce Bootstrapped On-Policy Distillation, a two-stage training method. The first stage uses reference information to train an intermediate model, while the second stage uses that model as a teacher for a fresh student without reference information. We evaluate the method using language models of two different sizes on mathematical reasoning and code generation tasks. The results show improved average reasoning performance compared with the initial models and the intermediate teachers, together with smaller and less consistent improvements in code generation.

Critical strengths

At first glance, the method's first strength is its simplicity. It needs no external teacher, no new loss, and the final model has the same size and cost, so it could be adopted easily. Moreover, the authors make the useful observation that long OPSD training can collapse, as shown in the matched-budget ablation where 100 plus 100 steps of BOPD beat 200 steps of OPSD at both model sizes. I think this warning is valuable for people who use self-distillation in practice.

Critical weaknesses

The obvious weakness is the weak statistical support. Each benchmark has only 30 problems; every configuration seems to be trained only once. The method requires two training stages, which may increase the computational cost compared with standard self-distillation. I think that the paper needs a clearer comparison under equal training budgets to establish whether the improvements justify the additional computation. Generally, the results are not uniformly better across individual benchmarks.

Questions for authors

I suggest the authors compare BOPD with standard self-distillation using the same total computational budget. The authors should explain whether BOPD provides much larger improvements on some mathematical benchmarks than on others, and clarify whether the same benefits are expected for language models with different architectures or substantially larger sizes.

4 comments

We thank the reviewer for the constructive feedback. We conducted additional experiments to address the concerns regarding computational cost, statistical robustness, and generalization.

1. Computational Cost and Matched-Budget Comparison.

Our original submission compared BOPD (100+100 steps) with continued OPSD (200 steps) on Qwen3-4B. We further evaluate BOPD with a shorter 50+50-step schedule against 100-step OPSD. We also measure the training cost using four NVIDIA A100 GPUs. Results are reported as Avg@6 / Pass@6 (%).

Total StepsMethodAIME24AIME25HMMT25MeanGPU-hoursWall Time
100BOPD (50+50)26.11 / 46.6723.33 / 43.3316.67 / 26.6722.04 / 38.893.2500:48:42
OPSD25.00 / 50.0019.44 / 40.0013.89 / 26.6719.44 / 38.893.2700:49:03
200BOPD (100+100)32.78 / 60.0025.56 / 50.0011.11 / 23.3323.15 / 44.446.4001:35:59
OPSD12.22 / 33.337.78 / 20.007.22 / 13.339.07 / 22.226.4801:37:11

At 100 total steps, BOPD improves mean Avg@6 from 19.44% to 22.04%, with nearly identical training costs (3.25 vs. 3.27 GPU-hours). At 200 steps, BOPD achieves 23.15%, compared with 9.07% for continued OPSD, again at comparable costs (6.40 vs. 6.48 GPU-hours). These results demonstrate BOPD's effectiveness under approximately matched computational budgets.

2. Multi-seed Robustness.

We repeated the mathematical reasoning and code-generation experiments on Qwen3-4B and Qwen3-8B using three training seeds. The following tables report the averages across seeds with n=6.

Mathematical Reasoning

ModelMethodAIME24AIME25HMMT25Mean
Qwen3-4BOPSD24.63 / 44.4420.56 / 43.3310.93 / 22.2218.70 / 36.67
BOPD30.37 / 56.6725.37 / 54.4412.96 / 32.2222.90 / 47.78
Qwen3-8BOPSD25.93 / 48.8921.11 / 42.2211.30 / 25.5619.45 / 38.89
BOPD28.89 / 53.4422.59 / 45.5512.59 / 30.0021.36 / 43.00

Averaged across seeds, BOPD improves both Avg@6 and Pass@6 on all three mathematical benchmarks at both model sizes.

Code Generation

ModelMethodHumanEval+MBPP+LCB v5-newLCB v6-newMean
Qwen3-4BOPSD75.8159.8820.5622.1044.59
BOPD77.2461.9920.7624.7646.19
Qwen3-8BOPSD80.0863.3220.9624.9547.33
BOPD81.9166.0523.5525.5249.26

BOPD also improves mean coding performance, with gains on all four coding benchmarks.

3. Additional Mathematical Benchmarks

To address the limited number of evaluation problems and investigate benchmark-dependent performance, we further evaluated Qwen3-4B and Qwen3-8B on MATH-500, OlympiadBench, and Minerva Math with n=6.

ModelMethodMATH-500OlympiadBenchMinerva MathMean
Qwen3-4BOPSD81.27 / 93.8022.95 / 48.9638.85 / 52.2147.69 / 64.99
BOPD82.57 / 94.8036.05 / 64.2439.40 / 52.5752.67 / 70.54
Qwen3-8BOPSD81.30 / 93.4025.57 / 52.0838.54 / 50.3748.47 / 65.28
BOPD81.90 / 93.8040.28 / 66.9139.83 / 52.2154.00 / 70.97

BOPD improves Avg@6 over OPSD 47.69% → 52.67% on Qwen3-4B and 48.47% → 54.00% Qwen3-8B. Pass@6 also increases by 5.55 and 5.69 points, respectively. These results extend our evaluation beyond the original benchmarks of BOPD's improvements.

4. Generalization to Another Model Family.

We additionally evaluated BOPD on SmolLM3-3B with n=12.

MethodAIME24AIME25HMMT25Mean
Initial10.00 / 30.009.44 / 20.002.50 / 10.007.31 / 20.00
OPSD16.39 / 50.0013.89 / 30.006.67 / 20.0012.31 / 33.33
BOPD17.50 / 50.0012.78 / 36.679.17 / 26.6713.15 / 37.78

On SmolLM3-3B, BOPD improves mean Avg from 12.31% to 13.15% and mean Pass from 33.33% to 37.78% relative to OPSD, providing additional evidence of the method's applicability to another model family.

5. Scaling to Qwen3-14B

To further examine scalability, we extended our experiments to Qwen3-14B with n=6.

MethodAIME24AIME25HMMT25Mean
Initial26.67 / 50.0024.44 / 40.0012.22 / 30.0021.11 / 40.00
OPSD31.67 / 53.3323.89 / 43.3313.89 / 30.0023.15 / 42.22
BOPD33.89 / 56.6730.56 / 60.0015.56 / 30.0026.67 / 48.89

BOPD improves mean Avg@6 from 23.15% to 26.67% and mean Pass@6 from 42.22% to 48.89% relative to OPSD. These results provide evidence that BOPD also benefits a larger 14B model.

Reviewer_TqnUOpenReview ↗

New score - weak acceptance

The authors have properly addressed my concerns, and I am willing to increase my score to a weak acceptance rating.

We sincerely thank the reviewer for the constructive feedback and positive reassessment. We greatly appreciate your recognition of our efforts and your willingness to raise the rating.

We sincerely thank the reviewer for the constructive feedback and positive reassessment. Could you increase your rating score, as you are willing to weak accept? We greatly appreciate your recognition of our efforts and your willingness to raise the rating.

Reviewer_HbBrOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper observes that while OPSD initially improves model performance, continued training eventually leads to performance degradation. To address this issue, the authors propose Bootstrapped On-Policy Distillation (BOPD), a two-stage framework. BOPD first uses OPSD to train a task-adapted teacher from the initial model, then performs OPD using this model as a teacher to train a fresh student initialized from the original model. Experiments demonstrate that BOPD outperforms both conventional OPSD and the intermediate teacher.

Critical strengths

S1. The paper analyzes the performance degradation and instability of OPSD under prolonged training and hypothesizes that these issues are associated with increasing teacher–student distributional mismatch and on-policy entropy. I find this hypothesis reasonable. Given that OPSD is currently an active research direction as an effective post-training method, I believe this is a timely and relevant problem to investigate.

S2. The paper empirically demonstrates that the student trained in the second stage of the proposed two-stage distillation framework can outperform the teacher obtained from OPSD in the first stage. Although the paper lacks a sufficient analysis or explanation of why the student outperforms its teacher, I find this observation interesting.

Critical weaknesses

W1. The paper empirically demonstrates performance degradation during prolonged OPSD training and the Stage II student's ability to outperform the Stage I teacher, but provides limited explanations of the underlying mechanisms. For the former, the proposed explanation based on increasing teacher–student distributional mismatch and on-policy entropy remains largely conjectural. For the latter, the paper offers little explanation beyond improved teacher–student alignment.

W2. The paper attributes the performance degradation of OPSD to the growing mismatch between the evolving student's on-policy state distribution and the frozen privileged teacher. A more straightforward alternative would be to use the current student model with additional reference context as the teacher, but this baseline is not evaluated.

Comparisons with recent methods such as DualOPSD [1], which adapts the privileged teacher to the evolving student, and β\beta-OPSD [2], which adjusts the distillation target to improve training stability, are also missing. In particular, DualOPSD addresses a same issue of growing teacher–student mismatch, making it a directly relevant baseline. These comparisons would help clarify the contribution of BOPD.

Finally, BOPD requires approximately twice as many training steps as standard OPSD due to its additional distillation stage. Beyond performance comparisons with these baselines, an analysis of whether the gains justify the additional computational overhead is needed.

[1] DualOPSD: Adaptive Privileged Teachers for On-Policy Self-Distillation, https://arxiv.org/abs/2608.26019v1

[2] β\beta-OPSD : Deriving with Policy Optimization, Training with Self-Distillation, https://arxiv.org/abs/2607.28582

Questions for authors

Please see the weaknesses section.

4 comments

We thank the reviewer for the constructive feedback. We conducted additional experiments to investigate alternative teacher configurations, compatibility with recent OPSD methods, computational cost and mechanism analysis.

1. Dynamic Privileged Teacher

Following the reviewer's suggestion, we evaluated a dynamic OPSD baseline in which the privileged teacher uses the current student parameters with additional reference conditioning, rather than a frozen initial model with Qwen3-4B.

MethodAIME24AIME25HMMT25Mean
Frozen-teacher OPSD25.00 / 50.0019.44 / 40.0013.89 / 26.6719.44 / 38.89
Dynamic OPSD0.00 / 0.000.00 / 0.000.00 / 0.000.00 / 0.00

Results are reported as Avg@6 / Pass@6 (%). The dynamic-teacher baseline fails under this setting, achieving zero evaluated accuracy across all three benchmarks. Inspection of the generated responses reveals repetitive patterns, including repeated phrases such as "once more again", and the absence of the expected boxed final answers.

This observation is consistent with the original OPSD paper, which explicitly adopts a fixed initial teacher to stabilize training. Our results also show that directly replacing the frozen teacher with the current student can lead to severe generation degeneration.

2. Relationship to β-OPSD and DualOPSD

We agree that β-OPSD and DualOPSD are closely related to our work. These methods improve OPSD through different mechanisms, whereas BOPD adopts a two-stage framework that uses OPSD to construct a teacher, followed by reference-free OPD to train a fresh student. Therefore, these approaches are complementary rather than competing. In particular, improvements to OPSD can be incorporated into BOPD's teacher-construction stage to provide stronger supervision in Stage II.

To demonstrate this compatibility, we reproduced β-OPSD using its official implementation and used the resulting checkpoints as teachers for the second stage of BOPD.

MethodStepsAIME24AIME25HMMT25Mean
β-OPSD10030.00 / 46.6726.11 / 50.0016.11 / 26.6724.07 / 41.11
β-OPSD → BOPD100+10048.33 / 76.6744.44 / 63.3326.11 / 40.0039.63 / 60.00
β-OPSD20049.44 / 80.0042.78 / 70.0031.11 / 56.6741.11 / 68.89
β-OPSD → BOPD200+10058.89 / 80.0056.67 / 83.3338.89 / 56.6751.48 / 73.33

Results are reported as Avg@6 / Pass@6 (%). Starting from a 100-step β-OPSD teacher, the additional BOPD stage improves mean Avg@6 from 24.07% to 39.63% and mean Pass@6 from 41.11% to 60.00%. Using a 200-step β-OPSD teacher followed by 100 steps of BOPD achieves 51.48% Avg@6 and 73.33% Pass@6.

These results demonstrate that BOPD is not restricted to vanilla OPSD and can effectively incorporate improved teacher-construction methods.

3. Computational Cost and Matched-Budget Comparison.

Our original submission compared BOPD (100+100 steps) with continued OPSD (200 steps). We further evaluate BOPD with a shorter 50+50-step schedule against 100-step OPSD. We measure the training cost using four NVIDIA A100 GPUs. Results are reported as Avg@6 / Pass@6 (%).

Total StepsMethodAIME24AIME25HMMT25MeanGPU-hoursWall Time
100BOPD (50+50)26.11 / 46.6723.33 / 43.3316.67 / 26.6722.04 / 38.893.2500:48:42
OPSD25.00 / 50.0019.44 / 40.0013.89 / 26.6719.44 / 38.893.2700:49:03
200BOPD (100+100)32.78 / 60.0025.56 / 50.0011.11 / 23.3323.15 / 44.446.4001:35:59
OPSD12.22 / 33.337.78 / 20.007.22 / 13.339.07 / 22.226.4801:37:11

At 100 total steps, BOPD improves mean Avg@6 from 19.44% to 22.04%, with nearly identical training costs (3.25 vs. 3.27 GPU-hours). At 200 steps, BOPD achieves 23.15%, compared with 9.07% for continued OPSD, again at comparable costs (6.40 vs. 6.48 GPU-hours). These results demonstrate BOPD's effectiveness under approximately matched computational budgets.

4. Mechanism analysis

Why OPSD Fail

Our original analysis shows that prolonged OPSD training is accompanied by increasing teacher–student divergence and on-policy entropy. To better understand this mismatch, we further examined how the privileged teacher supervises correct student-generated solutions that differ from the reference. We found that the reference-conditioned teacher sometimes assigns lower probabilities to key steps in valid alternative solutions. When we replayed trajectories from training, we found that the reference-conditioned teacher sometimes assigned negative token-level advantage to key steps in correct student solutions that differed from the reference. For example, on an intersecting-chords problem, the student directly applied the intersecting-chords theorem, whereas the reference used similar triangles. Replaying the student's correct trajectory yielded a mean token-level advantage of −0.196 for the key formula, where advantage is defined as log⁡qt(yt)−log⁡pt(yt)\log q_t(y_t)-\log p_t(y_t). Similarly, on a chocolate-bar problem, the student used the invariant that each break increases the number of pieces by one, while the reference used a different counting approach. The corresponding mean advantage was −1.284.

In our stratified sample of correct training trajectories, approximately 19.4% followed valid alternative solution paths. These observations suggest that reference-conditioned supervision can conflict with valid student reasoning, potentially reducing the probability of correct continuations that differ from the reference. Such conflicts provide a possible explanation for the growing teacher–student mismatch during OPSD training.

Why does BOPD outperform its OPSD teacher?

We further analyzed the original Qwen3-4B results across 90 problems from AIME24, AIME25, and HMMT25. A problem is considered solved if at least one of six sampled responses is correct.

Outcome for M0 and M1ProblemsSolved by M2
Both solved2625/26
Only M0M_0 solved42/4
Only M1M_1 solved94/9
Neither solved519/51

Here, M0M_0 is the initial model, M1M_1 is the OPSD teacher, and M2M_2 is the BOPD student.

OPSD expanded problem-solving coverage, while performance declined on some problems the base model had previously solved. OPSD solved nine problems that M0M_0 missed and lost coverage on four that M0M_0 solved. BOPD recovered two of those four, retained four of the nine OPSD gains, and solved nine problems that both models missed. We further investigated how BOPD solved problems that neither the base model nor the OPSD teacher had solved.

To understand these gains, we examined the models’ reasoning traces. We found M0M_0 may start with a correct approach and make a mistake later, while M1M_1 may fail on the same problem because it takes the wrong reasoning path from the outset. In a seating-arrangement problem, M0M_0 correctly interpreted the constraint as forbidding three consecutive occupied seats but made a counting error. In contrast, M1M_1 initially misinterpreted the constraint as forbidding any two adjacent occupied seats.However, when provided with M0M_0's correct intermediate prefix, M1M_1 successfully completed the solution by counting blocks of occupied seats.

This example illustrates an important property of on-policy distillation: even when the teacher fails to generate a correct solution from scratch, it may still provide useful guidance on intermediate states reached by the student. Since BOPD initializes Stage II from M0M_0 and queries M1M_1 on student-generated prefixes, the student can potentially exploit useful teacher predictions that are not reflected in the teacher's own sampled trajectories.

BOPD can also mitigate local conflicts between reference-conditioned supervision in OPSD and valid alternative solution paths. Holding the student and evaluated prefixes fixed, we replaced the reference-conditioned OPSD teacher with the reference-free bootstrapped teacher M1M_1. The mean token-level advantage increased from −1.284 to −0.536 for the chocolate-bar invariant and from −0.072 to −0.032 for the counting step in a doubles-handshake problem. These results suggest that the bootstrapped teacher can provide less conflicting supervision on certain valid alternative reasoning steps. Together with the problem-level analysis, they offer a possible explanation for why a fresh student trained through on-policy distillation can outperform its teacher.

Dear Reviewer,

Thank you so much for your thoughtful initial reviews!

With only two days left before the discussion window closes, I wanted to send a friendly reminder to check out the authors' rebuttal. They have put a lot of effort into answering the specific questions you raised.

Whenever you have a spare moment today, please drop a brief reply in the thread to let the authors know if their responses have addressed your concerns, or if any questions remain. Your timely feedback makes a huge difference in ensuring a fair and constructive review process!

Warm regards

I thank the authors for their response and new experiments. Majority of my concerns are addressed. I will maintain my original score (weak accept).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.