- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
BOPD first builds a teacher through reference-conditioned self-distillation, then transfers its behavior without references to a fresh copy of the original model. Qwen3-4B/8B experiments examine reasoning, coding, general-capability retention, and training stability.
Critical strengths
- Average mathematics performance exceeds both the starting model and the intermediate teacher at both scales, supporting the usefulness of the two-stage procedure.
- The approach requires no architectural changes or additional inference overhead. Ablations also document when reference conditioning and repeated distillation become counterproductive.
Critical weaknesses
- The causal explanation remains unresolved. Divergence and entropy trends correlate with deterioration but do not isolate privileged-context mismatch as its source. Changing rollout distributions and insufficient controls for teacher configuration and loss clipping leave optimization-related explanations open.
- The empirical advantage lacks convincing robustness. BOPD exceeds the best reported 4B OPSD checkpoint by only 0.93 percentage points on Avg@6, while its 8B coding improvement over initialization is 0.09 points. Neither uncertainty estimates nor validation-selected baseline comparisons are provided. Appendix F’s inconsistent data provenance and evaluation batches further limit conclusions about repeated distillation.
Questions for authors
- Which Table 3 comparisons share identical training samples and evaluation settings, given the discrepancies disclosed in Appendix F?
- Evidence request: Does the improvement persist across training seeds against tuned, validation-selected OPSD under equal compute, including a non-thinking student paired with a thinking teacher?
- What is the exact clipped training loss, and what evidence separates its optimization effects from the proposed privileged-context explanation?
5 comments
5. Non-thinking Student with Thinking Teacher
We further evaluated BOPD with a non-thinking student and a thinking teacher on Qwen3-4B and Qwen3-8B with n=6.
| Model | Method | AIME24 | AIME25 | HMMT25 | Mean |
|---|---|---|---|---|---|
| Qwen3-4B | Initial | 71.67 / 83.33 | 66.67 / 86.67 | 45.56 / 63.33 | 61.30 / 77.78 |
| OPSD | 73.89 / 80.00 | 65.00 / 80.00 | 37.78 / 56.67 | 58.89 / 72.22 | |
| BOPD | 75.56 / 86.67 | 68.33 / 80.00 | 46.11 / 56.67 | 63.33 / 74.45 | |
| Qwen3-8B | Initial | 76.67 / 83.33 | 68.89 / 83.33 | 47.22 / 63.33 | 64.26 / 76.66 |
| OPSD | 77.22 / 83.33 | 67.22 / 80.00 | 46.67 / 63.33 | 63.70 / 75.55 | |
| BOPD | 79.44 / 86.67 | 72.22 / 80.00 | 47.22 / 56.67 | 66.29 / 74.45 |
At 100 total steps, BOPD improves mean Avg@6 from 19.44% to 22.04%, with nearly identical training costs (3.25 vs. 3.27 GPU-hours). At 200 steps, BOPD achieves 23.15%, compared with 9.07% for continued OPSD, again at comparable costs (6.40 vs. 6.48 GPU-hours). These results demonstrate BOPD's effectiveness under approximately matched computational budgets.
6. Loss Clipping and Optimization Effects
Exact Clipped Loss
We follow the official OPSD implementation for pointwise loss clipping. Let and denote the student and frozen teacher distributions at each student-generated prefix. The training objective is
Here, denotes the set of valid completion positions. The cap is applied to each signed vocabulary-level contribution before summation. Both stages of the main BOPD experiments use .
Loss Clipping Ablation
To investigate the effect of loss clipping on prolonged OPSD training, we evaluated five configurations on Qwen3-4B: (1) 100 steps with clipping, (2) 200 steps with clipping, (3) 100 steps with clipping followed by 100 steps without clipping, (4) 100 steps without clipping, and (5) 200 steps without clipping. The entirely unclipped configurations were trained from the initial model using the same training seed (42).
| Configuration | AIME24 | AIME25 | HMMT25 | Mean |
|---|---|---|---|---|
| OPSD (Clip 100) | 25.00 / 50.00 | 19.44 / 40.00 | 13.89 / 26.67 | 19.44 / 38.89 |
| OPSD (Clip 200) | 12.22 / 33.33 | 7.78 / 20.00 | 7.22 / 13.33 | 9.07 / 22.22 |
| OPSD (Clip 100 → No Clip 100) | 23.33 / 33.33 | 20.56 / 36.67 | 8.33 / 16.67 | 17.41 / 28.89 |
| OPSD (No Clip 100) | 23.89 / 43.33 | 21.11 / 33.33 | 10.56 / 26.67 | 18.52 / 34.44 |
| OPSD (No Clip 200) | 22.78 / 43.33 | 18.33 / 36.67 | 11.67 / 20.00 | 17.59 / 33.33 |
| BOPD (100+100) | 32.78 / 60.00 | 25.56 / 50.00 | 11.11 / 23.33 | 23.15 / 44.44 |
Removing loss clipping substantially mitigates the performance degradation of prolonged OPSD training. After 100 steps of clipped OPSD, disabling clipping for the subsequent 100 steps improves mean Avg@6 from 9.07% (continued clipped OPSD) to 17.41%, although it remains below the step-100 performance of 19.44%. Similarly, when OPSD is trained entirely without clipping, its performance still decreases from 18.52% at step 100 to 17.59% at step 200. Neither configuration surpasses BOPD (23.15%).
These results suggest that removing loss clipping can alleviate OPSD's performance degradation but does not yield further improvements. In contrast, BOPD achieves additional performance gains through its second-stage distillation.
7. Mechanism analysis
Why OPSD Fail
Our original analysis shows that prolonged OPSD training is accompanied by increasing teacher–student divergence and on-policy entropy. To better understand this mismatch, we further examined how the privileged teacher supervises correct student-generated solutions that differ from the reference. We found that the reference-conditioned teacher sometimes assigns lower probabilities to key steps in valid alternative solutions. When we replayed trajectories from training, we found that the reference-conditioned teacher sometimes assigned negative token-level advantage to key steps in correct student solutions that differed from the reference. For example, on an intersecting-chords problem, the student directly applied the intersecting-chords theorem, whereas the reference used similar triangles. Replaying the student's correct trajectory yielded a mean token-level advantage of −0.196 for the key formula, where advantage is defined as . Similarly, on a chocolate-bar problem, the student used the invariant that each break increases the number of pieces by one, while the reference used a different counting approach. The corresponding mean advantage was −1.284.
In our stratified sample of correct training trajectories, approximately 19.4% followed valid alternative solution paths. These observations suggest that reference-conditioned supervision can conflict with valid student reasoning, potentially reducing the probability of correct continuations that differ from the reference. Such conflicts provide a possible explanation for the growing teacher–student mismatch during OPSD training.
Why does BOPD outperform its OPSD teacher?
We further analyzed the original Qwen3-4B results across 90 problems from AIME24, AIME25, and HMMT25. A problem is considered solved if at least one of six sampled responses is correct.
| Outcome for and | Problems | Solved by |
|---|---|---|
| Both solved | 26 | 25/26 |
| Only solved | 4 | 2/4 |
| Only solved | 9 | 4/9 |
| Neither solved | 51 | 9/51 |
Here, is the initial model, is the OPSD teacher, and is the BOPD student.
OPSD expanded problem-solving coverage, while performance declined on some problems the base model had previously solved. OPSD solved nine problems that missed and lost coverage on four that solved. BOPD recovered two of those four, retained four of the nine OPSD gains, and solved nine problems that both models missed. We further investigated how BOPD solved problems that neither the base model nor the OPSD teacher had solved.
To understand these gains, we examined the models’ reasoning traces. We found may start with a correct approach and make a mistake later, while may fail on the same problem because it takes the wrong reasoning path from the outset. In a seating-arrangement problem, correctly interpreted the constraint as forbidding three consecutive occupied seats but made a counting error. In contrast, initially misinterpreted the constraint as forbidding any two adjacent occupied seats.However, when provided with 's correct intermediate prefix, successfully completed the solution by counting blocks of occupied seats.
This example illustrates an important property of on-policy distillation: even when the teacher fails to generate a correct solution from scratch, it may still provide useful guidance on intermediate states reached by the student. Since BOPD initializes Stage II from and queries on student-generated prefixes, the student can potentially exploit useful teacher predictions that are not reflected in the teacher's own sampled trajectories.
BOPD can also mitigate local conflicts between reference-conditioned supervision in OPSD and valid alternative solution paths. Holding the student and evaluated prefixes fixed, we replaced the reference-conditioned OPSD teacher with the reference-free bootstrapped teacher . The mean token-level advantage increased from −1.284 to −0.536 for the chocolate-bar invariant and from −0.072 to −0.032 for the counting step in a doubles-handshake problem. These results suggest that the bootstrapped teacher can provide less conflicting supervision on certain valid alternative reasoning steps. Together with the problem-level analysis, they offer a possible explanation for why a fresh student trained through on-policy distillation can outperform its teacher.
We thank the reviewer for the constructive feedback and for recognizing the value of our two-stage distillation framework. We conducted additional experiments to address the concerns regarding experimental consistency, robustness, fair comparison, and mechanistic interpretation.
1. Training Data and Evaluation Consistency
We apologize for the inaccurate description in Appendix F. All distillation runs in Table 3 use the same set of 3,200 training examples and the same main training hyperparameters. All configurations follow the same evaluation protocol, including benchmarks, decoding settings, and metrics. We will correct and clarify these descriptions in Appendix F.
2. Multi-seed Robustness
We repeated the mathematical reasoning and code-generation experiments on Qwen3-4B and Qwen3-8B using three training seeds. The following tables report results averaged across seeds with n=6.
Mathematical Reasoning
Averaged across seeds, BOPD improves both Avg@6 and Pass@6 on all three mathematical benchmarks at both model sizes.
Code Generation
BOPD improves mean coding performance by 1.60 points at 4B and 1.93 points at 8B, with positive average gains across all four coding benchmarks.
3. Validation-selected OPSD
For the OPSD baseline, we follow the empirically supported configuration from the original OPSD paper. Its ablations investigate different distillation objectives, including forward KL, reverse KL, and JSD, and as full-vocabulary and sampled-token distillation, etc. Based on these findings and our own preliminary checks, we adopt forward KL with full-vocabulary distillation and the corresponding recommended training settings. Thus, our OPSD baseline follows an established and empirically validated training recipe.
We further evaluated checkpoints from steps 0 to 200 at 20-step intervals and selected the best checkpoint using held-out validation data. The step-120 checkpoint achieved the highest validation accuracy.
The validation-selected OPSD checkpoint achieves 19.63% mean Avg@6, compared with 23.15% for the original BOPD. Furthermore, using the selected step-120 checkpoint as the Stage I teacher yields 23.89% Avg@6 and 50.00% Pass@6 after an additional 100-step OPD stage. These results suggest that improvements in OPSD teacher construction can also benefit subsequent BOPD training.
4. Computational Cost and Matched-Budget Comparison
Our original submission compared BOPD (100+100 steps) with continued OPSD (200 steps). We further evaluated BOPD with a shorter 50+50-step schedule against 100-step OPSD and measured training costs using four NVIDIA A100 GPUs with n=6.
Under matched total training steps, BOPD improves mean Avg@6 by 2.60 points at 100 steps and 14.08 points at 200 steps, with nearly identical measured GPU-hour costs. These results show that BOPD's gains are not solely attributable to additional training steps.