- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper shows that vanilla OPSD often degrades thinking models, since solution-conditioned supervision applied at every token suppresses the student’s exploration. The authors use two frozen self-teachers: one conditioned on the reference solution (as in OPSD), and one without it. They classify each token position by the top-3 entropy of the solution-free teacher: at high-entropy (fork) states, the target is the solution-free teacher with temperature softening; at low-entropy (lock) states, the target is the solution-conditioned teacher; intermediate (neutral) states use the solution-free teacher without softening. On five thinking models and three math benchmarks, the method improves avg@16 by up to 8.26 points over OPSD and up to 2.36 points over the base model.
Critical strengths
-
The motivation is clear and data-driven: Fig. 1 shows that solution conditioning overturns the self-teacher’s top token mainly at high-entropy forks and rarely at locks, directly justifying when to apply privileged supervision.
-
The method is simple, adding only one extra teacher pass and an entropy gate with interpretable thresholds.
Critical weaknesses
- No comparison with recent OPSD variants. Purified OPSD[1] and AD-OPSD address the same thinking-collapse problem and are discussed in related work, but only vanilla OPSD is used as a baseline,
- Limited experimental scope. Experiments cover only math reasoning with a 4,096-token completion limit during training, so it is unclear whether the findings transfer to other domains or to longer reasoning traces where exploration matters most.
- Modest gains over Base and limited evaluation scale. Improvements over the base model are small for several models and the small-scale results in Appendix E are mixed, while each benchmark contains only 30 problems and no confidence intervals or multiple seeds are reported, so it is hard to assess whether the gains over Base are significant.
- Generality of the token-level analysis. The key observation motivating the method (Fig. 1) is based on 60 Qwen3-4B rollouts only; it is unclear whether the same fork/lock pattern holds for the other models, which weakens the justification for using a fixed τ_lock across all models.
- No computational cost analysis. The method requires two teacher forward passes instead of one, but no wall-clock or memory comparison with vanilla OPSD is reported.
[1] Shen, Zhanming, et al. "Purified OPSD: On-Policy Self-Distillation Without Losing How to Think." arXiv preprint arXiv:2607.02234 (2026).
[2] Peng, Keqin, et al. "Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation." arXiv preprint arXiv:2607.10805 (2026).
Questions for authors
Does FS-OPSD improve reasoning, or only preserve it?
The method’s supervision at every position is either the solution-free self-teacher (which is the student’s own initial distribution, softened) or the solution-conditioned self-teacher at locks. Neither source seems to provide a signal that could strengthen the student’s reasoning beyond its initial capability: at forks the student is anchored to its own prior, and at locks the solution only reweights candidates the student already prefers. This raises the possibility that the gains come mainly from the student learning to produce more concise or more decisive trajectories, rather than from improved reasoning.
Could the authors clarify where new capability is expected to come from under this design? Relatedly, does FS-OPSD change the average response length compared to Base and vanilla OPSD? If responses become shorter, this would support the interpretation above; if not, some analysis of what actually changes in the student’s trajectories would help substantiate the claim that exploration is preserved.
5 comments
Response (Part 2)
W3: Modest gains over Base and limited evaluation scale. Improvements over the base model are small for several models and the small-scale results in Appendix E are mixed, while each benchmark contains only 30 problems and no confidence intervals or multiple seeds are reported, so it is hard to assess whether the gains over Base are significant.
Here we would like to provide 95% paired-bootstrap confidence intervals for our main results (Table 1). The differences below are measured in percentage points of avg@16, averaged across AIME 2024, AIME 2025, and HMMT 2025.
| Model | FS-OPSD − Base | FS-OPSD − vanilla OPSD |
|---|---|---|
| Qwen3-4B | +2.22 [+0.42, +3.96] | +8.26 [+5.56, +11.04] |
| Qwen3-8B | +0.41 [−1.81, +2.71] † | +7.08 [+4.44, +9.93] |
| R1-Distill-Qwen-7B | +2.36 [+0.28, +4.51] | +3.82 [+1.60, +6.04] |
| R1-Distill-Llama-8B | +2.15 [+0.14, +4.31] | +2.84 [+0.56, +5.28] |
| GLM-Z1-9B-0414 | +0.62 [−1.81, +3.13] † | +4.51 [+1.53, +7.64] |
FS-OPSD yields positive point estimates over Base across all five main models. The 95% paired-bootstrap confidence intervals support statistically significant gains for Qwen3-4B (+2.22 percentage points), R1-Distill-Qwen-7B (+2.36), and R1-Distill-Llama-8B (+2.15), providing evidence of improvements beyond the initial model’s performance. As a complementary comparison, FS-OPSD improves over vanilla OPSD by 2.84–8.26 percentage points across all five models, with every 95% confidence interval excluding zero.
These smaller gains should be viewed in the context of the problem studied: vanilla OPSD reduces average performance below Base on all five models. FS-OPSD consistently improves on this training approach and achieves statistically supported gains beyond Base in three models, demonstrating the practical value of adapting privileged supervision to the reasoning state.
W4: Generality of the token-level analysis. The key observation motivating the method (Fig. 1) is based on 60 Qwen3-4B rollouts only; it is unclear whether the same fork/lock pattern holds for the other models, which weakens the justification for using a fixed τ_lock across all models.
Here we would like to provide two more fork/lock pattern analysis for R1-Distill-Qwen-7B and R1-Distill-Llama-8B.
| Model | Switch:Fork | Neutral | Lock | Drop:Fork | Neutral | Lock |
|---|---|---|---|---|---|---|
| Qwen3-4B | 43.52 | 19.52 | 0.862 | 9.40 | 1.35 | 0.066 |
| R1-Distill-Qwen-7B | 37.29 | 11.99 | 0.220 | 5.85 | 0.278 | 0.0057 |
| R1-Distill-Llama-8B | 36.56 | 10.97 | 0.248 | 5.84 | 0.326 | 0.0093 |
On these two models, the switch and drop share the same pattern as Qwen3-4B, indicating the same fork/lock pattern holds for different models.
As for , the value is first derived mathematically (Appendix B). Then we combine concentration change point for calibration and choose . The value is model-independent. We evaluated candidate thresholds from 0.19 to 0.26 in increments of 0.01. As shown below, all three evaluated models attain their lowest observed residual at , supporting our choice of a shared threshold across these models.
| Qwen3-4B | R1-Distill-Qwen-7B | R1-Distill-Llama-8B | |
|---|---|---|---|
| 0.19 | 0.53 | 0.50 | 0.55 |
| 0.20 | 0.42 | 0.41 | 0.46 |
| 0.21 | 0.36 | 0.37 | 0.42 |
| 0.22 | 0.31 | 0.32 | 0.37 |
| 0.23 | 0.38 | 0.34 | 0.39 |
| 0.24 | 0.38 | 0.34 | 0.38 |
| 0.25 | 0.40 | 0.40 | 0.45 |
| 0.26 | 0.42 | 0.43 | 0.49 |
Response (Part 3)
W5: No computational cost analysis. The method requires two teacher forward passes instead of one, but no wall-clock or memory comparison with vanilla OPSD is reported.
We would like to provide computational cost comparison with vanilla OPSD. During training, we use vLLM as the inference engine, which makes the memory usage almost the same for both vanilla OPSD and FS-OPSD methods. Below we report training time required for different models.
Training:
| Model | vanilla OPSD | FS-OPSD |
|---|---|---|
| Qwen3-4B | 3h49min | 4h9min |
| Qwen3-8B | 4h59min | 5h32min |
| R1-Distill-Qwen-7B | 4h3min | 4h16min |
| R1-Distill-Llama-8B | 4h32min | 4h54min |
| Qwen3-4B | 5h7min | 5h34min |
As can be seen from these two tables, our method requires only a small amount of additional training time compared to vanilla OPSD (less than 10%). But we have a relatively much better result than vanilla OPSD.
Thank you for the review! We address each weakness and question below.
Since the source code of Purified OPSD [1] and AD-OPSD [2] was not open-sourced, we did not previously compare our work against these two papers. Following the reviewer's suggestion, we have now implemented them based on the descriptions provided in the papers. And the results are shown below.
As shown in the table, our method achieves comparable performance to Purified OPSD while requiring 8.5% less wall-clock training time (FS-OPSD: 4h9min and Purified OPSD: 4h32min). Compared with AD-OPSD, our method improves average accuracy by 5.14%, with consistent gains across three benchmarks. These additional comparisons demonstrate that FS-OPSD remains competitive with recent OPSD variants beyond the vanilla OPSD baseline.
[1] Shen, Zhanming, et al. "Purified OPSD: On-Policy Self-Distillation Without Losing How to Think." arXiv preprint arXiv:2607.02234 (2026).
[2] Peng, Keqin, et al. "Diagnosing and Mitigating Thinking Collapse in On-Policy Self-Distillation." arXiv preprint arXiv:2607.10805 (2026).
First we would like to provide further experiment results on math reasoning with a 8192-token completion limit during training. The results are shown below.
In this table we can see that the thinking collapse issue of vanilla OPSD happens the same in longer reasoning traces and our method significantly alleviates this problem. So the findings can transfer to longer reasoning traces.
Second we would like to provide further experiment results conducted on other science domains (physics, chemistry, biology). We seleted 10000 problems from MegaScience dataset, including 4000 physics, 2500 math, 2500 chemistry and 1000 biology, as our training set. Then we evaluate trained model on GPQA-Diamond, MMLU-Pro (categories: math, physics, chemistry and biology) and UGPhysics. We report avg@16 for GPQA-Diamond, avg@1 for MMLU-Pro and UGPhysics. The results are shown below.
In this table we can see that the thinking collapse issue of vanilla OPSD happens in other science domains. Our method achieves higher performance than base model. So the findings can transfer to other domains.