- Rating
- 3(+1)
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes B-OPSD, which aims to improve self-distillation by constructing a better self-teacher. Specifically, B-OPSD first performs several steps of vanilla OPSD training to obtain a future version of the student as the self-teacher. It then restores the student to its initial checkpoint and uses this future self-teacher to provide privileged information and OPD-style supervision. The authors develop concrete methods for settings both with and without ground-truth answers. Experiments on Qwen3-4B and Qwen3-8B across three mathematical reasoning benchmarks show that B-OPSD consistently outperforms vanilla OPSD in both settings. The paper further investigates B-OPSD’s effectiveness and advantages through ablation studies, comparisons under matched training budgets, and visualizations of training gains. Overall, the proposed approach is interesting, demonstrates effectiveness in the mathematical reasoning domain, and offers useful insights into self-distillation.
Critical strengths
- Prior work on self-distillation has primarily focused on constructing suitable privileged information. B-OPSD instead explores how to build a better self-teacher, proposing to use a future version of the student as a stronger self-teacher. The motivation is compelling, and the approach does not require introducing a stronger external teacher, offering an interesting direction for further improving OPSD.
- B-OPSD demonstrates notable performance gains over OPSD across two model sizes, three mathematical reasoning benchmarks, and settings both with and without ground-truth answers. The experimental finding that privileged information needs to be matched to the self-teacher is also interesting, and the analysis presented in Figure 2 is intuitive.
Critical weaknesses
- The paper’s central finding is that a future version of the model can provide better supervision as a self-teacher. While Figure 2 offers an intuitive explanation, the results do not sufficiently clarify which properties of the supervision are improved by the future self-teacher, or why these improvements allow the restarted student to outperform continued training. A deeper mechanistic investigation or an appropriate theoretical analysis is needed. It would also be valuable to discuss the conditions under which the approach is effective and those under which it fails to provide benefits.
- Regarding the experimental setup, we have two concerns. First, what motivates the use of LoRA for training? Although LoRA is used in several existing OPSD studies, it remains unclear whether self-distillation would remain effective under the more conventional full-parameter fine-tuning setup, or whether its effectiveness depends on LoRA. Could the authors provide additional experiments to investigate this question? Second, the current evaluation is limited to three benchmarks in the mathematical reasoning domain. We would like to see experiments on additional benchmarks and non-mathematical domains, such as coding, to assess whether B-OPSD’s performance gains and advantages generalize more broadly.
Questions for authors
In addition to the questions raised in the weaknesses, we would appreciate clarification on the following points and are happy to revising our final score based on the authors’ responses.
- The paper states that thinking mode is disabled during all training and evaluation. Could the authors confirm whether thinking mode is also disabled when constructing the self-teacher? Recent work, What Does Privileged Information Add to On-Policy Self-Distillation?, suggests that the effectiveness of self-distillation may stem from a mismatch in thinking modes between the self-teacher and the student.
- Prior work on the failure modes of self-distillation, Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?, reports that performance can deteriorate in later stages of training, accompanied by shorter model outputs. Figure 3 also shows performance degradation for OPSD. Could the authors extend the B-OPSD performance curves to more training steps and report how output length evolves during training for both OPSD and B-OPSD?
- Can the proposed approach of using a future version of the model to improve its current self be applied iteratively? Recent work, RISE: Recursive Improvement via Self-Extrapolating Policy Distillation, introduces an iterative policy distillation procedure. We are curious whether B-OPSD could support a similar process of iterative improvement. For example, could the checkpoint at step 100 of the blue curve in Figure 3 serve as a second-generation future self-teacher, with the student reset to its initial checkpoint for another round of B-OPSD?
- B-OPSD involves several conditional rules for selecting the self-teacher and the privileged information. How would performance change if these rules were removed and the future self-teacher were consistently used to provide both privileged information and supervision?
Sorry for the late reply. Thanks for your detailed response, and I will raise my score from 2 to 3.
Just as a friendly reminder (which does not affect my score or overall evaluation): the two links you provided in response to Q2 are inaccessible to me (and maybe other reviewers), so I would suggest updating them if possible.
P.S. If any of you are also serving as reviewers for other submissions that are still waiting for a response, I hope you can also take a moment to reply to those authors! :-D