acceptodds
Under review as a conference paper at ICLR 2027

Diagnosing On-Policy Self-Distillation for Reasoning Language Models

14 views

Abstract

On-policy self-distillation (OPSD) has attracted growing interest as a promising approach to improve the reasoning ability of language models. Without external rewards nor a separate stronger teacher, the self-teacher with privileged information could provide dense signals on student's trajectories. However, its behavior in language reasoning remains unclear, with reported outcomes ranging from modest gains to behavioral collapse. In this work, we diagnose OPSD for mathematical reasoning across models spanning 0.6B–8B parameters. We conduct controlled experiments and token-level analyses to fully delve into OPSD. We point out that teacher's signal is shaped by reasoning-mode alignment and the complete teacher prefix, rather than by privileged semantics alone. OPSD improves reasoning only in narrow compatibility regimes. Otherwise, it produces ineffective length growth, stable degradation, or behavioral collapse. Token-level analysis shows that teacher's signal is not stable and does not predict downstream performance. Based on these results, we argue that OPSD is a sensitive algorithm rather than a generally reliable reasoning-improvement post-training method.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Post Training, Self-Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gYMaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper systematically studies when OPSD helps mathematical reasoning and when it causes degradation. During training, the student generates its own reasoning trajectories. A frozen copy of the initial model receives privileged information, such as a reference solution or answer, and provides token-level supervision on those same trajectories. The authors primarily study Qwen3 and OLMo-3, examining reasoning modes, rollout length, training duration, teacher context, and initialization, alongside different losses and choices of supervision positions. The evaluation reports Avg@8, Pass@8, and output length, and uses formatting and training dynamics to distinguish ineffective length growth, stable degradation, and behavioral collapse. The authors find that OPSD gains depend on the model and configuration: neither richer privileged information nor a stronger teacher signal alone determines training utility. Some training metrics are more useful as warnings of collapse than as predictors of final performance.

Critical strengths
  1. The experimental coverage is systematic. The reasoning-mode and no-privilege controls help identify mode-transfer effects, while the rollout-length, context, and initialization experiments distinguish several failure modes. These findings provide useful guidance for applying OPSD.
  2. The appendix includes repetitions across independent training and decoding seeds, cross-dataset validation, LoRA controls, and replications of selected OPSD variants. Together with the analyses of accuracy, output length, and formatting, these experiments provide substantial evidence for degradation and collapse.
Critical weaknesses

My main concern is the mechanistic attribution to teacher-prefix mismatch. Full solution, Answer only, and Unrelated solution use different teacher templates and apply length filtering under their respective templates. These conditions may therefore vary instruction wording, context length, target-problem position, and the retained training examples simultaneously. The observation that unrelated context also affects the supervision signal supports a contribution beyond task-relevant semantics, but does not establish how much each factor contributes to the training outcome. Section 5.3 acknowledges that these are complete-prefix interventions. The mechanistic claims in the abstract and conclusion should respect this scope or be supported by more tightly controlled comparisons.

Questions for authors

Do Full, Answer, and Unrelated use exactly the same training problems in the same order? Can the existing results separate semantic relevance from the effects of template, length, and target-problem position? If these effects cannot be separated, please clarify the scope of attribution supported by the current controls and adjust the mechanistic claims accordingly.

1 comment
AuthorsOpenReview ↗

Response to reviewer

Thanks for your rigorous reading and recognition. Our work is based on large and broad training and evalution. We provide a revised version and hope to solve your concerns and questions.

To questions:

Q1: Do Full, Answer, and Unrelated use exactly the same training problems in the same order?

A1: Thanks for pointing out. In the revised version, we retrain on the same dataset in exactly the same order. The conclusion stays the same: Full solution > Unrelated > Answer only (Section 5.3, Page 5, Table 3). The old results are kept in the appendix as a robustness check.

Q2: Can the existing results separate semantic relevance from the effects of template, length, and target-problem position.

A2: They can be separated sometimes. We additional do some similar experiments you've mentioned in Appendix E.1(Page 32, Table 30), which keeps Answer only and changes only the privileged answer, so the template is fixed and the length is fixed or only slightly different. A wrong answer yields better results than the correct answer. This shows that prefix distribution difference contributes more than semantic information in OPSD. In our main text, more detailed priviledged information always brings longer length. To minimize the impact of length, we keep the template of Answer only, Full solution and CoT+solution same. The conclusion supported by these controls is that what the privileged information brings is a context-distribution difference rather than semantic information, and a proper distribution difference can bring more gains.

To weakness:

W1: Concerning about mixture of teacher prefix.

A1: We agree that these are complete-prefix interventions, as Section 5.3 states(Page 5). The mechanistic claims stay within the scope in the answers above.

We sincerely hope this reply and the revised paper resolve your concerns and questions. We are willing to answer if anything remains unclear. If they do resolve your concerns and questions, we would be grateful if you could raise the score.

Reviewer_1HW2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors analyze a recent popular line of work investigating on policy distillation where a teacher has access to privileged information to supervise the student rollout. The authors claim that this technique largely fails to produce reliable gains across several different models and scales, including investigating its effects across non-thinking and thinking models.

Critical strengths
  • The paper does a good job of outlining the problem, and I very much appreciate work in the "negative results" direction as it provides a nice counterbalance to "viral" research directions that may not be principled.
  • The paper has decent empirical breadth and depth experimenting with several models and scales, and making fairly precise claims.
Critical weaknesses
  • Figure 1 needs to be heavily cleaned up. Figures, especially the first one in the paper, are supposed to be expository and helpful. This one is not and requires a lot of zooming in to even read the plot.
  • The second claimed contribution "OPSD gains are modest and confined to model- and configuration-specific regimes" is not really novel to this work. Several other works - including some cited by the authors, have found that OPSD results are over-hyped.
  • Related to the point above, the paper's positioning relative to prior work is not very clear/well fleshed out. The authors simply say "These works deeply analyze a part of OPSD but lack broader explorations and global diagnosis." when talking about prior literature. In my opinion, the burden is on the authors in this case to clearly and specifically state why their work adds something valuable beyond the analysis of previous work.
  • Many results/tables do not come with confidence intervals, which is especially important in a work like this.
  • The results section is pretty muddled to be honest. The paper lacks a clear story for the experimental section. It reads more like "we did this experiment ... then we did this other experiment" and this reader was left without a coherent picture for how all the results connect.
Questions for authors

What are the unique contributions of your work in analyzing why OPSD style training often doesn't work well? I'm willing to increase my score if this can be made clear, and the manuscript can be cleaned up.

2 comments
AuthorsOpenReview ↗

Response to reviewer

Thanks for your recognition, especially "breadth and depth experimenting", and constructive suggestions,. Due to the page limit and the writing, we are sorry that the manuscript was muddled and confused you. We hope the revised paper is easier to read and understand.

To questions:

Q1: Unique contributions of this work about OPSD's failure.

A1: Thanks for the question, which is important to emphasize our work. To our knowledge, this work is the first to fully diagnose OPSD in many aspects.

  • For aspects that already have concurrent studies, we confirm their conclusions, such as the difference between Reasoning models and Instruct models, and the rethinking of privileged information. We also transfer a conclusion from OPD to OPSD, such as the early stop of the student rollout.

  • Some aspects are first studied. We apply OPSD to at most 11 models and observe the scaling of OPSD. We are the first to find that there is no unified KL loss for different models, train a model from a pretrained checkpoint and study the performance of OPSD, characterize privileged information as different distributions. The analysis of different positions, and the use of dynamics to diagnose collapse and differences across models are also our unique contributions.

Q2: Some muddled writing.

A2: We sincerely thanks for pointing out. We add an overview at the start of Section 5(Page 4), and each subsection now states its purpose or importance. The experiments proceed in order: a valid matched-mode comparison, training configurations, key designs of OPSD, and comparison of present method. The order is step by step, as in the main text.

To weakness:

W1: Heavy Figure.

A1: We modify Figure 1(Page 3) and Figure 4(Page 7). We enlarge the fonts and the figures so they are easier to read. We keep Figure 1 because it contains the pipeline of OPSD and the factors in our diagnosis.

W2 & W3: Similar conclusion and overclaim of OPSD.

A2: We acknowledge these recent studies. We borrow their ideas, extend the experiments, and verify their conclusions. For example, in Section 5.5 and 6.2, we verify the difference of Reasoning model and Instruct model, which is similar with [1]. We also further state these differences in Related Work(Section 2 Related Work, Page 2).

For the overclaim, we deeply agree with that. One of purpose of this paper is to fully delve into OPSD and find which part of OPSD really works or leads to fail.

W4: Confidence Intervals

A4: Thanks for your suggestions. We report confidence intervals for the main experiments in Appendix F.3 (Page 40). We also evaluate some experiments on two more seeds, 1024 and 65536 (Figure 4 at Page 7, Table 5(a) at Page 6). The confidence intervals and the three seeds reduce the error and the evaluation noise.

W5: Muddled experiments and writing.

A5: Same as Q2.

We hope the revised paper and this reply make the paper easier to understand. We are willing to answer if anything remains unclear. If they do resolve your concerns and questions, we would be grateful if you could raise the score.

[1]: Kaur et al., Rethinking On-Policy Self-Distillation for Thinking Models, arXiv July 2026

AuthorsOpenReview ↗

Response to Reviewer 1HW2

Hi,

We would like to kindly follow up on our rebuttal responses and the corresponding revisions to the manuscript. We understand that you may be busy, but with less than 24 hours remaining in the rebuttal period, we would greatly appreciate it if you could take a moment to review our responses and revisions.

We sincerely hope that our clarifications and revisions have addressed your concerns. If there are any remaining questions or points that require further clarification, we would be more than happy to respond promptly and make further improvements.

Thank you very much for your time and consideration!

Reviewer_CPhKOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies on-policy self-distillation (OPSD) for mathematical reasoning. In OPSD, a copy of the student model acts as the teacher, but it gets more information about the task (e.g. the answer key or the reference solution). The student outputs solutions and the teacher scores this output. The paper is a diagnosis study that runs experiments across 11 models (model sizes range from 0.6B to 8B) and studies OPSD failure cases related to what they call ineffective length growth, stable degradation, and behavioral collapse.

Critical strengths
  1. I appreciate the comparison across many models, which covers different model families and includes instruct and reasoning versions.
  2. Some analyses point towards practical guidelines. For example, supervising only the first 256 tokens does better than supervising the last 256 (Table 2). That seems to have cost efficiency implications. These analyses need more rigorous investigation though with a more formalized benchmark (see weakness 1).
Critical weaknesses
  1. [main weakness] The definitions in Section 4 are each one short paragraph, and they are all qualitative ("little decline in accuracy," "substantial evaluation length growth," "large performance degradation"). It is unclear how each run gets assigned to one of these categories (if it is just based on a threshold, what is that threshold for each category?). I would expect a precise rule (e.g., accuracy change below X points and length growth above Y% equals large performance degradation). This is especially important for a diagnosis paper.

  2. Several of the diagnosed conditions have already been explored in the OPD literature. For example, [1] studies ineffective length growth in OPD in general and proposes several training stabilization strategies (mainly a reference-KL constraint.) I think [1] already investigates this problem comprehensively and offers solutions, yet the paper only cites it in the appendix and does not benchmark its mitigation strategies. Similarly, [2] studies the degradation problem of self-distillation. It would have been helpful to discuss both works in detail (and explain what the proposed diagnosis adds beyond what these papers already showed.)

  3. The experiments are a bit small for a diagnosis paper. Models only go up to 8B, and evaluation is on about 120 problems in total, with mostly a single training run per condition. There is established research on how noisy evaluation on such small math benchmarks can be [3], where scores move by several points just from decoding settings like temperature (or even changing the seed). Many differences between settings here are 1 to 3 points, which may mean answering one more question correctly, so I don't think these comparisons can tell us which conditions actually matter for OPSD. I would follow [3]'s suggestions about using many evaluation seeds (30+) on small benchmarks and at least reporting the mean and std.

[1]: Luo et al., Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models, arXiv April 2026

[2]: Kim et al., Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?, arXiv March 2026

[3]: Hochlehnert et al., A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility, COLM 2025

Questions for authors
  1. Are "ineffective length growth" (abstract) and "ineffective deliberation" (Section 4) the same failure mode? If so, I would suggest sticking to one of these two terms and defining how to measure it.
  2. What is a "narrow compatibility regime"? How do we check whether a given model or OPSD algorithm only works in this narrow regime?
5 comments
AuthorsOpenReview ↗

Response to reviewer

Thanks for your rigorous reading and suggestions, which are significant to diagnose OPSD. We provide a revised version and hope to solve your concerns and questions.

To questions:

Q1: Question about "ineffective length growth" (abstract) and "ineffective deliberation" (Section 4).

A1: Yes and thanks for pointing out. "Ineffective length growth" and "ineffective deliberation" referred to the same failure mode. In the revision we drop both terms(Setion 4 at Page 3).

Q2: What is a "narrow compatibility regime"? How to check?

A2: A narrow compatibility regime means that OPSD improves over the base model only under a small set of jointly compatible settings, and otherwise produces degradation or collapse. In the revised Section 5.8(Page 7), we summarize this from the controlled experiments. OPSD tends to work better on smaller models or fully trained Instruct models, with shorter student rollouts and fewer than 100 training steps. No single KL objective or privileged context works for every model.

To check whether a given model or OPSD algorithm stays inside this regime, see whether it matches the experimental conclusions above. Also check whether the token signal is more concentrated in high-entropy regions, and whether the entropy and gradient norm stay stable. OPSD is compute-friendly, so the cost of this check is low.

To weakness:

W1: Quantitative defination of "Degradation" and "Collapse".

A1: Thanks for you rigorous suggestions. Collapse means the reasoning ability is totally ruined, which is apparent because it means absolutely 0 accracy and max rollout length. Degradation is an accuracy drop but the output is reasonable. There is no strict defination of Degradation, but we can classify. For example, we can say "accuracy declines 5% and length increases 10%" as slight degradation, "accuracy declines above 20% and length increases 50%" as heavy degradation. We report the difference in Section 4 at Page 3.

W2: Related work and its benefit to our work.

A2: Thanks for pointing us to Luo et al. and Kim et al. We discuss the two papers separately.

Luo et al. study OPD with a stronger teacher. They first apply SFT and then OPD, and they stabilize length inflation with a reference-KL constraint and mixture distillation. Because of page and compute limits, this paper focuses on diagnosing vanilla OPSD rather than OPD, so we do not verify those methods.

Kim et al. argue that more privileged information can degrade self-distillation. We verify this view in our diagnosis, and a comprehensive diagnosis naturally cites it. We borrow their idea, add an Unrelated teacher context (Section 5.3, Page 5), and extend the answer-only setting with answer transformations in Appendix E.1 (Page 32). These comparisons are run at a larger scale.

Overall, this paper is a broad diagnosis of OPSD. For similar work on one aspect, we verify the finding, borrow the idea, and explore it further, rather than propose a repair. We also study the post-training stage, find that shorter student rollouts perform better, and find that different KL objectives behave differently across models. To our knowledge, these results are first reported here, and they support the conclusion that OPSD is a sensitive algorithm. If there are more time and computation, we would like to take a deep research about related work.

W3: Many evaluation seeds to decrease the error and noise.

A3: We have added 3-seed evaluations for some experiments(Figure 4 at Page 7, Table 5(a) at Page 6, Appendix F.1 at Page 37) and the conclusions stay the same. We add the check at last of Appendix to report some experiments over 10 seeds. We also report confidence intervals in the appendix(Appendix F.3 at Page 40). In the next few days, we will extend more experiments to 10 seeds and add them to the paper.

Due to compute limits, we can scale models only up to 8B, and we study at most 11 models. This already exceeds many papers that diagnose OPD and OPSD [3, 4], and our experiments are more comprehensive, which makes the results more convincing. Reporting Avg@8 and Pass@8 over 8 samples itself reduces the noise of a single generation. We will expand as many experiments as possible to multiple seeds.

We sincerely hope this reply and the revised paper resolve your concerns and questions. We are willing to answer if anything remains unclear. If they do resolve your concerns and questions, we would be grateful if you could raise the score.

[1]: Luo et al., Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models, arXiv April 2026

[2]: Kim et al., Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?, arXiv March 2026

[3]: Li et al., Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe, arXiv April 2026

[4]: Kaur et al., Rethinking On-Policy Self-Distillation for Thinking Models, arXiv July 2026

Reviewer_CPhKOpenReview ↗

Response to authors

I thank the authors for taking the time to answer my questions and provide further clarification. Unfortunately, issues related to the definitions of the diagnosis terms (the bulk of the paper) and the lack of clear comparison to related work still remain. I hope to clarify that here.

Q1: Both of the terms are still in the document. Please ctrl-f to find them.

Q2: I see the updated list of conditions for the narrow compatibility regime, but that still does not answer my question, as there is no clear way to check if a given model or algorithm suffers from this.

W1: I appreciate the attempt to make these terms clearer, but they are still qualitative definitions, and they are not sufficient to declare OPSD “not a promising algorithm” (Section 5.8). Specifically, degradation is still defined qualitatively in the paper, and the thresholds in the response are arbitrary and post-hoc (and not even reflected in the updated paper). There is still no rule for how large a drop counts as degradation or how to separate it from evaluation noise (or why a certain threshold distinguishes noise from a qualitative degradation).

W2: I still do not understand why there is no direct comparison to Kim et al., and why it is not referenced in the section where the authors “borrow their idea.” It seems like it is first mentioned in the appendix (page 16)? A paper with this much overlap should definitely be cited in the main paper where it is most relevant.

W3: I agree with the authors that it is harder to run larger models, but the response does not really alleviate the concern that there is a lack of empirical evidence that the main results on 120 problems would generalize. I also want to clarify that the added seeds are evaluation seeds, while training is still a single run per condition. Several of the comparisons with small gaps (e.g., Tables 2, 3 and 4) are also still reported for a single evaluation seed.

AuthorsOpenReview ↗

Response to Reviewer CPhK (1/2)

Thanks for your time and follow-up constructive comments. We've updated the third revision, which may be helpful to address the concerns.

Q1: We've deleted the terms in the revision paper and replaced them with "degradation". We sincerely appreciate your pointing this out!

Q2: The phrase "narrow compatibility regime" was confusing, so we revise it to say that OPSD works only in a limited range of models and training configurations. To make the check concrete, given a model or OPSD variant, one can inspect the following factors against our experimental conclusions:

  • Checkpoint type: Instruct vs Reasoning / Think. Gains are more often observed on Instruct (or smaller) models than on strong Reasoning checkpoints, and Student mode should be matched to Teacher mode.
  • Student rollout length: shorter rollouts (e.g., 128--256) are preferred, longer rollouts more often degrade.
  • Training steps: fewer than 100 steps are preferred in our setting; longer training can worsen the trend.
  • KL objective, privileged context, and post-training stage: no single choice works for all models.

If most of these factors fall outside the preferred range above, OPSD is unlikely to perform better than the baseline and is more likely to degrade or collapse. Because OPSD is compute-friendly(training and evaluation cost less than 20 GPU hours), we can check how to choose kl, etc. We have modified expressions like "narrow regime" accordingly, and apologize for the earlier confusion.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of our revisions

We thank all three reviewers for their careful reading, constructive criticism and suggestions. All of the feedback is helpful to strengthen our experimental evidence and clarify the scope of our contributions.We provided the first modified version at 13 Nov and updated the second modified version at this time.

Our strengths are as follows:

  1. We design numerous experiments on many aspects of OPSD and find out when OPSD works or fails.

  2. Based on evaluations, we design analyses of token-level signal to explain the results.

  3. Our experiments are conducted on at most 11 models and cover many aspects of OPSD, which are lacking in the prior work. Our conclusions are robust and coherent.

Our main modifications of the paper are as follows:

  1. Thanks for the concern from Reviewer gYMa.
  • We rerun the experiments in Table3 of Section 5.3 at Page 5. The original version trains with randomly selected data, and we use exactly the same dataset in the same order to train and evaluate. We have updated these results in both revised versions.
  1. Thanks for the suggestions and questions from Reviewer 1HW2.
  • We modify Figure 1 at Page 3 and Figure 4 at Page 7.
  • We add explanatory sentences and conclusion in Section 5 to make the experiments more fluent and coherent.
  • We add Confidence Interval in Appendix F.3 at Page 40 to make the results and conclusions more convincing and robust.
  • We add the differences from prior work in Section 2.
  1. Thanks for the constructive criticism and questions from Reviewer CPhK.
  • We unify the terms like "ineffective length growth" and "ineffective deliberation" in the paper, which may confuse the reader.
  • We point out the "narrow compatibility regime" in Section 5.8 at Page 7, which is one of our main contributions.
  • We add evaluations over more seeds to strengthen the robustness of our conclusion(Table 1(c) at Page 4, Table 5(a) at Page 6, Figure 4 and Table 6 at Page 7, etc.). We add detailed results at the last section of Appendix.

Besides the suggestions from reviewers, we also fix some grammar errors, provide detailed metrics definitions(Table 7 at Page 8), etc.

Compared to the first modified version, the second modified version only changes in providing more seeds evaluation and some words.

All of these added experiments strengthen our experimental evidence and conclusion. We believe that our work is more robust and helpful to deeply analyse when and how OPSD works. We sincerely hope that our revision and comments could help you

P.S. We updated the third revised version on Nov 17. Compared with the previous two versions, we removed repetitive phrasing such as "ineffective length growth", revised some wording in the introduction, related work, and other sections, and added a citation to Kim et al in main text. The overall and detailed conclusions remain unchanged. These edits are only for smoother and more rigorous presentation. We thank the reviewers for their follow-up comments. We are willing to answer any further questions.

P.S. We updated the fourth revised version. We delete the overclaim "distribution difference is more important than semantic information" and replace it with "both of them are important and distribution gap base on priviledged information can gains more than baseline". Specifically, we revise Section1 Introduction and Section 6.2 Priviledged Information Effects. We also add some more-seed experiments at the last section of Appendix. The remaining conclusion and experiments keep fixed.

AuthorsOpenReview ↗

Response to all reviewers.

We sincerely thank all reviewers for their thoughtful comments and valuable feedback. We have made every effort to carefully address each point and have revised the manuscript accordingly.

As the rebuttal period is drawing to a close, we would sincerely appreciate it if you could share any remaining questions or concerns. We genuinely hope to have the opportunity to address them, as your feedback would help us further clarify and improve this work. We remain more than willing to respond promptly to any further questions or suggestions.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.