acceptodds
Under review as a conference paper at ICLR 2027

On-Policy Self Distillation As Automatisation: When and Why Thinking Collapses in Reasoning Models

Abstract

On-policy self-distillation (OPSD) is a promising post-training technique in which a model teaches itself by distilling from a copy which is granted , such as the ground-truth answer to the task. It outperforms verifier-based methods like GRPO while making models more reasoning token efficient. However, on some tasks OPSD cuts reasoning so aggressively that accuracy drops sharply, a phenomenon called . Using the framework of usable, or -, information, we find that the longer and more relevant to the answer the privileged context is, the more the student shortens its reasoning, yet this shortening turns into collapse only the model generated that context itself. To explain , we draw on a classic distinction from cognitive neuroscience between slow, deliberate thinking () and fast, practiced responses (). OPSD can make reasoning automatic: the model stops extracting usable information about the correct answer as it reasons, and terminates earlier. Such automatic reasoning cannot adapt its effort to the task, leading to performance degradation as task difficulty increases, even outside the training domain. We additionally uncover a previously undocumented failure mode of OPSD: students inherit the teacher's trust in context, which makes them easier to misalign when that context is malicious, a risk for robustness and safety. Thinking collapse is thus practice-driven automatisation rather than a training fault.

Primary area:
interpretability and explainable AI
Keywords:
on-policy self distillation, cognitive control, information theory, context reliance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_91VMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper interprets thinking collapse in on-policy self-distillation (OPSD) as over-automatisation: students acquire efficient but inflexible reasoning procedures from privileged-context teachers. Experiments with two model families show that rollout-based contexts can shorten reasoning and reduce accuracy on difficult tasks, including outside the training domain. A synthetic shortest-path task separates context informativeness from length, while pointwise usable-information measurements relate context access and reasoning depth to answer likelihood. Further experiments examine adaptation to unseen problems and susceptibility to malicious context. The paper argues that these observations connect reasoning compression, reduced flexibility, and context reliance within a common cognitive framework.

Critical strengths
  1. I find the difficulty-dependent transfer results compelling, particularly the contrast between preserved performance on easy ZebraLogic puzzles and substantial degradation on XL puzzles in Figure 1. This establishes a practically important limitation of OPSD that would be missed by reporting reasoning efficiency or average accuracy alone.

  2. The synthetic intervention on context informativeness and padding length is a useful attempt to move beyond qualitative comparisons of privileged contexts. Together with the malicious-context evaluation, it provides concrete evidence that context design affects both reasoning behavior and downstream robustness.

Critical weaknesses
  1. I find automatisation an interesting organizing hypothesis, but the main text does not sufficiently distinguish it from learning a premature stopping policy or adopting context-dependent shortcuts. Shorter traces, poorer difficult-task performance, and reduced gains in answer likelihood are compatible with these alternatives, so I am not yet convinced that the cognitive framework explains the observations beyond redescribing them.

  2. The padded answer-only controls address length, but they do not match rollout contexts in solution content, reasoning style, redundancy, or alignment with the model’s output distribution. Consequently, the conclusion that self-generation causes “over-practice” seems stronger than the evidence warrants; I would find a comparison against matched solutions from another model substantially more persuasive.

  3. The PVI analysis supports association more clearly than mechanism. Equation 4 changes both the supplied context and the sampled reasoning trace, while Figure 4 pools settings that already differ in informativeness and padding. Likewise, Figure 5 measures gold-answer likelihood along each model’s own traces, so its differences do not establish whether unproductive reasoning causes early termination or accompanies an already degraded policy.

  4. The proposed connection between collapse and safety vulnerability is incomplete. Figure 6 identifies FS as the most vulnerable condition, although FS does not exhibit the pronounced collapse observed for RC/RU in the primary Qwen experiments. I consider the safety finding valuable independently, but this ordering weakens the impression that lost reasoning flexibility and malicious-context reliance are manifestations of one established mechanism.

Questions for authors
  1. What intervention would distinguish over-automatisation from a learned stopping bias in the collapsed students? Testing whether an enforced additional reasoning budget restores accuracy would help determine whether the main impairment concerns termination or the reasoning procedure itself.

  2. Does the collapse persist when self-generated and externally generated solution contexts are matched for length, correctness, and reasoning style? This comparison would clarify whether self-generation has a distinctive effect beyond the properties of the resulting text.

  3. How does the proposed framework explain FS producing the largest safety degradation while RC/RU produce the strongest thinking collapse? Clarifying whether these outcomes reflect separable processes would make the scope of the central explanation more convincing.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 91VM (1/3): stopping policy and budget forcing

We thank the reviewer for finding the difficulty-dependent transfer compelling and the synthetic and malicious-context experiments useful. Following the review, we (i) tested whether the collapsed students learned a stopping bias, by tracking how ready they are to end their reasoning at every step and by forcing them to reason as long as Base; (ii) trained students whose teacher is given correct rollouts written by other models, and a control in which the model's own rollout is cut to the length of another model's; (iii) measured how close each rollout is to the model's own reasoning; (iv) computed the teacher PVI of every math context; and (v) retrained all students on a single training set shared by every context, re-running almost every evaluation in the paper. The new results support the automatisation account over the alternatives, and we address each point below.

W1+Q1

(a) The main text does not sufficiently distinguish it from learning a premature stopping policy or adopting context-dependent shortcuts. [...] I am not yet convinced that the cognitive framework explains the observations beyond redescribing them.

We tested both alternatives directly: the collapsed students do not learn to stop earlier, and their shortening is not tied to any context. A stopping policy would make a student readier than Base to end its reasoning while it is still working on the problem. On the 300 problems of the new test set, we fed each student's own rollouts back to it and tracked, at every position of the reasoning, the rank of the end-of-thinking token </think> in its next-token distribution (new Appendix C.6, Fig. 20). Students of the model's own rollouts (rollout-correct, RC; rollout-unfiltered, RU) rank it lower than Base, not higher, throughout their reasoning (typically around 84,000th, against 69,000th for Base), and the more a student collapses, the lower it ranks the token. They do not stop prematurely; they stop because they have learned to allocate less computation to the problem. As for the reviewer's second alternative, context-dependent shortcuts would not transfer the way collapse does: students are evaluated without any context, yet they shorten on problems and domains never seen in training (ZebraLogic, MBPP+). This is what automatisation of a procedure predicts (Section 2): a compiled routine runs wherever it is triggered. The framework also makes a prediction of its own: an automatised routine no longer scales its effort with the problem, so a collapsed student should cut the hardest problems the most. We test this with a new measure, the effort elasticity (new Fig. 5b): the slope of a student's log reasoning length against Base's across test problems, with Base's length as a measure of difficulty. A slope of 1 means every problem is shortened by the same fraction, so effort still grows with difficulty; a slope below 1 means the hardest problems are cut the most. Only the students of rollouts fall below 1 (0.76–0.77 for RC, RU and RC-R1; 0.92 for RC-G4), while those of answers (AO) and solutions (FS) keep about 1 (1.00–1.12), including the full solution padded to the length of RC's context (1.06). Neither context length nor a stopping policy, ruled out above, accounts for this pattern. The collapsed students thus reason through a shorter procedure, rather than stopping early or exploiting cues in the context.

(b) Testing whether an enforced additional reasoning budget restores accuracy would help determine whether the main impairment concerns termination or the reasoning procedure itself.

We ran the suggested intervention: a longer budget does not restore accuracy, because the collapsed students repeat themselves instead of reasoning further. On ZebraLogic, we forced every student to reason at least as long as Base does on each puzzle, replacing every earlier end-of-thinking token with "Wait" (budget forcing, as in s1 [1]; new Appendix C.7, Fig. 21, Table 6). The collapsed students are forced on about 9 puzzles out of 10, and their accuracy falls instead of recovering (RC: from 63.0% to 46.7%). On the puzzles where forcing does not cause a repetition loop, no student gains accuracy. What separates the students is how they use the extra budget. Forced to add 1,000–2,000 tokens past their first stop, RC and RU fall into a loop in 46% and 62% of their rollouts, against at most 3% for the students of answers and solutions forced to add as much on puzzles of the same size. A student that had stopped too early would use the extra budget to keep reasoning; the collapsed students instead gain nothing from it and often loop. This agrees with the direct measurement in (a): they are further from stopping than Base throughout their reasoning, and the end-of-thinking token climbs the ranking only in its last percent (new Appendix C.6, Fig. 20b). The impairment therefore concerns the reasoning procedure itself, not its termination.

AuthorsOpenReview ↗

Response to Reviewer 91VM (2/3): rollouts from other models and length-matched controls

W2+Q2

(a) The padded answer-only controls address length, but they do not match rollout contexts in solution content, reasoning style, redundancy, or alignment with the model's output distribution. [...] I would find a comparison against matched solutions from another model substantially more persuasive.

We trained two Qwen3-8B students whose teacher has access to a correct rollout from another model, and showed that the closer a rollout is to the model's own output distribution, the stronger the collapse. The teacher's privileged context is a correct rollout from DeepSeek-R1 (RC-R1: its reasoning from the dataset followed by the worked solution) or from Gemma-4-E4B (RC-G4). Both also collapse, but less than with the model's own rollouts: on AIME-24, pass@1 drops by 27.7 points for RC, 18.5 for RC-R1 and 8.9 for RC-G4, and on the hardest ZebraLogic puzzles accuracy falls from 38% for Base to 5–7% for RC and RU, against 15–20% for RC-R1 and RC-G4 (updated Fig. 1). We measured the alignment the reviewer mentions as the likelihood of each rollout under base Qwen3-8B: the model finds its own rollouts the most likely, then DeepSeek-R1's, then Gemma-4-E4B's, the same order as their collapse (new appendix Fig. 17). Both students also cut the hardest problems the most, with an effort elasticity below 1 (W1 (a)). Collapse thus arises whenever the teacher is given a rollout, and grows with how close the rollout is to the model's own reasoning; we revised the abstract, introduction and conclusion accordingly.

(b) Does the collapse persist when self-generated and externally generated solution contexts are matched for length, correctness, and reasoning style?

We added two length-matched controls: a worked solution padded to the length of the model's own rollouts, and the model's own rollout cut to the length of Gemma-4-E4B's. The full-solution + random (FS+R) context pads the dataset's worked solution with random tokens, so it is as long as RC's (median 6.0k vs 5.4k tokens; new appendix Fig. 16), correct and complete. Its student shortens reasoning by 49% on the test set, close to RC's 60%, yet does not collapse on AIME-24 and keeps Base's elasticity (1.06). In the second control, the teacher is given the model's own correct rollout, with its reasoning truncated so that, on every problem, it is as long as Gemma-4-E4B's rollout; the final answer is kept whole. Correctness and length then match RC-G4, and only the author differs. Despite more than halving the context, this student behaves more like RC than like RC-G4, and cutting the rollout reduces collapse only in part: AIME-24 pass@1 drops by 19.4 points, against 27.7 for the full rollout and 8.9 for Gemma-4-E4B's rollout of the same length, and on the hardest ZebraLogic puzzles accuracy is 10%, against 20% for RC-G4. On AIME-24, AIME-25 and ZebraLogic, its accuracy and rollout length lie 53–71% of the way from RC-G4 to RC (new Appendix C.5, Table 5). At equal length and correctness, the model's own reasoning collapses the student about twice as much as another model's, so self-generation has an effect beyond length and correctness.

AuthorsOpenReview ↗

Response to Reviewer 91VM (3/3): Eq. 4, Fig. 5 and safety results

W3

(a) Equation 4 changes both the supplied context and the sampled reasoning trace, while Figure 4 pools settings that already differ in informativeness and padding.

We clarified that Eq. 4 measures the total effect of the context on the teacher, and added a test on math in which every context is a single point. In OPSD the teacher conditions on the context and reasons before every token it scores, so what the student is distilled toward is the information the teacher extracts by reasoning with the context; Eq. 4 measures exactly this, with the reasoning trace as the route through which the context acts. In the graph task the intervention is on the context alone (information density and padding, with the problem fixed), and Fig. 3 shows the effect of each manipulation separately; Fig. 4 then asks whether one quantity summarises both. On math, each of the nine contexts is one point (new Fig. 5a): the more usable information a context gives the teacher, the more its student shortens (Spearman ρ = 0.75). We therefore present PVI as a measure of how much the context lets the teacher, and hence the student, shorten its reasoning, not of whether the shortening turns into collapse: the padded full solution has the highest PVI of any context, yet does not collapse.

(b) Figure 5 measures gold-answer likelihood along each model's own traces, so its differences do not establish whether unproductive reasoning causes early termination or accompanies an already degraded policy.

We agree that Fig. 5 (now Fig. 5c) is correlational, and two new experiments now separate the two readings. If the reasoning were unproductive because the students end it too early, they would be readier than Base to stop while still reasoning, and would recover when forced to reason longer. Instead, they rank the end-of-thinking token lower than Base at every point of their reasoning, and when forced to reason as long as Base they repeat themselves, gaining no accuracy and little information about the answer (W1; new Appendix C.6–C.7, Figs. 20–21). Unproductive reasoning thus accompanies an already degraded procedure rather than causing early termination.

W4+Q3

Figure 6 identifies FS as the most vulnerable condition, although FS does not exhibit the pronounced collapse observed for RC/RU in the primary Qwen experiments. [...] How does the proposed framework explain FS producing the largest safety degradation while RC/RU produce the strongest thinking collapse?

With the retrained students, the rollout students are now the most vulnerable on Qwen3-8B on JailbreakBench and BeaverTails, and Section 2 explains why collapse and context reliance can still separate. On JailbreakBench with a harmful reference, RC, RU, RC-R1 and RC-G4 raise the attack success rate by 5.7–10.4 points over Base, above FS (+4.0; updated Fig. 6). The framework, however, does not require the two orderings to coincide. Section 2 distinguishes two things OPSD can automatise in the student's procedure: a curtailed reasoning budget, and a readiness to defer to whatever context is supplied. Every OPSD teacher answers from its context, so every student can inherit the second; only rollout contexts transfer the first strongly enough to collapse. A student can therefore defer to context without collapsing, as FS does on Gemma-4-E4B, where it is as compromised as the rollout students (appendix Fig. 22). Collapse and context reliance are thus two separable products of procedural automatisation, which is why their orderings need not match.

Summary

We tested the alternatives the reviewer raised, tracking how ready the students are to stop, forcing them to reason longer and adding length-matched and cross-model contexts, and retrained all students on a single shared training set. The collapsed students are further from stopping than Base throughout their reasoning and repeat themselves when forced to continue, collapse grows with how close the context is to the model's own reasoning, and context reliance is a separable part of the same procedural automatisation. We hope these additions address the reviewer's concerns.

References

[1] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. EMNLP, 2025.

Reviewer_tJ8DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies thinking collapse in OPSD, where a model trained to match a copy of itself that sees privileged context ends up reasoning too briefly, causing accuracy drop. The authors train Qwen3-8B and Gemma-4-E4B on math with 4 teacher contexts: the answer only, the dataset's worked solution, and the model's own rollout that's filtered for correctness or not. Only the rollout contexts cause collapse, and it hurts harder ZebraLogic puzzles much more than easy ones. A synthetic shortest path task varies how informative and how long the context is, and PVI predicts both accuracy and how much reasoning shortens. Students trained on the answer padded with random tokens shorten without losing accuracy, which the paper reads as collapse coming from practice on the model's own traces (automatisation), answering with minimal tokens instead of reasoning.

Critical strengths
  1. The shortest path task is an effective way to separate how informative the context is from just how long it is, which are entangled in real data. The padding result is interesting, that random filler shortens reasoning and hurts accuracy, as explained by Figure 3c. My understanding is that the model's urge to stop depends on absolute position, so padding pushes its reasoning to where it's already likely to stop.

  2. The difficulty gradient and the safety result are useful for OPSD users. Collapse barely shows on easy tasks like small ZebraLogic puzzles and MBPP+ but is severe on the hardest ones, so it's easy to miss with easy evals but revealed in this study. Students also follow a harmful reference in context more readily than the base model, which is a novel finding.

Critical weaknesses
  1. Experiment control: There is no isolated experiment to support the new claim "this shortening turns into collapse only when the model generated that context itself". The rollout contexts are both much longer than the worked solution and the model's own text, and the padded-answer students control for length but have nothing beyond the answer. There's no context that's long and informative but written by something other than the student, so "over-practice" on its own traces can't be separated from the teacher just seeing a long complete solution. This matters since self-generation is the paper's explanation for why collapse happens.

  2. Novelty: the main in-domain result sits close to [1], who train SDPO (the teacher sees the model's own successful rollout) and find shorter outputs with lower accuracy on math, mitigated when the thinking is removed from that context. Beyond that, the evidence for the information account comes from the shortest path task, which only looks at the teacher without training, and PVI is never computed for the math contexts the students are trained on. Therefore, the claim, how usable information would predict how much the students shorten, is not well-supported.

[1] Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?

Questions for authors
  1. Can you train on long (correct) rollouts from a different model, for example each model on the other model's rollouts? If those don't collapse, the self-generation claim would be strengthend.
  2. What's the PVI of each math context (AO, FS, RC, RU, padded), and does it order how much each student shortens?
5 comments
AuthorsOpenReview ↗

Response to Reviewer tJ8D (1/2): long, informative contexts not written by the student

We thank the reviewer for their time and consideration – we’re glad you found the shortest-path task an effective way to separate informativeness from length. Additionally we’re glad you thought the difficulty gradient and safety results will be useful for OPSD users. Following the review, we:

  • (i) trained the two controls the reviewer asked for, contexts that are long and informative but not written by the student: correct rollouts from other models and a full solution padded to the length of the model's own rollouts;
  • (ii) cut the model's own rollouts to the length of another model's, to separate who wrote a rollout from how long it is;
  • (iii) computed the teacher PVI of every math context and related it to how much each student shortens;
  • (iv) introduced a measure of how each student scales its effort with difficulty; and
  • (v) retrained all students on a single training set shared by every context, re-running almost every evaluation in the paper.

The new results strengthen our claim, and we address each point below.

W1+Q1

(a) There's no context that's long and informative but written by something other than the student, so "over-practice" on its own traces can't be separated from the teacher just seeing a long complete solution.

We added a full solution padded to the length of the model's own rollouts, and showed that seeing a long, complete solution shortens reasoning without causing collapse. We would like to note that the submitted paper already included long contexts not written by the student: the padded answer-only contexts (answer-only + random, AO+R; random + answer-only, R+AO; Section 5 and Appendix C.4 of the submission) are nearly as long as the rollout contexts and contain the gold answer, and their students shorten reasoning without losing accuracy on held-out problems, whereas the students of the model's own rollouts collapse. We have now extended this control to a complete solution. The new full-solution + random (FS+R) context pads the dataset's worked solution with random tokens, so it is as long as the model's own rollout-correct (RC) context (median 6.0k vs 5.4k tokens; new appendix Fig. 16) and contains a complete solution. On a new test set of 300 problems from the training distribution, its student shortens reasoning by 49%, close to RC's 60%, yet it does not collapse on AIME-24 (−6.3 pp pass@1, against −27.7 for RC). The two students shorten in different ways. We measure how each student scales its effort with difficulty by the slope of its log reasoning length against Base's across test problems (effort elasticity, new Fig. 5b): a slope of 1 means every problem is shortened by the same fraction, and a slope below 1 means the hardest problems are cut the most. FS+R keeps Base's allocation (1.06), like the answer-only (AO) and full-solution (FS) students, padded or not (1.00–1.12), while RC falls to 0.76. A long, complete solution thus shortens reasoning appropriately; only rollout contexts turn this shortening into collapse.

(b) Can you train on long (correct) rollouts from a different model, for example each model on the other model's rollouts?

We trained two Qwen3-8B students whose teacher is conditioned on a correct rollout written by another model, and showed that the closer a rollout is to the model's own reasoning, the stronger the collapse. Following the reviewer's suggestion, one teacher sees a correct rollout from Gemma-4-E4B (RC-G4); the other receives one from DeepSeek-R1 (RC-R1: its reasoning from the dataset followed by the worked solution). Both collapse, but less than with the model's own rollouts: on AIME-24, pass@1 drops by 27.7 points for RC, 18.5 for RC-R1 and 8.9 for RC-G4, and on the hardest ZebraLogic puzzles accuracy falls from 38% for Base to 5–7% for RC and rollout-unfiltered (RU), against 15–20% for RC-R1 and RC-G4 (updated Fig. 1). This order follows how close each rollout is to the model's own reasoning: base Qwen3-8B finds its own rollouts the most likely, then DeepSeek-R1's, then Gemma-4-E4B's (new appendix Fig. 17). Since Gemma-4-E4B's rollouts are also shorter, we added a length-matched control: the teacher is given the model's own correct rollout, with its reasoning cut and marked "[...]" so that, on every problem, it is as long as Gemma-4-E4B's; the final answer is kept whole (RC-len-G4; new Appendix C.5, Table 5). Despite more than halving the context, the student behaves more like RC than like RC-G4: its AIME-24 pass@1 drops by 19.4 points, against 27.7 for RC and 8.9 for RC-G4, and on AIME-24, AIME-25 and ZebraLogic its accuracy and rollout length lie 53–71% of the way from RC-G4 to RC. Collapse thus arises whenever the teacher is given a rollout, and is strongest when the rollout is the model's own, even at equal length; we revised the abstract, introduction and conclusion accordingly.

AuthorsOpenReview ↗

Response to Reviewer tJ8D (2/2): relation to Kim et al. and PVI of the math contexts

W2+Q2

(a) Novelty: the main in-domain result sits close to [1], who train SDPO (the teacher sees the model's own successful rollout) and find shorter outputs with lower accuracy on math, mitigated when the thinking is removed from that context.

We clarified how our study departs from [1]: we vary the properties of the context independently, estimate the information it gives the teacher, and follow collapse out of domain. We agree that [1] first documented this in-domain effect, and we follow them in training on mathematics. [1] vary one context, the correct solution with or without its thinking, and order them by conditional mutual information through information-ordering arguments, without estimating it. Removing the thinking removes both the reasoning and almost all of the tokens (640 of 13,054 in their Section 3), so information, length and the presence of reasoning change together. We vary them separately: the answer only, the dataset solution, the model's own and other models' rollouts, padded answers and solutions, and a synthetic task that sets information density and length independently. These controls show that information alone does not decide collapse: FS+R gives the teacher the most usable information of any context (point (b)), yet its student does not collapse. [1] observe collapse in mathematics but not in chemistry, and attribute the difference to task coverage: diverse, unseen problems need the uncertainty that self-distillation suppresses. Our results point to difficulty instead of domain or novelty: math-trained students also collapse on logic puzzles, the more so the harder the puzzle, while easy puzzles are unaffected, and they cut the hardest problems the most whether or not they trained on them (effort elasticity within 0.03 on training and test problems). They also come to rely more on harmful context. The in-domain shortening is thus our starting point; our contribution is to identify which property of the context turns it into collapse, and how far the collapse reaches.

(b) PVI is never computed for the math contexts the students are trained on. What's the PVI of each math context (AO, FS, RC, RU, padded), and does it order how much each student shortens?

We computed the teacher PVI of all nine math contexts on the training problems, and showed that it orders how much each student shortens. The teacher PVI measures how many more bits base Qwen3-8B assigns to the gold answer after conditioning on the context and reasoning, compared with reasoning without it:

AOFSAO+RRC-G4RUR+AORCRC-R1FS+R
1.75.68.08.28.48.79.210.512.1

(AO+R and R+AO pad the answer with random tokens, after or before it.) The more usable information a context gives the teacher, the more its student shortens on the new test set (Spearman ρ = 0.75 across contexts; new Fig. 5a), as on the graph task. PVI does not, however, decide whether the shortening is harmful: FS+R has the highest PVI and shortens about as much as the rollout contexts, but keeps Base's allocation of effort and does not collapse (point W1 (a)). Usable information thus sets how much the student shortens, while the content of the context decides whether this shortening turns into collapse.

Summary

We trained the controls the reviewer asked for, long and informative contexts not written by the student, and computed the teacher PVI of every math context. A padded full solution shortens reasoning without collapse, rollouts from other models collapse less than the model's own even when it is cut to the same length, and teacher PVI orders how much each student shortens, which sharpens our claim: collapse comes from giving the teacher a rollout, most strongly the model's own, not from length or information alone. We hope these additions address the reviewer's concerns.

References

[1] Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. Why does self-distillation (sometimes) degrade the reasoning capability of LLMs? arXiv:2603.24472, 2026.

AuthorsOpenReview ↗

Discussion closes in 2 days: long, informative contexts not written by the student

Dear Reviewer tJ8D,

thank you again for your review. With about two days left in the discussion, we wanted to make sure our responses reached you. We added long, informative contexts not written by the student (a padded full solution, and rollouts from DeepSeek-R1 and Gemma-4-E4B), clarified how our study departs from Kim et al., and computed teacher PVI on the math contexts. We are happy to answer any further questions. If these additions address your concerns, we would appreciate it if you would consider revisiting your score.

Thank you,

The Authors

Reviewer_ik2qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies "thinking collapse" in on-policy self-distillation (OPSD), where reasoning gets shortened so much that accuracy drops, and asks when and why it happens.

The authors frame OPSD as moving the model from controlled to automatic processing, borrowing from cognitive neuroscience, and use V-usable information to measure what a privileged context provides. They train Qwen3-8B and Gemma-4-E4B on math with four privileged contexts (answer only, dataset solution, and the model's own correct or unfiltered rollout), evaluate in and out of domain, and add a synthetic shortest-path task and safety benchmarks.

Some important findings: the authors report that only the long, self-generated rollout contexts cause collapse, which transfers out of domain and gets worse with task difficulty. The teacher's reasoning length under a context closely predicts the student's, and in the synthetic task both informativeness and length shorten the teacher's reasoning. The paper also reports that OPSD students follow harmful or biasing context more readily than the base model.

Critical strengths
  1. The empirical pattern of collapse is useful: it appears only with the long rollout contexts, transfers out of domain, and grows with task difficulty, which is practical guidance on when OPSD is risky.
  2. Teacher length predicting student length is a cheap diagnostic before training, and the synthetic task nicely separates informativeness from length.
  3. The writing is clear and easy to follow.
Critical weaknesses

There are a few things that I want to point out:

  1. In my opinion, the fundamental issue is that the paper's "why" is never isolated. The abstract says shortening turns into collapse "only when the model generated that context itself", but the rollout-context students differ from the others in more than who wrote the context. They are trained on a different set of problems, since problems without a correct rollout are replaced, so "only c varies" does not hold. The SFT control trains on the same self-generated rollouts and does not collapse, which goes against "practice" as the cause. As far as I can tell, every result in Sections 3 to 5 is also predicted by a simpler account where longer context leads to more shortening, so the automatisation framing does not add much explanatory power.

  2. I also have some concerns about the eval. Rollouts that never finish are excluded from both accuracy and length, and the skip rates are not reported, so I'm not sure in which direction this shifts the comparison, given that the base model reasons the longest. Relatedly, the "unseen" set used in Figures 2 and 5 is selected as problems the base model failed, yet the base model scores around 83% on it in the appendix. Is it possible that the selection mostly captured truncation rather than difficulty?

  3. The information-theoretic account is not measured where the main claims are made. Context PVI is only computed for the base model on the synthetic graph task, never for the math contexts the students were trained on. Also, the abstract says automatised reasoning "cannot adapt its effort", but the collapsed students' reasoning still grows several-fold from easy to hard puzzles. As tested, the account reads more like an analogy than an explanation.

Questions for authors
  1. How many problems were replaced for the rollout contexts? Does a student still collapse when its teacher reads another model's rollout, or a padded dataset solution, on the same problems?
  2. What are the skip rates per configuration, and do the results hold if non-terminating rollouts count as wrong?
  3. Can you report context PVI for the math contexts the students were trained on?
7 comments
AuthorsOpenReview ↗

Response to Reviewer ik2q (1/3): shared training set and rollouts from other models

We thank the reviewer especially for recognising that the privileged context determines whether OPSD collapses and that this can be predicted at inference time. In response to your feedback we have

  • (i) retrained the 12 students from our original submission and trained 8 new ones on a single training set shared by every context and both models;
  • (ii) added contexts that separate who wrote the context and how long it is from what it contains (rollouts from other models, padded solutions);
  • (iii) computed the teacher PVI of every math context;
  • (iv) introduced a measure of how each student scales its effort with difficulty;
  • (v) rebuilt the seen and unseen evaluation sets and now report skip rates throughout; and re-ran almost every evaluation in the paper.

Our results hold well, and we address each point below.

W1+Q1

(a) They are trained on a different set of problems, since problems without a correct rollout are replaced, so "only c varies" does not hold.

We rebuilt the training set so that every context uses exactly the same problems, and retrained all students on it. In the original setup, the rollout-correct (RC) students shared 59% (Qwen3-8B) and 54% (Gemma-4-E4B) of their training prompts with the answer-only (AO) and full-solution (FS) students, since problems without a correct rollout were replaced. To remove this confound, we built a single set of 4,765 problems shared by every condition and both models (100% prompt overlap). Rather than replacing problems, we sampled extra rollouts until each model had a correct one for RC, while keeping the original share of correct contexts for rollout-unfiltered (RU). We then retrained all students on this set for one epoch, so the privileged context c is now the only thing that varies. We have updated all the impacted evaluations in the paper to reflect this change.

Our main findings are untouched by such a modification, showing that privileged context – not training data composition – is the main cause of collapse (updated figures and discussions), with the model's own rollouts still causing by far the greatest collapse (see updated Fig. 1).

(b) The abstract says shortening turns into collapse "only when the model generated that context itself", but the rollout-context students differ from the others in more than who wrote the context.

We added a new condition where the privileged context is a correct rollout written by another model, and showed that the closer a rollout is to the model's own reasoning, the stronger the collapse. We thank the reviewer for this suggestion, which strengthens our claim. The teachers of the two Qwen3-8B students are given a correct rollout from DeepSeek-R1 (RC-R1) or from Gemma-4-E4B (RC-G4). Both also collapse, but less than with the model's own rollouts (RC): on AIME-24, pass@1 drops by 27.7 points for RC, 18.5 for RC-R1 and 8.9 for RC-G4 (updated Fig. 1). The order of collapse follows how close each rollout is to the model's own reasoning. In a new appendix figure (Fig. 17), base Qwen3-8B finds its own rollouts the most likely, and DeepSeek-R1's more likely than Gemma-4-E4B's, at every position and trace length. This similarity may explain why RC-R1 collapses more than RC-G4.

Because traces sampled from different models have different lengths, we ran a new experiment to remove that confound. The new experiment shows that this is not only a length effect: when Qwen3-8B's own rollout is cut to the length of Gemma-4-E4B's (RC-len-G4; new Appendix C.5), the Qwen3-8B student still behaves more like RC than like RC-G4 (AIME-24 pass@1 drops by 19.4 points, against 27.7 for RC and 8.9 for RC-G4). Collapse thus arises whenever the context is a rollout, and most strongly when it is the model's own. We revised the abstract, introduction and conclusion accordingly.

AuthorsOpenReview ↗

Response to Reviewer ik2q (2/3): context length and the SFT control

(c) As far as I can tell, every result in Sections 3 to 5 is also predicted by a simpler account where longer context leads to more shortening, so the automatisation framing does not add much explanatory power.

We added a padded full-solution control as long as the RC context, and a new measure of how each student allocates its effort across problems. We would like to note that the submitted paper already tested this account: the padded answer-only contexts (answer-only + random, AO+R; random + answer-only, R+AO; Section 5 and Appendix C.4 of the submission) are nearly as long as the rollout contexts and shorten the student's reasoning without losing accuracy on held-out problems, whereas RC and RU collapse. We have now strengthened this control. Context length alone does not predict collapse. The new control, full-solution + random (FS+R), pads the worked solution with random tokens to a context as long as RC's (median 6.0k vs 5.4k tokens). Like AO+R and R+AO, FS+R shortens reasoning without collapsing (−6.3 pp pass@1 on AIME-24, vs −27.7 for RC), while RC-R1 and RC-G4 hurt more despite much shorter contexts (context lengths of every condition in the new appendix Fig. 16). The two kinds of shortening also differ in kind. Base reasons longer on harder problems. A student that automatises appropriately should keep this: it can reason less overall, but should still spend more on the problems Base finds hard. We measure this with the slope of a student's log reasoning length against Base's across test problems (new Fig. 5b). A slope of 1 means every problem is shortened by the same fraction, so effort still grows with difficulty; a slope below 1 means the hardest problems are cut the most. Students of answer and solution contexts, padded or not, keep a slope of about 1 (1.00–1.12). Students of rollouts fall below (0.76–0.77 for RC, RU and RC-R1; 0.92 for RC-G4, in between), and only RC, RU and RC-R1 lose test-set accuracy. FS+R and RC, with similar context lengths, fall on either side (1.06 vs 0.76). Length thus sets how much the student shortens, but the content of the context decides whether this shortening is appropriate or turns into collapse.

(d) The SFT control trains on the same self-generated rollouts and does not collapse, which goes against "practice" as the cause.

We now define practice explicitly in Section 2 and distinguish it from imitation, which is what supervised fine-tuning (SFT) does. A skill becomes automatic when its answer is retrieved rather than computed [1], and practice is what drives this shift: the learner performs the task itself and is corrected by someone who already knows the solution. OPSD is practice: the student solves each problem on its own, and every step is scored by a teacher that has already seen a solution, which rewards reaching the answer without the deliberation the teacher no longer needs. SFT on the same rollouts is imitation: it trains the student to reproduce every step, deliberation included, so nothing pushes it to do less. Accordingly, SFT keeps Base's length (−1% on AIME-24, +3% on the new test set), while every OPSD student shortens on the new test set (18–60%). Practice automatises most when the teacher's context is a rollout close to the model's own reasoning, which matches the order of collapse (point (b)). The SFT control thus shows that the rollouts collapse the student only when they serve as the teacher's context in practice, not as targets to imitate.

AuthorsOpenReview ↗

Response to Reviewer ik2q (3/3): skip rates, effort elasticity and teacher PVI

W2+Q2

(a) Rollouts that never finish are excluded from both accuracy and length, and the skip rates are not reported, so I'm not sure in which direction this shifts the comparison, given that the base model reasons the longest.

We now report skip rates for every configuration, and recompute the most affected benchmark with skipped rollouts counted as failures. Skip rates (rollouts that reach the token limit) for every model, benchmark and student are in a new appendix table (Table 4). They are highest for Qwen3-8B Base on AIME-25 (12.6%), so we test the reviewer's concern there by counting every skipped rollout as a failure (new appendix Fig. 9). Excluding skipped rollouts does not affect the pattern of results: with skips counted as failures, RC and RU still lose 34–35 pp at pass@1 and stay significantly below Base at every k.

(b) Relatedly, the "unseen" set used in Figures 2 and 5 is selected as problems the base model failed, yet the base model scores around 83% on it in the appendix. Is it possible that the selection mostly captured truncation rather than difficulty?

In response to your thoughts, we have added an effort elasticity analysis that targets difficulty more precisely as an independent variable, and it cleanly separates appropriately from inappropriately automatised students. The effort elasticity takes how long Base reasons on each problem as its difficulty, and measures how a student's reasoning scales with it: the slope of the student's log reasoning length against Base's (Fig. 5b). The students of answers and solutions, padded or not, keep an elasticity of 1.00–1.12: they shorten every problem by a similar fraction and keep their accuracy, which is appropriate automatisation. RC, RU and RC-R1 fall to 0.76–0.77: they cut the hardest problems the most and are the only students that lose accuracy on the new test set, which is over-automatisation. We also rebuilt the evaluation sets: a test set of 300 problems drawn from the training distribution (same selection and mix, no overlap with training) and a matched set of 300 training problems. The elasticity is the same on both (within 0.03; new appendix Fig. 18). Over-automatisation is thus a property of the student, not of the novelty of the problem. The seen/unseen comparison, which no longer separates the students on these sets, is now in the appendix (Fig. 18).

W3+Q3

(a) Context PVI is only computed for the base model on the synthetic graph task, never for the math contexts the students were trained on.

We computed the teacher PVI of every math context and showed that it predicts how much each trained student shortens, reproducing the behaviour seen in the synthetic graph task. On the training problems, the more usable information a context gives the teacher, the more its student shortens on the new test set (Spearman ρ = 0.75 across contexts; new Fig. 5a). The result of the graph task thus carries over from the teacher at inference to the student after training, and from a synthetic task to math.

(b) Also, the abstract says automatised reasoning "cannot adapt its effort", but the collapsed students' reasoning still grows several-fold from easy to hard puzzles. As tested, the account reads more like an analogy than an explanation.

We now measure how each student scales its effort with difficulty, and show that collapsed students scale it less than proportionally. Our claim is not that collapsed students spend the same effort on every problem, but that they do not scale it as much as difficulty requires. The effort elasticity (Fig. 5b, see W1+Q1) measures this directly. RC, RU and RC-R1 scale their effort less than proportionally with difficulty (0.76–0.77, against 1.00–1.12 for the other students), so they cut the hardest problems the most. We made the abstract more precise on this point. On ZebraLogic, the same students also stop gaining information from their own reasoning early (Fig. 5c). Both predictions of the account are thus measured directly on the students: they reason less where it matters most, and gain less from the reasoning they do.

Summary

We retrained every student on a single shared training set, added contexts that separate who wrote the context and how long it is from what it contains, and re-ran almost every evaluation. Our findings hold and are now clearer: collapse comes from rollout contexts, most strongly the model's own, not from context length alone; the teacher PVI that predicts teacher shortening on the graph task also predicts how much the student shortens on math; and only students of rollouts cut their effort most on the hardest problems. We hope these additions address the reviewer's concerns.

References

[1] Gordon D. Logan. Toward an instance theory of automatization. Psychological Review, 95(4):492–527, 1988.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions: new controls, direct tests of the account, and PVI on math

We thank the reviewers for their feedback and the Area Chair for handling our submission. The reviewers raised overlapping concerns, which we group below with the revisions that address them. All changes are marked in blue in the revised PDF.

To test the reviewers' concerns, we retrained the 12 students of the submission and trained 8 new ones on a single training set shared by every context and both models, and re-ran almost every evaluation in the paper. Our main findings hold.

1. Does collapse come from who wrote the context, or from its length and information? (ik2q W1, tJ8D W1, 91VM W2)

  • Same problems for every context: the rollout students no longer train on different problems; all students now share 4,765 problems (100% overlap). Collapse is unchanged (Fig. 1).
  • Rollouts from other models: teachers given correct rollouts from DeepSeek-R1 or Gemma-4-E4B collapse less than with the model's own (AIME-24 pass@1: −18.5 and −8.9, against −27.7), in the order of how likely base Qwen3-8B finds each rollout (Fig. 17).
  • Length-matched controls: a full solution padded to the length of the model's own rollouts shortens reasoning but does not collapse (−6.3; Fig. 16 for context lengths), and the model's own rollout cut to the length of Gemma-4-E4B's still costs 19.4 points, against 8.9 (Appendix C.5).

Collapse comes from giving the teacher a rollout, most strongly the model's own, not from context length or information alone. The new controls reinforce our central claim and add finer detail: the closer the rollout to the model's own reasoning, the stronger the collapse. The abstract, introduction and conclusion now include this result.

2. Is automatisation more than a redescription? (ik2q W1, W3; 91VM W1, W3)

  • Effort no longer scales with difficulty: a new measure, the effort elasticity, shows that only students of rollouts cut the hardest problems the most (0.76–0.77, against 1.00–1.12 for answers and solutions, padded or not; Section 5.2, Fig. 5b).
  • Not a stopping policy: collapsed students rank the end-of-thinking token lower than Base throughout their reasoning (Appendix C.6), and forcing them to reason as long as Base does not restore accuracy (Appendix C.7).
  • Not a context-specific shortcut: students are evaluated without context, yet the shortening transfers to unseen domains and is the same on training and test problems (Appendix C.4, Fig. 18).

Collapsed students run a shorter reasoning procedure: they neither stop early nor rely on cues in the context.

3. Is the information account measured where the claims are made? (ik2q W3, tJ8D W2, 91VM W3)

  • PVI on math: we computed the teacher PVI of all nine math contexts; it predicts how much each student shortens (Spearman ρ = 0.75; Fig. 5a).
  • What PVI does and does not explain: PVI sets how much a student shortens, not whether the shortening turns into collapse: the padded full solution has the highest PVI yet does not collapse. We clarified that Eq. 4 measures the total effect of the context on the teacher.

Usable information sets how much a student shortens; what the context contains decides whether it collapses.

4. Evaluation (ik2q W2)

  • Skip rates: reported for every model, benchmark and student (Table 4); counting skipped rollouts as failures leaves the results unchanged (Fig. 9).
  • Evaluation sets: rebuilt as a 300-problem test set from the training distribution and a matched set of 300 training problems.

The results do not depend on how unfinished rollouts are handled.

5. Relation to Kim et al. [1] (tJ8D W2)

  • We clarified that Kim et al. [1] vary one context, the solution with or without its thinking, so information, length and reasoning change together, and attribute degradation to information richness and task coverage. Our controls separate these factors and contradict Kim et al’s accounts: the most informative context does not collapse, and collapse follows difficulty, transfers out of domain, and is the same on seen and unseen problems.

Collapse depends on the difficulty of the problem, not on the information in the context or on task coverage.

6. Collapse and safety (91VM W4)

  • With the retrained students, the rollout students are the most vulnerable on Qwen3-8B on JailbreakBench and BeaverTails (Fig. 6). Section 2 now explains why collapse and context reliance are separable products of the same automatisation, so their orderings need not match.

Collapse and reliance on context are two separable outcomes of the same automatisation.

We hope these revisions address the reviewers' concerns, and we thank the Area Chair for considering them.

[1] Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. Why does self-distillation (sometimes) degrade the reasoning capability of LLMs? COLM, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.