acceptodds
Under review as a conference paper at ICLR 2027

Learning without Overwriting:  A Theory of Self-Distillation and Supervised Fine-Tuning in Continual Reasoning

Abstract

On-policy self-distillation (OPSD) of large language models (LLMs) has demonstrated the ability to improve reasoning capabilities while preserving previously acquired knowledge. Despite substantial empirical success, the dynamics of OPSD in continual reasoning remain incompletely understood. Modeling LLM reasoning as search over a graph, we provide a unified theoretical analysis of both the dynamics of post-training—OPSD and supervised fine-tuning (SFT) in continual learning—and the impact of pre-training on subsequent performance. Our findings establish three key insights with an optimization guarantee: (i) OPSD with hints from correct outputs enables continual learning without forgetting by sparse yet effective gradient descent updates induced by the hint structure. (ii) SFT on correct reasoning paths can lead to catastrophic forgetting due to dense updates along the training paths, which overwrite the information previously acquired. (iii) Diversity in pre-training is crucial for enabling a post-trained model to reach a correct output when a rollout starts from an intermediate state. Our results, supported by theoretical analysis, show that reliable continual reasoning depends on how post-training updates interact with the reasoning structure established during pre-training.

Primary area:
learning theory
Keywords:
Learning theory, Catastrophic forgetting, Self-distillation, Supervised fine-tuning, Reasoning graph, on-policy post-training, off-policy post-training, Chain-of-Thought, Continual Learning, LLM

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_6kE7OpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper seeks to develop theoretical foundations of on-policy self-distillation (OPSD) and supervised fine-tuning (SFT) by translating the problem of LLM reasoning to a search problem over a graph. The analysis establishes that correct outputs serve as valuable hints to OPSD in enabling continual learning without forgetting; on the other hand, SFT could result in catastrophic forgetting on correct reasoning paths. A third finding reveals that diversity in pre-training is critical for post-training model performance. Limited experimental evaluations are presented to demonstrate that observations are in agreement with the developed theoretical analysis.

Critical strengths
  1. Developing theoretical foundations of OPSD is an interesting challenge that has potential for widespread impact and improved understanding of the fundamental machinations of LLM reasoning.

  2. Rather than developing an all-encompassing theoretical foundation, the paper focuses on one somewhat less understood aspect of the role of hints in shaping learning dynamics. The authors subsequently translate this to a search problem on a directed acyclic graph (DAG), which is a well-understood mathematical and computational structure.

Critical weaknesses
  1. It is not clear how the material presented in Sec. 3.3 on theoretical implications for pre-training diversity retains conditions in Assumption 2 (except for a coverage lower bound) that is stated earlier. An explanation would be helpful in understanding the nuance between assuming pretrained diversity and developing theoretical insights on this aspect.

  2. The experimental evaluations presented in the main paper are particularly limited- both in terms of the quantity as well as detail afforded to understanding the presented results beyond stating that the observations are consistent with the developed theory.

  3. While the abstract and introduction extensively mention OPSD and SFT in the context of LLM reasoning, the experimental evaluations presented directly start from a graph instance, rather than concretely demonstrating that the constructed graph corresponds to a realistic realization of LLM behavior.

  4. The paper (Sec. 4) and appendix (Appendix C) indicate experimental evaluations on five tasks labeled Task 1 - Task 5, but there seems to be no information provided on what each of these tasks specifically entails. Additional detail will be helpful.

Questions for authors
  1. Please see Weaknesses, above.

  2. The authors acknowledge that Assumption 1 is quite strong, which I agree with. It would however be helpful if the reasonability of the assumption can be better supported (either through a reference or grounding in instances of actual LLM use).

5 comments
AuthorsOpenReview ↗

Response to Reviewer 6kE7

Thanks you for reviewing our paper and for a lot of helpful comments in the experiment section. However, we're afraid that there might be some misunderstandings in the theoretical part. We hope that our responses would address all your concerns. The followings are detailed explanations.

Q1-1:

It is not clear how the material presented in Sec. 3.3 on theoretical implications for pre-training diversity retains conditions in Assumption 2 (except for a coverage lower bound) that is stated earlier.

We constructs two following paragraphs for this comment, and hope they'll address your concern.

Modification of Assumption 3

We have clarified the relationship between the two assumptions and added an intuitive explanation, as detailed below.

  • Clarify that Assumption 3 retains all conditions of Assumption 2 except the coverage requirement in Equation (5), which is relaxed at a single boundary node. This controlled comparison isolates the role of coverage in Section 3.3.

  • Add some intuitive explanation for Assumption 3. While the model can reach all correct boundary nodes with sufficient probabilities in Assumption 2, Assumption 3 allows a rare boundary node that may remain unvisited during training.

Separation due to diversity

Section 3.3 clarifies the role of pre-trained coverage in intermediate restartability. Under Assumption 2, SD from the original start yields high accuracy from all relevant starts after sufficient training. It is proved in Proposition 3 on page 40. In contrast, however, Proposition 1 shows that, under Assumption 3, when M′M' is so large that it satisfies Eq. 32 on page 8, high original-start accuracy can coexist with poor intermediate restart performance even with lots of learning iterations. Remarks 3 and 4 on page 44 quantify the separation between such M′M' and MM.

Q1-2:

The experimental evaluations presented in the main paper are particularly limited-both in terms of the quantity as well as detail afforded to understanding the presented results beyond stating that the observations are consistent with the developed theory.

Thank you for your helpful advices. We regret the poor quantity of our experimental section. Promoting to help readers understand our objectives and setups in experiments, we added the following points in our original paper.

  • Experimental setup: To help readers notice our experiments are limited in toy models, I emphasized the experimental setup we conducted. Although our experiments are toy, conducted surveys are valuable since results show that tracing the sparse updates in SD and conflict between different tasks in SFT would be helpful to capture the observed retention and forgetting, which is our main explanation in the experiments.

  • Results: This section would allow readers to capture what Fig. 2 implies. This section mainly focuses on whether the bound is non-vacuous—for example, whether an upper bound on a probability is strictly less than 1. Figure 2 plots experimental results and shows that our theoretical bounds are not useless, but effective to trace the dynamics of self-distillation and SFT.

Previous work also validate their theory with synthetic experiments (Kim & Wei, 2026 [1]) and even sometimes does not include numerical experiments (Kim et al, 2025 [2]), but all of them are widely accepted and regarded as valuable work. From that perspective, we believe our paper with theories validated by numerical experiments is a strong contribution.

References are listed at the end of the comments.

AuthorsOpenReview ↗

Response to Reviewer 6kE7

Q1-3

While the abstract and introduction extensively mention OPSD and SFT in the context of LLM reasoning, the experimental evaluations presented directly start from a graph instance, rather than concretely demonstrating that the constructed graph corresponds to a realistic realization of LLM behavior.

We thanks you for your important comment.

Prior theoretical studies of reasoning graphs have made meaningful contributions to the field while relying primarily on synthetic experiments (Kim et al. [2]; Wei & Kim [1]). Consistent with this line of work, we believe that our original experiments already provide sufficiently strong empirical support for our theoretical findings through controlled validation of the predicted mechanisms. While we are confident in our experimental results, we acknowledge the reviewer’s valid concerns about the insufficient detail in our current explanations and the connection between our experimental setup and real-world settings. To broaden our theoretical insights into the practical models and datasets, we conducted experiments with Qwen3-1.7B on the MATH dataset. For further details, please refer to Section 4 and Appendix C in our paper.

Graph construction grounded in LLM behavior.

Following Wang et al. [3] and Minegishi et al. [4], we cluster reasoning-prefix representations and connect consecutive states in CoT trajectories generated by Qwen3-1.7B on algebra and probability problems. The graph contains lots of potentially conflicting paths. The resulting graph thus captures transitions observed in actual LLM-generated reasoning.

Controlled continual learning.

We train the same model to navigate this graph, first on algebra and then on probability. Starting from a shared algebra-trained model, SD and SFT undergo 16 updates using a fixed high-interference training pool. SD locally promotes one goal-reachable alternative edge at an erroneous branching state, whereas SFT supervises full, cycle-free successful paths.

Support for our theoretical insights.

Averaged over four seeds, SD (λ = 0) improves probability performance by 25.18 percentage points while preserving algebra performance (+0.54 points). The parameter λ controls gradient localization in self-distillation. SFT improves probability by 14.22 points but reduces algebra performance by 23.74 points. Together with the smaller KL divergence at unhinted states, these results support our theoretical account of localized distributional updates and reduced forgetting (Figure 3 on page 10).

Table 3. Four-seed mean results at update 16, corresponding to Figure 3 in main paper.

MetricTaskSD (λ=0)(\lambda = 0) ±1σ\pm 1 \sigmaSFT ±1σ\pm 1\sigma
Performance change (pp)Probability (New)+25.18±2.17+25.18 \pm 2.17+14.22±0.30+14.22 \pm 0.30
Performance change (pp)Algebra (Old)+0.54±0.39+0.54 \pm 0.39−23.74±0.87-23.74 \pm 0.87
KL(πbefore∥πafter)\mathrm{KL}(\pi_{\mathrm{before}} \parallel \pi_{\mathrm{after}}) (nats)Probability0.0798±0.02000.0798 \pm 0.02000.4261±0.01680.4261 \pm 0.0168
KL(πbefore∥πafter)\mathrm{KL}(\pi_{\mathrm{before}} \parallel \pi_{\mathrm{after}}) (nats)Algebra0.0175±0.00270.0175 \pm 0.00270.3468±0.02230.3468 \pm 0.0223

These results strengthen our confidence that two mechanisms help explain the observed learning dynamics: (i) hint-based updates localize changes in SD, and (ii) conflicts between similar tasks induce forgetting. We hope these findings address your concerns and further strengthen our paper.

Q1-4

The paper (Sec. 4) and appendix (Appendix C) indicate experimental evaluations on five tasks labeled Task 1 - Task 5, but there seems to be no information provided on what each of these tasks specifically entails. Additional detail will be helpful.

Thank you for pointing this out. We acknowledge that this point was not explicitly stated in the original manuscript, and we apologize for the omission. If we understand your comment correctly, you are asking for clarification on what each of Tasks 1–4 consists of. Each task is a searching task to reach its correct terminal cluster, so the goal in our experiment is to have the model learn all effective transitions through continual learning without forgetting any valuable paths with different tasks. The above clarification has been incorporated to Section 4 and Appendix C of the revised paper. We believe this explanation helps clarify our experimental setup and results.

References are listed at the end of next comment

AuthorsOpenReview ↗

Response to Reviewer 6kE7

Q.2

The authors acknowledge that Assumption 1 is quite strong, which I agree with. It would however be helpful if the reasonability of the assumption can be better supported (either through a reference or grounding in instances of actual LLM use).

We appreciate your careful and constructive consideration of this point. Assumption 1 simplify the inter-cluster transitions and captures the share of essential computations across tasks. As you might know, transition-uniqueness assumption just simplifies the multiple transitions from boundary nodes, which might not be a controversial point. Thus, we think the vague point for you is the sharing assumption so the followings are focused on it.

  • Hanna et al. [5]: Shared local computations across tasks.

    Hanna et al. [5] identify a greater-than circuit in GPT-2 for year-span prediction and show that it also supports price-range prediction (Section 5, pp.9–10; Appendix F, Fig.26, p.25). Ablating edges outside the original circuit yields 88.9% loss recovery on the price-range task. Although the task contexts differ, both require selecting values above a given bound, and the same circuit supports this local computation. These findings provide empirical motivation for Assumption 1, which models shared local reasoning objectives through common correct inter-cluster transitions at shared boundary nodes.

  • Merullo et al. [6]: Shared computational mechanisms across tasks.

    They identify circuits whose central role is selecting and copying input tokens to the output, with approximately 78% head overlap between two different tasks; Indirect Object Identification (IOI) and Colored Objects (Fig.3, p.6). Copy ability plays an essential role in solving them since it is not acquired by the model during training period. An IOI-guided intervention enhancing those circuits related to copying raises Colored Objects accuracy from 49.6% to 93.7%, with downstream effects consistent with the IOI mechanism (Fig.5, p.8). In our graph, such token-copying operations correspond to inter-cluster transitions. Assumption 1 models their consistent computational role across tasks through shared correct edges at common boundary nodes.

To complement the above evidence, we also provide an intuitive example. Medical decision-making and statistical model analysis can both reach a reasoning state in which a prior distribution and a likelihood are available. Applying Bayes’ theorem then leads to a state in which the posterior has been determined. Here, states describe what information has been established at each stage of reasoning. Assumption 1 represents this shared progression between reasoning states as a common correct inter-cluster transition across tasks.

Furthermore, immediately after Theorem 4 (p.7), we discuss settings beyond Assumption 1 where tasks have conflicting correct edge choices at shared boundary nodes. Even in these settings, SD can be less prone to forgetting than SFT because hints localize its updates, preserving old correct paths outside the updated region.

Receiving your review, previous work supporting Assumption 1 has been added. We hope that these references and explanations would be helpful for readers and provide further support for Assumption 1.

[1] Kim & Wei. Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently. ICML, 2026

[2] Kim et al. Metastable dynamics of chain-of-thought reasoning: Provable benefits of search, RL and distillation. ICML, 2025

[3] Wang et al. Understanding Reasoning Ability of Language Models From the Perspective of Reasoning Paths Aggregation. ICLR workshop, 2024

[4] Minegishi et al. Topology of Reasoning: Understanding Large Reasoning Models through Reasoning Graph Properties. NeurIPS, 2026

[5] Hanna, Liu & Variengien. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. NeurIPS, 2023

[6] Merullo et al. Circuit Component Reuse Across Tasks in Transformer Language Models. ICLR, 2024

Reviewer_v2TFOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3
Read review
Paper summary

This paper established a graph-based theoretical framework for On-Policy Self Distillation and SFT in continue learning. In contrast to SFT, it reveals how OPSD with hints from correct outputs enables continual learning without forgetting, and studies how diversity in pretraining affects reasoning from intermediate states. The paper also makes synthetic experiments to study the retention and forgetting between SD and SFT in continual learning, which is supportive to their theoretical bounds.

Critical strengths

S1. I have not actively searched for other literatures understanding the advantages of OPSD over SFT, but I do know that OPSD alleviates forgetting in continue learning. Based on the contributions claimed in the paper, I consider the theoretical part of this work is highly valuable.

S2. Both the modeling of CoT and the theoretical formulation are sufficient and solid.

S3. The experiment results are interesting and very supportive to the theory.

Critical weaknesses

W1: I think the writing should be improved further.

  • I don't think six lines are enough for Section 4. The conclusion of this experiment is not well discussed.
  • the discussion following eq11 addresses the effect of ϵrel\epsilon_{\mathrm{rel}} on ξℓ,n(K)(δ)\xi_{\ell,n}^{(K)}(\delta). I think the expression in thm1 could be conveyed simply by stating that ξℓ,n(K)(δ)\xi_{\ell,n}^{(K)}(\delta) is monotonic to ϵrel\epsilon_{\mathrm{rel}}

W2: There is still a gap between the theoretical setting and the real world scenario.

Questions for authors

Q1: Following the discussion of thm1, Preserving the student’s pre-trained diversity also leaves some of its initial errors intact. My understanding is OPSD is more conservative in modifying the model’s distribution, which leads it to preserve both existing knowledge and initial errors. I wonder what adjustment we can do to balance between retaining existing knowledge and correcting errors based on your theory? For example, if we can sacrifice some previously knowledge (not catastrophic forgetting in sft) to achieve a lower error rate on the current task, what adjustments could we make?

Q2: Beyond explaining the forgetting phenomenon, which most people are already familiar with, can your theoretical framework yield any conclusions that are not well known?

Q3: Could experiments be extended to real world LLMs across different tasks in continue learning? I think the current experiments are sufficient for a self-contained theoretical paper and it is already a strong submission. However, a controlled study of forgetting in OPSD and SFT on real datasets will raise my assessment of this paper.

2 comments
AuthorsOpenReview ↗

Response to Reviewer v2TF

Thank you for your positive and professional feedback-we truly appreciate the time and care you invested in reviewing. Your review helps us not only to refine our paper, but also to identify our limitations and future perspectives. Below, we address your concerns and suggestions for improvement in detail. We hope that our responses resolve your concerns and strengthen your confidence in the contributions of our work.

W1:

I think the writing should be improved further.

Thank you for this helpful comment. We have carefully revised the main paper according to your comments.

W2:

There is still a gap between the theoretical setting and the real world scenario.

Q3:

Could experiments be extended to real world LLMs across different tasks in continue learning?

A controlled study of forgetting in OPSD and SFT on real datasets will raise my assessment of this paper.

Thank you for your comments pointing out our current limitations. We believe that these could be resolved by experimenting the practical situations and hope that our responses help demonstrate the significance of our contributions.

Practical experiments: As you noted, our original experiments offered limited evidence of practical relevance. To support our theoretical findings, we conducted additional experiments with Qwen3-1.7B on the MATH dataset. Following Wang et al. [1] and Minegishi et al. [2], we constructed a CoT graph from mathematical reasoning trajectories. The graph has a variety of potentially conflicting paths. We trained the model to search over this graph, first on algebra and then on probability.

We used this continual learning setup to examine localized updates in self-distillation and forgetting under SFT due to conflicts between tasks. The experimental results are presented below.

Table 2. Four-seed mean results at update 16, corresponding to Figure 3 in main paper.

MetricTaskSD (λ=0)(\lambda = 0) ±1σ\pm 1 \sigmaSFT ±1σ\pm 1\sigma
Performance change (pp)Probability (New)+25.18±2.17+25.18 \pm 2.17+14.22±0.30+14.22 \pm 0.30
Performance change (pp)Algebra (Old)+0.54±0.39+0.54 \pm 0.39−23.74±0.87-23.74 \pm 0.87
KL(πbefore∥πafter)\mathrm{KL}(\pi_{\mathrm{before}} \parallel \pi_{\mathrm{after}}) (nats)Probability0.0798±0.02000.0798 \pm 0.02000.4261±0.01680.4261 \pm 0.0168
KL(πbefore∥πafter)\mathrm{KL}(\pi_{\mathrm{before}} \parallel \pi_{\mathrm{after}}) (nats)Algebra0.0175±0.00270.0175 \pm 0.00270.3468±0.02230.3468 \pm 0.0223

The table supports our theoretical insights: balanced changes in performance reflect forgetting due to conflicts between similar tasks, while the small KL divergence is consistent with knowledge retention through localized updates. Further details are provided in the revised main paper in Section 4 & Appendix C. We hope that these clarifications and further results address your concerns, and we would appreciate your reconsideration of our work in light of this evidence.

Q1:

I wonder what adjustment we can do to balance between retaining existing knowledge and correcting errors based on your theory? For example, if we can sacrifice some previously knowledge to achieve a lower error rate on the current task, what adjustments could we make?

We thank you for your thoughtful question. We believe it helps us to deepen our understandings to the coverage of current theories. Two possible approaches could be considered: adding negative hints that suppress transitions in incorrect directions, and providing hints with a broader scope.

  • Negative hint The former follows the idea discussed immediately after Theorem 1. Specifically, we believe that negative hints of the form

    \begin{cases} {\color{blue}{\epsilon_{\text{rel}}}}& (\tau_i < T, y_{\tau_i + 1} = v_i),\\\\ {\color{blue}{1}} & (\tau_i < T, y_{\tau_i + 1} \neq v_i),\\\\ Z_{\mathbf{W}, i}(x) & (\tau_i \geq T), \end{cases}$$ where $c_i = (u_i, v_i)$ and $\tau_i = \inf\\{s \in \\{0, \ldots, T\\} \mid y_s = u_i\\}$, would help the model avoid passing through incorrect boundary nodes. A similar calculation to that in Lemma 4 would yield localized updates that discourage the model from visiting such nodes.
  • Hints with broader scopes The latter approach is to provide positive hints that are not restricted to boundary nodes. This idea is motivated by Theorems 1 and 4. Theorem 1 shows that localized hints enable the model to learn effectively through localized parameter updates. On the other hand, Theorem 4 suggests that modifying intra-cluster transitions can create conflicts between different tasks and cause forgetting. Together, these observations suggest that providing hints with a broader scope may involve a trade-off between improving intra-cluster transitions and avoiding forgetting.

We hope that the above two would be helpful to address your question.

We listed references in the next.

AuthorsOpenReview ↗

Response to Reviewer v2TF

Q2:

Comments

Beyond explaining the forgetting phenomenon, which most people are already familiar with, can your theoretical framework yield any conclusions that are not well known?

We thank you for this question. Our framework provides two additional insights:

  • Rollout diversity (Section 3.3, p. 8). Self-Distillation Enables Continual Learning [3] and Reinforcement Learning via Self-Distillation [4] emphasize the pretrained model’s In-context learning ability. Our analysis identifies an additional requirement: sufficient rollout diversity. Without it, relevant transitions may remain unlearned, causing failures when reasoning resumes from intermediate states despite high accuracy from the original starting state.

  • The mechanism underlying localized updates. Our analysis further suggests that the key ingredients of successful self-distillation are hints that compensate for the model’s insufficient capabilities and the implicit regularization from using the model as its own teacher. Importantly, although some people argued that reverse-KL form is appropriate (Liu et al. 2026 [5]) since the decomposition of KL-divergence could be represented as that of RL, localized updating in Lemma 4 seems not to be original in reverse KL-divergence form. It does not use any specific properties for reverse-KL. This suggests that locality may extend beyond reverse KL, or even KL-based losses, provided the loss gradient vanishes at teacher–student agreement. Consistently, Shenfeld et al [3] reports that forward KL performs best in its experiments, despite its theoretical motivation for reverse KL.

[1] Wang et al. Understanding Reasoning Ability of Language Models From the Perspective of Reasoning Paths Aggregation. ICLR workshop, 2024

[2] Minegishi et al. Topology of Reasoning: Understanding Large Reasoning Models through Reasoning Graph Properties. NeurIPS, 2026

[3]Shenfeld et al. Self-Distillation Enables Continual Learning. ICML, 2026

[4]Hübotter et al. Reinforcement Learning via Self-Distillation. ICML, 2026

[5]Liu et al. Self-Distilled Policy Gradient. arXiv preprint arXiv:2606.04036, 2026.

Reviewer_X7q4OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the benefit of Self-Distillation (SD) compared to SFT from a theoretical perspective: it studies the graph model and argues that SD can preserve the old reasoning traces while learn the new ones, while SFT will overwrite the existing traces which may be needed by previous tasks. Specifically, it establishes that SD can learn a new task through selective updates with high probability, while SFT can learn all steps needed for a task. In addition, SD can preserve the earlier task’s learning guarantee with high probability. In contrast, SFT can suppress old solution paths that conflict with the new tasks. Numerical experiments are conducted to corroborate the theoretical findings.

Critical strengths
  • This paper provides high-probability guarantees for the properties of SD under the graph models: SD preserves the existing reasoning paths or patterns and improves the performance on the new tasks via updates on selective parameters. In contrast, SFT improves the performance on the new task and can overwrite the reasoning paths of previous tasks. This provides theoretical explanations on the observed phenomenon in the prevailing on-policy SD empirical works, though it is under a structured graph model.
  • It shows that SD can achieve high accuracy when starting from original point, while leaving the rarely visited reasoning regions still unlearned. This theoretically demonstrates the importance of the coverage during the pretraining stage for more reliable reasoning in post training.
Critical weaknesses
  • The theoretical guarantees provides limited evidence for practical applicability. The graph model is constructed upon strong assumptions, which are not evaluated on real LLMs. While this is a theoretical work, it tries to explain the commonly used SD and SFT properties in practical LLMs, thus, it would be good if the authors can demonstrate their setups can faithfully model the reasoning trajectories of practical LLMs.
Questions for authors
  • Can the authors provide evidence that the realistic hints produce localized updates?
  • Can the authors provide evidence that the assumptions are satisfied in practical LLMs beyond the toy examples, and how sensitive are the results to the violations of the assumptions? This would greatly strengthen the practical meaning of this theoretical work.
4 comments
AuthorsOpenReview ↗

Response to Reviewer X7q4

We would like to thank you for your valuable questions and helpful comments. We believe that you have pointed out the theoretical limitation to the real-world conditions and expressed concern about it. Below, we address your comments.

The theoretical guarantees provides limited evidence for practical applicability. The graph model is constructed upon strong assumptions, which are not evaluated on real LLMs.

Thank you for your constructive comments. We understand your concern to be that the evidence supporting our assumptions and theories is limited. To address this concern, we provide further clarification and additional experiments using an LLM in realistic settings. We address your questions individually below. Here, we first describe the additional experiments.

Practical experiments: As you noted, our original experiments offered limited evidence of practical relevance. To support our theoretical findings, we conducted further experiments with Qwen3-1.7B on the MATH dataset. Following Wang et al. [1] and Minegishi et al. [2], we constructed a CoT graph from mathematical reasoning trajectories. We trained the model to search this graph, first on algebra and then on probability. The graph has lots of potentially conflicting paths between them. We used this continual learning setup to examine localized updates in self-distillation and forgetting under SFT due to conflicts between tasks.

The experimental results are presented below.

Table 1. Four-seed mean results at update 16, corresponding to Figure 3 in main paper.

MetricTaskSD (λ=0)(\lambda = 0) ±1σ\pm 1 \sigmaSFT ±1σ\pm 1\sigma
Performance change (pp)Probability (New)+25.18±2.17+25.18 \pm 2.17+14.22±0.30+14.22 \pm 0.30
Performance change (pp)Algebra (Old)+0.54±0.39+0.54 \pm 0.39−23.74±0.87-23.74 \pm 0.87
KL(πbefore∥πafter)\mathrm{KL}(\pi_{\mathrm{before}} \parallel \pi_{\mathrm{after}}) (nats)Probability0.0798±0.02000.0798 \pm 0.02000.4261±0.01680.4261 \pm 0.0168
KL(πbefore∥πafter)\mathrm{KL}(\pi_{\mathrm{before}} \parallel \pi_{\mathrm{after}}) (nats)Algebra0.0175±0.00270.0175 \pm 0.00270.3468±0.02230.3468 \pm 0.0223

The table supports our theoretical insights: balanced changes in performance reflect forgetting due to conflicts between similar tasks, while the small KL divergence is consistent with knowledge retention through localized updates. Further details are provided in the revised main paper. We hope that these clarifications and additional results address your concerns, and we would appreciate your reconsideration of our work in light of this evidence.

Q1:

Can the authors provide evidence that the realistic hints produce localized updates?

We appreciate this insightful question. We consider hints, including feedback on rollouts, to complement difficult reasoning steps while omitting information that is trivial to the model. Such hints are therefore expected to localize changes in the model’s reasoning.

  • Hübotter et al. (2026) [3]; This work provided empirical illustrations supporting this intuition: feedback-conditioned self-distillation produces learning signals concentrated at error-relevant tokens (Fig. 4, p. 4; Fig. 9, p. 10; Fig. 22, p. 50). In their experiments, model-generated programs are evaluated by a fixed testing program, which returns feedback pointing to errors in the code. Conditioned on this feedback, the model re-evaluates its original rollout to identify where corrections are needed.

  • Yeo et al. (2026) [4]: Their approach generates corrective feedback for a small subset of failure-relevant steps, selecting at most three steps per failed trajectory. Each feedback message targets only the mistake at its selected step, and self-distillation is restricted to the corresponding action span (Sec. 2–3.1; Fig. 5).

  • Jin et al. (2026) [5]: They suggested the method which incorporates on-policy self-distillation into GRPO and provides additional corrective hints only at identified error steps. While the self-teacher receives environment feedback at every step, these additional hints specify how individual erroneous actions should be corrected, illustrating localized hint provision (Sec. 2.2; Fig. 1).

These selective corrections are closely related to our problem setup, in which predictors learn transitions they have not yet mastered.

Our hint construction follows the above methods, critical hints for the model and we therefore believe that our modelling would be an acceptable one.

References are listed at the end of the next comment.

AuthorsOpenReview ↗

Response to Reviewer X7q4

Q2:

Can the authors provide evidence that the assumptions are satisfied in practical LLMs beyond the toy examples, and how sensitive are the results to the violations of the assumptions?

We thank the reviewer for highlighting the paper limitation. At first, we would like to answer the concern to assumptions.

Validity for Graph Modelling

Modelling LLM reasoning as graph search is an established approach in theoretical analysis, as discussed in Appendix A:

  • Bachmann & Nagarajan (2024) [6]: Use path-star graphs to show how teacher-forced next-token prediction can favor shortcuts that hinder planning.

  • Kim et al. (2025) [7]: Model CoT reasoning as a metastable Markov process on clustered graphs.

  • Wei & Kim (2026) [8]: Study graph pathfinding to compare the backtracking capabilities of RLVR and SFT.

Following this line of work, we model LLM reasoning as a Markov process on a reasoning graph.

Validity for Assumption 1

Transition uniqueness in Assumption 1 simplifies the graph model. We focus on the sharing of essential computations across tasks, motivated by the following findings:

  • Hanna et al. [9]: A greater-than circuit supports year-span and price-range prediction, with imperfect generalization.

  • Merullo et al. [10]: IOI and Colored Objects share selection and copying mechanisms, with targeted interventions raising Colored Objects accuracy from 49.6% to 93.7%.

Our model represents such essential but imperfectly executed computations as inter-cluster transitions, and Assumption 1 captures their sharing across tasks through common correct transitions at shared boundary nodes.

Requirements behind Assumption 2

Assumption 2 specifies sufficient pre-training conditions for the learning guarantees in our framework. These conditions describe what pre-training should achieve: reliable basic reasoning and sufficient coverage of relevant boundary nodes, while essential transitions remain to be learned. Section 3.3 further motivates the coverage requirement by showing that insufficient coverage can leave relevant transitions unlearned and impair reasoning from intermediate states, even after successful learning from the original start.

Sensitivity is closely related to Assumption 2. The sensitivity for MM—lack of diversity would hide the failure—is discussed in our main papers, so we'd like to focus on the values of ϵ1\epsilon_1 and ϵ2\epsilon_2. When ϵ1\epsilon_1 is large, frequent intra-cluster failures can prevent SD from learning successfully. Since hints specifying correct inter-cluster transitions do not update intra-cluster parameters, these pretraining errors persist and can keep the failure probability high. When ϵ2\epsilon_2 is too small, the model cannot sufficiently explore correct cluster paths or learn their transitions within the training budget. These conditions therefore require the pretrained model to already possess the potential to generate correct reasoning paths.

[1] Wang et al. Understanding Reasoning Ability of Language Models From the Perspective of Reasoning Paths Aggregation. ICLR workshop, 2024

[2] Minegishi et al. Topology of Reasoning: Understanding Large Reasoning Models through Reasoning Graph Properties. NeurIPS, 2026

[3] Hübotter et al. Reinforcement Learning via Self-Distillation. ICML, 2026

[4] Yao et al. HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents. ACL ARR, 2026

[5] Jin et al. AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL. arXiv preprint arXiv:2608.24114, 2026

[6] Bachmann & Nagarajan. The pitfalls of next-token prediction. ICML, (2024)

[7] Kim et al. Metastable dynamics of chain-of-thought reasoning: Provable benefits of search, RL and distillation. ICML, 2025

[8] Kim & Wei. Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently. ICML, 2026

[9] Hanna, Liu & Variengien. How does GPT-2 compute greater-than?: Interpreting mathe- matical abilities in a pre-trained language model. NeurIPS, 2023

[10] Merullo et al. Circuit Component Reuse Across Tasks in Transformer Language Models. ICLR, 2024

Thank the authors for the detailed response! The explanations on the localized update look largely reasonable to me. The authors acknowledged and clarified the limitations of their assumptions, in particular Assumption 2. I kindly ask the authors to elaborate the boundary of the area where their theoretical characterization is valid in the revision, as the limited empirical results do not seem to be strong enough.

Given that this paper is in the learning theory track which puts more emphasis on the theoretical aspect, and the clarifications from the author, my score is updated accordingly.

AuthorsOpenReview ↗

Response to Reviewer X7q4

Thank you for your response! We have revised the manuscript accordingly by adding a brief discussion of sensitivity to the pre-training conditions immediately after Assumption 2.

Comments on the paper

1 comment

Dear Area Chair,

We would like to thank you for the time and care you have devoted to evaluating our submission and rebuttal amid the unforeseen circumstances. To support your evaluation, we have prepared a summary of the rebuttal period.

Paper Overview

On-policy self-distillation (OPSD) in LLM post-training is well known for its effectiveness in mitigating catastrophic forgetting in continual learning. To investigate why, we analyzed LLM reasoning as search over a directed acyclic graph. Through this analysis, we theoretically established the following three results.

(i) OPSD with hints from correct outputs enables continual learning without forgetting through sparse yet effective updates induced by the hint structure.

(ii) SFT on correct reasoning paths can lead to catastrophic forgetting due to dense updates along the training paths, which overwrite previously acquired information.

(iii) Diversity in pre-training is crucial for enabling a post-trained model to reach a correct output when a rollout starts from an intermediate state.

To assess our theoretical results, we conducted two types of experiments: a toy experiment that helps validate the theoretical bounds derived in our main paper, and an experiment in a real-world setting that we believe further supports our theoretical intuitions. The results of the latter experiment are presented below.

Table 1. Results at update 16 (mean ±\pm one standard deviation across four seeds), corresponding to Figure 3 in the main paper (added during the rebuttal period).

MetricTaskSD (λ=0\lambda = 0)SFT
Performance change (pp)Probability (New)+25.18±2.17+25.18 \pm 2.17+14.22±0.30+14.22 \pm 0.30
Performance change (pp)Algebra (Old)+0.54±0.39+0.54 \pm 0.39−23.74±0.87-23.74 \pm 0.87
KL(πbefore∥πafter)\mathrm{KL}(\pi_{\mathrm{before}} \parallel \pi_{\mathrm{after}}) (nats)Probability0.0798±0.02000.0798 \pm 0.02000.4261±0.01680.4261 \pm 0.0168
KL(πbefore∥πafter)\mathrm{KL}(\pi_{\mathrm{before}} \parallel \pi_{\mathrm{after}}) (nats)Algebra0.0175±0.00270.0175 \pm 0.00270.3468±0.02230.3468 \pm 0.0223

These results indicate that our theoretical insights (i) and (ii) provide a reasonable explanation for the observed learning dynamics.

Initial Ratings

3 (X7q4), 3 (v2TF), 1 (6kE7).

Final Ratings

4 (X7q4), 4 (v2TF), 3 (6kE7).

Discussions with Reviewers during the Rebuttal Period

  • Reviewer X7q4: "Thank the authors for the detailed response! The explanations on the localized update look largely reasonable to me. The authors acknowledged and clarified the limitations of their assumptions, in particular Assumption 2. I kindly ask the authors to elaborate the boundary of the area where their theoretical characterization is valid in the revision, as the limited empirical results do not seem to be strong enough. Given that this paper is in the learning theory track which puts more emphasis on the theoretical aspect, and the clarifications from the author, my score is updated accordingly."

  • Reviewer v2TF: "Thank you for your comments. I have raised my score."

  • Reviewer 6kE7: "Thank you for your careful and detailed response. Your response has been very helpful in clarifying the scope of the current work. I have accordingly updated my score."

Based on the reviewers' current feedbacks, all of them now hold a positive view of our paper.

Strengths Recognized by All Reviewers

  • A valuable theoretical contribution to understanding on-policy self-distillation, with implications for continual learning and the mechanisms underlying LLM reasoning.
  • A structured graph-based formulation of chain-of-thought reasoning, enabling rigorous analysis of how hints shape self-distillation updates and learning dynamics.

Main Concerns and Questions Resolved

Table 2. Main concerns and questions raised by the reviewers and our solutions.

ConcernReviewerSolution
1. The gap between theoretical guarantees and real-world learning dynamicsX7q4, v2TF, 6kE7We conducted additional experiments in a real-world setting using Qwen3-1.7B on the MATH dataset. The results are presented in Table 1 and appear to support our theoretical insights.
2. The validity of the theoretical assumptionsX7q4, 6kE7We clarified how prior work supports the assumptions used in our analysis. We also added more detailed explanations to help readers understand these assumptions.

We sincerely thank all reviewers for their constructive and helpful feedback, which has helped us strengthen our paper. We are confident that our revisions and additional experiments have addressed all concerns. We would appreciate it if our rebuttal were fully considered in your final decision.

Best regards,

The authors of Paper #28436

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.