- Rating
- 3(+2)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper seeks to develop theoretical foundations of on-policy self-distillation (OPSD) and supervised fine-tuning (SFT) by translating the problem of LLM reasoning to a search problem over a graph. The analysis establishes that correct outputs serve as valuable hints to OPSD in enabling continual learning without forgetting; on the other hand, SFT could result in catastrophic forgetting on correct reasoning paths. A third finding reveals that diversity in pre-training is critical for post-training model performance. Limited experimental evaluations are presented to demonstrate that observations are in agreement with the developed theoretical analysis.
Critical strengths
-
Developing theoretical foundations of OPSD is an interesting challenge that has potential for widespread impact and improved understanding of the fundamental machinations of LLM reasoning.
-
Rather than developing an all-encompassing theoretical foundation, the paper focuses on one somewhat less understood aspect of the role of hints in shaping learning dynamics. The authors subsequently translate this to a search problem on a directed acyclic graph (DAG), which is a well-understood mathematical and computational structure.
Critical weaknesses
-
It is not clear how the material presented in Sec. 3.3 on theoretical implications for pre-training diversity retains conditions in Assumption 2 (except for a coverage lower bound) that is stated earlier. An explanation would be helpful in understanding the nuance between assuming pretrained diversity and developing theoretical insights on this aspect.
-
The experimental evaluations presented in the main paper are particularly limited- both in terms of the quantity as well as detail afforded to understanding the presented results beyond stating that the observations are consistent with the developed theory.
-
While the abstract and introduction extensively mention OPSD and SFT in the context of LLM reasoning, the experimental evaluations presented directly start from a graph instance, rather than concretely demonstrating that the constructed graph corresponds to a realistic realization of LLM behavior.
-
The paper (Sec. 4) and appendix (Appendix C) indicate experimental evaluations on five tasks labeled Task 1 - Task 5, but there seems to be no information provided on what each of these tasks specifically entails. Additional detail will be helpful.
Questions for authors
-
Please see Weaknesses, above.
-
The authors acknowledge that Assumption 1 is quite strong, which I agree with. It would however be helpful if the reasonability of the assumption can be better supported (either through a reference or grounding in instances of actual LLM use).
5 comments
Response to Reviewer 6kE7
Q1-3
While the abstract and introduction extensively mention OPSD and SFT in the context of LLM reasoning, the experimental evaluations presented directly start from a graph instance, rather than concretely demonstrating that the constructed graph corresponds to a realistic realization of LLM behavior.
We thanks you for your important comment.
Prior theoretical studies of reasoning graphs have made meaningful contributions to the field while relying primarily on synthetic experiments (Kim et al. [2]; Wei & Kim [1]). Consistent with this line of work, we believe that our original experiments already provide sufficiently strong empirical support for our theoretical findings through controlled validation of the predicted mechanisms. While we are confident in our experimental results, we acknowledge the reviewer’s valid concerns about the insufficient detail in our current explanations and the connection between our experimental setup and real-world settings. To broaden our theoretical insights into the practical models and datasets, we conducted experiments with Qwen3-1.7B on the MATH dataset. For further details, please refer to Section 4 and Appendix C in our paper.
Graph construction grounded in LLM behavior.
Following Wang et al. [3] and Minegishi et al. [4], we cluster reasoning-prefix representations and connect consecutive states in CoT trajectories generated by Qwen3-1.7B on algebra and probability problems. The graph contains lots of potentially conflicting paths. The resulting graph thus captures transitions observed in actual LLM-generated reasoning.
Controlled continual learning.
We train the same model to navigate this graph, first on algebra and then on probability. Starting from a shared algebra-trained model, SD and SFT undergo 16 updates using a fixed high-interference training pool. SD locally promotes one goal-reachable alternative edge at an erroneous branching state, whereas SFT supervises full, cycle-free successful paths.
Support for our theoretical insights.
Averaged over four seeds, SD (λ = 0) improves probability performance by 25.18 percentage points while preserving algebra performance (+0.54 points). The parameter λ controls gradient localization in self-distillation. SFT improves probability by 14.22 points but reduces algebra performance by 23.74 points. Together with the smaller KL divergence at unhinted states, these results support our theoretical account of localized distributional updates and reduced forgetting (Figure 3 on page 10).
Table 3. Four-seed mean results at update 16, corresponding to Figure 3 in main paper.
| Metric | Task | SD | SFT |
|---|---|---|---|
| Performance change (pp) | Probability (New) | ||
| Performance change (pp) | Algebra (Old) | ||
| (nats) | Probability | ||
| (nats) | Algebra |
These results strengthen our confidence that two mechanisms help explain the observed learning dynamics: (i) hint-based updates localize changes in SD, and (ii) conflicts between similar tasks induce forgetting. We hope these findings address your concerns and further strengthen our paper.
Q1-4
The paper (Sec. 4) and appendix (Appendix C) indicate experimental evaluations on five tasks labeled Task 1 - Task 5, but there seems to be no information provided on what each of these tasks specifically entails. Additional detail will be helpful.
Thank you for pointing this out. We acknowledge that this point was not explicitly stated in the original manuscript, and we apologize for the omission. If we understand your comment correctly, you are asking for clarification on what each of Tasks 1–4 consists of. Each task is a searching task to reach its correct terminal cluster, so the goal in our experiment is to have the model learn all effective transitions through continual learning without forgetting any valuable paths with different tasks. The above clarification has been incorporated to Section 4 and Appendix C of the revised paper. We believe this explanation helps clarify our experimental setup and results.
References are listed at the end of next comment
Response to Reviewer 6kE7
Q.2
The authors acknowledge that Assumption 1 is quite strong, which I agree with. It would however be helpful if the reasonability of the assumption can be better supported (either through a reference or grounding in instances of actual LLM use).
We appreciate your careful and constructive consideration of this point. Assumption 1 simplify the inter-cluster transitions and captures the share of essential computations across tasks. As you might know, transition-uniqueness assumption just simplifies the multiple transitions from boundary nodes, which might not be a controversial point. Thus, we think the vague point for you is the sharing assumption so the followings are focused on it.
-
Hanna et al. [5]: Shared local computations across tasks.
Hanna et al. [5] identify a greater-than circuit in GPT-2 for year-span prediction and show that it also supports price-range prediction (Section 5, pp.9–10; Appendix F, Fig.26, p.25). Ablating edges outside the original circuit yields 88.9% loss recovery on the price-range task. Although the task contexts differ, both require selecting values above a given bound, and the same circuit supports this local computation. These findings provide empirical motivation for Assumption 1, which models shared local reasoning objectives through common correct inter-cluster transitions at shared boundary nodes.
-
Merullo et al. [6]: Shared computational mechanisms across tasks.
They identify circuits whose central role is selecting and copying input tokens to the output, with approximately 78% head overlap between two different tasks; Indirect Object Identification (IOI) and Colored Objects (Fig.3, p.6). Copy ability plays an essential role in solving them since it is not acquired by the model during training period. An IOI-guided intervention enhancing those circuits related to copying raises Colored Objects accuracy from 49.6% to 93.7%, with downstream effects consistent with the IOI mechanism (Fig.5, p.8). In our graph, such token-copying operations correspond to inter-cluster transitions. Assumption 1 models their consistent computational role across tasks through shared correct edges at common boundary nodes.
To complement the above evidence, we also provide an intuitive example. Medical decision-making and statistical model analysis can both reach a reasoning state in which a prior distribution and a likelihood are available. Applying Bayes’ theorem then leads to a state in which the posterior has been determined. Here, states describe what information has been established at each stage of reasoning. Assumption 1 represents this shared progression between reasoning states as a common correct inter-cluster transition across tasks.
Furthermore, immediately after Theorem 4 (p.7), we discuss settings beyond Assumption 1 where tasks have conflicting correct edge choices at shared boundary nodes. Even in these settings, SD can be less prone to forgetting than SFT because hints localize its updates, preserving old correct paths outside the updated region.
Receiving your review, previous work supporting Assumption 1 has been added. We hope that these references and explanations would be helpful for readers and provide further support for Assumption 1.
[1] Kim & Wei. Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently. ICML, 2026
[2] Kim et al. Metastable dynamics of chain-of-thought reasoning: Provable benefits of search, RL and distillation. ICML, 2025
[3] Wang et al. Understanding Reasoning Ability of Language Models From the Perspective of Reasoning Paths Aggregation. ICLR workshop, 2024
[4] Minegishi et al. Topology of Reasoning: Understanding Large Reasoning Models through Reasoning Graph Properties. NeurIPS, 2026
[5] Hanna, Liu & Variengien. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. NeurIPS, 2023
[6] Merullo et al. Circuit Component Reuse Across Tasks in Transformer Language Models. ICLR, 2024
Response to Reviewer 6kE7
Thanks you for reviewing our paper and for a lot of helpful comments in the experiment section. However, we're afraid that there might be some misunderstandings in the theoretical part. We hope that our responses would address all your concerns. The followings are detailed explanations.
Q1-1:
We constructs two following paragraphs for this comment, and hope they'll address your concern.
Modification of Assumption 3
We have clarified the relationship between the two assumptions and added an intuitive explanation, as detailed below.
Clarify that Assumption 3 retains all conditions of Assumption 2 except the coverage requirement in Equation (5), which is relaxed at a single boundary node. This controlled comparison isolates the role of coverage in Section 3.3.
Add some intuitive explanation for Assumption 3. While the model can reach all correct boundary nodes with sufficient probabilities in Assumption 2, Assumption 3 allows a rare boundary node that may remain unvisited during training.
Separation due to diversity
Section 3.3 clarifies the role of pre-trained coverage in intermediate restartability. Under Assumption 2, SD from the original start yields high accuracy from all relevant starts after sufficient training. It is proved in Proposition 3 on page 40. In contrast, however, Proposition 1 shows that, under Assumption 3, when M′ is so large that it satisfies Eq. 32 on page 8, high original-start accuracy can coexist with poor intermediate restart performance even with lots of learning iterations. Remarks 3 and 4 on page 44 quantify the separation between such M′ and M.
Q1-2:
Thank you for your helpful advices. We regret the poor quantity of our experimental section. Promoting to help readers understand our objectives and setups in experiments, we added the following points in our original paper.
Experimental setup: To help readers notice our experiments are limited in toy models, I emphasized the experimental setup we conducted. Although our experiments are toy, conducted surveys are valuable since results show that tracing the sparse updates in SD and conflict between different tasks in SFT would be helpful to capture the observed retention and forgetting, which is our main explanation in the experiments.
Results: This section would allow readers to capture what Fig. 2 implies. This section mainly focuses on whether the bound is non-vacuous—for example, whether an upper bound on a probability is strictly less than 1. Figure 2 plots experimental results and shows that our theoretical bounds are not useless, but effective to trace the dynamics of self-distillation and SFT.
Previous work also validate their theory with synthetic experiments (Kim & Wei, 2026 [1]) and even sometimes does not include numerical experiments (Kim et al, 2025 [2]), but all of them are widely accepted and regarded as valuable work. From that perspective, we believe our paper with theories validated by numerical experiments is a strong contribution.
References are listed at the end of the comments.