acceptodds
Under review as a conference paper at ICLR 2027

Circuit Emergence Does Not Establish Functional Emergence: Reasoning Distillation Can Recruit Latent Attention-Head Signals

Abstract

Reasoning post-training can make an attention head causally important even when ablating the same head in the base model has little effect. This leaves open two possibilities: the head may have acquired new task-relevant function, or the post-trained network may have learned to use information that was already present. We study this distinction in Qwen2.5-Math-7B and DeepSeek-R1-Distill-Qwen-7B, whose attention heads align exactly. We scan all 784 query heads and test selected heads on disjoint holdouts. The clearest case, L26H25, has no detectable positive causal effect in the base model but a large effect after distillation. However, its base-model activation recovers 98.8% of the distilled causal effect after RMS matching. Token-position shuffling and a same-layer control head recover only 8.2% and 6.8%. Rolling back the local Q/K/V/O pathway removes only about 21% of the amplification, while transplanting the distilled local pathway into the base model does not create the distilled role. A 1.5B aligned pair shows both local-rewrite and reader-dominant cases. On MATH, ablating three reasoning-specific heads preserves final accuracy but increases restricted generation length by 16.0%. These results show that newly causal circuit membership can arise through recruitment of task-relevant signals already present in the base model.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, reasoning models, attention heads, knowledge distillation, causal interventions, model diffing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DTdyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether post-training introduces new knowledge into a model or instead unlocks capabilities that are already present. The authors study attention heads that have little causal influence in base models but become significant after post-training. They find that, although such heads do exist, they appear to reflect circuit emergence rather than functional emergence: post-training seems to make use of knowledge already present in the base model rather than introducing genuinely new knowledge.

Critical strengths
  • The paper is well written and easy to follow.

  • The research question is relevant and useful for future research.

  • The experiments are generally convincing. The authors openly discuss several limitations, but these do not seem to substantially undermine their main findings.

Critical weaknesses
  • While the paper focuses on specific heads to draw its conclusions, I was left with an important question that I did not see addressed: can we generalize these findings to other heads? For the heads studied, such as L26H25, the evidence suggests circuit emergence rather than functional emergence. However, could some heads exhibit genuine functional emergence, and if so, how could we identify them? The paper's narrow focus is partly a strength, as it allows the authors to investigate a specific phenomenon in depth without making the paper difficult to follow. However, it also leaves many other heads unexplored. As a result, I do not think the paper fully answers the broader question of whether post-training introduces new knowledge and capabilities or primarily makes use of existing ones. Instead, it provides evidence that post-training leverages pre-training knowledge for at least some heads.
Questions for authors

N/A

2 comments

You're right that focusing on a few heads left the broader picture unclear. We ran two follow-ups. Among 16 discovery-ranked candidates, seven showed amplified causal effects with substantial base-to-distilled transfer. We also tested an equal-probability sample of 112 heads from all 784 on 96 new GSM8K probes. Three met our criteria. This measures the frequency of that specific pattern, not the fraction of new capabilities explained by recruitment; one of the three (L0H3) was already strongly causal before distillation.

We also replicated another near-zero-base case, L23H14 (base effect −0.013, distilled effect 0.142; raw transfer retention 110%).

To test whether the finding extends beyond isolated arithmetic probes, we examined three heads at different steps of 64 human-written GSM8K solutions. Aligned base-to-distilled transfer outperformed position-shifted controls in all nine head-by-position comparisons (Holm-adjusted p ≤ 0.0099). These experiments use teacher forcing rather than free-running generation. (Section 4.6; Appendix E.)

Your question about genuine functional emergence is important, and we agree that it could occur. Low transfer alone would not establish it, since recoding or changed upstream inputs could also explain the failure. But our positive finding goes beyond showing that existing information might be reused: L26H25's base-model activation recovers nearly all of its distilled causal effect, despite having almost no causal influence in the base model. Our new downstream interventions further support the role of the receiving network.

This gives a concrete example of how a reasoning head can become newly causal without learning its task-relevant signal from scratch. We do not claim that all heads behave this way or that distillation cannot create new functions. (Sections 4.2–4.3; Appendices C, D and G.)

Dear Reviewer DTdy,

We wanted to briefly follow up before the discussion period ends.

We added a broader survey of attention heads, another apparent-emergence case, and multi-step reasoning experiments addressing your concerns about generalizability. We also clarified the scope of our conclusions about functional emergence.

We'd be grateful if you could consider these new results when revisiting your assessment, and would be happy to discuss any remaining concerns.

Thank you for your time!

Reviewer_56g6OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates whether attention heads that become causally important after reasoning distillation acquire new task-relevant information or make use of signals already present in the base model. For selected heads, replacing distilled-model activations with the corresponding base-model activations largely preserves their causal contribution. I find this observation interesting. However, the evidence is concentrated in a small number of heads, and the experiments do not fully establish the proposed downstream-reader explanation. The organization and presentation could also be clearer, particularly in separating the empirical findings from their mechanistic interpretation. Overall, this is a worthwhile direction, but the paper would benefit from further investigation and revision.

Critical strengths
  1. The paper asks an interesting question and makes a useful distinction between newly acquired causal importance and newly acquired task-relevant information.

  2. The experimental design is thoughtful, combining cross-model activation patching, parameter interventions, and negative controls.

Critical weaknesses
  1. The detailed evidence comes from a small number of selected heads, with L26H25 providing the clearest apparent-emergence case. Although the authors acknowledge mechanistic heterogeneity, the prevalence of latent-signal recruitment remains unclear. Evaluating a broader set of heads would help establish how representative this finding is.

  2. Base→distilled activation transfer provides convincing evidence for pre-existing usable signals. However, the stronger downstream-reader interpretation remains underdetermined: local rollback retains distilled upstream inputs, and unsuccessful transplantation does not isolate upstream from downstream contributions. Although the paper correctly avoids interpreting the remaining amplification as a literal downstream fraction, more direct separation of upstream context and downstream computation would strengthen this mechanistic attribution.

Questions for authors
  1. Section 3.3 describes bidirectional activation patching, but the reported activation-transfer results primarily focus on base→distilled transfer. Could the authors provide the corresponding distilled→base results and explain how the asymmetry informs their interpretation?

  2. How common is latent-signal recruitment among heads that become more causally important after distillation? Could the authors evaluate a broader set of heads to estimate its prevalence?

  3. Do these findings suggest practical guidance for improving reasoning distillation, or should they primarily be understood as a diagnostic insight for interpreting changes in circuit membership?

4 comments

Our original rollback experiment was not enough to isolate a downstream reader. To address that, we swapped L27 directly between the models. Replacing distilled L27 with base weights reduced L26H25's causal effect by 0.103 on the original 64 probes and 0.116 on the same 96-problem cohort used in the 16-head survey; the reverse swap increased it by 0.135 and 0.120. On the latter set, the L26H25 increase was 0.029 greater than for active same-layer L26H26 (95% paired CI [0.015, 0.044]). L27 attention and MLP both contribute. Since other heads respond too, we describe this as a causal downstream contribution, not an isolated reader.

You also asked about distilled-to-base activation transfer. In the unchanged base model, patching distilled L26H25 activations shifted the clean margin by just +0.0315 (raw) or +0.0294 (RMS-matched). Conversely, base-to-distilled transfer recovered 98.4% and 98.9% of the distilled causal effect. We should have made this asymmetry clearer in the original paper.

For breadth, seven of 16 discovery-ranked candidates and three of 112 probability-sampled heads passed our amplification-and-transfer criteria. Neither number establishes what fraction of newly causal heads are explained by recruitment. We see the practical value primarily as diagnostic: ablation should be paired with aligned transfer and controls before claiming new head-level function, rather than taken as a recipe for better distillation. (Sections 4.2–4.3; Appendices C, D and G.)

Dear Reviewer 56g6,

We wanted to briefly follow up before the discussion period ends.

We added reciprocal L27 interventions, reverse activation transfer, and a broader head survey directly addressing your main concerns.

We'd really appreciate hearing whether these additional results address your concerns and whether they change your assessment of the paper.

Thank you for your time!

Thank you for the response and the additional experiments. They help clarify the results, but I am still not convinced by the main interpretation.

The L27 swaps explain only about a quarter of L26H25’s amplification, and they recover a similar fraction for L26H26. Most of the change is therefore still unexplained.

I also find the transfer test difficult to interpret. Almost every tested head transfers well, including heads the authors describe as having changed locally.

More broadly, the strongest examples are still just one or two heads in a single math-specialized model pair. These are interesting observations, but it is unclear how much they add to the existing understanding that fine-tuning can reuse pretrained capabilities, especially when the results offer no clear guidance for distillation. I still find the contribution too narrow for ICLR.

I regret that I must maintain my current rating.

Thank you for taking another look. We'd like to clarify a few points, particularly regarding the scope of the contribution.

First, we agree that the L27 swaps do not fully explain L26H25's amplification, and we do not claim they do. They establish a causal downstream contribution, not a complete decomposition. While L26H26 also responds to the swap, the increase for L26H25 was 0.029 larger on the 96-problem cohort (95% paired CI [0.015, 0.044]). We do not interpret this as evidence of an isolated reader. More importantly, our central result does not require a complete downstream explanation: L26H25 has almost no causal effect in the base model, yet its base-model activation recovers nearly all of its distilled causal effect after RMS matching, while disrupted-transfer controls recover much less.

Second, we agree that successful activation transfer alone is insufficient evidence of recruitment. We also do not see successful transfer in heads with local parameter changes as contradictory: local rewriting and preservation of a usable pre-existing signal are not mutually exclusive. Our finding concerns the combination of newly acquired causal importance, substantial aligned transfer, and negative controls.

To address your original concerns about generality, we expanded the analysis to 16 discovery-ranked and 112 probability-sampled heads, identified another near-zero-base case (L23H14), added transfer controls on a separate 1.5B model pair, and tested transfer at multiple reasoning steps. We acknowledge that these experiments do not establish prevalence across model families, but the phenomenon is not limited to a single isolated head.

Finally, we'd like to address the concern that the contribution is too narrow for ICLR. We believe its significance extends beyond the particular heads studied. Our result is not simply that fine-tuning can reuse pretrained capabilities. It demonstrates a specific limitation of interpreting changes in causal circuit membership: a head can become strongly causal without needing to acquire its task-relevant signal from scratch. Even a well-controlled counterexample matters here, because causal importance alone cannot establish the emergence of a new head-level function.

There is relevant precedent for focused mechanistic investigations. Prakash et al. (ICLR 2024) studied how fine-tuning enhances an existing entity-tracking mechanism, while Makelov et al. (ICLR 2024) demonstrated how causally effective interventions can lead to misleading mechanistic interpretations. Our work addresses a distinct question about the relationship between causal circuit emergence and the acquisition of task-relevant signals after reasoning distillation.

We therefore see the contribution primarily as a diagnostic insight for mechanistic interpretability, rather than a method for improving distillation. Its value does not depend on a complete mechanistic decomposition or immediate performance gains.

We appreciate your reconsideration and hope these clarifications help the Area Chair assess the contribution alongside the additional experimental evidence.

Reviewer_JTBdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors examine if an attention head becoming important after reasoning training really mean it learned a new function.They compare a Qwen math base model with its DeepSeek reaosning distilled versions, and found that there's a important head after distillation. However, if we migrate the activation of the head of the original LLM to the distilled model, it can recover 98%+ causal effect (upon magnitude matching). This means that training mainly teaches the model to use an existing rather than creating new information in the head. They further validated this on 128 math problems, and they detect no final-accuracy difference when contol head is ablated.

Critical strengths
  1. The conceptual contribution is very clear. This work calls out that causal importance need not imply new task-relevant information.
  2. I think the evidence chain for the main conclusion is solid. The authors perform cross-model activation transfer, bidirectional parameter intervention, alignment control, and held-out evaluation. They all support the explanation that the head only reuse the information it has already gained before,
Critical weaknesses
  1. I think the generalizibility of the achieved conclusion is my concern. The evidences come from a few heads in one aligned model pair on arithmetic probes; the smaller-model study lacks a control experiment. Maybe more experiments on braoder model classes would make the study more convincing.
Questions for authors
  1. Which downstream components changed to make the pre-existing signals causally important? Can direct interventions on those components verify this?
  2. I'm curious, do the findings extend to multi-step reasoning? Can base-model head activations still recover the distilled model’s causal effects on those tasks?
2 comments

Your multi-step question pointed to a gap in our original evaluation. On 64 human-written GSM8K solutions, we tested three heads at the first, middle and last calculation. Aligned base-to-distilled transfer beat a position-shifted control in all nine head-by-position comparisons (Holm-adjusted p ≤ 0.0099). For L26H25, RMS-matched recovery was 89.7%, 56.2% and 59.8%. This shows that the base signal remains useful at later positions, but the experiment uses teacher forcing, not freely generated reasoning. (Section 4.6; Appendix E.)

For the downstream question, replacing distilled L27 with base weights reduced L26H25's causal effect by 0.116 on the 96-problem cohort also used in the 16-head survey; the reverse swap into base increased it by 0.120. Swapping L27 attention or MLP separately also changed the effect, although we have not isolated one reader. (Section 4.3; Appendix D.)

We also added 1.5B transfer controls for four candidate and four comparison heads on 128 original and 96 additional problems. Ten of the additional problems had appeared in earlier screening; on the remaining 86, candidate recovery was 88.0–96.6%, and aligned transfer beat both shuffled and different-head controls after Holm correction. Some comparison heads were themselves causal, so the negative controls are the disrupted transfers, not necessarily inactive heads. Finally, we expanded the 7B analysis to 16 selected and 112 probability-sampled heads. Generalization to unaligned model classes remains open. (Sections 4.2 and 4.5; Appendices C, F and G.)

Dear Reviewer JTBd,

We wanted to briefly follow up before the discussion period ends.

Our rebuttal includes new multi-step reasoning experiments, direct downstream interventions, and additional 1.5B controls addressing your questions.

We'd be happy to discuss any remaining concerns, and would really appreciate it if you could take these new results into account when revisiting your assessment.

Thank you again for your helpful review!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Note on Revisions and Limited Reviewer Discussion

Dear Area Chair and Reviewers,

With the discussion period approaching its deadline, we would like to raise a concern about the limited reviewer discussion of our revised submission and summarize the substantial new evidence added in response to the reviews.

We posted detailed rebuttals on November 7 and followed up with all three reviewers on November 17. As of this comment, Reviewers JTBd and DTdy have not responded to our rebuttals or follow-ups. Reviewer 56g6 replied today, and we appreciate their engagement; we have provided a further response addressing the remaining concerns.

Consequently, despite conducting several new experiments specifically requested by the reviewers, we have had very limited opportunity to discuss their implications or establish which original concerns remain unresolved.

We would particularly like to highlight the extent of the new evidence:

  • Generality: We expanded the 7B analysis to 16 discovery-ranked and 112 probability-sampled heads, with seven and three, respectively, meeting our joint amplification-and-transfer criteria. We also identified a second near-zero-base case, L23H14.
  • Downstream mechanisms: Reciprocal L27 interventions changed L26H25's causal effect by −0.116 and +0.120 on the same 96-problem cohort, establishing a causal downstream contribution without claiming complete mechanistic attribution.
  • Additional model-pair evidence: On a separate aligned 1.5B model pair, candidate-head transfer recovered 88.0–96.6% of the distilled effect on 86 previously unscreened problems, outperforming disrupted-transfer controls after Holm correction.
  • Multi-step reasoning: Across three heads and three calculation positions in human-written GSM8K solutions, aligned transfer outperformed position-shifted controls in all nine comparisons after Holm correction. We explicitly acknowledge the teacher-forcing limitation.

These experiments directly address major questions raised in the original reviews. They also strengthen the evidence surrounding our central finding: a head can become strongly causal after distillation even when its base-model activation recovers nearly all of its distilled causal effect after RMS matching (98.8%), while disrupted-transfer controls recover only 8.2% and 6.8%.

We believe this is a meaningful mechanistic and diagnostic contribution, rather than merely another observation that fine-tuning can reuse pretrained capabilities. It demonstrates why newly acquired causal importance cannot, by itself, establish the acquisition of a new head-level function.

We recognize that some limitations remain, particularly regarding generalization across model families and complete mechanistic attribution. However, the revised submission contains substantially more evidence than was available during the initial reviews.

Our concern is that the lack of substantive discussion with two reviewers has left us without an opportunity to determine whether their original concerns have been addressed by these results.

We would therefore respectfully ask the Area Chair to take the limited discussion into account when assessing the unchanged ratings, and to give the revised manuscript and additional experimental evidence independent consideration.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.