- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates whether post-training introduces new knowledge into a model or instead unlocks capabilities that are already present. The authors study attention heads that have little causal influence in base models but become significant after post-training. They find that, although such heads do exist, they appear to reflect circuit emergence rather than functional emergence: post-training seems to make use of knowledge already present in the base model rather than introducing genuinely new knowledge.
Critical strengths
-
The paper is well written and easy to follow.
-
The research question is relevant and useful for future research.
-
The experiments are generally convincing. The authors openly discuss several limitations, but these do not seem to substantially undermine their main findings.
Critical weaknesses
- While the paper focuses on specific heads to draw its conclusions, I was left with an important question that I did not see addressed: can we generalize these findings to other heads? For the heads studied, such as L26H25, the evidence suggests circuit emergence rather than functional emergence. However, could some heads exhibit genuine functional emergence, and if so, how could we identify them? The paper's narrow focus is partly a strength, as it allows the authors to investigate a specific phenomenon in depth without making the paper difficult to follow. However, it also leaves many other heads unexplored. As a result, I do not think the paper fully answers the broader question of whether post-training introduces new knowledge and capabilities or primarily makes use of existing ones. Instead, it provides evidence that post-training leverages pre-training knowledge for at least some heads.
Questions for authors
N/A
2 comments
Dear Reviewer DTdy,
We wanted to briefly follow up before the discussion period ends.
We added a broader survey of attention heads, another apparent-emergence case, and multi-step reasoning experiments addressing your concerns about generalizability. We also clarified the scope of our conclusions about functional emergence.
We'd be grateful if you could consider these new results when revisiting your assessment, and would be happy to discuss any remaining concerns.
Thank you for your time!
You're right that focusing on a few heads left the broader picture unclear. We ran two follow-ups. Among 16 discovery-ranked candidates, seven showed amplified causal effects with substantial base-to-distilled transfer. We also tested an equal-probability sample of 112 heads from all 784 on 96 new GSM8K probes. Three met our criteria. This measures the frequency of that specific pattern, not the fraction of new capabilities explained by recruitment; one of the three (L0H3) was already strongly causal before distillation.
We also replicated another near-zero-base case, L23H14 (base effect −0.013, distilled effect 0.142; raw transfer retention 110%).
To test whether the finding extends beyond isolated arithmetic probes, we examined three heads at different steps of 64 human-written GSM8K solutions. Aligned base-to-distilled transfer outperformed position-shifted controls in all nine head-by-position comparisons (Holm-adjusted p ≤ 0.0099). These experiments use teacher forcing rather than free-running generation. (Section 4.6; Appendix E.)
Your question about genuine functional emergence is important, and we agree that it could occur. Low transfer alone would not establish it, since recoding or changed upstream inputs could also explain the failure. But our positive finding goes beyond showing that existing information might be reused: L26H25's base-model activation recovers nearly all of its distilled causal effect, despite having almost no causal influence in the base model. Our new downstream interventions further support the role of the receiving network.
This gives a concrete example of how a reasoning head can become newly causal without learning its task-relevant signal from scratch. We do not claim that all heads behave this way or that distillation cannot create new functions. (Sections 4.2–4.3; Appendices C, D and G.)