acceptodds
Withdrawn from ICLR 2027

A Systematic Study of Functional Attention Heads in Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A Transformer spreads its computation over many attention heads, and interpretability work now ties individual heads to specific functions. Induction heads copy repeated patterns, retrieval heads fetch information from long contexts, and truthfulness heads separate true from false answers. These labels are used as if they transferred across models, settled early in training, and could be turned up at will. Each label, though, was found in one model, with one detector and one intervention. This paper tests the assumptions together. We use six head-level signals, five base models, two Pythia pretraining trajectories, and matched-control interventions on Qwen, Llama, and Mistral. One answer recurs: a repeatable signal fixes a head type's coarse structure, not its fine identity. (1) Distribution: local-pattern heads precede induction heads in every model. No complete layer order is shared. (2) Attribution: induction and retrieval detectors pick largely the same heads, 20 and 19 of 30 on Qwen and Llama. The retrieval cost of an “induction” mask falls on those shared heads, not on either label's exclusive heads. (3) Formation: detector strength saturates thousands of steps before top-head membership settles. Fine-tuning changes membership without moving layer distributions. (4) Control: patching the selected retrieval heads restores most of the retrieval loss, and the labels improve KV-cache allocation. Doubling a head's output does not improve its task. A label therefore transfers three things: the relative order of head types, the task contribution of the selected set, and a ranking for memory allocation. It does not transfer absolute depth, an exclusive function, head membership, or gains from amplification. Reuse a label's coarse structure, and re-derive its membership and coefficients for each model.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, attention heads, functional specialization, language models, causal interventions

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.20 over 5 reviews, above 2% of the others.
Reviewer_5ToHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper examines whether functional labels assigned to attention heads, such as induction, retrieval, and truthfulness, remain meaningful across models, training stages, and interventions. Through comparisons across models, analyses of pretraining trajectories, and head-level interventions, the paper argues that these labels capture certain coarse structural and task-related regularities, but do not reliably identify specific heads, establish exclusive functions, or guarantee performance improvements through amplification.

Critical strengths

The paper investigates an interesting and relevant problem in mechanistic interpretability: what can legitimately be inferred from a functional label assigned to an attention head? Examining whether labels established using a particular model, detector, and intervention remain meaningful in other settings is a valuable research direction. The distinction between coarse structural regularities and the identities of individual heads is potentially useful for interpreting and reusing head-level analyses. Bringing together questions about distribution, attribution, formation, and control could also provide a useful framework for studying these issues systematically.

Critical weaknesses
  1. The organization and formulation of the research questions need substantial improvement.

Although the paper addresses several interesting questions, their motivation and precise formulation are not sufficiently developed. The relationship between each research question, its experimental setup, and the resulting conclusion is often difficult to follow. In particular, the paper should clearly establish what property is expected to transfer, how that transfer is measured, and what evidence would support the corresponding claim. Organizing each section around an explicit question, experimental rationale, and conclusion would make the argument easier to assess.

  1. The model coverage is limited relative to the breadth of the conclusions.

The main model comparison includes Qwen2.5-7B, Llama-2-7B, Llama-3.1-8B, Mistral-7B, and Pythia-6.9B. This provides useful coverage across model families, but the models are concentrated in a narrow parameter range of approximately 7–8B. Since the paper’s central contribution is an empirical characterization of the transferability of head-level labels, the dependence of its findings on model scale remains insufficiently explored. Comparisons across substantially different model sizes would strengthen the conclusions. Alternatively, the paper should more explicitly delimit its claims to the evaluated settings. The scope of conclusions drawn from the two Pythia pretraining trajectories should likewise be distinguished from conclusions supported by the broader model comparison.

  1. Too much essential evidence is deferred to the appendix.

The main paper does not present enough of the evidence needed to assess its central claims, with too many important result tables relegated to the appendix. The authors should select the most informative tables and incorporate them into the main text, particularly those supporting the principal comparisons across models and the intervention results. Each table should be directly connected to a research question and accompanied by a clear interpretation. The appendix is appropriate for exhaustive numerical results, additional breakdowns, and implementation details, but the main paper should contain enough evidence for readers to evaluate its core arguments.

  1. The pretraining configuration underlying Figure 3 is insufficiently clear.

Reporting training steps alone does not provide enough context to interpret the timing of head formation or compare the training trajectories. The paper should specify the effective global batch size, sequence length, and corresponding number of training tokens, and clarify which training settings are shared across the trajectories. Providing cumulative training tokens alongside optimization steps would make the results easier to interpret. This is especially relevant to the claim that detector strength saturates thousands of steps before top-head membership stabilizes: the reader needs sufficient training context to understand the magnitude and significance of that separation.

  1. The experimental choices require clearer justification.

Beyond the missing configuration details, the paper does not sufficiently explain why its experimental settings provide appropriate tests of the stated hypotheses. For each research question, the authors should justify the chosen detectors, head-selection criteria, interventions, and evaluation measures, and explain how the resulting observations support the corresponding interpretation. Detector overlap, the causal contribution of a selected head set, and the effect of output amplification address different questions and warrant different conclusions. Without a clearer connection between the experimental design and the claims, it is difficult to assess whether the broader interpretations are adequately supported.

Questions for authors

None, it is just not a conference-level paper.

Reviewer_Lq6NOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies different types of attention heads, including previous-token, duplicate-token, induction, retrieval, truthfulness heads, The analysis compares their distribution in a model, the formation during training, and their effects on the model outputs. The main finding is that local-pattern heads precede induction heads, and the function of attention heads may overlap, meaning that different naming of heads does not imply exclusive functionality.

Critical strengths

None

Critical weaknesses
  1. Lack of novelty. The finding that "local-pattern heads precede the induction heads" is already well known [1,2]. The finding that different names of attention heads may have overlapping function is also known [3,4]. The literature review (Section 4) does not show how this paper is different from earlier work, either.

  2. Lack of readability. This paper is poorly rewritten (likely heavily edited or generated by AI). This paper contains many confusing terms, disconnected sentences, or unclear motivations. For example:

  • "A catalogue of parts is where a mechanistic account begins, not where it ends." (L47) What does this mean?
  • "Neuroscience makes the same ..." (L48) Why a long sentence about neuroscience?
  • "So is whether the effect of ablating a labeled head belongs to that label when the same head carries another, whether heads form in the order they depend on one another, and whether strengthening a detected head strengthens its function. Every practical use above assumes an answer." (L51) Convoluted sentence, it is unclear to me what this sentence conveys.
  • Figure 1 needs clarification. What do these plots mean?
  • "A label therefore transfers as a procedure rather than as a set of heads." (L445) What procedure?
  • "Functional labels for attention heads are reproducible enough to be useful and too loose to be mechanistic identities." (L460) What does it mean? How is attention head related to reproducibility? Why is loose?

[1] Olsson, Catherine, et al. "In-context learning and induction heads." arXiv preprint arXiv:2209.11895 (2022).

[2] Wang, Kevin, et al. "Interpretability in the wild: a circuit for indirect object identification in GPT-2 small." arXiv preprint arXiv:2211.00593 (2022).

[3] Kissane, Connor, et al. "Interpreting attention layer outputs with sparse autoencoders." arXiv preprint arXiv:2406.17759 (2024).

[4] Gould, Rhys, et al. "Successor heads: Recurring, interpretable attention heads in the wild." International Conference on Learning Representations. Vol. 2024. 2024.

Questions for authors

Figure 1 is unclear. What do these plots mean?

Reviewer_VstSOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how functional attention heads are organized across models and how they change during training. It brings together several existing head detectors and examines head locations, overlap between functional labels, stability across checkpoints, and the effects of interventions. The results suggest that some relative layer orderings are consistent, but the exact heads are less stable. Induction and retrieval heads also overlap substantially, making their ablation effects difficult to attribute to either label alone. Increasing head outputs does not consistently improve performance, but the detector scores are somehow useful for KV-cache allocation (not stable), showing the application potential of explainability findings. The main contribution is a systematic check of what these functional labels actually tell us and how far they can be reused.

Critical strengths
  1. The paper brings together several lines of research on functional attention heads in a comparative study. Examining them under a common framework yields useful observations that are harder to obtain from studying each head type separately.
  2. Using functional-head scores to guide KV-cache allocation is an interesting application. Although the gains are limited and depend on the cache budget, the experiment suggests a practical use for these scores and a direction worth exploring further.
Critical weaknesses
  1. The study feels scattered. The connections between the four research questions are not sufficiently developed, and the results do not build toward a clear account of how functional heads are organized. For example, RQ1 studies layer distributions across models, while RQ3 studies head formation during Pythia training, but the paper does not clearly show how the training observations explain the layer distributions. The paper seems to be a bag of mini findings about attention heads, but they are not connected in a systematic way.

  2. Many conclusions establish limits on existing interpretations without offering a clear alternative explanation. Negative results can be valuable, but several findings here depend on the model, detector, or intervention setting, making it difficult to identify what general knowledge the study establishes. For example, the retrieval loss from removing shared induction or retrieval heads is clear on Llama-3, but weaker on Mistral and absent on Qwen in that test. Similarly, the amplification experiments mainly show that doubling head outputs does not reliably help, but do not explain why or establish when amplification could work. These results make it difficult to identify a clear general conclusion. I would like to see stronger validation of a smaller set of findings, or a deeper analysis explaining the observed differences.

Questions for authors
  1. 047-050 Neuroscience makes the same move after a cell-type atlas. It asks where each type sits, whether two markers label the same cells, when the types develop, and whether driving a type changes behavior, because those answers decide what the atlas can be used for. what’s the source of this statement?
  2. What evidence connects the four RQs beyond their shared focus on attention heads? do the training results in RQ3 help explain the distributions or overlap observed in RQ1 and RQ2?
  3. Retrieval scores are computed only on examples where retrieval succeeds. How sensitive are the selected heads and their overlap with induction heads to this choice? Could the overlap partly reflect the scoring procedure?
  4. What do the authors see as the main new insight from this study beyond the known limitations of functional-head labels? Which result most clearly changes our understanding of how attention heads are organized?
  5. How should we interpret the differences across models and settings? Do they point to genuinely different functional organization, or mainly to limitations of the detectors? This distinction seems important to the paper’s overall conclusion.
Reviewer_hVcWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the functional organization of attention heads in language models. Building on head labels identified in prior work, it examines their relationships and consistency across models and training stages. The study is organized around four questions: where different head types occur within the network (Distribution), whether their behavioral contributions can be attributed to distinct functional labels (Attribution), how their signals and identities develop during training (Formation), and whether amplifying selected heads improves the associated capabilities (Control). Through detector-based measurements and controlled interventions across several language models, the authors find that functional labels capture some recurring structure but do not consistently identify fixed head populations, exclusive functions, or reliable targets for amplification.

Critical strengths

The paper presents an extensive set of experiments investigating the relationships among functional attention-head labels within language models, reflecting substantial empirical effort.

Critical weaknesses
  1. Although the paper poses relevant questions about functional attention heads, the significance of its specific findings is not sufficiently developed. For example, RQ1 identifies a few recurring relative-depth relationships and finds no complete ordering shared across models, but provides limited explanation of why these patterns arise or what determines their transferability. Similarly, RQ3 establishes a separation between detector strength and head-membership stability without conclusively answering whether heads form in dependency order. The contribution would be stronger with deeper mechanistic explanations and clearer implications for interpreting or modifying models. I

  2. The experimental coverage is uneven, and the rationale for selecting particular combinations of detectors, models, and interventions is sometimes not explained. RQ2 focuses on induction-prefix and retrieval because their selected heads overlap substantially, but this pair was chosen after inspecting the results, and the clearest shared-head effect occurs on one model. This limits how broadly the attribution findings can be generalized. Likewise, the choice of induction-attention rather than induction-prefix for the held-out RQ4 experiment is insufficiently motivated. A clearer account of these choices and their implications for the scope of each conclusion would strengthen the study.

  3. The presentation makes the experimental logic difficult to follow. The terminology is difficult to follow when first introduced. The abstract refers to “six head-level signals,” while in abstract and introduction there is no summary or reference what these six signals are. Although Figure 1 lists the six detectors and section 2.1 defines them, an earlier explanation of these terms would improve readability. Furthermore, Figure 1 is dense, with small labels and schematic elements that require more explanation. Also, Figure 8 in the appendix is not rendered.

Questions for authors

Please see weakness.

Reviewer_9DAKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper purports to investigate reproducibility of “head types” across models. I.e., do concepts like induction heads, duplicate detection heads, etc reproduce across models in terms of where they occur in the model, when they emerge in training, or what function they perform in the model.

Critical strengths

I like the premise of the study. Understanding how consistent these concepts are across models is interesting and would bring the field forward.

Critical weaknesses

This paper is very very difficult to read, and I consider myself to be very experienced in reading and writing mech interp papers. I have a strong suspicion that it is AI generated. Perhaps I am wrong, but if so, the criticisms still hold. For a top-tier competitive conference, the abstract and intro should be very clear and compelling. Given that, based on this intro and discussion, I have no idea what the paper is really about, it is not worth my time to give a deep read of the results and methods. If the authors want this published, they should do a complete rewrite of the intro to make it clear and compelling to a human reader.

By hard to read, I mean that there is a plethora of jargon and noun compounds that sound like scientific terms of art but are in fact unique to this paper and meaningless to an expert reader. This is very common in AI writing. For example, in the intro and abstract alone, the following phrases and sentences are not meaningful to me:

  • local-pattern heads
  • retrieval cost
  • “induction” mask
  • label’s exclusive heads
  • detector strength
  • top-head membership
  • retrieval loss
  • KV-cache allocation
  • Task (in the context of “doubling a head’s output does not improve its task”)
  • ranking for memory allocation

Additionally, several seemingly load-bearing sentences in the intro and discussion don’t make any sense. E.g.:

  • “a repeatable signal fixes a head type’s coarse structure, not its fine identity.”
  • “It does not transfer absolute depth, an exclusive function, head membership, or gains from amplification. Reuse a label’s coarse structure, and re-derive its membership and coefficients for each model.”
  • First line of discussion: “A repeatable label licenses coarse claims and not fine ones. It identifies heads that carry the matched task. Masking them costs the task, patching them back restores most of the loss on both tested models, and ranking heads by the label tells a memory allocator which heads to keep.”

Perhaps the experiments are fine. But it would be embarrassing for ICLR to publish a paper written this way. The authors should rewrite substantially in order to warrant a careful scientific review from a human expert.

Questions for authors

n/a, see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.