acceptodds
Under review as a conference paper at ICLR 2027

Which Attention Heads are Like the Human Head? Not the Ones that Compute

Abstract

Brain–AI alignment is often interpreted as a sign that model and brain perform similar computations. Whether the aligned units are causally involved in model computation is rarely checked. On an abstract pattern-completion task (AAABAAA → B), we compare LLM attention-head representations with human EEG and test how ablating those heads affects task performance. Alignment and causation dissociate: brain-aligned heads contribute to performance, but their removal is substantially less disruptive than removal of heads selected via attribution patching. We compare two head sets that prior interpretability work defines without reference to the brain: concept vectors (CVs), which represent abstract patterns across formats, and function vectors (FVs), selected for their contribution to correct-answer prediction. Brain alignment shows little association with FV scores, while its association with CV scores varies across models. Among brain-aligned heads, we find recurring attention profiles: one emphasizes distinctive elements (novelty heads), the other repeating elements (repetition heads). The novelty family tracks salience and attends to the same elements that humans look at, yet its removal is less damaging than random ablation on average. Repetition heads contribute modestly to performance and are associated with abstract-pattern representation (CVs). Across 17 models spanning 3B–72B parameters, FV-ranked removal is substantially more disruptive than brain-ranked removal. Brain alignment thus captures how the model reads the stimulus, and only faintly captures how it represents the pattern and solves the task.

Primary area:
interpretability and explainable AI
Keywords:
NeuroAI, Brain–AI alignment, EEG, mechanistic interpretability, attention heads, causal intervention, abstract reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vVvtOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether attention heads whose representations resemble human neural activity are also important for model performance. Using an abstract sequence completion task, the authors compare representational similarity matrices derived from human EEG responses and LLM attention heads, then examine the effects of ablating heads with different degrees of brain alignment. The reported results suggest that greater representational similarity to human neural activity does not necessarily imply greater causal importance for task performance.

Critical strengths

Studying whether alignment with the brain influences model computation is an interesting and well motivated question. The intervention-based approach is promising. Ablating selected attention heads and comparing the effects with random head removal provides is a nice control. Adapting human abstract sequence completion experiments for model evaluation is a nice setting to compare human with LLMs.

Critical weaknesses

The quantitative reporting and interpretation are insufficiently clear to support the central conclusion. I found it difficult to reconstruct the definitions and calculations of the brain, patching, and concept scores, understand why these measures were selected, or assess their distributions and reliability. Figure 1 is particularly difficult to interpret: the scales of “brain-like” and “causal,” their numerical values, and the relevant reference points or ceilings are unclear. The magnitude and uncertainty of the ablation effects also need clearer reporting.

The paper is very hard to follow. I find that the empirical study is not carefully designed and is insufficiently explained, with evaluations and choices described very vaguely. The takeaway is also unclear. The writing conveys that insufficient thought has been put into the design of the experiments. I am not convinced. I am unsure what the implications of this paper are.

Questions for authors

If the model behaviors are already different from humans in the sequence completion tasks, then what is the reason to compare their internal representations?

The paper lacks a sufficient explaination on what are the tasks used to evaluate the models and the humans. The number of task examples.

Why are attention heads the right unit to compare with human FRPs eeg signals?

1 comment

We thank the reviewer for taking the time to provide feedback. We appreciate your comment that the question is well motivated and the intervention-based approach with random controls promising. We took the concerns about clarity seriously and have substantially rewritten the paper.

Definitions of scores and Figure 1. The scores are now defined in the main text. The new Figure 1 shows schematically how the brain score is computed (each head's representational dissimilarity matrix across the eight patterns is correlated with the matrix from human frontal FRPs) and where each head set sits in the model. Section 2 describes how the EEG target is built and tests its reliability. Section 3 defines each interpretability score: function vector (FV) and concept vector (CV) heads, and the symbol abstraction, symbolic induction, and retrieval heads of Yang et al. (2025). Table 1 (Appendix A.2) lists the data and intervention used for each.

Task description and number of items. Section 2.1 now describes the human task. Each of 25 participants solved 400 problems over five sessions, 50 for each of eight abstract patterns (e.g., AAABAAA → B), choosing the missing eighth icon from four options. Models receive text versions with words or symbols in place of the icons. For the models, the brain score uses 200 multiple-choice items, and all ablations are evaluated on 400 new multiple-choice items (50 per pattern), each preceded by three solved examples (Section 4). Table 1 gives the items used to identify each interpretability-defined head set.

If model behavior differs from humans, why compare internal representations? Model–brain representational alignment is widely reported and interpreted even when behavior is not matched. Our goal is to test what such alignment tells us about model computation. RSA compares the structure of representations across patterns, which can be shared even when accuracy differs. [Average human accuracy is 82% and model accuracy ranges from 27% to 89% (Table 2)]

Why attention heads? Attention heads are the unit at which interpretability work often defines mechanistic function: function vectors, concept vectors, and the symbolic heads of Yang et al. (2025) are all sets of attention heads. Comparing brain alignment at this level lets us read a brain score against units with known function and causal relevance. Each head yields a representation that can be compared with the one extracted from human EEG via RSA. A brain score can therefore be read against heads whose function is known, rather than against anonymous units. The Introduction now states this motivation.

Implications. Brain alignment, as measured here, should not be taken as evidence that the aligned units carry out the model's computation for the task. Claims based on alignment should be checked with causal interventions, and the framework we propose shows how to do this.

We hope the revised manuscript is substantially clearer, and we would be grateful if the reviewer would take another look.

Reviewer_SEupOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work reveals novel alignment between the attention heads in LLM and the fixation-related potential measured by EEG via an abstract pattern-completion task.

Critical strengths

The analysis looks solid and comprehensive. The framework proposed in this work carry scientific value and are interesting to me. The analysis framework they establish, “a brain-alignment score can be read against heads whose function is known rather than against anonymous units.”, in which the generability is also the major strength of this work.

Critical weaknesses

Two major weaknesses:

  1. The claim in this work is too general than it can be. First, this work is built upon EEG, but EEG is coarse and shallow from the perspective of brain activity. Second, this work only focuses on fixation-related potential, and uses only the activity in the frontal lobe. Third, this work is built upon a very specific task, the abstract pattern-completion task, which itself is not general enough to represent the claim it bears here. For example, one of the first and major discovery in this work is “Brain scores do not predict patching scores.” It should be “Fixation-related potential measured by EEG does not predict patching scores.” If the modality becomes iEEG or fMRI, it is hard to say whether "Brain scores do not predict patching scores" still hold. Please make the claim specific and clear to what it can actually bear.

  2. Does the brain-aligned heads with repetition profile match the heads with the strongest repetition profile globally in each LLM? If you randomly select 20 heads from an LLM, you can always say in these 20 heads, some of them show more repetition profile, some show more novelty profile, and the rest show mixture of both. I think the paragraph from line 259 addresses this issue, but also exposes the issue. “repetition carriers accounted for 22.6% of the top 20 FRP heads versus 17.5% of all heads” which is weak alignment. “Novelty carriers were common but not overrepresented (44.4% versus 46.7%, respectively)” is even weaker. This actually echoes what I just argued, randomly select 20 heads from an LLM, then you can always find some repetition carriers, novelty carriers, or neither.

The first can be addressed by properly modifying the framing. However, the second one is fatal. It carries the major insight and contribution. However, from its current state, the major claim, “Brain-aligned heads reveal repetition and novelty profiles.” is correct, but trivial. Anyone who randomly pick 20 heads will get similar result shown in Figure 4. Please justify the non-triviality.

Questions for authors

My major questions are listed in the Critical Weaknesses. If they can be addressed properly, I can raise my score. Overall, I like the writing and the framework proposed by this work. The point is that, the current major result needs to be non-trivial.

Other questions I have:

  1. To make the contribution layered and structured, I suggest to split the findings into brain-relevant contribution, brain-irrelevant contribution. Major observations in section 4 are actually brain-irrelevant. They are more like interpretability on LLM itself.

  2. In line 156, the paragraph talks about the accuracy of each LLM under such prompt. I am wondering whether the correctness is also confounded by the frequency of the correct “word”. For example, (a) pen pen pen quill pen pen pen ? (b) quill quill quill pen quill quill quill ? It is more likely, by my guess, that (b) would enjoy higher accuracy since “pen” itself is a much more common and everyday word, compared to “quill”. So, if the model is just confused, it may just output the most common word. In this case, it happens to be "pen", compared to "quill". I can understand once the number of prompts goes up, such effect would be cancelled out and therefore the result would be something we want, by design. But, if the word frequency itself does affect the accuracy, trying to cancel it out (always use words with similar frequency) may make the result more robust and more interpretable.

  3. The deployed LLM reach about 50% to 80% of accuracy under the designed prompts. For the wrong answer, it is easy to exclude them since they are wrong. However, for the seemingly correct ones (the 50%-80% cases), does the model in each case figure out the actual pattern and get the correct answer? Or, the model is just guessing by word frequency (this is my question 2)? Furthermore, even if the word frequency is controlled, how do you know the model is really catching the pattern and making the correct prediction for the answer? It could still guess it correctly by luck. Measuring the logits may be the way to address it, and I know the paper's AIE score can be used for it. However, the AIE is used only to measure the patching in this paper, not to evaluate how sure the model is on the answer. Also, even if the model is so sure about the answer, and the answer looks to be correct, how can this be uniquely explained by the correct recognition of the structured pattern? For example, in the case of ABABCDC->D, or ABBAABB->A, the reason a model say D for the first pattern, and A for the second pattern may just be because of it is counting, and it wants each letter to appear in exact the same times (making each letter appear twice in the first pattern and four times in the second). If so, the model don't even need to care about the structure of the pattern. It just needs to count. If so, the attention heads cannot serve as the function you want in this paper. Please also report the per-pattern accuracy, and try to characterize how sure the model is on each answer, and why you think the model is recognizing the pattern in the way you want, and use it as a control variable in this work.

  4. The current head removal is done by zeroing them out. However, removal can also be done by randomizing the head activation, or freezing them. Are those different ablations also showing similar pattern demonstrated in figure 3?

  5. Please annotate the values of X and Y axis in figure 1b.

2 comments

We thank the reviewer for the constructive review and for appreciating the framework of reading brain alignment against heads with known function. That framework is now the core of the revised paper.

Scope of claims. We agree that our results can only be read in light of this measure and this task, and we have narrowed the claims accordingly. Throughout the revision, we state results in terms of what we measured: frontal fixation-related potentials (FRPs) recorded with EEG during an abstract pattern-completion task. For example, the abstract now reads “We recorded EEG while participants solve an abstract pattern-completion task (e.g., AAABAAA → B), and score the alignment of every attention head in 17 LLMs with frontal fixation-related potentials (FRPs) via representational similarity analysis (RSA).”, and in Intro we write “... brain-aligned heads, i.e., attention heads whose activations correlate strongly with frontal-electrode FRPs”. A new Discussion subsection states this limitation and notes that other electrode groups, recordings (iEEG, fMRI), or tasks could identify different heads. It also notes a similar dissociation in object classification: in convolutional networks, a kernel’s contribution to predicting brain activity was essentially uncorrelated with its contribution to classification, for fMRI, EEG, and intracortical recordings (Scholte et al., 2026). Whether this holds for abstract reasoning remains to be tested.

Non-triviality of the novelty and repetition profiles. We agree with the reviewer’s analysis. Novelty and repetition carriers are not meaningfully overrepresented among the top brain-aligned heads, so their presence there is not a brain-specific finding. We have removed this analysis, together with the eye-tracking analysis that served mainly to characterize novelty heads. The paper’s main contribution is now the comparison of brain-aligned heads with head sets of known function: FV and CV heads, and we add three head types from prior work (symbol abstraction, symbolic induction, and retrieval heads; Yang et al., 2025). This comparison now uses a more reliable brain RDM and layer-matched random controls (see our reply to C4dC). It also addresses the reviewer’s suggestion to separate brain-relevant from brain-irrelevant contributions, since the brain-independent characterization of attention profiles is no longer part of the paper.

Word frequency. Answer and distractor words are drawn at random on every trial from a fixed list of common single-token English words. Their frequency on the Zipf scale (wordfreq; van Heuven et al., 2014; Speer, 2022) is 3.6–7.4. For reference, pen is 4.4 and quill 3.1. Frequency is therefore balanced across patterns by design. We also tested the concern directly in Llama-3.1-8B and 70B, comparing trials within each pattern. Accuracy did not rise with the answer’s frequency relative to the distractors (r = −.13 and −.02, n.s.). Errors were not biased toward common words: the most frequent wrong option was chosen on 24% and 37% of errors, against 33% expected by chance. We will add this analysis for all 17 models to the appendix.

Per-pattern accuracy, confidence, and counting strategies. Correct answers are not lucky guesses. On correct trials, Llama-3.1-8B and 70B assign a median probability of .55 and .77 to the answer among the four options (chance .25), against .24 on errors. The counting strategy the reviewer describes (completing the sequence so that every word appears equally often) gives the correct answer in six of the eight patterns. ABCAABC? separates the two accounts: counting predicts B or C, but both models answer A, as the pattern requires (24/25 and 25/25 trials), and with their highest confidence. Errors elsewhere mostly pick an already over-represented word, contrary to counting. As the reviewer suggested, we also tested per-pattern accuracy, confidence and counting success as controls. None correlates with the human FRP RDM (|ρ| ≤ .27), and partialling them out leaves the brain scores nearly unchanged (ρ = .95–.96 with the raw scores).

Alternative ablations. We repeated the ranked removals in Llama-3.1-8B and 70B with mean ablation (each head’s output frozen at its average over trials) and resample ablation (replaced by its output on a trial with a different pattern), ablating the random controls the same way (Appendix A5). The main result is unchanged. Removing brain-aligned heads costs no more accuracy than removing random heads from the same layers (−0.4 to 0.7 points over the first 5% of heads), while every interpretability-defined set remains far more damaging (11–55 points). Their order shifts with the method, so we make no claim about it. The gap between them and the brain-aligned heads holds under every method.

Figure 1b. Figure 1 now shows our main finding.

We hope these revisions address the reviewer’s concerns, and we would be grateful if the reviewer would reconsider their score in light of them.

I appreciate the newly conducted analysis and most of my concerns are properly addressed in the rebuttal. However, one thing I still want to note is that, in this version, one of the major claim and analysis is removed from the paper (the novelty and repetition profiles). It should be removed due to the result, but after the removal, the current state of the paper is more like a workshop paper amount of contribution. I would suggest considering upcoming workshop to submit, or discovering novel insights and analysis that could facilitate interpretability for the next round of submission. The current state of the paper is a promising and can be strengthened for future acceptance. I will maintain my current score.

Reviewer_C4dCOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether attention heads whose representations resemble human EEG during an abstract pattern completion task are the heads that LLMs rely on to solve it. Across 17 models, removing the most brain aligned heads hurts accuracy far less than removing heads selected by attribution patching, and less than removing heads that encode the pattern across formats. Brain aligned heads show two recurring attention profiles, one attending to the rare element and one to repeated elements, and the first matches human gaze but contributes little to the task.

Critical strengths

Strengths

  1. The paper directly tests a common reading of brain alignment, namely that units resembling brain activity are the ones doing the model's computation. It ranks attention heads by their similarity to human EEG, removes them, and compares the damage with random removal and with heads chosen by causal or concept criteria. This connects brain alignment studies with interpretability methods that already measure causal importance.

  2. The main findings hold across a broad set of models. The ablation comparison in Figure 3 covers 17 models from 3B to 72B parameters, including base, instruction tuned and distilled variants, and the novelty and repetition profiles found in Llama-3.1-70B reappear under independent clustering in 17 and 13 of these models.

  3. The paper goes beyond ablation to examine what the brain aligned heads do. It separates two attention profiles and tests each further, showing for example that removing repetition heads lowers accuracy while the pattern information in later concept heads stays intact (Appendix A.6.1), and that novelty heads also matter little on eight other tasks.

Critical weaknesses

Weaknesses

  1. (Solved) The headline claim that brain aligned heads are not the ones that compute rests on a contrast that is expected by design. FV heads are selected by their causal contribution to this same task (Section 2), while the brain score uses no information about task performance, so removing FV heads is bound to hurt more. Against random removal, brain ranked removal is more damaging in 15 of 17 models, which points to weaker but real causal involvement rather than a dissociation.

  2. (Not Solved) The reliability of the brain score is not tested. Each head's score is a correlation over only 28 pattern pairs, and the top 20 heads are picked from up to 5,120 heads, so some of them may score high by chance. The paper reports no split half reliability or noise ceiling for the human RDM and no permutation test for head scores, although an earlier report on the same data notes that the FRPs might have been too noisy [1]. If the brain score is mostly noise, brain ranked removal would look close to random for that reason alone.

  3. (Solved) The random controls do not match the layers of the removed heads. As Appendix A.4.1 states, the random sets match only the number of heads. Head sets chosen by different scores can sit in different layers, so part of the gap between FV, brain and random removal, and the finding that novelty heads hurt less than random, may come from where the heads are rather than what they encode. The authors list layer matched controls as future work in Section 6.

references:

[1] Christopher Pinier, Claire E. Stevenson, and Michael D. Nunez. Moderate evidence for large language models reflecting human neurocognition during abstract reasoning. In Conference on Cognitive Computational Neuroscience, 2025. URL https://2025.ccneuro.org/abstract_pdf/Pinier_2025_Moderate_evidence_large_language_models_reflecting.pdf.

Questions for authors

With random controls matched to the layer distribution of each ranked set, do brain ranked and novelty ranked removal still differ from random in the way shown in Figure 3?

2 comments

We thank the reviewer for the careful review. All three weaknesses raised were valid, and addressing them has changed, and we believe strengthened, the paper.

Circularity of the FV comparison. We agree that FV heads are selected by their causal contribution to this task, so their larger ablation effect is partly expected by design. The revised paper therefore compares brain-ranked heads with two kinds of head sets that are not selected by causal effects on our task. First, concept vector (CV) heads are scored representationally, without any intervention. A head scores highly if its output at the final position separates the 8 patterns across symbol sets (abstract symbols, German words, English words) and question formats (open-ended, multiple choice). Second, we added the three symbolic-mechanism head types of Yang et al. (2025): symbol abstraction, symbolic induction, and retrieval heads. Following the original work, we identify these heads on a different task, the ABA/ABB identity-rule task, by causal mediation between prompt pairs. We estimate these scores with attribution patching. The estimates agree closely with the authors' released activation-patching scores for Llama-3.1-70B (r = .93–.99) and with exact patching of 40 heads per type in each of the 17 models (r = .88–.99). Ablating each of these head sets is more disruptive than ablating brain-ranked heads [Figure 3]. The comparison therefore no longer depends on a score selected for the outcome being measured.

Reliability of the brain score. The reviewer's concern was justified. We added a split-half reliability analysis. We pooled trial-level FRPs across participants and sessions and randomly split the trials into two non-overlapping halves, balancing each pattern's trial count between them. We then averaged waveforms by pattern within each half, built one RDM per half, and correlated their 28 pairwise distances, repeating this 1,000 times. The original RDM was not reliable (median split-half Spearman ρ = -0.09). We replaced it with an RDM constructed hierarchically within participants and sessions, which has a median split-half ρ = 0.58 and recomputed all brain scores and analyses with it. Details are in the new section “Eeg Methods and Brain RDM Reliability”.

Layer-matched controls. We now compare every ranked ablation against random head sets matched to that set's layer distribution [Fig 3]. Under this control, every interpretability-defined head set remains more disruptive to ablate than brain-ranked heads, while brain-ranked ablation is statistically indistinguishable from random (see Section 4.1). In the original submission, brain-ranked ablation was more damaging than size-matched random ablation in 15 of 17 models. That difference reflected both the unreliable RDM and the layers the selected heads occupied. The reviewer read the original result as weaker but real causal involvement rather than a dissociation. With the reliable RDM and layer-matched controls, we no longer find evidence of causal involvement beyond random.

Question: do brain-ranked and novelty-ranked removal still differ from random under layer-matched controls? Brain-ranked removal does not differ from layer-matched random removal. We removed the novelty-ranked ablation from the paper along with the novelty/repetition analysis (see our reply to SEup).

We hope these changes address the reviewer's concerns, and we would be grateful if the reviewer would take them into account in their assessment.

Thanks for the rebuttal! However, my main concerns about practical use of the brain score have not been fully explained and proofed, and I'm not convinced by the new results. I'll maintain my score.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback that has helped us greatly strengthen the paper. We have re-run a number of analyses and substantially revised the paper. We hope that the reviewers can find the time to take these revisions into consideration.

The main changes are summarized below, and we answer each reviewer in detail in individual replies.

Main differences are:

  1. Focus and scope (reviewers: SEup, vVvt): The paper now focuses on its main message: brain alignment measured with frontal FRPs does not preferentially identify the heads that are causally important for the task. Following reviewer SEup’s recommendation, we restrict our conclusions to frontal FRPs on this task, and the Discussion considers how the results may relate to other brain measures (e.g., fMRI). We removed the novelty/repetition-head and eye-tracking analyses, which were not meaningfully related to brain alignment.

  2. Interpretation circularity and controls (reviewers: C4dC, SEup): We agree with reviewer C4dC that comparing brain-ranked heads only with FV heads, which are selected using this task, is partly circular. We also agree with reviewer SEup that the paper's main strength is comparing brain alignment with heads of known function. We therefore added three head types identified in prior interpretability work on an abstract sequence completion task (symbol abstraction, symbolic induction, and retrieval heads; Yang et al., 2025). Following reviewer C4dC’s suggestion we use layer-matched controls, so random controls remove the same number of heads from each layer. Under these controls, ablating brain-aligned heads does not differ significantly from random, whereas every interpretability-defined head set is more disruptive than random in at least 16 of 17 models. There is also no overlap between brain-aligned heads and the five sets of attention heads beyond what would be expected by chance.

  3. Brain RDM reliability (reviewer C4dC): A new split-half analysis (1,000 random trial splits, balanced by pattern and session) showed that the original brain RDM, built from FRPs pooled across participants and sessions, was unreliable (median ρ = −0.09). We replaced it with a hierarchical RDM, which we build within each session in its native channel space and then average across sessions and participants. The new RDM is reliable (median ρ = 0.58), and we recomputed all brain scores with it. The main result holds.

  4. Clarity (reviewer vVvt): We rewrote the Methods to define and motivate the brain score and each interpretability score (FV, CV, and symbolic-mechanism heads; Yang et al., 2025).

In addition, we revised the abstract, redrew Figure 1, and restructured the paper around the main result.

We hope that the reviewers agree that the paper has greatly improved and deserves to be published given the novel contribution that EEG-aligned attention heads are not the ones that compute the solution to this abstract pattern completion task.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.