- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates whether attention heads whose representations resemble human neural activity are also important for model performance. Using an abstract sequence completion task, the authors compare representational similarity matrices derived from human EEG responses and LLM attention heads, then examine the effects of ablating heads with different degrees of brain alignment. The reported results suggest that greater representational similarity to human neural activity does not necessarily imply greater causal importance for task performance.
Critical strengths
Studying whether alignment with the brain influences model computation is an interesting and well motivated question. The intervention-based approach is promising. Ablating selected attention heads and comparing the effects with random head removal provides is a nice control. Adapting human abstract sequence completion experiments for model evaluation is a nice setting to compare human with LLMs.
Critical weaknesses
The quantitative reporting and interpretation are insufficiently clear to support the central conclusion. I found it difficult to reconstruct the definitions and calculations of the brain, patching, and concept scores, understand why these measures were selected, or assess their distributions and reliability. Figure 1 is particularly difficult to interpret: the scales of “brain-like” and “causal,” their numerical values, and the relevant reference points or ceilings are unclear. The magnitude and uncertainty of the ablation effects also need clearer reporting.
The paper is very hard to follow. I find that the empirical study is not carefully designed and is insufficiently explained, with evaluations and choices described very vaguely. The takeaway is also unclear. The writing conveys that insufficient thought has been put into the design of the experiments. I am not convinced. I am unsure what the implications of this paper are.
Questions for authors
If the model behaviors are already different from humans in the sequence completion tasks, then what is the reason to compare their internal representations?
The paper lacks a sufficient explaination on what are the tasks used to evaluate the models and the humans. The number of task examples.
Why are attention heads the right unit to compare with human FRPs eeg signals?
We thank the reviewer for taking the time to provide feedback. We appreciate your comment that the question is well motivated and the intervention-based approach with random controls promising. We took the concerns about clarity seriously and have substantially rewritten the paper.
Definitions of scores and Figure 1. The scores are now defined in the main text. The new Figure 1 shows schematically how the brain score is computed (each head's representational dissimilarity matrix across the eight patterns is correlated with the matrix from human frontal FRPs) and where each head set sits in the model. Section 2 describes how the EEG target is built and tests its reliability. Section 3 defines each interpretability score: function vector (FV) and concept vector (CV) heads, and the symbol abstraction, symbolic induction, and retrieval heads of Yang et al. (2025). Table 1 (Appendix A.2) lists the data and intervention used for each.
Task description and number of items. Section 2.1 now describes the human task. Each of 25 participants solved 400 problems over five sessions, 50 for each of eight abstract patterns (e.g., AAABAAA → B), choosing the missing eighth icon from four options. Models receive text versions with words or symbols in place of the icons. For the models, the brain score uses 200 multiple-choice items, and all ablations are evaluated on 400 new multiple-choice items (50 per pattern), each preceded by three solved examples (Section 4). Table 1 gives the items used to identify each interpretability-defined head set.
If model behavior differs from humans, why compare internal representations? Model–brain representational alignment is widely reported and interpreted even when behavior is not matched. Our goal is to test what such alignment tells us about model computation. RSA compares the structure of representations across patterns, which can be shared even when accuracy differs. [Average human accuracy is 82% and model accuracy ranges from 27% to 89% (Table 2)]
Why attention heads? Attention heads are the unit at which interpretability work often defines mechanistic function: function vectors, concept vectors, and the symbolic heads of Yang et al. (2025) are all sets of attention heads. Comparing brain alignment at this level lets us read a brain score against units with known function and causal relevance. Each head yields a representation that can be compared with the one extracted from human EEG via RSA. A brain score can therefore be read against heads whose function is known, rather than against anonymous units. The Introduction now states this motivation.
Implications. Brain alignment, as measured here, should not be taken as evidence that the aligned units carry out the model's computation for the task. Claims based on alignment should be checked with causal interventions, and the framework we propose shows how to do this.
We hope the revised manuscript is substantially clearer, and we would be grateful if the reviewer would take another look.