acceptodds
Under review as a conference paper at ICLR 2027

From Embeddings to Attention Heads: Input-Agnostic Tracing of Embedding-MLP-Attention Circuits in LLMs

Abstract

Input-agnostic mechanistic interpretability explains neural computation from model parameters without repeated forward passes. Existing input-agnostic methods still face a tradeoff: analyses of interactions between modules are mostly limited to toy transformers, while studies of standard LLMs usually focus on isolated modules, leaving key residual-stream interactions underexplored. We address this gap by introducing a task-conditioned zero-pass framework for tracing embeddingnative gated-MLP-neuronattention-head interactions in standard LLMs. Our method requires no task-example forward passes for activation/gradient collection. To our knowledge, prior input-agnostic work has not traced these interactions at this scale in a standard pretrained LLM. Experiments against input-dependent methods, including state-of-the-art causal path patching methods, show that our method achieves a highly favorable balance between causal effectiveness and computational efficiency.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, input-agnostic circuit tracing, MLP neuron analysis, efficient interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hy9aOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a method for identifying MLP neurons that contribute to a target attention head's behavior by analyzing a language model's weights (without running forward passes on task examples). The key idea is to expand the computation of later modules in terms of earlier ones and ultimately the embedding matrix, in the same spirit as the original mathematical framework for transformer circuits, but additionally considering MLP neurons in addition attention.

Validation is limited to showing that, on a single model (OpenLLaMA-7B) and two tasks (probing for syntactic dependencies and IOI), restoring a set of 200 neurons selected by the method preserves model behavior better than some baselines.

Critical strengths
  • Interpreting a model primarily by inspecting weights rather than using task data is an interesting angle
  • I appreciate the ablations and the comparison of FLOP costs per baseline method
Critical weaknesses
  1. Limited experimental validation. Results are restricted primarily to a single model (OpenLLaMA-7B) and two tasks (dependency probing and IOI). Moreover, for attribution-style work it is standard to report a frontier of restoration (or ablation) effect as the number of selected components varies, rather than a single cutoff the paper only reports results at a fixed threshold of 200 neurons. The method additionally doesn't not improve much over the path-patching baseline.

  2. Clarity of presentation. The writing makes the paper very difficult to evaluate. Concretely, details of the task setup, the baselines, and the details of method itself (how cutoffs for embedding and head-composition scores were chosen, choice of vocabulary per task).

  3. Much of the prose reads as though it was primarily language-model-generated, and is hard to follow due to heavy use of jargon and vacuous statements. I acknowledge that some of this is a matter of taste, but I think it is worth raising as a separate critique from the weakness of the results themselves.

Questions for authors

None

4 comments
AuthorsOpenReview ↗

(1/3) Response to W1

Thank you for your reviews. Our reply is as follows:

For your concerns about the limited experimental validation in W1:

Results are restricted primarily to a single model (OpenLLaMA-7B) and two tasks ...

Actually, we've reported the results of the factual recall task and OpenLLaMA-3B in Appendix D.3 and D.4. We apologize that these contents might not be easily noticeable. To further resolve your concern, we've adapted our method to more models and tasks. On the more representative dependency probing task, we ran it on Llama-2-7b, Llama-2-13b, Yi-6B and Phi-3-mini-4k, and compared it with random neurons and Probeless. We also adapted it to four more tasks: plural, past tense, gerund and adverb. The results are as follows:

UUAS / ULAS ↑Clean runCorrupted runRandomProbelessOurs
OpenLLaMA-3B48.78 / 26.8843.68 / 9.7743.74 / 9.7643.79 / 10.1845.60 / 15.76
Llama-2-7b43.86 / 27.9438.36 / 12.9438.33 / 12.9138.35 / 13.1640.94 / 17.45
Llama-2-13b48.59 / 32.6241.85 / 18.8041.88 / 18.8743.03 / 20.1144.52 / 25.00
Yi-6B47.01 / 30.5439.51 / 15.1339.51 / 15.1140.55 / 15.5541.52 / 17.69
Phi-3-mini-4k45.66 / 29.0543.76 / 19.7543.77 / 19.7343.61 / 19.8744.91 / 22.49
ΔLogit vs. Corrupt. ↑IOIFactual recallPluralPast tenseGerundAdverb
Probeless0.15470.15000.17560.23390.22040.2663
Path patching1.01340.30700.37740.43080.44590.3854
Ours1.18680.43410.39630.52560.55910.8183

Our method also adapts well to these models and tasks, and beats the baselines on all of them. We have added both tables to Section 4.5 (Tables 3 and 4) and the descriptions of these models and tasks to Section 4.1 (with task examples in Appendix A.1) in our rebuttal version paper (added contents are marked orange; please check it).

For your concern about the frontier over the number of selected neurons:

... it is standard to report a frontier ... as the number of selected components varies ...

We ran our method on dependency probing with OpenLLaMA-7b while varying the number of selected neurons per relation from 50 to 3200, and compared it with the same number of random neurons (averaged over three seeds, on the development set):

Neurons per relationOurs (UUAS / ULAS)Random (UUAS / ULAS)ULAS recovered
5045.02 / 21.8941.51 / 14.7248%
10045.69 / 22.8441.51 / 14.7054%
200 (default)45.82 / 23.0941.53 / 14.7356%
40045.98 / 23.3941.52 / 14.7757%
80046.15 / 23.7241.48 / 14.7360%
160046.33 / 24.0541.46 / 14.7862%
320046.54 / 24.4241.64 / 15.0464%
Clean / Corrupted run47.22 / 29.7741.49 / 14.73

(ULAS recovered is our gain over random neurons, as a share of the gap between the clean run and random neurons)

Our method stays far above random neurons at every size, and 200 lies at the bending point of this curve. The restoration rises quickly up to 100–200 neurons per relation, while after 200, each doubling of the neuron set only brings a marginal ULAS gain. This size is also at the same scale as previous neuron-level works: Probeless (Antverg & Belinkov, 2022) evaluates its rankings with up to 150 top neurons, and Tang et al. (2024) take 1% of all neurons as language-specific neurons, while our 800 neurons over the four relations are only about 0.23% of the MLP neurons in OpenLLaMA-7b. What's more, such a small set keeps the explanation budget low (the number of neurons a human needs to read through), so we can still inspect the selected neurons one by one, as in the output-vocabulary projections of the dependency neurons in Appendix E.2 and the lexicons of the traced GPT-2 neurons in Appendix D.5. We have added these results to the new Appendix H.

For your concern about the gap to path patching:

The method additionally doesn't not improve much over the path-patching baseline.

Path patching needs to run on many data examples, with many patched forward passes per example, so we chose it mainly as an expensive baseline to verify whether our method can achieve comparable or even higher performance than actual patching. Its exhaustive neuron-level version is computationally prohibitive, so we already had to use a reduced adaptation of it (Section 4.2 and Appendix F). Even so, on all six logit-based tasks in the table above, our method outperforms this adaptation, with a large margin on adverb. Our method also reaches such a frontier of computation and task performance. As shown in Figure 1, it stays close to path patching on dependency probing with over two orders of magnitude less computation than even this reduced adaptation (14.53 vs. 6276.48 TFLOPs), and it is the cheapest of all the methods there.

References:

  • Antverg & Belinkov, 2022. On the Pitfalls of Analyzing Individual Neurons in Language Models.
  • Tang et al., 2024. Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models.
AuthorsOpenReview ↗

(2/3) Response to W2

For your concern about the clarity of the settings in W2:

Clarity of presentation. The writing makes the paper very difficult to evaluate. Concretely, details of the task setup, the baselines, and the details of method itself (how cutoffs for embedding and head-composition scores were chosen, choice of vocabulary per task).

We're sorry that the task setup, the baselines and the details of our method were only briefly described in the main text and scattered across Appendix A, which made the paper difficult to evaluate. We hope the following clarification of the settings might help reduce this difficulty.

For the task setup, the target heads and the corrupted positions are as follows (all on OpenLLaMA-7b):

TaskTarget headsCorrupted positionsMetric
Dependency (det, prep, cc, aux)Top-10 heads per relation by MI with the relation labelThe function word of each arcUUAS / ULAS on all dev sentences
IOITop-10 heads attending from the end position to IO most oftenIO nameLogit difference at the end position
Factual recall, plural, past tense, gerund, adverbTop-10 heads attending from the end position to the subject most oftenSubject spanLogit difference at the end position

In the corrupted run, we add Gaussian noise N(0,0.12)\mathcal N(0, 0.1^2) to the input embeddings at these positions, and the patched run restores the selected neurons at the same positions.

For the baselines, their ranking scores are described in Section 4.2, and all of them share the following settings with our method:

SettingShared by all methods
Target headsThe heads in the table above
Neuron budget200 neurons per relation (or task)
Corruption and restored positionsAs described above
EvaluationThe same examples and seeds

So none of these settings is chosen to favour our method, and each of our result tables comes from one configuration shared by all methods. The only difference is that the input-dependent baselines use task examples to select neurons while our method uses none, which, if anything, favours the baselines.

For the method itself, the cut-off hyperparameters are klexk_{lex} and kpairk_{pair}. klexk_{lex} is the number of top tokens under the embedding-facing score (Eq. 5) kept as a neuron's activation lexicon, which is then matched with the task vocabulary. kpairk_{pair} is the number of top-scoring neuron pairs kept from the head-composition matrix (Eq. 8), from which we count how often each neuron composes strongly with the target head. We found that, like the curve after the top-200 neurons in our reply to W1, they only have marginal effects on the performance (Appendix D.1 for kpairk_{pair}). So we chose the setting in our paper. The task vocabularies are collected automatically from the data, so no manual choice of words is involved:

TaskTask vocabularyExamples
detDependent-side words of the relation in the training split (22 words)the, a, an, this, some, no, those
prepSame as above (115 words)of, in, for, from, with, into, about
ccSame as above (18 words)and, but, or, nor, yet, plus
auxSame as above (42 words)will, has, would, is, could, can, may
IOI~1,000 common English first names (a separate list from the names used in the prompts)Jacob, Emily, Michael, Hannah, Sarah
Factual recallSubject words of the taskLeslie Moonves
Plural, past tense, gerund, adverbSubject words of the task (the base forms)cast (plural), cap (past tense), portion (gerund), merry (adverb)

To avoid further misunderstanding, we have added a short summary of the target heads, the task vocabularies and the settings shared by all methods to Section 4.1 of our rebuttal version paper, together with the corruption method in Section 4.3 and the tables above in Appendix A.1. We have also corrected the description of kpairk_{pair} in Appendices A.2 and D.1.

AuthorsOpenReview ↗

(3/3) Response to W3

For your concern about the writing in W3:

Much of the prose reads as though it was primarily language-model-generated, and is hard to follow due to heavy use of jargon and vacuous statements. ...

We apologize that some sentences were vague, or used terms that a reader has to "decode" first. We went through the whole paper and revised them. The main revisions are as follows:

  • In Abstract: "We address this gap by introducing a task-conditioned zero-pass framework for tracing embedding→native gated-MLP-neuron→attention-head interactions in standard LLMs." → "We address this gap with a method that traces how input embeddings influence later attention heads through individual MLP neurons in standard LLMs, using only the model weights and a task vocabulary." Also, "a highly favorable balance between causal effectiveness restored task performance and computational efficiency".
  • In Introduction: "It seeks to uncover universal functional mechanisms within the model by analyzing parameter distributions and matrix properties, ..." → "It explains model computation from the weights alone, so no inputs or intermediate activations are needed."
  • In Introduction: "... this module-centric perspective largely abstracts away the cross-module interactions through which many model behaviors may emerge." → "These works assign functions to single components, while many behaviors come from several modules together, e.g. an MLP neuron writing a feature that a later head reads."
  • In Introduction: "As a result, they provide valuable conceptual tools but do not readily scale to modern transformers ..." → "Since these compositions leave out MLPs, they do not directly apply to modern transformers."
  • In Introduction: "... it makes MLPs part of the analyzable mechanism, and it ties weights analysis to the actual residual-stream signals that those weights write and later heads read." → "... each MLP neuron becomes an explicit term of the target head's attention score, and this term uses the vector that the neuron actually writes into the residual stream."
  • In Introduction: "To evaluate discovered neurons causally, We further designed a perturb-then-restore-based benchmark." → "To compare the neurons found by different methods, we corrupt the input at task-relevant positions, restore only the selected neurons, and measure how much of the original behavior returns."
  • In Contributions: "First, it opens a new regime of input-agnostic mechanistic interpretability: from toy models to ..." → "First, it extends input-agnostic analysis of module interactions from attention-only toy models to standard LLMs with gated MLPs." Also, "Third, it achieves an exceptionally favorable balance ... requiring only negligible computation by comparison." → "Third, the neurons it finds restore task behavior comparably to path patching, with over two orders of magnitude less computation (Figure 1)."
  • In Section 3: "... the method breaks the barrier posed by MLP non-linearity and also identifies concrete contributors written into the residual stream." → "We write each MLP output as a sum of one vector per neuron, so the non-linearity only enters through a scalar activation strength, and each neuron's vector can be followed into later heads."
  • In Section 4.1: "This task is a vital mechanistic interpretability task since it is one of the most well-known “in-the-wild” circuit discoveries ..." → "IOI is one of the best-studied circuits found in a pretrained model, with known roles for its attention heads."
  • In Section 4.5: "... showing that end-to-end tracing is not only a methodological spotlight of our framework. It is also a real source of performance gain." → "... showing that combining the two scores improves the restoration." The closing sentences that only restate the results are shortened to plain statements such as "This answers RQ3.", and the closing "We hope this provides a useful step toward ..." of Conclusion is removed.

We also switched some wordings that might be vague or abstract: role vocabulary and task-specific vocabulary → task vocabulary, writers → kinds of vectors added, input-side signal → input information, superposed head functions → mixed head functions, dual gating → using both scores, repair mechanism → compensation, tailored adaptation → adapted version, and bitter tradeoff → tradeoff, while zero-pass and native are removed. All revised sentences are marked orange in our rebuttal version paper.

AuthorsOpenReview ↗

Request for Feedback on Rebuttal

Thank you for your review. We have responded to the weaknesses you raised in our rebuttal. As the discussion period is closing, could you please let us know whether any concerns remain? We would be happy to clarify them.

Reviewer_kjBpOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The authors address analyses of interactions between modules that are typically limited to toy transformers, while studies of standard LLMs usually focus on isolated modules. They propose a task-conditioned zero-pass framework for tracing embedding to native gated-MLP-neuron to attention-head interactions in LLMs, thereby requiring no task-example forward passes. They evaluate the causal importance of MLP-mediated paths to attention heads, demonstrating that this method can identify some meaningful internal structure.

Critical strengths
  • The paper overall addressed a relevant problem of identifying relevant internal structure in language models in an input-agnostic setting.

  • The paper includes some relevant remarks on which conditions need to be fulfilled in order to have a pair of neurons be activated (l.290-297). This nicely connects to other works on neurons interactions, i.e., “a pair of neurons (j,j′) is assigned relevance if the following three conditions are met: (i) it jointly activates, (ii) some pairs of neurons in the layer above jointly react, (iii) these reacting pairs are themselves relevant.” [Ebe22].

  • Demo code for reproducibility was provided.

Critical weaknesses
  • Limited evaluation on only one model (OpenLLaMA-7b) on two tasks (syntactic dependency probing and IOI).

  • Lack of relevant baselines. Authors state they “follow the dependency probing setting of [Liu26]”, yet do not include the associated baselines (which seem quite competitive on the UAS/LAS metrics). The included gradient baseline is known to not be robust, gradient robustification has been shown to achieve improved performance and would be a more informative, see [Jaf25, Aro26a,Aro26b] Methodological novelty seems to be primarily be proposing (i) embedding-facing score (Eq. 5, in which x is just substituted by the embedding matrix), and (ii) MLP–attention composition score that is a recording of terms of the standard model forward pass (see Eqs. 6-8). It remained unclear to me what the main proposed method/framework (“ours”) is comprised of.

  • The “Input-agnostic” interpretability claim seems to overstate what is done in the experiments. In Section 4.1, the authors describe “we instantiate the proposed framework in a task-specific way.” (l. 288), meaning they utilize the input task samples to select relevant neurons. I acknowledge that Appendix E provides studies when no additional prior task knowledge is available, but given this is a key motivation for the paper, this should be a setting in the main and evaluation benchmark.

Related work

  • [Ebe22] O. Eberle, et al., "Building and Interpreting Deep Similarity Models," in TPMAI, doi: 10.1109/TPAMI.2020.3020738.
  • [Liu26] Hongxu Liu, Jing Ma, Xiaojie Wang, Caixia Yuan, and Fangxiang Feng. An information-theoreticparameter-free bayesian framework for probing labeled dependency trees from attention score. ICLR 2026.
  • [Jaf25] FR Jafari et al. RelP: Faithful and Efficient Circuit Discovery in Language Models via Relevance Patching, Mechanistic Interpretability Workshop, NeurIPS 2025
  • [Aro26a] Arora, Aryaman, et al. "Language model circuits are sparse in the neuron basis." arXiv preprint arXiv:2601.22594 (2026).
  • [Aro26b] Arora, A., Wu, Z., Steinhardt, J., & Schwettmann, S. ADAG: Automatically describing attribution graphs. COLM 2026
Questions for authors
  • How would the proposed approach extend to free-from generative settings and more complex tasks?
  • How could the approach be extended to capture more complex interaction patterns, e.g. beyond pair-wise interactions?
5 comments
AuthorsOpenReview ↗

(1/4) Response to W1 and W2 (baselines)

Thank you for your valuable comments! Our replies are as follows:

For your concern about the limited evaluation in W1:

Limited evaluation on only one model (OpenLLaMA-7b) on two tasks (syntactic dependency probing and IOI).

Actually, we've reported the results of the factual recall task and OpenLLaMA-3B in Appendix D.3 and D.4. We apologize that these contents might not be easily noticeable. To further resolve your concerns, we've done more experiments adapting our method to other models and tasks. We adapted it to Llama-2-7b, Llama-2-13b, Yi-6B and Phi-3-mini-4k on the more representative dependency probing task, and compared it with random neurons and Probeless. We also further adapted it to other tasks, including plural, past tense, gerund and adverb. The results are as follows:

UUAS / ULAS ↑Clean runCorrupted runRandomProbelessOurs
OpenLLaMA-3B48.78 / 26.8843.68 / 9.7743.74 / 9.7643.79 / 10.1845.60 / 15.76
Llama-2-7b43.86 / 27.9438.36 / 12.9438.33 / 12.9138.35 / 13.1640.94 / 17.45
Llama-2-13b48.59 / 32.6241.85 / 18.8041.88 / 18.8743.03 / 20.1144.52 / 25.00
Yi-6B47.01 / 30.5439.51 / 15.1339.51 / 15.1140.55 / 15.5541.52 / 17.69
Phi-3-mini-4k45.66 / 29.0543.76 / 19.7543.77 / 19.7343.61 / 19.8744.91 / 22.49
ΔLogit vs. Corrupt. ↑Factual recallPluralPast tenseGerundAdverb
Probeless0.15000.17560.23390.22040.2663
Path patching0.30700.37740.43080.44590.3854
Ours0.43410.39630.52560.55910.8183

Our method also adapts well to these models and tasks, and beats the baselines on all of them. We have added both tables to Section 4.5 (Tables 3 and 4) and the descriptions of these models and tasks to Section 4.1 in our rebuttal version paper (added contents are marked orange; please check it).

For your concern about the baselines in W2:

Lack of relevant baselines. Authors state they “follow the dependency probing setting of [Liu26]”, yet do not include the associated baselines ...

We apologize for the confusion about the baselines of [Liu26]. Our method uses the probing method of [Liu26] only to obtain the target attention heads of each dependency label, and then traces the embedding->neuron->head interactions, so the output of our method is a set of neurons. The baselines in [Liu26] are competitors for locating the heads, while the baselines in our paper are competitors for locating the neurons once the heads are fixed (traced by the method of [Liu26]). The two groups of baselines are therefore solving different problems. We have added this clarification to Section 4.1 in our rebuttal version paper.

For your concern about the robustness of gradient-based baselines in W2:

The included gradient baseline is known to not be robust, gradient robustification has been shown to achieve improved performance ... see [Jaf25, Aro26a,Aro26b]

To address your concern, we've done two experiments. We first directly applied RelP, a more robust gradient-based method, to our setting and compared its dependency probing performance (UUAS/ULAS) with ours. We then ran further analyses verifying that these input-based methods tend to find general neurons that activate for all tasks, while the neurons found by our method are more task-specific and explainable.

Thank you for pointing us to [Jaf25, Aro26a, Aro26b]. For the first experiment, we adapted RelP to our embedding->neuron->head setting: the attention scores of the target heads serve as the metric from which the gradients are computed, and since RelP requires corrupted inputs, we corrupt the dependency tokens in the same way as in our restoration patching evaluation and our path patching baseline, so that all of them share the same corruption. The results on OpenLLaMA-7b are as follows:

MethodUUASULAS
Ours45.8423.04
RelP43.0220.02

We can see that RelP is really a stronger and more robust competitor than the gradient baseline in our paper, while our method still has a margin over it. We have added RelP to the baselines in Section 4.2 and its results to Table 1 in our rebuttal version paper.

AuthorsOpenReview ↗

(2/4) Response to W2 (continued)

For the second experiment, we revisited what our method emphasizes compared with input-based attribution and circuit methods. Input-based methods score real activations when the model is fed inputs, so the circuits they discover unavoidably contain many generally important components: like neurons that activate on all inputs and matter for the model's overall prediction, without being specific to the task. To verify this, we conducted the following two analyses. First, we measured the intersection (by IoU) between the neuron sets selected for different dependency labels, since generally important neurons would be selected for every label and give a large IoU. Second, we examined where the selected neurons actually activate: we ran the model over the training sentences and, for each selected neuron, collected the words at the positions where it activates most strongly as its activation vocabulary. If a neuron truly serves the label, this vocabulary should be filled with the label's dependent words (e.g., determiners for det); if it is a generally important neuron, the vocabulary will be crowded with unrelated words from all over the sentences. We used word types to count how many distinct words appear in this vocabulary. We also used embedding coherence (the average cosine similarity between the embeddings of these words): a crowded vocabulary gives many word types and a low coherence. As shown by the tables below, our method has an advantage over input-based methods, including RelP, on both analyses. We also ran these tests on the text8 dataset (which we ran our Appendix C.1's embedding-facing neuron activation correlation analysis on), and the same patterns hold.

MethodMean IoU across labels
Ours0.047
RelP0.096
Probeless0.107
Path patching0.071
MethodOverlap with label wordsWord typesEmbedding coherence
Ours0.38130.34
RelP0.02300.11
Probeless0.02360.06
Gradient attribution0.00240.14
Path patching0.08290.08
Random neurons0.02350.05

For your concern about the methodological novelty and what "ours" is comprised of in W2:

Methodological novelty seems to be primarily be proposing (i) embedding-facing score ... and (ii) MLP–attention composition score ... It remained unclear to me what the main proposed method/framework (“ours”) is comprised of.

Our method is a two-stage method built on the two scores you've mentioned. The first stage is the embedding-facing stage, which finds the neurons likely to activate at the task positions: for a given task vocabulary, we score each neuron by the overlap between its activation lexicon (the top tokens under Eq. 5) and the task vocabulary, and keep the top-scoring neurons as candidates. The second stage is the head-composition stage, which finds the candidates that can strongly influence the target heads: for a given target head set, we score each candidate by how often it appears among the top-scoring neuron pairs under Eq. 8. Finally, we rank the candidates by the product of the two scores and take the top set.

For the novelty, we think there are several points: (1) Previous weight-based methods mainly treat the non-linear MLP as a barrier, while with the neuron-level MLP decomposition and the product of the two scores, we circumvent it and trace a multi-hop circuit input-agnostically (previous methods mainly analyzed single modules or one-hop interactions). (2) We agree that the model computation can be expanded in many ways, each measuring a module-to-module interaction. However, previous expansions often give vector- or matrix-shaped scores (like the virtual heads in Transformer Circuits) that might be hard to attribute (as we discussed in the introduction), while our neuron-head-neuron term is a scalar that multiplies directly with the scalar embedding-facing score, so no norm-like reduction is needed along the whole pipeline. (3) The product of the two scores is also non-trivial: it follows the real computation (mm and the term that becomes B(h)B^{(h)} are exactly multiplied in Eq. 7), and restores better than either score alone (Table 2; we added these clarifications to the Introduction in our rebuttal version paper):

SettingUUASULAS
Embedding-facing only42.5220.20
Attention-composition only43.7621.47
Ours44.0222.19
AuthorsOpenReview ↗

Response to W3 and Q1

For your concern about the "input-agnostic" claim in W3:

The “Input-agnostic” interpretability claim seems to overstate what is done in the experiments. ... I acknowledge that Appendix E provides studies when no additional prior task knowledge is available, but ... this should be a setting in the main and evaluation benchmark.

Thank you for acknowledging the settings in our Appendix E. Actually, the "Attention-composition only" variant in Table 2 shares exactly the same spirit as the Appendix E setting (it ranks neurons by the composition score of Equation 8, which is exactly the coarse filtering score of Appendix E, and it requires no task vocabulary and no embedding-facing score). So your suggestion is actually already adopted in our paper, and this variant has comparable performance to our full embedding-facing + head-composition setting. Furthermore, when the task vocabulary is included, our method still does not require running the model forward and collecting activations (this is what we refer to as "input-agnostic" in the Abstract, l. 18-19, and Section 1, l. 47-49 of our paper). Our starting point is that, sometimes, a task vocabulary is easier to obtain (like enumerating the words having a specific dependency relationship, or person names) than collecting abundant sentences and running the model on them. We have added the relationship between the two variants to Section 3.1, and this explanation of "input-agnostic" to the Introduction in our rebuttal version paper.

For your question about extending to free-form generation and more complex tasks in Q1:

How would the proposed approach extend to free-from generative settings and more complex tasks?

Actually, we think free-form generation might be a better fit for our method: input-based tracing requires a clear metric value to attribute on (like the logits at the answer position), so it often faces problems when the answer is spread over multiple tokens: each token gives their own circuit, and fusing the circuits attributed at different tokens is difficult (as recent work suggests [Wu26, Sun25]). Our input-agnostic setup circumvents this problem, since neither of its two inputs is tied to a specific output position. As long as we can find a good set of heavily used heads and a good task vocabulary, Eqs. 5 and 8 score the neurons without any forward pass, and the procedure does not change with the generation length.

Also, finding such heads or vocabulary might not be hard. For the heads, existing literature has already found sparse sets of heads that free-form generation and multi-step reasoning heavily rely on, like the retrieval heads [Wu25], the retrieval heads kept under full attention in long-context decoding [Xia25], the retrieval-reasoning heads [Fu25], the inductive-reasoning heads in chain-of-thought [Dut24], and the receiver heads in mathematical reasoning traces [Bog25]. For the vocabulary, even when the task has no clear templated ground-truth answers, we can still find words that are frequent in the task, especially for tasks like symbolic or mathematical reasoning, whose numbers and operators can be listed without any task example (that's also the difference between "requiring a task vocabulary" and "requiring real task inputs"). And when even such a vocabulary is unclear, the vocabulary-free "Attention-composition only" variant (Table 2) still applies, with performance comparable to our full setting.

References:

[Wu26] Wu et al. Query Circuits: Explaining How Language Models Answer User Prompts. ICML 2026. [Sun25] A. Sun. Circuit Stability Characterizes Language Model Generalization. ACL 2025. [Wu25] Wu, et al. Retrieval Head Mechanistically Explains Long-Context Factuality. ICLR 2025. [Xia25] Xiao et al. DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads. ICLR 2025. [Fu25] Fu et al. Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning. ICLR 2025. [Dut24] Dutta et al. How to Think Step-by-Step: A Mechanistic Understanding of Chain-of-Thought Reasoning. TMLR 2024. [Bog25] Bogdan et al. Thought Anchors: Which LLM Reasoning Steps Matter? arXiv:2506.19143, 2025.

Reviewer_82AxOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors make an earnest attempt at scaling weight-based interpretability for language models. Their methodology relies on directly and straightforwardly calculating the contributing effects from embeddings, MLP and attention to a residual stream vector, and noting that we can do this for the entire vocabulary. They call the resulting (per token in the vocabulary) item the “embedding-facing score” and also analyze an MLP-attention score. They go on to show that compared to baselines including a random neuron set control and previous techniques like path patching and gradient attribution, their method performs better at established interpretability benchmarks where taking total TFLOPS cost into account, and performs better in balancing overall efficiency/performance. Interestingly, they find a proximity effect, in that neurons from nearby later layers compose more tightly with later heads. Overall, this is a valuable and well-supported contribution to weight-based interpretability methods.

Critical strengths
  • The authors validate the efficacy of their weight-only methodology on important in-the-wild circuits like IOI. They employ clear and reasonable baselines, including comparison against previous methods like gradient attribution, and a version of path patching improved for computational feasibility. Their validation on noisier factual recall tasks and the ablations in Appendix D are appreciated by this reviewer.
Critical weaknesses
  • The methodology relies on analyzing terms that become astronomically large in practice. For example, a language model with at least 20 layers, model dimension 1024, and MLP expansion factor 4 would have on the order of 6.71B interaction terms in equation (6) when analyzed at layer 20. For the neuron-neuron terms, an overwhelming majority, the authors propose analyzing them via equation (5) in the embedding basis, which in turns increases this number by the size of the vocabulary squared. The methodology does not appear to be tractable in practice without further justification: for example, if we laid out all these terms in a single vector and took a softmax, would the largest entries stand out, or would this be close to a uniform distribution for most models/layers? And if the former, are these larger entries meaningful as explanations, and can they be identified in advance without evaluating all terms? A proper justification for this objection would increase this reviewer’s score.
  • It would be useful to add a Limitations section that addresses the above, and any other notable limitations of the technique.
Questions for authors
  • See the question posed in the Weaknesses section.
  • How does LayerNorm affect the results in Appendix C? This seems fairly important to account for given many common LLMs use LayerNorm for normalization, though the reviewer accedes that most frontier open weight models use RMSNorm in practice.
  • Can you show a more explicit example for IOI, such as recovering neurons or interactions previously determined as important to the IOI task for the heads that you analyzed? Doing this on GPT-2 where such answers (and the role of each heads) are known would be a useful addition to Appendix A.
4 comments
AuthorsOpenReview ↗

(1/2) Response to Weaknesses

Thank you for your valuable comments! Our replies are as follows:

For your concern about the number of interaction terms in W1:

The methodology relies on analyzing terms that become astronomically large in practice. ... The methodology does not appear to be tractable in practice without further justification ...

Thank you for pointing this out, and yes, your estimate for the layer-20 head is theoretically exactly correct. And actually, we used several technical details to reduce the computation far below this estimate. First, after calculating the embedding-facing scores (this process can also be parallelized, i.e., applying the MLP input weight matrices on the whole embedding matrix, so the computation is like an ordinary model forward pass with a long context), we summarize each neuron into a single scalar (the intersection between its activation lexicon and the role vocabulary, as we described in Eq. 13 of Appendix A.2). This means there actually won't be a vocabulary-squared count multiplied with the number of neuron pairs in Eq. 6. We also retained only the neurons with high embedding-facing scores, so the number of neurons in Eq. 6 is also limited. And when calculating the neuron composition frequency, we also applied a hierarchical top-k: we first calculate the local top-k for each previous layer (candidates->head<-previous layer), then perform the top-k globally. Actually, we've already done the TFLOPs and time analysis, and reported them in Section 4.5 and Appendix A.2, which supports that our method is tractable in practice.

For your concern about whether these terms are nearly uniform, we further analyzed the distribution of the composition scores (Eq. 8). Since each score sums many products, unstructured weights would give a Gaussian distribution by the central limit theorem, while our results are far from it:

... would the largest entries stand out, or would this be close to a uniform distribution for most models/layers? And if the former, are these larger entries meaningful as explanations ...?

Per headGaussian referenceObserved
Largest score / std6–756–305
Scores beyond 10 × std≈ 010410^4–10610^6

The largest scores lie an order of magnitude beyond the Gaussian expectation, and every head has a long tail of such entries. And the comparison with random neurons in Table 1 answers whether these stand-out entries are significant: restoring randomly selected neurons gives no noticeable recovery, while restoring the neurons from the top-scoring pairs brings the performance much closer to the clean run. We've added both the explanations above and this analysis to the new Appendix G in our rebuttal version paper (please check it), with a pointer to it in Section 4.5. Also, as you suggested, we added related discussions in the limitation section (Appendix A.2), and thank you again for it.

AuthorsOpenReview ↗

(2/2) Response to Questions

For your question about how our model adapts to LayerNorm:

How does LayerNorm affect the results in Appendix C? ...

LayerNorm differs from RMSNorm only by a fixed mean-centering and a bias, i.e., Dγx/σ(x)→DγPx/σ(Px)+βD_\gamma x/\sigma(x) \to D_\gamma P x/\sigma(Px) + \beta with P=I−11⊤/dP = I - \mathbf{1}\mathbf{1}^\top/d. The centering PP can be folded into the WQKW_{QK} weights (PDγWQKDγPP D_\gamma W_{QK} D_\gamma P), the same as what we do for DγD_\gamma in Eq. 20, so the within-pair ranking is preserved exactly as in Eq. 20. The bias, on the other hand, leaves all neuron-pair scores unchanged and only adds terms involving a single neuron: if we write x^\hat x for the normalized part DγPx/σ(Px)D_\gamma P x/\sigma(Px), the score just becomes (x^s+β)⊤WQK(x^t+β)(\hat x_s + \beta)^\top W_{QK} (\hat x_t + \beta), so after expanding it, the only pairwise term x^s⊤WQKx^t\hat x_s^\top W_{QK} \hat x_t still stays in the same form as Eq. 20, while the other terms involve only one side or none. We added this discussion to Appendix C.4 (now titled "Effect of RMSNorm/LayerNorm") in our rebuttal version paper.

For your suggestion about an explicit IOI example on GPT-2:

Can you show a more explicit example for IOI ...? Doing this on GPT-2 where such answers (and the role of each heads) are known would be a useful addition ...

Thank you for this suggestion! We applied our method to GPT-2 small with the same hyper-parameters as in the paper (GPT-2's MLP has no gate, so we simply drop the gate term in Eq. 5, and its LayerNorm is folded as in our reply on LayerNorm), and traced the neurons composing with the three name mover heads (9.9, 9.6, 10.0) of Wang et al. (2023) on both the key and the query side. On the key side, the top-ranked neurons are name features with name-dominated lexicons, and the first MLP layer contributes the most neurons of all layers, which is consistent with the previous finding that GPT-2's first MLP acts as an extended embedding (Wang et al., 2023; Hanna et al., 2023; McDougall et al., 2024). Several of them also fire much less on the repeated name (S2), which may reflect the duplicate token heads marking S2 (Wang et al., 2023) and makes S2 less attractive to the name movers. On the query side, the neurons concentrate in layers 7–9, right after the S-inhibition heads that write into the name movers' queries (Wang et al., 2023).

To check whether these neurons actually matter to the name movers, we attributed their attention logit difference (end attending to IO minus end attending to S2) to every neuron, roughly by (mrIO−mrS2) vr⊤WQK xend(m_r^{\text{IO}} - m_r^{\text{S2}})\, v_r^\top W_{QK}\, x_{\text{end}} for a key-side neuron rr (similarly for the query side). Our key-side neurons carry more than an order of magnitude more of this attribution than random neurons. Some typical examples are:

SideNeuron (layer, index)Lexicon examplesActivation at IO / S / S2 / endAttribution rank (of 36,864)
key(5, 33)kumar anderson silva davies1.22 / 1.15 / -0.01 / -0.024
key(6, 1534)barry oliver pierre manuel graham1.79 / 1.79 / 0.67 / -0.0413
key(2, 1823)paul david michael keith larry kevin1.43 / 1.36 / 0.51 / -0.0322
query(8, 2010)vladimir adrian alex donald george0.38 / 0.30 / -0.14 / 0.776
query(9, 370)david ronald william paul joseph patrick0.41 / 0.40 / -0.11 / 1.9424

All of them rank near the top among all neurons of GPT-2 small, with the key-side ones firing less on the repeated name and the query-side ones firing mainly at the end position. We added this example as Appendix D.5 in our rebuttal version paper, with a short reminder in Section 4.5.

References:

Wang et al., 2023. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small.

Hanna et al., 2023. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model.

McDougall et al., 2024. Copy Suppression: Comprehensively Understanding a Motif in Language Model Attention Heads.

AuthorsOpenReview ↗

Request for Feedback on Rebuttal

Thank you again for your valuable suggestions. We have provided responses to the weaknesses and questions in your review. With the discussion period nearing its end, could you please let us know whether any concerns remain? We would be happy to clarify them.

Thank you for providing responses to my questions, and particularly for the detailed and helpful analysis on IOI in GPT-2. While some of my concerns have been addressed, I am still not convinced that the methodology is able to scale significantly beyond the scope of models considered in the paper, and will maintain my present positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.