- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes a method for identifying MLP neurons that contribute to a target attention head's behavior by analyzing a language model's weights (without running forward passes on task examples). The key idea is to expand the computation of later modules in terms of earlier ones and ultimately the embedding matrix, in the same spirit as the original mathematical framework for transformer circuits, but additionally considering MLP neurons in addition attention.
Validation is limited to showing that, on a single model (OpenLLaMA-7B) and two tasks (probing for syntactic dependencies and IOI), restoring a set of 200 neurons selected by the method preserves model behavior better than some baselines.
Critical strengths
- Interpreting a model primarily by inspecting weights rather than using task data is an interesting angle
- I appreciate the ablations and the comparison of FLOP costs per baseline method
Critical weaknesses
-
Limited experimental validation. Results are restricted primarily to a single model (OpenLLaMA-7B) and two tasks (dependency probing and IOI). Moreover, for attribution-style work it is standard to report a frontier of restoration (or ablation) effect as the number of selected components varies, rather than a single cutoff the paper only reports results at a fixed threshold of 200 neurons. The method additionally doesn't not improve much over the path-patching baseline.
-
Clarity of presentation. The writing makes the paper very difficult to evaluate. Concretely, details of the task setup, the baselines, and the details of method itself (how cutoffs for embedding and head-composition scores were chosen, choice of vocabulary per task).
-
Much of the prose reads as though it was primarily language-model-generated, and is hard to follow due to heavy use of jargon and vacuous statements. I acknowledge that some of this is a matter of taste, but I think it is worth raising as a separate critique from the weakness of the results themselves.
Questions for authors
None
4 comments
(2/3) Response to W2
For your concern about the clarity of the settings in W2:
Clarity of presentation. The writing makes the paper very difficult to evaluate. Concretely, details of the task setup, the baselines, and the details of method itself (how cutoffs for embedding and head-composition scores were chosen, choice of vocabulary per task).
We're sorry that the task setup, the baselines and the details of our method were only briefly described in the main text and scattered across Appendix A, which made the paper difficult to evaluate. We hope the following clarification of the settings might help reduce this difficulty.
For the task setup, the target heads and the corrupted positions are as follows (all on OpenLLaMA-7b):
| Task | Target heads | Corrupted positions | Metric |
|---|---|---|---|
| Dependency (det, prep, cc, aux) | Top-10 heads per relation by MI with the relation label | The function word of each arc | UUAS / ULAS on all dev sentences |
| IOI | Top-10 heads attending from the end position to IO most often | IO name | Logit difference at the end position |
| Factual recall, plural, past tense, gerund, adverb | Top-10 heads attending from the end position to the subject most often | Subject span | Logit difference at the end position |
In the corrupted run, we add Gaussian noise to the input embeddings at these positions, and the patched run restores the selected neurons at the same positions.
For the baselines, their ranking scores are described in Section 4.2, and all of them share the following settings with our method:
| Setting | Shared by all methods |
|---|---|
| Target heads | The heads in the table above |
| Neuron budget | 200 neurons per relation (or task) |
| Corruption and restored positions | As described above |
| Evaluation | The same examples and seeds |
So none of these settings is chosen to favour our method, and each of our result tables comes from one configuration shared by all methods. The only difference is that the input-dependent baselines use task examples to select neurons while our method uses none, which, if anything, favours the baselines.
For the method itself, the cut-off hyperparameters are and . is the number of top tokens under the embedding-facing score (Eq. 5) kept as a neuron's activation lexicon, which is then matched with the task vocabulary. is the number of top-scoring neuron pairs kept from the head-composition matrix (Eq. 8), from which we count how often each neuron composes strongly with the target head. We found that, like the curve after the top-200 neurons in our reply to W1, they only have marginal effects on the performance (Appendix D.1 for ). So we chose the setting in our paper. The task vocabularies are collected automatically from the data, so no manual choice of words is involved:
| Task | Task vocabulary | Examples |
|---|---|---|
| det | Dependent-side words of the relation in the training split (22 words) | the, a, an, this, some, no, those |
| prep | Same as above (115 words) | of, in, for, from, with, into, about |
| cc | Same as above (18 words) | and, but, or, nor, yet, plus |
| aux | Same as above (42 words) | will, has, would, is, could, can, may |
| IOI | ~1,000 common English first names (a separate list from the names used in the prompts) | Jacob, Emily, Michael, Hannah, Sarah |
| Factual recall | Subject words of the task | Leslie Moonves |
| Plural, past tense, gerund, adverb | Subject words of the task (the base forms) | cast (plural), cap (past tense), portion (gerund), merry (adverb) |
To avoid further misunderstanding, we have added a short summary of the target heads, the task vocabularies and the settings shared by all methods to Section 4.1 of our rebuttal version paper, together with the corruption method in Section 4.3 and the tables above in Appendix A.1. We have also corrected the description of in Appendices A.2 and D.1.
(3/3) Response to W3
For your concern about the writing in W3:
Much of the prose reads as though it was primarily language-model-generated, and is hard to follow due to heavy use of jargon and vacuous statements. ...
We apologize that some sentences were vague, or used terms that a reader has to "decode" first. We went through the whole paper and revised them. The main revisions are as follows:
- In Abstract:
"We address this gap by introducing a task-conditioned zero-pass framework for tracing embedding→native gated-MLP-neuron→attention-head interactions in standard LLMs."→ "We address this gap with a method that traces how input embeddings influence later attention heads through individual MLP neurons in standard LLMs, using only the model weights and a task vocabulary." Also, "a highly favorable balance betweencausal effectivenessrestored task performance and computational efficiency". - In Introduction:
"It seeks to uncover universal functional mechanisms within the model by analyzing parameter distributions and matrix properties, ..."→ "It explains model computation from the weights alone, so no inputs or intermediate activations are needed." - In Introduction:
"... this module-centric perspective largely abstracts away the cross-module interactions through which many model behaviors may emerge."→ "These works assign functions to single components, while many behaviors come from several modules together, e.g. an MLP neuron writing a feature that a later head reads." - In Introduction:
"As a result, they provide valuable conceptual tools but do not readily scale to modern transformers ..."→ "Since these compositions leave out MLPs, they do not directly apply to modern transformers." - In Introduction:
"... it makes MLPs part of the analyzable mechanism, and it ties weights analysis to the actual residual-stream signals that those weights write and later heads read."→ "... each MLP neuron becomes an explicit term of the target head's attention score, and this term uses the vector that the neuron actually writes into the residual stream." - In Introduction:
"To evaluate discovered neurons causally, We further designed a perturb-then-restore-based benchmark."→ "To compare the neurons found by different methods, we corrupt the input at task-relevant positions, restore only the selected neurons, and measure how much of the original behavior returns." - In Contributions:
"First, it opens a new regime of input-agnostic mechanistic interpretability: from toy models to ..."→ "First, it extends input-agnostic analysis of module interactions from attention-only toy models to standard LLMs with gated MLPs." Also,"Third, it achieves an exceptionally favorable balance ... requiring only negligible computation by comparison."→ "Third, the neurons it finds restore task behavior comparably to path patching, with over two orders of magnitude less computation (Figure 1)." - In Section 3:
"... the method breaks the barrier posed by MLP non-linearity and also identifies concrete contributors written into the residual stream."→ "We write each MLP output as a sum of one vector per neuron, so the non-linearity only enters through a scalar activation strength, and each neuron's vector can be followed into later heads." - In Section 4.1:
"This task is a vital mechanistic interpretability task since it is one of the most well-known “in-the-wild” circuit discoveries ..."→ "IOI is one of the best-studied circuits found in a pretrained model, with known roles for its attention heads." - In Section 4.5:
"... showing that end-to-end tracing is not only a methodological spotlight of our framework. It is also a real source of performance gain."→ "... showing that combining the two scores improves the restoration." The closing sentences that only restate the results are shortened to plain statements such as "This answers RQ3.", and the closing"We hope this provides a useful step toward ..."of Conclusion is removed.
We also switched some wordings that might be vague or abstract: role vocabulary and task-specific vocabulary → task vocabulary, writers → kinds of vectors added, input-side signal → input information, superposed head functions → mixed head functions, dual gating → using both scores, repair mechanism → compensation, tailored adaptation → adapted version, and bitter tradeoff → tradeoff, while zero-pass and native are removed. All revised sentences are marked orange in our rebuttal version paper.
Request for Feedback on Rebuttal
Thank you for your review. We have responded to the weaknesses you raised in our rebuttal. As the discussion period is closing, could you please let us know whether any concerns remain? We would be happy to clarify them.
(1/3) Response to W1
Thank you for your reviews. Our reply is as follows:
For your concerns about the limited experimental validation in W1:
Actually, we've reported the results of the factual recall task and OpenLLaMA-3B in Appendix D.3 and D.4. We apologize that these contents might not be easily noticeable. To further resolve your concern, we've adapted our method to more models and tasks. On the more representative dependency probing task, we ran it on Llama-2-7b, Llama-2-13b, Yi-6B and Phi-3-mini-4k, and compared it with random neurons and Probeless. We also adapted it to four more tasks: plural, past tense, gerund and adverb. The results are as follows:
Our method also adapts well to these models and tasks, and beats the baselines on all of them. We have added both tables to Section 4.5 (Tables 3 and 4) and the descriptions of these models and tasks to Section 4.1 (with task examples in Appendix A.1) in our rebuttal version paper (added contents are marked orange; please check it).
For your concern about the frontier over the number of selected neurons:
We ran our method on dependency probing with OpenLLaMA-7b while varying the number of selected neurons per relation from 50 to 3200, and compared it with the same number of random neurons (averaged over three seeds, on the development set):
(ULAS recovered is our gain over random neurons, as a share of the gap between the clean run and random neurons)
Our method stays far above random neurons at every size, and 200 lies at the bending point of this curve. The restoration rises quickly up to 100–200 neurons per relation, while after 200, each doubling of the neuron set only brings a marginal ULAS gain. This size is also at the same scale as previous neuron-level works: Probeless (Antverg & Belinkov, 2022) evaluates its rankings with up to 150 top neurons, and Tang et al. (2024) take 1% of all neurons as language-specific neurons, while our 800 neurons over the four relations are only about 0.23% of the MLP neurons in OpenLLaMA-7b. What's more, such a small set keeps the explanation budget low (the number of neurons a human needs to read through), so we can still inspect the selected neurons one by one, as in the output-vocabulary projections of the dependency neurons in Appendix E.2 and the lexicons of the traced GPT-2 neurons in Appendix D.5. We have added these results to the new Appendix H.
For your concern about the gap to path patching:
Path patching needs to run on many data examples, with many patched forward passes per example, so we chose it mainly as an expensive baseline to verify whether our method can achieve comparable or even higher performance than actual patching. Its exhaustive neuron-level version is computationally prohibitive, so we already had to use a reduced adaptation of it (Section 4.2 and Appendix F). Even so, on all six logit-based tasks in the table above, our method outperforms this adaptation, with a large margin on adverb. Our method also reaches such a frontier of computation and task performance. As shown in Figure 1, it stays close to path patching on dependency probing with over two orders of magnitude less computation than even this reduced adaptation (14.53 vs. 6276.48 TFLOPs), and it is the cheapest of all the methods there.
References: