acceptodds
Withdrawn from ICLR 2027

Every Component Is a Lookup: One Linear Graph for Interaction, Composition and Attribution

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Interpretability methods for transformers are typically built around separate questions: which components interact, how information routes to the output, and which input tokens contribute. Because these methods rely on different assumptions, their answers are difficult to relate. We argue that two architecturally motivated assumptions suffice to address all three questions: attention and MLPs share a key-value form, , in which selects over values , and components read from an additive residual stream, the sum of component outputs. Holding these selections at their forward-pass values turns the model into a computational graph, of which component interactions, composition paths, and token attribution are different readouts. We develop UNPACK, a backward attribution procedure over this graph, and validate each readout against the external check its purpose-built counterpart is trusted for: interaction scores predict ablation effects across models from 160M to 6.9B parameters, recovered routes reproduce established circuits down to the key, query, or value branch the circuit specifies, and token attribution passes the same faithfulness test as dedicated attribution methods. The results suggest that these two assumptions suffice for the interpretability questions above. On a task with a known circuit, we find that contribution and causal effect can differ, and that the difference has a recognisable signature: components that matter for the task change their contribution when the task is removed from the input, while components that act like a bias term do not.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, attribution, path attribution, residual stream, key-value memories, faithfulness, circuits

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qZmmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces UNPACK, a view of Transformers that freezes non-linearities at their forward-pass values, allowing for easy attribution to all residual-stream components. Logits can be attributed to attention heads (via Q, K, V) and MLP neurons. The paper claims that this unifies component interactions, paths, and token attribution. The method is validated against the well-known IOI and greater-than circuits.

Critical strengths

The method requires just a single forward pass, and no gradients.

The method is intuitive: It’s clear to see where attributions come from.

Splitting attention heads into the Q, K, and V branches is useful.

Critical weaknesses

Major: Novelty. The authors do not elaborate how their proposed framework improves upon the well-known views of Elhage et al. 2021 and Geva et al. 2021. Elhage et al. already introduced the linearization of attention, various papers used frozen LayerNorm, and Geva et al. 2021 introduced viewing MLPs as key value pairs. This framing has been used in many circuits papers (e.g. Ameisen et al. 2025). The paper should more clearly state which of its contributions are novel.

The headline results overstate the agreement: Table 2 excludes layer 0 (which is the layer where the method does worst on). Table 9 shows the best-placed member which is lenient (compared to table 10), e.g. claim 2 shows the best-ranked component as rank 10 where chance would predict rank 11.

Minor: Table 13 uses only 15 Pile prompts which makes the results potentially noisy.

Minor: The method does not reflect causal effect (e.g. the layer 0 components documented in the paper: “layer-0 components receive high credit on every prompt”).

Questions for authors

What is the novel contribution of UNPACK compared to existing practices in the literature (see weakness 1)?

Reviewer_fXWeOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes that 2 simple, architecturally motivated assumptions, are sufficient for mechanistic interpretability and design a method around these.

In particular, the assumptions are:

  1. Both MLPs and Attention share a key-value form
  2. Every downstream component reads from an additive residual stream

When taken together, one can run a forward pass, fix the values at each component in the transformer, and decompose to both answer:

  1. Which components feed other components? (Interaction scores)
  2. How do the components compose to produce the answer? (Circuit detection)
  3. How are the input tokens responsible for the answer? (Attribution)

Their main contribution is thus a unified view of several prudent questions in mechanistic interpretability, where a single forward pass can be used to answer all 3. Their main methodological contribution is their recursive credit assignment: given credit I assigned to a component C, they distribute this among the components feeding component C according to these components alignment with a readout direction d. They also use a specific normalization procedure, SafeDenom, to stabilize this credit assignment when 2 contributions cancel.

The authors validate their findings on GPT-2 and increasing sizes of models from the Pythia-deduped family. Their results are:

  1. For interaction scores, across the Pythia family, their interaction scores are useful predictors of the impact that removing a component's contribution to the forward pass. They get Spearman correlations between 0.72 and 0.96.
  2. For circuit detection, on GPT-2, they show that UNPACK can recover the heads involved in a known circuit, and that the involved heads receive appropriately high attribution in the expected branches.
  3. For attribution, they compare UNPACK with 2 Layerwise Relevance Propagation based methods and Information Flow Routes (+ a random control), and show that the token attributions provided by UNPACK are more faithful.

Taken together, the experiment results demonstrate the validity of the authors' approach, and suggest that an architecturally motivated approach may yield a simplifying picture of mechanistic interpretability: they demonstrate that interaction, composition, and attribution can indeed be thought of as a common framework.

Critical strengths

The paper's main strength is in providing a unifying framework for 3 interesting questions of mechanistic interpretability: how do components interact, what circuit do they form, and which tokens matter?

Their approach is theoretically justified and relies on 2 well-established architectural assumptions (although they do make a few extra assumptions in their method). They develop a recursive credit assignment method and move backwards through the output logits to establish the full computational graph that produced them. From that graph, 3 different readouts provide answer to each of the 3 questions they pose. They validate their approach on models that span millions to billions of parameters and across 3 different experimental settings that test each a specific question.

In general, I am in favor of methods and frameworks that unify and simplify methods across a field, and I consider this the main strength of the paper.

Critical weaknesses

Unfortunately, the paper's core contributions are already well-known and established within the field of mechanistic interpretability. A significant amount of papers already leverage the additive decomposition of the Transformer architecture. A Mathematical Framework for Transformer Circuits and Interpretability in the Wild: A Circuit for Indirect Object Identification in GPT-2 Small, both of which the paper cites, contain the core ingredients already: projecting the residual-stream components onto a target logit-difference direction and decomposing the network contributions. There is some algorithmic novelty in the recursive credit assignment and specific choice of normalization and method, but even that feels very close to Information Flow Routes, which the paper also cites, which recursively traces important routes and similarly does not need patching or counterfactuals. Tracing Attention Computation Through Feature Interactions, which the paper does not engage with, explicitly decomposes query-key interactions between residual-stream components and combines those to form attribution graphs. Given these similarities, I would expect to see a significant investigation into why UNPACK's particular credit assignment matters.

A major weakness of this method is in its handling of feature interactions and weakness to especially self-repair. Several heads in the IOI circuit are for instance only involved once another head is ablated. That is, their write onto the I,S difference direction only appears once another head's contribution is missing from the residual stream. While having a method that does not require ablation or counterfactuals is tempting, such methods can by definition not discover contributions like these. So while UNPACK can describe the computation that occurred, it cannot describe the model's full computation. The paper does acknowledge such a limitation, but it is worth noting that it still severely diminishes the claim that their 2 assumptions are "sufficient" for mechanistic interpretability. Similarly, joint interactions through the QK-attention score pose a credit-assignment problem which the authors resolve by assigning equal credit (1/3) to the Q, K, and V branches -- an additional assumption where the weights are chosen by the authors and do not follow directly from residual stream additivity.

Questions for authors
  • Why do you restrict your analysis to Pythia and GPT-2 only? Given that the paper is meant to provide a unifying framework for several separate questions in mechanistic interpretability, I would expect to see some analysis of more modern models. On the contrary, if this is meant to be a theoretical contribution, I would expect slightly more justification for choices of normalization and constants (like the 1/3 attributions for Q, K, and V)
  • Where do you think UNPACK provides value that is not already provided by existing interpretability and explainability method. In particular, why do you think it is advantageous that you do not have to use either counterfactuals or gradients.
Reviewer_dDe4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors suggest a method for finding attribution graphs in LLMs. Their main method relies on the fact that attention heads and MLPs read from and write to the residual stream linearly. They define the attribution score of each component as the dot product of its output and an unembedding vector of the output token. The edge between each pair of components is again defined using a dot product.

In the experiments section they show that on the Pile dataset, there is a correlation between the importance of an edge according to their method and the change in the model’s perplexity after ablating that edge from the graph. They also compare their method to the IOI circuit (https://arxiv.org/abs/2211.00593) and show that their method can recover similar connections.

Critical strengths

I think that theoretically their method is supposed to be faster than other popular attribution methods (path patching, edge attribution patching, mask learning, information flow, etc.) because it only requires one forward pass and dot products operations . However, They barely test their method or compare it to other methods, so I find it hard to identify key strengths.

Critical weaknesses

-The comparison with other methods should be stronger. There is a whole line of work on circuits discovery and attribution methods. I would expect the authors to run experiments that are common in this line of research to proof the strength of their method - like circuit faithfulness or completeness and compare their method to other methods [1,2].

  • I think the novelty of this paper is not significant enough. The introduction and the motivation are not fully accurate - other attribution techniques (like path patching, edge attribution patching (EAP), EAP-IG, AtP*, and mask learning) aim to answer the same question as this method (what are the important component sand what are the important connections between them ), and since their method is not really compared to other techniques, Its hard to judge it.

references: [1] https://arxiv.org/abs/2403.17806 [2] https://arxiv.org/abs/2504.13151

Questions for authors

See me comment on the comparison for other methods in the Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.