- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces UNPACK, a view of Transformers that freezes non-linearities at their forward-pass values, allowing for easy attribution to all residual-stream components. Logits can be attributed to attention heads (via Q, K, V) and MLP neurons. The paper claims that this unifies component interactions, paths, and token attribution. The method is validated against the well-known IOI and greater-than circuits.
Critical strengths
The method requires just a single forward pass, and no gradients.
The method is intuitive: It’s clear to see where attributions come from.
Splitting attention heads into the Q, K, and V branches is useful.
Critical weaknesses
Major: Novelty. The authors do not elaborate how their proposed framework improves upon the well-known views of Elhage et al. 2021 and Geva et al. 2021. Elhage et al. already introduced the linearization of attention, various papers used frozen LayerNorm, and Geva et al. 2021 introduced viewing MLPs as key value pairs. This framing has been used in many circuits papers (e.g. Ameisen et al. 2025). The paper should more clearly state which of its contributions are novel.
The headline results overstate the agreement: Table 2 excludes layer 0 (which is the layer where the method does worst on). Table 9 shows the best-placed member which is lenient (compared to table 10), e.g. claim 2 shows the best-ranked component as rank 10 where chance would predict rank 11.
Minor: Table 13 uses only 15 Pile prompts which makes the results potentially noisy.
Minor: The method does not reflect causal effect (e.g. the layer 0 components documented in the paper: “layer-0 components receive high credit on every prompt”).
Questions for authors
What is the novel contribution of UNPACK compared to existing practices in the literature (see weakness 1)?