- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper concerns interpretability of looped-transformers. More specifically, it studies the role and functionality of attention heads, mainly using activation-patching. In looped-transformers, the head can react differently to interventions across different iterations through the network. The authors then ask whether this implies that the attention head has different roles through these iterations. First, the authors observe that a patched head can have an immediate effect on the logits at the current call but have no effect on the final answer since later calls suppress it. Second, they then focus on small toy-models they train themselves and show that an unrelated random donor patching can still improve the clean-versus-corrupt score, mainly by suppressing the corrupted state, whereas matched donors promote the clean state. They then show that there are different ways to organize measurements of intervention effects across calls, with each making certain patterns easier or harder to observe. Finally, they ask whether knowing additional information such as whether the model has reached the final answer can help explain better the effect of the head.
Critical strengths
-
The general question of extending current mechanistic-interpretability methods to looped-transformers is interesting
-
The authors are very honest about their results and report a lot of negative results throughout the paper. They show that patching and intervention methods can lead to false interpretation when applied to looped-transformers, which can be useful for the community.
-
The toy-model setting is useful to understand the behavior of looped-transformers across iterations.
Critical weaknesses
-
The paper is hard to read and very dense because it introduces many terms before making the basic experimental design understandable.
-
The main experiment focuses on a single pre-trained model and studies the behavior on a single attention head. For the other experiments they train their own toy models on a small synthetic task. This limits the scope of the paper.
-
A significant part of the paper is dedicated to C2 and the fact that unrelated donor patching can still improve the clean-versus-corrupt score by suppressing the corrupted state. This is important to faithfully interpret activation-patching results but it is not specific to looped-transformers, which questions its presence in the main text.
-
The authors study specific role-switching interpretations of attention heads with a series of negative results showing that changing effects does not establish change in functions. This leaves the main question unresolved as to what can distinguish whether a head changes function or not in different states.
Questions for authors
Clarification:
- Why is the observation about unrelated donor effects specific to looped-transformers ? Could it not be instead because of the task ?
Suggestion:
-
Why not considering other mechanistic interpretability methods in the context of looped-transformers ? The paper focuses on activation-patching for attention heads. Why attention heads only ? What about steering vectors, J-lens ?
-
What evidence could establish a change in function ?