- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper challenges a prevailing view in mechanistic interpretability: a successful activation intervention is evidence that a component's causal role has been identified. They show that this correspondence often breaks down. It is possible, for an example, to construct two different ways to divide up a model's computation that gives exactly the same outputs, yet when tested on held-out examples, the two descriptions give opposite answers regarding which components matter more. They claim that to avoid this, you must test a sufficient amount of examples, and they provide a mathematical way to determine how much remains untested.
As a concrete example, they consider functions f_1 and f_2, where f_1 is a(x) + b(x) and f_2 is a(x) + W\phi(x) + b(x) - W\phi(x). Under this scenario, both outputs are a(x) + b(x) but the contributions attributed to the individual components differ. They claim that this construction scales, and that in general -- for finite number of examples in an experiment -- it is possible to create such constructions that agree on all the examples already investigated but disagree on held-out ones.
They demonstrate this first theoretically through the following sequence of demonstrations:
- Two different decompositions can produce exactly the same model behavior
- Because finite experiments do not cover all directions in the model activation space, W\phi can be chosen such that the modification is "invisible"
- This difference between tested and untested examples can be large enough to flip component importance rankings
- This problem disappears when new inputs lie entirely within the space covered by the experiments
They also introduce R, the exposure ratio, which estimates how much a ranking could change based on the original gap between component importances.
They next run 4 sets of experiments:
-
They use GPT-2 Small and pick the main pair of Name-movers (L9H9 / L9H6) from the attention-heads and that were identified and previously studied in the IOI task, and apply the re-distribution described above. They show that they reverse the ranking on all 24 held-out comparisons whilst preserving the results on the 512 probe-prompts.
-
They take a pretrained JumpReLU SAE trained on Gemma-2 2B activations, and modify it to obtain an alternate SAE that has the same number of features, exactly the same reconstructions, gives identical outcomes on tested interventions when the relevant features are active, but produces reversed feature-importance rankings on several held-out prompts for those same features.
-
They test how many examples are necessary to eliminate the non-identifiability, and find that in general the tested directions must nearly span the relevant activation space
-
They show that their exposure ratio R works and tracks when rankings can be reversed
In general, the main contribution of the paper is identifying and demonstrating this non-identifiability of particular decompositions, and finding the conditions necessary to mitigate it.
Critical strengths
- The researchers discover that activation decompositions can be misleading, and that 2 different decompositions that appear to be identical on tested experiments can be different on held-out examples. This is worth keeping in mind when doing interpretability work that relies on activation interventions to obtain component rankings, such as in circuit detection.
Critical weaknesses
- The setting they describe is narrow, and in both of their experimental settings is immediately detectable from the model itself. In the GPT-2 setting, their intervention does not manifest as a valid architecture, and in the JumpReLU setting, the modified SAE has different sparsity than the original one.
- The paper is extremely poorly written, and it was generally difficult to parse what the authors meant and what their experimental results actually were. I do not think it is publishable in its current form and would require a substantial rewrite.
Questions for authors
What is the setting in which you consider your theoretical contribution to be useful? Given that your interventions directly modify the model, it seems like the scope is rather narrow. Can you identify a "real" setting in which your work may prevent what would otherwise be an error in an interpretability result?