acceptodds
Under review as a conference paper at ICLR 2027

When Activation Interventions Fail to Identify Component Rankings

Abstract

Mechanistic interpretability often treats a successful activation intervention as evidence that a component's causal role has been identified. We show this inference can fail for the reported ranking itself: for a fixed finite intervention family, a second decomposition can match every observed outcome exactly yet reverse the ordering of two components on a held-out contrast, while leaving every residual-stream activation and output unchanged. The ambiguity is governed by coverage, the portion of the reported contrast outside the span of the probed directions, measurable on the original model before any alternative is built. On GPT-2 Small, a construction reverses the IOI name-mover ordering that single-position interchange measures on all 24 held-out pairs while agreeing with a 512-prompt probe family, and generalizes to unseen templates. On Gemma-2 2B, an edited same-architecture JumpReLU SAE matches reconstruction and every probe outcome where the intervened feature is inactive, while reversing the feature-pair ordering on most active prompts. Coverage separates evidence styles: single-position families can leave the ordering steerable until probe count nears the model width (768 on IOI here), while position-resolved families and the per-context trace of a discovery sweep drive the uncovered portion to or near zero at this width. Successful interventions alone do not establish identification; whether the evidence covers the claim does.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, activation interventions, activation patching, causal attribution, identifiability, component rankings, circuit analysis, sparse autoencoders, representation interpretability, causal interventions

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QaxWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper challenges a prevailing view in mechanistic interpretability: a successful activation intervention is evidence that a component's causal role has been identified. They show that this correspondence often breaks down. It is possible, for an example, to construct two different ways to divide up a model's computation that gives exactly the same outputs, yet when tested on held-out examples, the two descriptions give opposite answers regarding which components matter more. They claim that to avoid this, you must test a sufficient amount of examples, and they provide a mathematical way to determine how much remains untested.

As a concrete example, they consider functions f_1 and f_2, where f_1 is a(x) + b(x) and f_2 is a(x) + W\phi(x) + b(x) - W\phi(x). Under this scenario, both outputs are a(x) + b(x) but the contributions attributed to the individual components differ. They claim that this construction scales, and that in general -- for finite number of examples in an experiment -- it is possible to create such constructions that agree on all the examples already investigated but disagree on held-out ones.

They demonstrate this first theoretically through the following sequence of demonstrations:

  1. Two different decompositions can produce exactly the same model behavior
  2. Because finite experiments do not cover all directions in the model activation space, W\phi can be chosen such that the modification is "invisible"
  3. This difference between tested and untested examples can be large enough to flip component importance rankings
  4. This problem disappears when new inputs lie entirely within the space covered by the experiments

They also introduce R, the exposure ratio, which estimates how much a ranking could change based on the original gap between component importances.

They next run 4 sets of experiments:

  • They use GPT-2 Small and pick the main pair of Name-movers (L9H9 / L9H6) from the attention-heads and that were identified and previously studied in the IOI task, and apply the re-distribution described above. They show that they reverse the ranking on all 24 held-out comparisons whilst preserving the results on the 512 probe-prompts.

  • They take a pretrained JumpReLU SAE trained on Gemma-2 2B activations, and modify it to obtain an alternate SAE that has the same number of features, exactly the same reconstructions, gives identical outcomes on tested interventions when the relevant features are active, but produces reversed feature-importance rankings on several held-out prompts for those same features.

  • They test how many examples are necessary to eliminate the non-identifiability, and find that in general the tested directions must nearly span the relevant activation space

  • They show that their exposure ratio R works and tracks when rankings can be reversed

In general, the main contribution of the paper is identifying and demonstrating this non-identifiability of particular decompositions, and finding the conditions necessary to mitigate it.

Critical strengths
  • The researchers discover that activation decompositions can be misleading, and that 2 different decompositions that appear to be identical on tested experiments can be different on held-out examples. This is worth keeping in mind when doing interpretability work that relies on activation interventions to obtain component rankings, such as in circuit detection.
Critical weaknesses
  • The setting they describe is narrow, and in both of their experimental settings is immediately detectable from the model itself. In the GPT-2 setting, their intervention does not manifest as a valid architecture, and in the JumpReLU setting, the modified SAE has different sparsity than the original one.
  • The paper is extremely poorly written, and it was generally difficult to parse what the authors meant and what their experimental results actually were. I do not think it is publishable in its current form and would require a substantial rewrite.
Questions for authors

What is the setting in which you consider your theoretical contribution to be useful? Given that your interventions directly modify the model, it seems like the scope is rather narrow. Can you identify a "real" setting in which your work may prevent what would otherwise be an error in an interpretability result?

Reviewer_8AAcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

work shows a non-identifiability issue in mechanistic interpretability. they prove that a finite set of activation interventions cant uniquely identify a component ranking. they build an alternative decomposition that matches all observations on a probe set but reverses the component ordering on held-out contrasts. this ambiguity is governed by the coverage of the evaluation contrast relative to the probe span. they show the failure empirically on gpt-2 and gemma-2.

Critical strengths

theorem 1 is an important result for the community. framing causal intervention failures as a fundamental non-identifiability problem instead of just estimation noise provides good theoretical grounding. the concept of coverage and the exposure ratio R offer a practical metric to detect when an intervention family fails to identify the underlying causal structure.

Critical weaknesses

The exact cancellation for the constructive witness relies heavily on the parallel-write structure of the components. direct application to sequential pre-norm blocks is mathematically murkier because of intervening nonlinearities. empirical demonstrations are on relatively small scales like gpt-2 and gemma-2 2b.

Questions for authors

For sequential architectures where the exact cancellation through the MLP nonlinearity fails can we still bound the worst-case reversal margin using a modified version of your coverage metric? how does the intrinsic dimension of the feature manifold impact the practical cost of achieving full coverage in frontier models?

Reviewer_tPfrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

unclear to me what the main contributions are as the paper is confusing from the second sentence of the abstract:

We show this inference can fail for the reported ranking itself: for a fixed finite intervention family, a second decomposition can match every observed outcome exactly yet reverse the ordering of two components on a held-out contrast, while leaving every residual- stream activation and output unchanged.

what does reported ranking refer to here? the rest of paper is full of such confusing phrasing that are clearly agent slop.

Critical strengths

None.

Critical weaknesses

Clear AI slop

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.