acceptodds
Under review as a conference paper at ICLR 2027

INTERVENTION IS NOT IDENTIFICATION: ACTIVATION EDITS CAN PRODUCE THE PREDICTED EFFECT FOR THE WRONG REASON

Abstract

An activation intervention can be exact, task-shaped, and sham-specific without identifying the mechanism that produced its effect. We show this in selective rule updating, where a current binding B and applicability variable S determine the required action A= B⊕S. Qwen2.5-14B-Instruct executes the operator perfectly when both operands are explicit (400/400). A free-reasoning scaffold raises accuracy to 89.6%, but it changes the computational trajectory; direct full-context accuracy is 46.2%. A bitwise-verified layer-28 replacement of the decoded B coordinate moves a supervised layer-35 readout of Aby +0.0288 and reverses across applicability conditions as the task predicts. The effect survives identity, zero-edit, and equal-norm orthogonal controls. It also persists when the measured A coordinate is held fixed at the edit site (paired held-minus-free difference +0.0011, 95% CI [−0.0021,+0.0043]). These checks establish a real, structured internal effect. They do not identify its mechanism. Under the pre-specified decomposition, contraction toward the downstream readout mean accounts for 74.3% of the interaction, while the baseline-independent remainder is unresolved. On cases where contraction and task-directed correction predict opposite signs, the displacement is near zero. Recipient dependence exceeds the task-directed bound by 4.58×and 4.18×. A structured task-irrelevant variable is more decodable than the target, simple raw-vector carry-through explains essentially none of the downstream interaction, and no preferred answer changes. Across 1,000 matched synthetic draws per condition, the directed estimator is approximately unbiased with 94.2% empirical coverage and separates injected directed effects from pure contraction under the specified generating process. The result separates a causal intervention effect from the mechanism used to explain it.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, activation patching, activation intervention, causal identification, probing classifiers, steering, interpretability illusion, language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_uR3DOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates whether activation intervention effects are sufficient to identify the underlying computational mechanism. It conducts controlled activation-editing experiments on a selective rule-updating task alongside diagnostic analyses and synthetic calibration to distinguish task-directed effects from readout contraction.

Critical strengths

The paper examines whether observed activation intervention effects are sufficient to identify the underlying computational mechanism. This question is important for avoiding overinterpretation of intervention results and improving the rigor of mechanistic claims.

Critical weaknesses
  1. Limitations in presentation and organization. The manuscript is poorly organized, with excessive numerical detail, unclear methodological exposition, and fragmented results. Key findings lack a consolidated quantitative summary, while the figures are difficult to read. These deficiencies obscure the connection between the evidence and conclusions, hindering assessment of the paper’s contribution.

  2. Incomplete empirical validation. The evaluation is restricted to one model, one constructed task, and one primary intervention setting, without a model-level positive control. Coupling between answer codes and presentation format prevents independent behavioral validation, while synthetic calibration does not establish the reliability of the complete intervention pipeline.

Questions for authors

Please find the weakness above.

Reviewer_XyNQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether a task-shaped activation intervention identifies the computation responsible for its effect. On a selective rule-updating task with Qwen2.5-14B-Instruct, a constrained binding-coordinate edit changes a downstream action readout in the predicted direction and survives several controls. Additional diagnostics find substantial baseline-dependent contraction and no statistically resolved positive directed remainder. Synthetic experiments validate the estimators under specified generating processes.

Critical strengths
  1. The paper carefully distinguishes successful intervention from mechanistic identification. The matched action-coordinate hold and disagreement-subset analysis provide useful controls beyond an equal-norm orthogonal sham.

  2. The synthetic calibration quantifies estimator bias, coverage, and decision rates under explicit assumptions. The paper also candidly reports its low direct-task accuracy, lack of answer flips, and limited experimental scope.

Critical weaknesses
  1. The full diagnostic pipeline lacks a model-level positive control. Synthetic calibration establishes recovery under the chosen generating processes, but does not establish that the complete intervention-and-readout pipeline recognizes genuine task-directed computation in a transformer. This matters particularly because direct-task accuracy is 46.2%. That accuracy does not establish the absence of relevant internal computation, but it limits what this case demonstrates about identifying mechanisms in successful task execution.

  2. Some conclusions exceed the scope of the diagnostics. The recipient-geometry bound assumes the fixed-gain replacement model in Eq. 5; violating it does not exclude recipient-dependent task-directed mechanisms. Similarly, better decoding of a structured irrelevant variable does not by itself establish that the action readout lacks specificity. The manuscript acknowledges several qualifications, but broader statements about transport, selectivity, and the effect being “absent” should consistently reflect these limitations.

Questions for authors
  1. Can you provide a model-level positive control in which task-directed computation is independently supported, and report the full battery’s outcome?

  2. How do the geometry conclusions change when the gain in Eq. 5 varies across recipients?

  3. What evidence distinguishes lack of specificity of the action readout from the representation simply encoding both task-relevant and irrelevant variables?

Reviewer_m5PjOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

It was very challenging to understand what this paper was really trying to do since the two task variables were never defined, explained or illustrated with an example. First the context: when trying to locate or understand a behavior inside a model, interventions on activations are usually used to assert that a given internal representation is used by the model to produce that behavior. If editing the representation leads to the expected effect, the mechanism is usually considered as confirmed. Now this paper claims to construct a case that illustrate that an intervention that produces the expected effect does not necessarily mean that the intended mechanism was behind it. Instead simpler explanations can mimic the same effect. This was shown on a single task and a single model.

Critical strengths

None

Critical weaknesses

The basics of writing are missing: The abstract and introduction (which are supposed to be the navigation compass of the reader in the paper) are written as if the reader has already read the paper, using jargon (that is invented by this paper*) before it's defined. The abstract leads immediately with unintelligible bulk of results. An example of the wrong ordering of ideas: right after saying "reaches 400/400 when B and S are explicit" (the reader still does not understand what those are), it follows up with "A free reasoning raises accuracy to 89.6%", while the raise is wrt to the 46.2% that comes after.

The following terms come undefined: "selective rule updating" "current binding," "applicability variable," and "required action" "the operator" (⊕ is never named) "explicit," "full-context examples," and "rows" "elicitation regime" and "scaffold" "state and applicability accuracies" the identity control and zero edit controls are also not defined.

  • free-reasoning seems to be a term for chain of thought like "thinking" on. Similarly binding usually means binding entities to attributes. But here the authors use it for something else, that the reader cannot assess after reading the abstract AND the introduction, which is a very bad sign for the writing quality.

Fig2 is unreadable with too small fonts and unsuitable size despite having space. Fig1 is only slightly better...

I kept following trying to understand the dataset and whether the authors will give me an example of A B and S, i had high hopes for Sec3.1, but here is what it says:

The frozen dataset contains 1024 examples over 128 worlds and eight structural families. Each example describes a small structured world from which the model must infer B and S before applying A=B⊕S. All mechanistic results use Qwen2.5-14B-Instruct(Yangetal.,2024).

Then it repeats what was said in the intro almost verbatim: "The model scores 400/400 when both operands are explicit but 473/1024(0.4619) on the direct full-context task."

There is no reference to an appendix that describes all this. Actually, there are no appendices in this 7-page long submission.

Questions for authors

No answer can materially affect my assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.