acceptodds
Withdrawn from ICLR 2027

Beyond the Fitted Answer: Causal Dissection and Downstream Consequences of Learned Activation Interventions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Activation interventions are a widely used method to associate language model (LM) representations with interpretable variables. However, success on the objective used to fit an intervention neither establishes that the intervention produced the intended downstream consequences, nor does it identify which internal components caused the behavioural changes. We investigate these questions by constructing a Theory of Mind (ToM) belief-tracking dataset comprised of short narratives and associated questions about agents’ beliefs and their consequences. Rank-16 activation patches are trained under the objectives of 1) transferring the location answer from a source narrative and 2) implementing a consistent remapping of locations. Evaluations of the activation patches are carried out by measuring the differences in matched target rules, held-fixed downstream consequence tests, and with unfitted principal component analysis (PCA) controls. Answer accuracy is shown to be insufficient to establish a coherent intervention. On the consequence test set, a PCA intervention followed by a fixed output relabelling achieves 100% accuracy on the target question under the alternative mapping, yet fails to answer all follow-up questions correctly for any narrative. By contrast, holding each patch constant across questions and subsequent observations, we find that the learned remapping led to the LMs behaving as if the underlying events had been rewritten, showing high agreement with follow on consequences, including those not trained for. Nevertheless, comparisons with PCA controls reveal no additional accuracy benefit from fitting for the intended source-answer transfer objective on these consequence-based evaluations. To test the causal contribution of attention keys to the behavioural difference between learned and PCA interventions, we interchange the attention keys induced by the two interventions. Queries and values evolve within the recipient. Exchanging the PCA-induced attention keys with the learned patch attention keys led to increased agreement with the trained patches’ outputs by up to 75.4 percentage points, averaged over multiple seeds, whereas the reciprocal exchanges favoured PCA outputs. On paired colour-lookup questions, performance also exceeds an exact upper bound achievable by any single table-independent transformation of the PCA colour answer. Together, these results establish the tested downstream scope of learned remapping and provide causal evidence that attention keys induced by the learned patch contribute to the behavioural differences between learned and PCA interventions under the tested multilayer intervention setting.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic interpretability, Activation patching, Causal abstraction, Attention mechanisms, Large language models, Language models, Theory of mind

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_eFsCOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates causal effects of interventions in large language models in a theory-of-mind-style dataset. I could not discern from the paper how the actual study design, dataset, and conclusions look like.

Critical strengths

The poor writing quality makes it hard to me to have enough confidence to judge the merits of the paper.

Critical weaknesses

The writing quality is extremely poor. The abstract seems to be partially a description of the experimental methodology, which is an unconventional choice, which is not necessarily bad per se, although it misses the high-level point and the emphasis on the contribution. I read both abstract and conclusion multiple times, and could not figure out what kind of study the authors conducted and what the high-level conclusion is. The experimental design details, what the three different settings actually mean (what is the operation? what would be a story? what does no component exchange mean?), and details in the tables are insufficiently explained to be sufficiently judged.

Unfortunately, the paper lacks several critical aspects, which make it hard to assess the contribution. Namely, the description of the dataset is missing - there is a brief summary of some parameters in Sec. 4, but no examples, prompt, etc. They refer to theory of mind and exchanging locations, but what does that mean exactly? What type of dataset is it? Whether there are any examples, the dataset size, etc., are not described in the manuscript, despite the fact that the paper has space.

Questions for authors

N/A

Reviewer_AscUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether successful learned activation interventions induce coherent downstream consequences beyond the fitted answer, and which internal components causally contribute to the resulting behavioral changes. The authors compare rank-16 DAS interventions trained for source-answer transfer and alternative location remapping against unfitted PCA controls. This paper further conduct reciprocal attention-key exchanges between learned and PCA-patched executions to study the contribution of attention keys.

Critical strengths

The paper raises an important distinction between fitting an intervention objective and establishing coherent downstream behavioral effects.

Critical weaknesses
  1. The intended DAS intervention provides almost no improvement over unfitted PCA. In Section 5.3 and Table 4, the intended source-answer transfer objective shows essentially no meaningful advantage from learning the DAS subspace. On the five-shift evaluation, learned-minus-PCA differences are only -0.18 percentage points for Qwen and +0.93 percentage points for Mistral.
  2. The manuscript also has noticeable presentation issues that make it appear insufficiently polished. First, many tables have lengthy captions positioned extremely close to the tabular content, with little visual separation. This occurs repeatedly throughout the paper and makes the results difficult to scan. The density of captions, table contents, and surrounding text would benefit from more careful typesetting. Second, there are obvious editing artifacts. A particularly noticeable example is the caption of Table 11, "This table is the same as above but with 10 but uses the prefill.."
Questions for authors

See Critical Weaknesses

Reviewer_86B5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that an activation intervention succeeding on its fitting objective does not show what it changed downstream. It tests this on a synthetic theory-of-mind belief-tracking task in Qwen2.5-72B and Mistral-24B, fitting rank-16 DAS patches under two objectives: transfer the source answer, or apply a fixed permutation of the six locations. Unfitted PCA patches serve as the control. The paper reports three findings:

  1. Holding each patch fixed across eight follow-up questions, the permutation-trained patch behaves as if the story event were rewritten, while unfitted PCA already achieves the intended source transfer.
  2. Exchanging the critical-token attention keys between the learned-patch and PCA-patch runs largely transfers their behavior in both directions.
  3. A colour-lookup test exceeds a proven bound for any simple relabelling of the PCA output.
Critical strengths

S1. Testing downstream consequences is a useful, falsifiable methodology. The paper holds a patch fixed and checks its predictions on untrained questions against explicit candidate accounts (Table 9). It shows that the permutation patch acts coherently like an event rewrite and that unfitted PCA already matches the intended objective, a cautionary result for DAS-style claims.

Critical weaknesses

W1. The mechanistic claim is coarse. There are three problems with it:

  • Only keys are exchanged. Without matched value-, query-, or residual-stream exchanges, it is unclear whether keys are special or whether any large edit at the critical token would transfer behavior.
  • The effect is not localized: no block band recovers more than about 11 of the 75 pp addition effect (Table 19).
  • For addition, the null control gives exactly the same result as no exchange (Tables 16–17), so it is a weak baseline.

The paper also does not say what these results add beyond the lookback and routing accounts it cites (Prakash et al. 2026; Steele et al. 2026).
W2. Presentation. The paper is hard to follow. Many terms are introduced with little intuition (cores, addition/removal, fidelity vs. correctness, collision cases, native), and the abstract is dense with numbers. A schematic and a worked example story would help considerably.
Minor fixes:

  • l. 263: "patchestrail" → "patches trail".
  • l. 609: "can witnesses" → "can witness".
  • Table 11 caption ("This table is the same as above but with 10 but uses the prefill..") is garbled; presumably "Same as Table 10, but with the Answer: prefill."
  • Appendix A.2 says ranks 1, 4, 16 and 64 were compared, but Table 8 shows only ranks 4, 16 and 64.
  • Appendix A.1: "1,369/1,500 and 1168/1500 pairs for Qwen and Mistral pairs": inconsistent number formatting and a repeated "pairs".
Questions for authors
  1. Do value-only or query-only exchanges under the same protocol transfer behavior as well? This would show whether keys are uniquely important.
  2. How close are the learned-patch activations at the critical token to those of the naturally rewritten story T? Could the patch simply be writing the permuted location token, which would predict the same downstream answers as an event rewrite?
  3. Would the lookback account of Prakash et al. (2026) already predict the key-exchange results? If so, what is the new insight?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.