acceptodds
Under review as a conference paper at ICLR 2027

What Can Activation-Patching Evidence Establish?

Abstract

Successful activation patching does not necessarily establish the functional claim it is used to support. Even when expected outputs and measured responses are restored, unmeasured claim-changing alternatives can remain indistinguishable. We formalize this gap as target-relative observability: relative to a declared class of admissible responses, evidence is sufficient only when every response consistent with the observations agrees on the target. On finite-dimensional linear response spaces, we characterize claim-changing blind directions, minimum target-specific measurement completion, and sharp error amplification; an activation-specific corollary shows when a finite probe panel cannot certify unrestricted local response preservation. Prospectively, among center-restoring GPT-2 patches, full-gradient discrepancy predicts withheld finite-response failures better than public-direction screening by 0.080/0.055 within-layer AUC on IOI/GT, with recurring but model-dependent gains across six external model families. Controlled studies show that variance, generic geometric matching, and span coverage can each misrank evidential value, while reference queries can transfer shared nonlinear structure. Reliable validation need not recover the whole response; it must rule out the admissible alternatives that can change the claim.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, activation patching, causal interventions, identifiability, observability, measurement design

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_Ha1tOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper analyzes activation patching and shows that evidence is sufficient only when every admissible response consistent with the observations agrees on the target. But I do not understand this paper well.

Critical strengths

The paper has a complete overall structure.

Critical weaknesses

The paper is hard to understand, and its organization needs significant improvement. The main text does not clearly explain what activation patching is or how it is performed.

Questions for authors

Could you explain the background and motivation more clearly, and give a simple example of activation patching early in the paper? For readers unfamiliar with this method, it is difficult to understand the abstract’s opening sentence, “Successful activation patching does not necessarily establish the functional claim it is used to support,” without a clear explanation of the procedure and the claim being discussed.

Reviewer_neo4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that a patch that restores the clean output does not by itself support a functional claim. It formalizes this as target-relative observability: observations support a target only if the kernel of the observation map lies in the kernel of the target. On GPT-2 Small IOI and Greater-Than, many well-restored patches fail a hidden test on random activation displacements. A full-gradient score predicts these failures better than a public-direction score (within-layer AUC +0.080 and +0.055), and this mostly recurs on six other model families. Controlled studies show that variance, geometric matching, and coverage can misrank measurement value.

Critical strengths
  1. The framing is useful. Asking what the evidence must rule out for a given claim separates restoration, identification, and stable reconstruction.
  2. The main test is rigorous. Scores are frozen before outcomes, the cohort is prospective and disjoint, and intervals use prompt-cluster resampling.
Critical weaknesses
  1. The failure target is not linked to any functional claim. "Failure" is a response mismatch under random displacements of about 0.1% of the activation norm. Restoration was never meant to measure this, so the gap is expected by construction. The only task-level test (Appendix E.2) goes against the main story: on IOI answer error, GG (0.249) and DD (0.237) do no better than uniform acceptance (0.234), while center patched margin does best (0.146).
  2. The contribution is thin for the reading effort. The theory is classical, as the authors note, and the main result follows almost directly from the linear setup. The paper is dense with acronyms and cohorts, and Section 5 is loosely tied to the main question.
Questions for authors
  1. Can you show a case where a well-restored patch supports a wrong functional conclusion, for example an IOI circuit claim under semantic or natural input changes? This would most change my assessment.
  2. How do you read Appendix E.2, where GG and DD do not beat uniform acceptance on answer error?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.