acceptodds
Under review as a conference paper at ICLR 2027

Localization Does Not Guarantee Control: Gate Failure and Partial Activation Steering in a Causal Reasoning Model

Abstract

Does locating a behaviorally relevant activation site suffice to construct an effective intervention? We study this question in a frozen 25.7M-parameter transformer trained to answer interventional queries from mixed observational and interventional evidence. On a selected subset of confounded worlds, transplanting activations at mid-layer observational-record positions recovers 88.9% of a reference behavioral shift induced by evidence zeroing. We then evaluate a gated activation-editing pipeline and an ungated ablation on 50 held-out confounded worlds. The detector, calibrated with zero validation false triggers but only 0.5 recall, triggers on none of 2,400 target generations and on 6.9–7.3% of held-out in-distribution generations. Bypassing the gate enables partial correction: at the larger of two tested doses, a pooled contrast direction reduces the regression intercept from 0.3820 to 0.2637, a 31% reduction, while also reducing the gain from 0.8236 to 0.5721. An exact coefficient identity shows why: the pooled edit attenuates the whole response nearly proportionally, lowering offset and gain by the same factor instead of restoring world-specific sensitivity. These results separate failure to activate an intervention from insufficient correction after activation, and show that localization, gate coverage, and intervention efficacy must be evaluated separately.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, activation steering, causal reasoning, linear probes, preregistration, negative results, representation-behavior gap

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FistOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether identifying internal activations that influence a model’s behavior is enough to construct an effective intervention. It studies a small transformer trained to answer causal questions using both observational and interventional evidence. On a selected subset of examples, replacing activations associated with observational records reproduces much of the behavioral change caused by zeroing that evidence. The paper then test an editing pipeline with a detector that decides when to intervene. This detector never activates on the held-out confounded examples, leaving the model’s predictions unchanged. Applying the edits unconditionally reduces a systematic offset in the predicted causal effects, but also weakens their sensitivity to the true effects. The paper argues that locating relevant activations, determining when to intervene, and constructing an effective edit are separate problems that should be evaluated independently.

Critical strengths
  • The paper is transparent about its negative results. It clearly reports that the detector fails to activate on the target examples and that unconditional editing reduces both the prediction offset and sensitivity to the true causal effects.

  • Evaluating the edits with and without the gate provides a useful diagnostic distinction. It shows why an unchanged model output can reflect a failure to trigger the intervention, rather than evidence that the intervention itself has no effect.

  • The localization experiments include paired comparisons between observational and non-observational token positions. These controls provide meaningful evidence about where the reference behavioral change can be recovered.

  • The evaluation looks beyond a single favorable metric. Reporting both offset and gain exposes the trade-off introduced by editing.

Critical weaknesses

In general, the paper leaves important experimental details unclear, and the evidence is too narrow to support several of its explanatory claims.

  • The detector’s training objective is not clearly aligned with the desired intervention. It predicts whether zeroing observational evidence increases the response slope, which does not necessarily identify examples that need correction or would benefit from editing. Without reporting the prevalence of this label on the evaluation sets, the observed failure cannot be adequately attributed to poor generalization. The calibration set also being relatively small further limits the conclusions.

  • The localization and editing experiments use different evaluation populations. The reported 88.9% recovery applies to 48 selected worlds out of 608 screened, whereas editing is evaluated on a separate set. The paper therefore does not establish that transplantation succeeds on the same examples where pooled editing fails. Moreover, recovery toward the zeroing reference does not itself establish improved agreement with the true causal effect.

  • The proposed explanation involving lost positional information is not directly tested. Paper does not clearly specify what additional information the oracle donor contains beyond the zeroed activations already computed for the adaptive edit. Comparing row-wise and pooled interventions using the same donor information might help establishing whether pooling is responsible for the performance gap.

  • The claim that editing proportionally attenuates the whole response is stronger than the analysis supports. Similar reductions in the fitted offset and gain establish a change in the regression relationship, but do not show that individual predictions are approximately rescaled. The substantial change in correlation also indicates that simple rescaling is not a complete explanation. A paired analysis of baseline and edited predictions is needed.

  • The adaptive-versus-fixed comparison does not directly measure which method performs better. Mean differences in predicted slopes can be positive or negative without indicating an improvement in accuracy. The comparison should report the same accuracy-based endpoints, offset and gain estimates, and paired uncertainty intervals at matched intervention strengths.

  • The evidence for behavioral improvement is incomplete. The main ungated results lack uncertainty estimates.

  • The contribution remains limited relative to its framing. The experiments cover one small checkpoint and a restricted editing construction. Important implementation details (e.g. including the exact zeroing operation, donor construction, and data-generation mechanisms) are insufficiently specified. A stronger diagnosis of the observed failures is needed to establish a substantial contribution beyond the paper’s acknowledged prior work.

Questions for authors

What fraction of CONF50 and ID40 satisfies the detector’s actual positive label, and how do detector scores relate to whether editing improves accuracy? Can you distinguish label mismatch from threshold miscalibration or poor detector generalization?

Can you compare oracle transplantation, row-wise editing, and pooled editing on the same held-out worlds, reporting agreement with the true causal effects? What additional information does the oracle donor contain that is unavailable to the prompt-computable edits?

What per-world evidence supports the proportional-attenuation explanation beyond the similar changes in regression coefficients? Can you test this against a simple rescaling baseline and report dose-matched fixed-versus-adaptive comparisons using prediction errors and paired confidence intervals?

Reviewer_JmaDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors study a gated activation-editing pipeline in a frozen 25.7M-parameter transformer answering synthetic causal queries. An oracle transplant recovers about 89% of an evidence-zeroing reference shift on 48 eligible worlds out of 608 screened. The separately calibrated detector, however, never triggers on the 2,400 recorded target generations from 50 held-out confounded worlds; it triggers on roughly 7% of held-out ID generations. Removing the gate lets pooled activation edits reduce the fitted response intercept, but also reduces gain and positive-world correlation. A coverage-times-conditional-effect decomposition and an OLS coefficient identity are used to distinguish failure of the detector from limitations of the edit itself. I am also unconvinced by the argument that the edit only scales the model's responses. The offset and the gain go down by order the same amount, but these are fit quantities across the worlds. They do not establish that each world's response goes down by the same factor. It would be helpful to plot the original vs. edited responses and to fit and report how well a single scaling factor describes the change, with uncertainty.

Critical strengths

The trigger counts are particularly helpful. They explain why the full pipeline has no effect, and the gated/ungated comparison and fixed-direction control provide useful diagnostics. Reporting gain alongside offset also avoids treating a reduction in response magnitude as an unqualified improvement. There is a worthwhile negative result here.

The paper is careful about several details that matter for interpreting that result. It acknowledges the restricted setting and selected localization population, distinguishes repeated decoding events from independent worlds, reports paired localization intervals, and describes fixing thresholds before the corresponding measurements. This makes the failure easier to assess.

Critical weaknesses

My main reservation is that the gate experiment shows that this detector failed without establishing a broader failure mechanism. Calibration uses only nine positive examples. Its label, D > 0.05, identifies a positive zeroing-induced shift rather than confounding or a need for behavioral correction. Localization instead uses |D| > 0.05, and the target population is selected separately for confounding. Zero triggering on CONF50 could therefore come from proxy-label mismatch or limited calibration coverage. The paper acknowledges the small calibration set, but the true proxy-label distribution and detector errors on CONF50 are needed to diagnose what happened.

I am also not convinced by the argument that the edit only scales the model's responses. The offset and the gain go down by order the same amount, but these are fit quantities across the worlds. They do not establish that each world's response goes down by the same factor. It would be helpful to plot the original vs. edited responses and to fit and report how well a single scaling factor describes the change, with uncertainty.

The transplantation experiment and the pooled editing experiment used different causal scenarios, so it is unclear that the difference in performance arises from losing information when averaging the activations. I would like to see transplantation, fixed editing and adaptive editing all on the same scenarios, evaluated by prediction error, offset and gain. The adaptive vs. fixed comparison requires these quantities as well: having a higher (or lower) average response does not by itself indicate that one edit makes better predictions.

Questions for authors
  1. How many CONF50 worlds satisfy the detector's actual training label D > 0.05? Please show how D and detector scores compare with calibration, and report held-out recall for that same label. This would distinguish detector generalization failure from a mismatch between the label and the intended intervention population.

  2. Could the proportional-attenuation explanation be tested directly by fitting edited response slopes against base response slopes, with residuals or goodness of fit and paired uncertainty? Equation (7) alone shows coefficient changes, not scaling at the level of individual worlds.

  3. On the same worlds, how do full row-wise transplantation, pooled adaptive editing, and fixed editing compare in gain, offset, and prediction error? Please include the fixed-control coefficients and paired uncertainty for adaptive-minus-fixed differences. The sign of a mean response shift alone does not establish improvement.

  4. I ‌would ‌also ‌like to see zeroing direct evidence included in baselines. It already runs the activations from a evidence-zeroed run, so it would be nice to see how the predictions from that run compare to the edited predictions for the same causal scenarios, to get a sense of whether the activation editing is any better than just turning off the observational evidence.

Reviewer_4PYqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates the "decodability", "localizability", and "releasability" of mechanistic interpretability methods, focusing on a proposed contrastive causal gate (CCG) pipeline.

Critical strengths

The paper aims to address several well-motivated problems in the mechanistic interpretability literature: the relationship between localization and editing efficacy; how best to condition or "gate" interventions; and the generalizability of steering methods. Each of these is a major issue, and any work that meaningfully advances our understanding of these problems would be a welcome contribution to today's literature.

Critical weaknesses
  • The most important limitation of this work is that absolutely essential details to understand and review this paper are not available -- it is transparently clear that the authors have simply split a single research project into two submissions: this one, and the repeatedly-cited "concurrent submission (Anonymous, 2026)". Without being able to see this other paper, it is impossible to provide an accurate review.
    • In general, this manner of submission (splitting a single project into two, and submitting them concurrently) is highly inappropriate, in that it "games the system" to increase the authors' odds at having multiple accepted papers while doing the work of only a single paper. As noted above, doing so also makes it impossible to review this paper. (I would guess that the same problem is true of the paper containing the other half of this project; but of course I cannot say for certain, given that I cannot see it.)
  • Following from the first weaknesses bullet -- all experiments are conducted on an underspecified model, trained on an unspecified task, with an unclear motivation. As such, I cannot infer the goal of the experiments, nor the potential implications of the results. However, even if I do my best to charitably guess at the missing information, it seems likely that experiments conducted on a 25.7M-param, 41-token language model would carry only minimal implications for even similarly-sized bona fide language models (i.e., where "language" here is "natural language", such as English, Chinese, etc, rather than the unspecified synthetic vocabulary and structure that C1 is trained on), let alone even the modestly-sized language models of 7-8 years ago like BERT or GPT-2. Thus, the implications for any "real world" models (or interpretability methods targeted at such models) that are trained on naturalistic data -- even smaller models at, e.g., the 100M-10B param scale specialized for edge compute -- would likely be minimal.
Questions for authors
  • Following from the first weaknesses bullet -- there is critical information missing from this paper that is very likely contained by "concurrent submission (Anonymous, 2026)". An illustrative but incomplete set of questions that cannot be answered from this submission (but would hopefully be answerable given the contents of the concurrent submission) would be:
    • All experiments center on the small (25.7M-param, 41-token) "C1" model, but we have no information regarding the task this toy model is trained on. What is the task? What are the inputs and outputs? At a higher level of abstraction/motivation, what problem or idea is the task intended to capture?
      • (Note that some information is included in Appendix A -- this should be adequately summarized in the main paper, as reviewers should be able to broadly understand experiments and key results without reading appendices; but even after reading the appendices, most of the necessary information to answer these questions is either incomplete or completely missing.)
    • What are the "worlds" discussed throughout the experiments? (Again -- as in the last bullet, some information that should be in the main paper is included in the appendices; but this is still insufficient to understand how the posited "worlds" relate to the underlying task, training objective, and motivation of this work, whatever they may be.)
  • What is your research question, or the underlying problem you are trying to solve? What are your hypotheses? How do the experiments and results relate to the RQ/problem or hypotheses?
  • What implications do you believe your findings would carry for naturalistic ("real world") models? For now, let's not even worry about models with >=1B parameters -- even for a BERT-base-uncased or a GPT-2 (as two of the most commonly studied artifacts in the history of the mechanistic interpretability/"BERTology" literature), do you feel that your findings could plausibly carry any implications for such models (or any current interpretability methods applied to such models)? How so?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.