- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates whether identifying internal activations that influence a model’s behavior is enough to construct an effective intervention. It studies a small transformer trained to answer causal questions using both observational and interventional evidence. On a selected subset of examples, replacing activations associated with observational records reproduces much of the behavioral change caused by zeroing that evidence. The paper then test an editing pipeline with a detector that decides when to intervene. This detector never activates on the held-out confounded examples, leaving the model’s predictions unchanged. Applying the edits unconditionally reduces a systematic offset in the predicted causal effects, but also weakens their sensitivity to the true effects. The paper argues that locating relevant activations, determining when to intervene, and constructing an effective edit are separate problems that should be evaluated independently.
Critical strengths
-
The paper is transparent about its negative results. It clearly reports that the detector fails to activate on the target examples and that unconditional editing reduces both the prediction offset and sensitivity to the true causal effects.
-
Evaluating the edits with and without the gate provides a useful diagnostic distinction. It shows why an unchanged model output can reflect a failure to trigger the intervention, rather than evidence that the intervention itself has no effect.
-
The localization experiments include paired comparisons between observational and non-observational token positions. These controls provide meaningful evidence about where the reference behavioral change can be recovered.
-
The evaluation looks beyond a single favorable metric. Reporting both offset and gain exposes the trade-off introduced by editing.
Critical weaknesses
In general, the paper leaves important experimental details unclear, and the evidence is too narrow to support several of its explanatory claims.
-
The detector’s training objective is not clearly aligned with the desired intervention. It predicts whether zeroing observational evidence increases the response slope, which does not necessarily identify examples that need correction or would benefit from editing. Without reporting the prevalence of this label on the evaluation sets, the observed failure cannot be adequately attributed to poor generalization. The calibration set also being relatively small further limits the conclusions.
-
The localization and editing experiments use different evaluation populations. The reported 88.9% recovery applies to 48 selected worlds out of 608 screened, whereas editing is evaluated on a separate set. The paper therefore does not establish that transplantation succeeds on the same examples where pooled editing fails. Moreover, recovery toward the zeroing reference does not itself establish improved agreement with the true causal effect.
-
The proposed explanation involving lost positional information is not directly tested. Paper does not clearly specify what additional information the oracle donor contains beyond the zeroed activations already computed for the adaptive edit. Comparing row-wise and pooled interventions using the same donor information might help establishing whether pooling is responsible for the performance gap.
-
The claim that editing proportionally attenuates the whole response is stronger than the analysis supports. Similar reductions in the fitted offset and gain establish a change in the regression relationship, but do not show that individual predictions are approximately rescaled. The substantial change in correlation also indicates that simple rescaling is not a complete explanation. A paired analysis of baseline and edited predictions is needed.
-
The adaptive-versus-fixed comparison does not directly measure which method performs better. Mean differences in predicted slopes can be positive or negative without indicating an improvement in accuracy. The comparison should report the same accuracy-based endpoints, offset and gain estimates, and paired uncertainty intervals at matched intervention strengths.
-
The evidence for behavioral improvement is incomplete. The main ungated results lack uncertainty estimates.
-
The contribution remains limited relative to its framing. The experiments cover one small checkpoint and a restricted editing construction. Important implementation details (e.g. including the exact zeroing operation, donor construction, and data-generation mechanisms) are insufficiently specified. A stronger diagnosis of the observed failures is needed to establish a substantial contribution beyond the paper’s acknowledged prior work.
Questions for authors
What fraction of CONF50 and ID40 satisfies the detector’s actual positive label, and how do detector scores relate to whether editing improves accuracy? Can you distinguish label mismatch from threshold miscalibration or poor detector generalization?
Can you compare oracle transplantation, row-wise editing, and pooled editing on the same held-out worlds, reporting agreement with the true causal effects? What additional information does the oracle donor contain that is unavailable to the prompt-computable edits?
What per-world evidence supports the proportional-attenuation explanation beyond the similar changes in regression coefficients? Can you test this against a simple rescaling baseline and report dose-matched fixed-versus-adaptive comparisons using prediction errors and paired confidence intervals?