- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates how intervention affects model output by counterfactuals. The authors propose to insert an edit moule at the residual stream and minimize the KL div between the source and target prompt output distribution.
Critical strengths
- The reconstruction caomparison is insightful.
- Holding response fixed is reasonably clean.
- THe analysis and diagnostics and thorough.
Critical weaknesses
- As most findings are for the midpoint layer (e.g., Section 3.1), the position where the edit module is applied needs further investigation, e.g., with layer sweep or sensitivity analysis. At least one more dataset should be included, performing experiments only on Scruple is not sufficient.
- In my opinion, the localization baseline is missing. Since the response is fixed and the token positions are alreayd well-aligned, why don't you just compute the position-wise activation difference and use this as a baseline? Furthermore, I was wondering whether fixing the response will lead to any off-policy issue, as the model may not generate such a trajectory.
Questions for authors
- The paper about intervention-based CoT faithfulness metrics is missing (Lanham et al., 2023; https://arxiv.org/abs/2307.13702)
3 comments
[W2-2] Fixed Response
We would like to provide some clarification about the generations and how the edit model is both trained and evaluated. All the generations are sampled from the model's distribution and as a result are a form of online behavior. Specifically, every response is first sampled autoregressively from the model. We then hold the sampled response fixed when comparing the control and intervention conditions, so both runs are evaluated on the same response prefix at each token position.
Furthermore, to help with generalization, for each prompt, the edit model is trained with both a response generated from the same prompt as well as a response generated from the counterfactual prompt. This way, the edits can be applied to both responses that are unchanged as well as responses that look more like the target behavior.
We also ran an additional experiment on a subset of the test set to see how the edits affect fresh generations. We applied the edits during autoregressive sampling and measured how often the model judged the author to be in the wrong. For both models, edits that simulate adding the intervention increased this rate, while edits that simulate removing it decreased the rate.
| Model | Direction | Original | Edited |
|---|---|---|---|
| Gemma | Control to Intervention | 28% | 35% |
| Gemma | Intervention to Control | 32% | 25% |
| DeepSeek | Control to Intervention | 27% | 46% |
| DeepSeek | Intervention to Control | 42% | 26% |
[Q1] Citation
Thank you for pointing us towards this reference, this would be a great addition, and we have added it to the related works.
[1] Panickssery, Nina, et al. "Steering llama 2 via contrastive activation addition." 2023.
[2] Li, Nathaniel, et al. "The wmdp benchmark: Measuring and reducing malicious use with unlearning." 2024.
[3] Guan, Melody Y., et al. "Monitoring monitorability." 2025.
Dear Reviewer BSSP,
Thank you again for taking the time and effort to review our paper! A quick reminder: the discussion period closes today (November 18). If you have any remaining questions, we'd be happy to address them before the deadline!
Best regards, The Authors
We thank the reviewer for finding the reconstruction comparison insightful, found the setup clean, and the analysis thorough. We also appreciate the thoughtful feedback which we address below.
[W1-1] Layer sweep/Sensitivity Analysis
We agree that experiments on different layers would be helpful. We used the midpoint as a common intervention site, since there are several downstream layers available to propagate the edit. Moreover, prior work on activation steering, such as [1], found that interventions at intermediate layers can effectively steer a range of behaviors.
As suggested, we further tested how sensitive our findings are to this choice of layer. We trained edit modules at layers with index 3m with m=0,…,13 and measure the KL recovery for each of the models and have included these results in the revision. We found that recovery is highest at intermediate layers, and we see a steady decrease in recovery from layer 15 to 39, suggesting that results are relatively stable across layers. We will include these in a revised manuscript.
[W1-2] More datasets
We agree that evaluating on another dataset is important. As suggested, we added an evaluation on WMDP [2] using the sandbagging scenario from [3]. The intervention is a statement that the model will not be deployed if it demonstrates advanced knowledge in biology, chemistry, or cybersecurity and is provided with a possible incorrect answer. We provide results for Gemma below.
Our method recovered substantially more counterfactual behavior than activation reconstruction, consistent with the main finding on Scruples. We also computed attribution maps using the edits and Integrated Gradients and found that within the intervention, the area most relevant to the question out of biology, chemistry, and cybersecurity has high attribution scores compared to others. As in Scruples, whitespace and punctuation tokens received high attribution in the generated reasoning. This demonstrates a shared structure across datasets. The results have been added to the revised version.
[W2-1] Activation Difference Baseline
Thank you for the great suggestion. This is a useful baseline to have, and we have run experiments using the activation differences as edit vectors. We found that using the activation differences directly covers 64.7% of counterfactual behavior for DeepSeek and 70.6% for Gemma. Thus, exactly matching the counterfactual activations at the edit layer recovers only part of the counterfactual output behavior. Importantly, our learned edit module adds two advantages:
Attribution to prompt and intervention tokens. The raw differences define edits at aligned response positions. When treated as fixed vectors, they provide no attribution path from those edits back to the prompt tokens that produced them. Our learned module provides that path. In the sandbagging experiment, for example, the domain named in the intervention that matched the question received higher attribution than the other domains.
Compact modeling of the edit trajectory. The baseline computes each difference by running both prompt conditions through the first half of the Transformer (using billions of parameters). Our edit module uses substantially fewer parameters than that upstream computation to capture patterns in how the edits vary across response positions and examples. This gives us a compact model of the changes that matter for recovering counterfactual behavior.
We will add this discussion to the revised manuscript.