- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The authors propose Context-Preserving Activation Perturbation (CPAP), to construct counterfactual perturbations in the context of activation patching in representation space rather than input text. Through a combination of localization and optimization objectives, CPAP evaluation suggests that detection of relevant components is competitive in approximation classical activation patching results. The paper explores extending this approach to reasoning settings, for which the definition of counterfactual inputs remains an open challenge.
Critical strengths
- The main idea of defining counterfactuals for interpretability in latent instead of token space is a timely challenge to interpretability research.
- A reasonably broad set of tasks and models is selected for evaluation. The included metrics also cover a diverse range of solution properties, e.g. correlation scores, Jaccard similarity, and AUROC.
- Including the reasoning experiments was a relevant demonstration of CPAP (though they remained exploratory).
Critical weaknesses
- The proposed algorithm relies on defining editable token positions P, which may be a more relaxed requirement than directly defining counterfactual inputs, but, this to me appears a practical limitation (see Equation 4, and paragraph)
- Results in Table 2 may not be significant for some comparisons, e.g. are the differences between CPAP and OA significant in the Sentiment setting? While CPAP overall seems to correlate well with handcrafted corruptions, I think the claim “CPAP achieves the highest AUROC among automatic methods on all three tasks” may be misleading and does not seem to generalize to other evaluation metrics. If available, additional explanation and discussion would be important here.
- The formulation of the CPAP objective, in particular the gradient localization, does not seem to reliably improve performance (”gradient guidance has no consistent advantage at matched budgets”, l. 439). More fine-grained ablations on the methodological decisions in Section 2 (Equation 3-9) would be needed to better understand and disentangle, which objectives and constraints drive CPAP performance. The authors correctly acknowledge that future work should “develop more precise constraints” (l. 480), but given the lack of support for which constraints are crucial in these simpler settings, I would prioritize proper ablations before moving to more complex reasoning settings.
Questions for authors
- What is the computational cost of CPAP compared to the other automated methods (only OA is discussed)?
- What is the distinction between “Handcrafted Corruptions” and “Handcrafted Counterfactuals”?
- What is the motivation behind the heuristic for selecting probes from the ("32 cosine-nearest vocabulary embeddings", l.1370) only? What happens if a meaningful counterfactual requires to pick a semantically dissimilar token?
5 comments
Response to Reviewer sows — 'W2: Statistical Significance and Interpretation of Table 2' and 'W3: Disentangling Objective Terms and Gradient Search' (part 2/3)
W2: Statistical Significance and Interpretation of Table 2
A: We agree that CPAP's AUROC on Sentiment is comparable to that of OA, and the small difference does not establish statistical significance. Yet, we believe that the overall comparison favors CPAP for two reasons.
-
First, on Sentiment, CPAP demonstrates a modest advantage over OA on other evaluation metrics, including reference recovery (8/9 vs. 7/9) and Spearman correlation (0.762 vs. 0.681).
-
Second, considering AUROC across all three tasks, as shown in Table 2, CPAP consistently achieves higher values than OA on IOI (0.968 vs. 0.921), Sentiment (0.983 vs. 0.980), and Greater-than (1.000 vs. 0.883).
Taken together, these results suggest that CPAP performs better than OA overall, although the magnitude of its advantage varies across tasks and evaluation metrics.
We appreciate the reviewer's concern and have revised our statement to clarify that it refers specifically to the observed AUROC values, rather than statistically significant superiority across all metrics. We would also welcome any further suggestions and additional discussions.
W3: Disentangling Objective Terms and Gradient Search
A: We agree that the effects of the core objective and the search implementation need to be evaluated separately. We therefore added finer-grained ablations covering both parts of Eq. (3) and the search decisions in Eqs. (4)–(9).
1. Contribution of the CPAP components. CPAP's components are designed to serve two purposes: improving intervention effectiveness and enhancing search efficiency. The effectiveness/efficiency of each component is shown below:
| Component | Purpose | Evaluation metric | Result |
|---|---|---|---|
| Maximize upstream activation change (Eq. 3, part 1) | Effectiveness | Task-score recovery (downstream bound fixed) | 92.60% → 98.83% (+6.23 pp) |
| Constrain downstream residual change (Eq. 3, part 2) | Effectiveness | Task-score recovery (upstream maximization fixed) | 93.83% → 98.83% (+5.00 pp) |
| Restrict candidate vocabulary | Efficiency | Mean search time per input | 19.94 s → 5.01 s (−74.9%) |
| Check prediction changes before selection | Validity / Efficiency | Mean search time per input | 19.94 s → 5.00 s (−74.9%) |
| Gradient-guided proposals | Efficiency | Mean search time per input | 19.94 s → 10.33 s (−48.2%) |
The ablations were conducted on the Greater-than year-comparison task with Qwen3-8B.
The +6.23 pp and +5.00 pp task-score recovery gains separately support the upstream-maximization and downstream-constraint terms. In contrast, vocabulary restriction, early behavioral screening, and gradient proposals concern the cost and implementation of the search, rather than forming additional terms of the intervention objective.
2. Changes in the revised manuscript.
We replaced the broad ablation discussion in Section 3.4 with a component-level effectiveness/efficiency table (Table 5), distinguishing the two objective terms from the search heuristics. We clarified the shared selection rule for gradient and random proposals in Fig. 5(b) and the search-budget discussion.
Response to Reviewer sows questions (part 3/3)
Q1: Computational Cost Relative to Other Automatic Methods
A: We compared CPAP with zero ablation, Gaussian noise, resampling, and Optimal Ablation (OA) on 80 Qwen3-8B IOI inputs, evaluating 1,152 attention heads.
| Method | Construction (TFLOPs) | Component scan (TFLOPs) | Total (TFLOPs) | Time (s) |
|---|---|---|---|---|
| Zero ablation | 0 | 514.98 | 514.98 | 23.67 |
| Gaussian noise (10 draws) | 194.51 | 5,149.78 | 5,344.29 | 75.99 |
| Resampling (10 donors) | 93.86 | 5,149.78 | 5,243.64 | 86.04 |
| Optimal Ablation (OA) | 910,724.91 | 514.98 | 911,239.89 | 33,656.72 |
| CPAP (full budget) | 84,792.97 | 482.79 | 85,275.76 | 3,137.16 |
| CPAP (compact search, projected) | ≈13,800 | 482.79 | ≈14,300 | ≈530 |
FLOPs are estimated from dominant matrix operations. Timings use A100-80GB GPUs in FP32, with OA fitting and CPAP search times accumulated across workers.
CPAP is more computationally expensive than direct perturbation methods (zero, Gaussian, and resampling), but requires approximately 10× less computation and time than OA. Its cost can be further reduced by optimizing the search strategies (e.g., rounds, beam width, and proposals), as illustrated by the projected compact configuration (2,048 vs. 12,563 evaluations).
Q2: “Handcrafted Corruptions” versus “Handcrafted Counterfactuals”
A: We only use the term “corruptions” in our paper, following established terminology in the activation patching literature [1–5]. More generally, “corruption” is a broader term encompassing both token replacements and other perturbations, such as Gaussian noise [1, 2], whereas “counterfactual” typically refers to constructed alternative inputs that modify task-relevant information [1, 5]. However, in mechanistic interpretability, the two terms are often used interchangeably when referring to manually constructed, task-specific contrastive inputs [1, 5]. Therefore, in the context of our work, “Handcrafted Corruptions” and “Handcrafted Counterfactuals” refer to the same inputs, with no methodological distinction.
References
[1] Zhang, F., & Nanda, N. (2024). Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. ICLR 2024. https://arxiv.org/abs/2309.16042
[2] Heimersheim, S., & Nanda, N. (2024). How to Use and Interpret Activation Patching. https://arxiv.org/abs/2404.15255
[3] Conmy, A., et al. (2023). Towards Automated Circuit Discovery for Mechanistic Interpretability. NeurIPS 2023. https://arxiv.org/abs/2304.14997
[4] Syed, A., et al. (2024). Attribution Patching Outperforms Automated Circuit Discovery. BlackboxNLP 2024. https://aclanthology.org/2024.blackboxnlp-1.25/
[5] Miao, Y., & Kan, M.-Y. (2025). Discursive Circuits: How Do Language Models Understand Discourse Relations? EMNLP 2025. https://aclanthology.org/2025.emnlp-main.1657/
Q3: Motivation and Limits of the 32-Nearest-Neighbor Probe
A: The motivation is to select perturbation candidates that are similar to, but distinct from, the original token. In our evaluated tasks, handcrafted corruptions typically preserve the token type while changing its task-relevant content (e.g., replacing names with other names). We therefore adopted the 32 nearest neighbors as a simple and computationally efficient heuristic, providing a compact candidate pool that is likely to include such type-consistent alternatives.
We agree that this restriction may miss informative positions when meaningful counterfactuals require semantically dissimilar replacements. In such cases, the candidate pool can be expanded or diversified accordingly. Moreover, our recent experiments suggest that LLM-based position selection guided by task descriptions provides a promising alternative that does not rely on this fixed nearest-neighbor restriction. We will clarify this motivation and limitation and include the additional results in the revised manuscript.
Response to Reviewer sows — W1: Practical Dependence on Editable Positions (part 1/3)
Thanks for your concern and we have integrated automatic token position search method [1-2] with CPAP. The additional experiments and detailed results are as below:
1. Experiments with automatic token-position selection
Conclusion: CPAP remains effective integrated with automatically selected positions.
Result Table: | Task | Edit positions used by CPAP | Recovered | Precision | AUROC [95% CI] | ρ / J | |---|---|---:|---:|---:|---:| | IOI | Manual (paper) | 21/26 | 0.808 | 0.968 [0.937, 0.981] | 0.563 / 0.631 | | IOI | MiCE-style gradients | 19/26 | 0.760 | 0.9555 [0.9210, 0.9782] | 0.331 / 0.510 | | IOI | Input marginalization | 15/26 | 0.577 | 0.9133 [0.8590, 0.9405] | 0.186 / 0.370 | | IOI | LLM-based selection | 20/26 | 0.800 | 0.9648 [0.9273, 0.9818] | 0.500 / 0.606 | | Sentiment | Manual (paper) | 8/9 | 0.889 | 0.983 [0.976, 0.986] | 0.762 / 0.603 | | Sentiment | MiCE-style gradients | 8/9 | 1.000 | 0.9893 [0.9827, 0.9918] | 0.378 / 0.603 | | Sentiment | Input marginalization | 8/9 | 0.889 | 0.9868 [0.9770, 0.9885] | 0.359 / 0.603 | | Sentiment | LLM-based selection | 8/9 | 0.889 | 0.9802 [0.9778, 0.9860] | 0.729 / 0.603 | | Greater-than | Manual (paper) | 10/10 | 1.000 | 1.000 [0.993, 1.000] | 0.561 / 0.818 | | Greater-than | MiCE-style gradients | 10/10 | 0.800 | 0.9955 [0.9763, 0.9991] | 0.494 / 0.538 | | Greater-than | Input marginalization | 10/10 | 0.800 | 0.9936 [0.9300, 1.0000] | 0.530 / 0.538 | | Greater-than | LLM-based selection | 10/10 | 1.000 | 1.0000 [0.9945, 1.0000] | 0.561 / 0.818 |
Position-selection methods
p.s.: These works already show that selecting intervention positions is effective, so we pay more attention on the replacement and just adopt a simple variant of MiCE [2]. In our recent practice, LLM-based selection shows superior results with explicit task descriptions.
2. Changes in the revised manuscript.