acceptodds
Under review as a conference paper at ICLR 2027

Do We Need Handcrafted Corruptions? Context-Preserving Activation Perturbation in Representation Space

Abstract

Activation patching is widely used in mechanistic interpretability to identify causally important model components. Yet its reliability depends on constructing corrupted inputs that alter task-relevant information while preserving the surrounding context. Designing such inputs is difficult in complex tasks, especially when answer-related evidence is dispersed and related mentions must be edited consistently. To address this challenge, we revisit activation patching in representation space rather than input text and introduce Context-Preserving Activation Perturbation (CPAP), which automatically constructs activation perturbations without using handcrafted corrupted inputs. CPAP maximizes the change in an upstream module's activation while constraining changes in selected downstream residual representations, using their stability as a practical proxy for preserving the surrounding computational context. Our experiments yield two main findings. First, across three models and four benchmarks, CPAP closely matches the component rankings from corrupted-input patching, with consistently higher rank agreement than zero and learned constant ablations. For GPT-2 circuit localization, it achieves the highest AUROC among automatic methods on all three tasks (0.968–1.000). Second, CPAP easily extends to explaining model behavior across complex tasks and larger models (27B). In long-context question answering and mathematical reasoning, it automatically constructs activation perturbations and identifies key components without manually designed corrupted inputs.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic interpretability, Activation patching, Circuit discovery, Large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sowsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The authors propose Context-Preserving Activation Perturbation (CPAP), to construct counterfactual perturbations in the context of activation patching in representation space rather than input text. Through a combination of localization and optimization objectives, CPAP evaluation suggests that detection of relevant components is competitive in approximation classical activation patching results. The paper explores extending this approach to reasoning settings, for which the definition of counterfactual inputs remains an open challenge.

Critical strengths
  • The main idea of defining counterfactuals for interpretability in latent instead of token space is a timely challenge to interpretability research.
  • A reasonably broad set of tasks and models is selected for evaluation. The included metrics also cover a diverse range of solution properties, e.g. correlation scores, Jaccard similarity, and AUROC.
  • Including the reasoning experiments was a relevant demonstration of CPAP (though they remained exploratory).
Critical weaknesses
  • The proposed algorithm relies on defining editable token positions P, which may be a more relaxed requirement than directly defining counterfactual inputs, but, this to me appears a practical limitation (see Equation 4, and paragraph)
  • Results in Table 2 may not be significant for some comparisons, e.g. are the differences between CPAP and OA significant in the Sentiment setting? While CPAP overall seems to correlate well with handcrafted corruptions, I think the claim “CPAP achieves the highest AUROC among automatic methods on all three tasks” may be misleading and does not seem to generalize to other evaluation metrics. If available, additional explanation and discussion would be important here.
  • The formulation of the CPAP objective, in particular the gradient localization, does not seem to reliably improve performance (”gradient guidance has no consistent advantage at matched budgets”, l. 439). More fine-grained ablations on the methodological decisions in Section 2 (Equation 3-9) would be needed to better understand and disentangle, which objectives and constraints drive CPAP performance. The authors correctly acknowledge that future work should “develop more precise constraints” (l. 480), but given the lack of support for which constraints are crucial in these simpler settings, I would prioritize proper ablations before moving to more complex reasoning settings.
Questions for authors
  • What is the computational cost of CPAP compared to the other automated methods (only OA is discussed)?
  • What is the distinction between “Handcrafted Corruptions” and “Handcrafted Counterfactuals”?
  • What is the motivation behind the heuristic for selecting probes from the ("32 cosine-nearest vocabulary embeddings", l.1370) only? What happens if a meaningful counterfactual requires to pick a semantically dissimilar token?
5 comments
AuthorsOpenReview ↗

Response to Reviewer sows — W1: Practical Dependence on Editable Positions (part 1/3)

Thanks for your concern and we have integrated automatic token position search method [1-2] with CPAP. The additional experiments and detailed results are as below:

1. Experiments with automatic token-position selection

  • Conclusion: CPAP remains effective integrated with automatically selected positions.

  • Result Table: | Task | Edit positions used by CPAP | Recovered | Precision | AUROC [95% CI] | ρ / J | |---|---|---:|---:|---:|---:| | IOI | Manual (paper) | 21/26 | 0.808 | 0.968 [0.937, 0.981] | 0.563 / 0.631 | | IOI | MiCE-style gradients | 19/26 | 0.760 | 0.9555 [0.9210, 0.9782] | 0.331 / 0.510 | | IOI | Input marginalization | 15/26 | 0.577 | 0.9133 [0.8590, 0.9405] | 0.186 / 0.370 | | IOI | LLM-based selection | 20/26 | 0.800 | 0.9648 [0.9273, 0.9818] | 0.500 / 0.606 | | Sentiment | Manual (paper) | 8/9 | 0.889 | 0.983 [0.976, 0.986] | 0.762 / 0.603 | | Sentiment | MiCE-style gradients | 8/9 | 1.000 | 0.9893 [0.9827, 0.9918] | 0.378 / 0.603 | | Sentiment | Input marginalization | 8/9 | 0.889 | 0.9868 [0.9770, 0.9885] | 0.359 / 0.603 | | Sentiment | LLM-based selection | 8/9 | 0.889 | 0.9802 [0.9778, 0.9860] | 0.729 / 0.603 | | Greater-than | Manual (paper) | 10/10 | 1.000 | 1.000 [0.993, 1.000] | 0.561 / 0.818 | | Greater-than | MiCE-style gradients | 10/10 | 0.800 | 0.9955 [0.9763, 0.9991] | 0.494 / 0.538 | | Greater-than | Input marginalization | 10/10 | 0.800 | 0.9936 [0.9300, 1.0000] | 0.530 / 0.538 | | Greater-than | LLM-based selection | 10/10 | 1.000 | 1.0000 [0.9945, 1.0000] | 0.561 / 0.818 |

  • Position-selection methods

    • MiCE-style gradients rank positions by the sensitivity of the prediction score to each input token embedding, adapting the token-selection signal from MiCE [2].
    • Input marginalization scores a position by how much the prediction changes when averaged over plausible replacements for that token [1].
    • LLM-based selection uses a large language model to choose among eligible token positions in the original input, guided by a description of the behavior under study.
  • p.s.: These works already show that selecting intervention positions is effective, so we pay more attention on the replacement and just adopt a simple variant of MiCE [2]. In our recent practice, LLM-based selection shows superior results with explicit task descriptions.

2. Changes in the revised manuscript.

  • (i) Distinguish manually provided and automatically selected positions in the method description surrounding Eq. (4), Section 2.3;
  • (ii) Add an Automatic Edit-Site Selection discussion in Related Work;
  • (iii) Report automatic-position circuit localization alongside the manual-position comparison in the main experimental results;
  • (iv) Add Appendix B.14, including selector definitions, experimental protocols, circuit-localization results, and position-agreement measurements (Tables 21–22 in the revised PDF).

References

[1] Kim et al. (EMNLP 2020). Interpretation of NLP Models through Input Marginalization.

[2] Ross, Marasović, and Peters (Findings of ACL 2021). Explaining NLP Models via Minimal Contrastive Editing (MiCE).

AuthorsOpenReview ↗

Response to Reviewer sows — 'W2: Statistical Significance and Interpretation of Table 2' and 'W3: Disentangling Objective Terms and Gradient Search' (part 2/3)

W2: Statistical Significance and Interpretation of Table 2

A: We agree that CPAP's AUROC on Sentiment is comparable to that of OA, and the small difference does not establish statistical significance. Yet, we believe that the overall comparison favors CPAP for two reasons.

  • First, on Sentiment, CPAP demonstrates a modest advantage over OA on other evaluation metrics, including reference recovery (8/9 vs. 7/9) and Spearman correlation (0.762 vs. 0.681).

  • Second, considering AUROC across all three tasks, as shown in Table 2, CPAP consistently achieves higher values than OA on IOI (0.968 vs. 0.921), Sentiment (0.983 vs. 0.980), and Greater-than (1.000 vs. 0.883).

Taken together, these results suggest that CPAP performs better than OA overall, although the magnitude of its advantage varies across tasks and evaluation metrics.

We appreciate the reviewer's concern and have revised our statement to clarify that it refers specifically to the observed AUROC values, rather than statistically significant superiority across all metrics. We would also welcome any further suggestions and additional discussions.

W3: Disentangling Objective Terms and Gradient Search

A: We agree that the effects of the core objective and the search implementation need to be evaluated separately. We therefore added finer-grained ablations covering both parts of Eq. (3) and the search decisions in Eqs. (4)–(9).

1. Contribution of the CPAP components. CPAP's components are designed to serve two purposes: improving intervention effectiveness and enhancing search efficiency. The effectiveness/efficiency of each component is shown below:

ComponentPurposeEvaluation metricResult
Maximize upstream activation change (Eq. 3, part 1)EffectivenessTask-score recovery (downstream bound fixed)92.60% → 98.83% (+6.23 pp)
Constrain downstream residual change (Eq. 3, part 2)EffectivenessTask-score recovery (upstream maximization fixed)93.83% → 98.83% (+5.00 pp)
Restrict candidate vocabularyEfficiencyMean search time per input19.94 s → 5.01 s (−74.9%)
Check prediction changes before selectionValidity / EfficiencyMean search time per input19.94 s → 5.00 s (−74.9%)
Gradient-guided proposalsEfficiencyMean search time per input19.94 s → 10.33 s (−48.2%)

The ablations were conducted on the Greater-than year-comparison task with Qwen3-8B.

The +6.23 pp and +5.00 pp task-score recovery gains separately support the upstream-maximization and downstream-constraint terms. In contrast, vocabulary restriction, early behavioral screening, and gradient proposals concern the cost and implementation of the search, rather than forming additional terms of the intervention objective.

2. Changes in the revised manuscript.

We replaced the broad ablation discussion in Section 3.4 with a component-level effectiveness/efficiency table (Table 5), distinguishing the two objective terms from the search heuristics. We clarified the shared selection rule for gradient and random proposals in Fig. 5(b) and the search-budget discussion.

AuthorsOpenReview ↗

Response to Reviewer sows questions (part 3/3)

Q1: Computational Cost Relative to Other Automatic Methods

A: We compared CPAP with zero ablation, Gaussian noise, resampling, and Optimal Ablation (OA) on 80 Qwen3-8B IOI inputs, evaluating 1,152 attention heads.

MethodConstruction (TFLOPs)Component scan (TFLOPs)Total (TFLOPs)Time (s)
Zero ablation0514.98514.9823.67
Gaussian noise (10 draws)194.515,149.785,344.2975.99
Resampling (10 donors)93.865,149.785,243.6486.04
Optimal Ablation (OA)910,724.91514.98911,239.8933,656.72
CPAP (full budget)84,792.97482.7985,275.763,137.16
CPAP (compact search, projected)≈13,800482.79≈14,300≈530

FLOPs are estimated from dominant matrix operations. Timings use A100-80GB GPUs in FP32, with OA fitting and CPAP search times accumulated across workers.

CPAP is more computationally expensive than direct perturbation methods (zero, Gaussian, and resampling), but requires approximately 10× less computation and time than OA. Its cost can be further reduced by optimizing the search strategies (e.g., rounds, beam width, and proposals), as illustrated by the projected compact configuration (2,048 vs. 12,563 evaluations).

Q2: “Handcrafted Corruptions” versus “Handcrafted Counterfactuals”

A: We only use the term “corruptions” in our paper, following established terminology in the activation patching literature [1–5]. More generally, “corruption” is a broader term encompassing both token replacements and other perturbations, such as Gaussian noise [1, 2], whereas “counterfactual” typically refers to constructed alternative inputs that modify task-relevant information [1, 5]. However, in mechanistic interpretability, the two terms are often used interchangeably when referring to manually constructed, task-specific contrastive inputs [1, 5]. Therefore, in the context of our work, “Handcrafted Corruptions” and “Handcrafted Counterfactuals” refer to the same inputs, with no methodological distinction.

References

[1] Zhang, F., & Nanda, N. (2024). Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. ICLR 2024. https://arxiv.org/abs/2309.16042

[2] Heimersheim, S., & Nanda, N. (2024). How to Use and Interpret Activation Patching. https://arxiv.org/abs/2404.15255

[3] Conmy, A., et al. (2023). Towards Automated Circuit Discovery for Mechanistic Interpretability. NeurIPS 2023. https://arxiv.org/abs/2304.14997

[4] Syed, A., et al. (2024). Attribution Patching Outperforms Automated Circuit Discovery. BlackboxNLP 2024. https://aclanthology.org/2024.blackboxnlp-1.25/

[5] Miao, Y., & Kan, M.-Y. (2025). Discursive Circuits: How Do Language Models Understand Discourse Relations? EMNLP 2025. https://aclanthology.org/2025.emnlp-main.1657/

Q3: Motivation and Limits of the 32-Nearest-Neighbor Probe

A: The motivation is to select perturbation candidates that are similar to, but distinct from, the original token. In our evaluated tasks, handcrafted corruptions typically preserve the token type while changing its task-relevant content (e.g., replacing names with other names). We therefore adopted the 32 nearest neighbors as a simple and computationally efficient heuristic, providing a compact candidate pool that is likely to include such type-consistent alternatives.

We agree that this restriction may miss informative positions when meaningful counterfactuals require semantically dissimilar replacements. In such cases, the candidate pool can be expanded or diversified accordingly. Moreover, our recent experiments suggest that LLM-based position selection guided by task descriptions provides a promising alternative that does not rely on this fixed nearest-neighbor restriction. We will clarify this motivation and limitation and include the additional results in the revised manuscript.

Reviewer_jMBGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper focuses on the activation patching problem in mechanistic interpretability. Traditional activation patching methods typically rely on carefully handcrafted corrupted inputs. However, constructing such inputs can be challenging for complex tasks, especially when answer-related evidence is scattered across the input and multiple references need to be modified consistently. To solve this problem, this paper proposes a method called Context-Preserving Activation Perburtation (CPAP), which automatically constructs activation perturbations without requiring handcrafted corrupted inputs. Specifically, CPAP maximizes changes in upstream activations while constraining changes in downstream residual representations to preserve the surrounding computational context. To reduce computational costs, the authors restrict the search space to local token replacements and employ gradient-guided discrete search to identify promising candidate tokens. Extensive experiments across multiple models and tasks demonstrate that CPAP achieves component localization results that closely align with those obtained using handcrafted corrupted-input patching. Overall, this paper introduces an automated activation perturbation framework that aims to reduce the reliance on manually designed corrupted inputs in mechanistic interpretability.

Critical strengths
  1. The problem is well-motivated, and the proposed method is straightforward and intuitive. Traditional activation patching heavily relies on manually constructed corrupted inputs, which can be challenging to design for complex tasks. This paper addresses this practical limitation by introducing an automated activation perturbation method. The proposed optimization objective is simple and intuitive, aiming to maximize upstream activation changes while constraining downstream residual changes to preserve the surrounding computational context.
  2. The experimental evaluation is extensive and comprehensive. The authors evaluate the proposed method across multiple models of different scales, including GPT-2 Small, OLMo-2-1B, and Qwen3-8B, as well as various tasks such as IOI, Greater-than, Acronym, and Months. The paper also compares CPAP against several established baselines, including Zero Ablation, Gaussian Patching, Optimal Ablation, and Resampling Ablation.
Critical weaknesses
  1. The downstream residual preservation lacks enough theory guarantee. The proposed method uses the similarity of downstream residual representations as a proxy for preserving the surrounding computational context. However, the paper does not provide sufficient evidence that maintaining residual similarity actually preserves task-irrelevant semantic information or other model capabilities.
  2. The main experiments still rely on handcrafted corruption positions, which weakens the claim of a fully automated pipeline. Although the proposed method aims to eliminate the need for manually constructed corrupted inputs, the main experiments still use the token positions identified by existing handcrafted corruption strategies.
Questions for authors
  1. Could the authors provide further theoretical justification or empirical evidence showing that preserving downstream residual similarity effectively maintains task-irrelevant semantic information?
  2. Could the authors clarify how CPAP performs when the editable token positions are selected entirely automatically, without relying on handcrafted corruption strategies? It would be helpful to provide a systematic comparison between handcrafted and automatically selected positions across the main benchmarks to better support the claim of an automated pipeline.
3 comments
AuthorsOpenReview ↗

Response to Reviewer jMBG — W1 and Q1: Motivation and Evidence for the Downstream Residual Constraint (part 1/2)

We appreciate your concern and provide the theoretical and empirical evidence below:

1. Theoretical motivation.

Through a theoretical analysis of a toy nonlinear neural network, Grant et al. [1] demonstrate that off-manifold patching activations can induce substantial deviations in residual-stream representations, potentially activating unintended computational circuits and causing undesirable model behavioral changes. These findings motivate our use of downstream residual constraints to limit such unsuitable effects on subsequent computations.

2. Preliminary experimental evidence. Our preliminary experiments, we found that unsuitable patching methods can introduce substantial downstream residual changes on task-irrelevant (spurious) heads, potentially leading to false localization results. As shown in the table, zero ablation produced considerably larger residual changes (3.67–35.75%) than handcrafted patching (0.05–3.15%), while Gaussian patching also caused larger deviations. These observations further motivated our use of downstream residual constraints to limit such disruptive effects.

Values represent mean relative L2 changes in downstream residual representations.

TaskModelHeadPromptsReceiving blockHandcrafted patchZero ablationGaussian patchCPAP
Greater-thanGPT-2 Small1.37080.05%4.19%0.35%0.04%
Greater-thanGPT-2 Small6.67080.80%3.67%2.68%0.67%
Greater-thanOLMo-2-1B8.146480.94%4.03%3.76%0.78%
MonthsOLMo-2-1B13.1064133.15%35.75%2.99%3.21%

3. Additional Supporting Evidence

Similar findings have been reported across broader deep learning research. During inference, unintended changes in internal features have been associated with unstable predictions and adversarial vulnerability [2, 3]. During fine-tuning, representation degradation can impair generalization [4, 5], while abrupt representation changes in continual learning can contribute to catastrophic forgetting [6]. More directly, abnormal spectral dynamics of Transformer residual streams have been shown to predict training instability [7]. Together, these studies suggest that unintended changes in internal representations can reflect undesirable computational behavior, further motivating CPAP's downstream residual constraint.

4. Changes in the revised manuscript.

We rewrote the motivation at the start of Section 2.2 to connect representational divergence caused by causal interventions to the use of selected downstream residual checkpoints. We also added Appendix B.16 (Downstream Representation Diagnostics), which (i) includes the related paper mentioned above; (ii) reports the four task-irrelevant-head comparisons in Table 25 In the discussion, we explicitly retain the limitation that residual proximity does not prove complete semantic separation.

References

[1] Grant et al. (2026). Addressing Divergent Representations from Causal Interventions on Neural Networks. ICLR 2026. https://arxiv.org/abs/2511.04638

[2] Zheng et al. (2016). Improving the Robustness of Deep Neural Networks via Stability Training. CVPR 2016. https://openaccess.thecvf.com/content_cvpr_2016/html/Zheng_Improving_the_Robustness_CVPR_2016_paper.html

[3] Xie et al. (2019). Feature Denoising for Improving Adversarial Robustness. CVPR 2019. https://openaccess.thecvf.com/content_CVPR_2019/html/Xie_Feature_Denoising_for_Improving_Adversarial_Robustness_CVPR_2019_paper.html

[4] Aghajanyan et al. (2021). Better Fine-Tuning by Reducing Representational Collapse. ICLR 2021. https://arxiv.org/abs/2008.03156

[5] Kumar et al. (2022). Fine-Tuning Can Distort Pretrained Features and Underperform Out-of-Distribution. ICLR 2022. https://arxiv.org/abs/2202.10054

[6] Caccia et al. (2022). New Insights on Reducing Abrupt Representation Change in Online Continual Learning. ICLR 2022. https://arxiv.org/abs/2203.03798

[7] Kim et al. (2026). Residual Koopman Spectral Profiling for Predicting and Preventing Transformer Training Instability. UAI 2026. https://proceedings.mlr.press/v337/kim26d.html

AuthorsOpenReview ↗

Response to Reviewer jMBG — W2 and Q2: Use of Handcrafted Edit Positions in the Main Experiments (part 2/2)

Thank you for noting this point. We agree and have integrated automatic token position search method [1-2] with CPAP. The additional experiments and detailed results are as below:

1. Experiments with automatic token-position selection

  • Conclusion: CPAP remains effective integrated with automatically selected positions.

  • Result Table: | Task | Edit positions used by CPAP | Recovered | Precision | AUROC [95% CI] | ρ / J | |---|---|---:|---:|---:|---:| | IOI | Manual (paper) | 21/26 | 0.808 | 0.968 [0.937, 0.981] | 0.563 / 0.631 | | IOI | MiCE-style gradients | 19/26 | 0.760 | 0.9555 [0.9210, 0.9782] | 0.331 / 0.510 | | IOI | Input marginalization | 15/26 | 0.577 | 0.9133 [0.8590, 0.9405] | 0.186 / 0.370 | | IOI | LLM-based selection | 20/26 | 0.800 | 0.9648 [0.9273, 0.9818] | 0.500 / 0.606 | | Sentiment | Manual (paper) | 8/9 | 0.889 | 0.983 [0.976, 0.986] | 0.762 / 0.603 | | Sentiment | MiCE-style gradients | 8/9 | 1.000 | 0.9893 [0.9827, 0.9918] | 0.378 / 0.603 | | Sentiment | Input marginalization | 8/9 | 0.889 | 0.9868 [0.9770, 0.9885] | 0.359 / 0.603 | | Sentiment | LLM-based selection | 8/9 | 0.889 | 0.9802 [0.9778, 0.9860] | 0.729 / 0.603 | | Greater-than | Manual (paper) | 10/10 | 1.000 | 1.000 [0.993, 1.000] | 0.561 / 0.818 | | Greater-than | MiCE-style gradients | 10/10 | 0.800 | 0.9955 [0.9763, 0.9991] | 0.494 / 0.538 | | Greater-than | Input marginalization | 10/10 | 0.800 | 0.9936 [0.9300, 1.0000] | 0.530 / 0.538 | | Greater-than | LLM-based selection | 10/10 | 1.000 | 1.0000 [0.9945, 1.0000] | 0.561 / 0.818 |

  • Position-selection methods

    • MiCE-style gradients rank positions by the sensitivity of the prediction score to each input token embedding, adapting the token-selection signal from MiCE [2].
    • Input marginalization scores a position by how much the prediction changes when averaged over plausible replacements for that token [1].
    • LLM-based selection uses a large language model to choose among eligible token positions in the original input, guided by a description of the behavior under study.
  • p.s.: These works already show that selecting intervention positions is effective, so we pay more attention on the replacement and just adopt a simple variant of MiCE [2]. In our recent practice, LLM-based selection shows superior results with explicit task descriptions.

2. Changes in the revised manuscript.

  • (i) Distinguish manually provided and automatically selected positions in the method description surrounding Eq. (4), Section 2.3;
  • (ii) Add an Automatic Edit-Site Selection discussion in Related Work;
  • (iii) Report automatic-position circuit localization alongside the manual-position comparison in the main experimental results;
  • (iv) Add Appendix B.14, including selector definitions, experimental protocols, circuit-localization results, and position-agreement measurements (Tables 21–22 in the revised PDF).

References

[1] Kim et al. (EMNLP 2020). Interpretation of NLP Models through Input Marginalization.

[2] Ross, Marasović, and Peters (Findings of ACL 2021). Explaining NLP Models via Minimal Contrastive Editing (MiCE).

Thanks for the additional experimental results. I think they address most of my concerns. Since I had already given the paper a positive score, I will keep my overall rating unchanged, but I am increasing both the Informativeness and Soundness scores to 3.

Reviewer_puJDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Context-Preserving Activation Perturbation (CPAP), a way to construct perturbed activations for activation patching without handcrafting a corrupted input. The objective is to change an upstream module's activation as much as possible while keeping selected downstream residual representations close to their clean values, as a proxy for preserving the surrounding context. In practice the search runs over inputs. Tokens at a set of editable positions are replaced by embedding neighbors. A beam search then selects the candidate with the largest aggregate head change among those whose residual change stays below a per-input bound. The candidate must also change the model's prediction. The activations of this single donor input are then patched in one component at a time.

CPAP is compared with handcrafted corrupted-input patching, zero ablation, Gaussian noise, Optimal Ablation and resampling on IOI, Greater-than, Acronym and Months with GPT-2 Small, OLMo-2-1B and Qwen3-8B-Base. It has the highest Spearman agreement with handcrafted patching in 8 of 12 settings. It also has the highest AUROC among the automatic methods for recovering the published GPT-2 IOI, sentiment and Greater-than circuits, although on IOI and sentiment its interval overlaps with that of the next-best method. An exploratory second part uses CPAP with an automatic position search to find ACDC-style circuits in OLMo for BBH dance-partner tracking and GSM8K continuations. The paper also runs IOI and Acronym on Qwen3.5-27B with a single hand-chosen edit position.

Critical strengths

The main strength is that CPAP improves on standard automatic perturbations in most settings when the positions to corrupt are known. In Spearman agreement with handcrafted patching it outperforms zero ablation and Optimal Ablation in every setting and Gaussian noise in all but one. The paired comparison with resampling (Table 6) also favors CPAP on Greater-than and Months in all three models. It recovers the published Greater-than circuit perfectly, ahead of all the other automatic methods. This is useful because someone who knows which tokens carry the information no longer has to design replacement tokens by hand. I also want to highlight the well done evaluation. Each method is scored against the handcrafted reference on the same observations, and the circuit comparison uses common inputs. The baselines include strong ones (Optimal Ablation and resampling), and the GPT-2 results are checked against published circuits. The paper also states its limits. For example, it reports that gradient guidance does not help, describes its validation examples as previously inspected and says that its LongBench case was selected retrospectively. This makes me trust the reported numbers.

Critical weaknesses

The first weakness is that the experiments with a reference only automate the choice of replacement tokens. In every comparison against a reference (Tables 1, 2 and 4), the editable positions are those changed by the handcrafted corruption (lines 241-243) or chosen by hand. Yet finding what to edit is a large part of corruption design in the long-context settings that motivate the paper, where dispersed evidence "requires locating all relevant information across documents and consistently editing every mention" (lines 49-50). The automatic position search is only used in the exploratory studies of Section 3.3, which have no reference, and the only LongBench result is one retrospectively selected input where the search edited the prompt marker "Answer". The Qwen3.5-27B run also uses a hand-chosen position. So the evidence supports a narrower claim than working "without handcrafted corrupted inputs". The paper states most of these limits in the appendices, so this can be fixed by narrowing the abstract and introduction and is not on its own a reason for rejection.

The main reason my recommendation is not higher, is that it is unclear whether the context-preserving objective is what makes CPAP work. CPAP differs from the baselines in several ways that do not come from the objective. It substitutes real tokens at the handcrafted positions, edits only those positions and keeps only candidates that change the prediction, while resampling donors are not filtered (lines 891-892). Gradient guidance also adds little, since gradient and random search end up with the same tokens for almost all IOI and "greater-than" prompts at the largest search budget in the GPT-2 ablation. Any advantage over a random neighbor substitution must then come from the selection rule, but the paper does not include this control (a random nearest-neighbor substitution that passes the same behavioral check). It would be close to an automated version of the symmetric token replacement recommended by Zhang & Nanda (2024). The bound ablation in Table 11 gives some support, but the agreement does not change consistently with the bound, and the ablation only uses single-position edits on a few dozen prompts without intervals.

Questions for authors
  1. In the BBH and GSM8K studies of Section 3.3, which token positions did the automatic search select, and are these the tokens that carry the information the prediction depends on?
  2. How does CPAP compare, with paired intervals, to a random admissible substitution with the same positions, neighborhood and behavioral check, both with and without the residual bound?
  3. How does CPAP perform on the Table 1 tasks and the GPT-2 circuits when the editable positions come from the automatic position search instead of the handcrafted corruption? How often does the search pick the positions that the handcrafted corruption edits?
5 comments
AuthorsOpenReview ↗

Response to Reviewer puJD — W1: Scope of Automation and Automatic Edit-Position Selection (part 1/3)

Thanks for your valuable suggestion. We agree that involving the automatic token selection makes the overall framework more complete and coherent.

Originally, we decompose activation patching into two relatively independent steps: selecting intervention positions and determining replacement activations. Our work primarily focuses on the latter, as automatic position selection has already been explored in prior studies [1–3]. These approaches can be integrated directly with CPAP. We conduct additional experiments and detailed results are as below:

1. Experiments with automatic token-position selection

  • Conclusion: CPAP remains effective integrated with automatically selected positions.

  • Result Table: | Task | Edit positions used by CPAP | Recovered | Precision | AUROC [95% CI] | ρ / J | |---|---|---:|---:|---:|---:| | IOI | Manual (paper) | 21/26 | 0.808 | 0.968 [0.937, 0.981] | 0.563 / 0.631 | | IOI | MiCE-style gradients | 19/26 | 0.760 | 0.9555 [0.9210, 0.9782] | 0.331 / 0.510 | | IOI | Input marginalization | 15/26 | 0.577 | 0.9133 [0.8590, 0.9405] | 0.186 / 0.370 | | IOI | LLM-based selection | 20/26 | 0.800 | 0.9648 [0.9273, 0.9818] | 0.500 / 0.606 | | Sentiment | Manual (paper) | 8/9 | 0.889 | 0.983 [0.976, 0.986] | 0.762 / 0.603 | | Sentiment | MiCE-style gradients | 8/9 | 1.000 | 0.9893 [0.9827, 0.9918] | 0.378 / 0.603 | | Sentiment | Input marginalization | 8/9 | 0.889 | 0.9868 [0.9770, 0.9885] | 0.359 / 0.603 | | Sentiment | LLM-based selection | 8/9 | 0.889 | 0.9802 [0.9778, 0.9860] | 0.729 / 0.603 | | Greater-than | Manual (paper) | 10/10 | 1.000 | 1.000 [0.993, 1.000] | 0.561 / 0.818 | | Greater-than | MiCE-style gradients | 10/10 | 0.800 | 0.9955 [0.9763, 0.9991] | 0.494 / 0.538 | | Greater-than | Input marginalization | 10/10 | 0.800 | 0.9936 [0.9300, 1.0000] | 0.530 / 0.538 | | Greater-than | LLM-based selection | 10/10 | 1.000 | 1.0000 [0.9945, 1.0000] | 0.561 / 0.818 |

  • Position-selection methods

    • MiCE-style gradients rank positions by the sensitivity of the prediction score to each input token embedding, adapting the token-selection signal from MiCE [2].
    • Input marginalization scores a position by how much the prediction changes when averaged over plausible replacements for that token [1].
    • LLM-based selection uses a large language model to choose among eligible token positions in the original input, guided by a description of the behavior under study.
  • p.s.: These works already show that selecting intervention positions is effective, so we pay more attention on the replacement and just adopt a simple variant of MiCE [2]. In our recent practice, LLM-based selection shows superior results with explicit task descriptions.

2. Changes in the revised manuscript.

  • (i) Distinguish manually provided and automatically selected positions in the method description surrounding Eq. (4), Section 2.3;
  • (ii) Add an Automatic Edit-Site Selection discussion in Related Work;
  • (iii) Report automatic-position circuit localization alongside the manual-position comparison in the main experimental results;
  • (iv) Add Appendix B.14, including selector definitions, experimental protocols, circuit-localization results, and position-agreement measurements (Tables 21–22 in the revised PDF).

References

[1] Kim et al. (EMNLP 2020). Interpretation of NLP Models through Input Marginalization.

[2] Ross, Marasović, and Peters (Findings of ACL 2021). Explaining NLP Models via Minimal Contrastive Editing (MiCE).

[3] Liu et al. (2026). Jacobian Scopes: A Unified Geometric Framework for Token-Level LLM Attribution.

AuthorsOpenReview ↗

Response to Reviewer puJD — W2: Contribution of the Context-Preserving Objective (part 2/3)

1. Effectiveness and Ablation of CPAP Components

CPAP's components are designed to serve two purposes: improving intervention effectiveness and enhancing search efficiency. The effectiveness/efficiency of each component is shown below:

ComponentPurposeEvaluation metricResult
Maximize upstream activation change (Eq. 3, part 1)EffectivenessTask-score recovery (downstream bound fixed)92.60% → 98.83% (+6.23 pp)
Constrain downstream residual change (Eq. 3, part 2)EffectivenessTask-score recovery (upstream maximization fixed)93.83% → 98.83% (+5.00 pp)
Restrict candidate vocabularyEfficiencyMean search time per input19.94 s → 5.01 s (−74.9%)
Check prediction changes before selectionValidity / EfficiencyMean search time per input19.94 s → 5.00 s (−74.9%)
Gradient-guided proposalsEfficiencyMean search time per input19.94 s → 10.33 s (−48.2%)

Ablations were conducted on the Greater-than year-comparison task and Qwen3-8B.

The +6.23 pp and +5.00 pp improvements in task-score recovery demonstrate that both terms in Eq. (3) contribute to intervention effectiveness. We guess all your confusions may originate from the term "random search" use in the paper. Therefore, we clarify this distinction in detail below.

2. Clarification of ''Random Search''

To clarify the distinction between the random search in our paper and random nearest-neighbor substitution you mentioned, we compare two concepts below:

  • Rand-NN (Random Nearest-Neighbor Substitution): Randomly samples neighbor tokens and selects one satisfying the prediction-change criterion.
  • Rand-CPAP (Random Search with CPAP Selection): Randomly samples neighbor tokens and selects the one maximizing upstream activation change while satisfying the prediction-change criterion and downstream residual bound.

First of all, we apologize for the misuse of "random search" in our original manuscript and will clarify in the revised manuscript. Besides, Rand-CPAP is more akin to a brute-force-style search, as it evaluates randomly sampled candidates and selects the best one under the CPAP objective. Therefore, our comparison in Fig. 5(b) was intended to examine whether gradient-guided search could achieve comparable performance to this brute-force-style search. The results show that both methods achieve similar performance.

Then, to further address the confusion, we compare CPAP with Rand-NN as a baseline and CPAP shows superior performance.

i. Case-level results: CPAP vs. Rand-NN. Although both methods satisfy the prediction-change criterion, CPAP recovers more reference circuit components.

Task and original promptCPAP replacementRand-NN replacementPrediction changeReference components recovered (CPAP vs. Rand-NN)
Sentiment · I thought this movie was flat, I criticized it. followed by Conclusion: This movie isflat → glossflat → dullBoth: terrible → great6/9 vs. 0/9
Acronym completion · The Verve Vulture Ziggurat (Verve → onductionVerve → reliesisseBoth: V → Z4/4 vs. 0/4
Greater-than · The quest lasted from the year 1175 to the year 111175 → 11301175 → 1118Both reverse the preference for an end year above 11758/10 vs. 5/10

ii. Dataset-level results: CPAP vs. Rand-NN. CPAP outperforms Rand-NN on both Greater-than and docstring completion.

TaskReference components recovered (CPAP vs. Rand-NN)AUROC (CPAP vs. Rand-NN)AUROC gain [paired 95% CI]
Greater-than (GPT-2)10/10 vs. 5–6/101.000 vs. 0.772+0.2285 [0.2185, 0.2930]
Docstring completion (four-layer attention model, five edited positions)2/2 vs. 1/2 at the final prediction position1.000 vs. 0.967+0.0333 [0.0278, 0.0333]

These results indicate that satisfying the prediction-change criterion alone is insufficient for effective activation perturbations. CPAP's upstream and downstream bound provide additional benefits beyond random nearest-neighbor substitution.

3. Changes in the revised manuscript.

  • (i) We added a component-by-component ablation table in Section 3.4 (Table 5) that separates objective effectiveness from search efficiency;
  • (ii) We added behavior-matched random-substitution comparisons, including bounded and unbounded controls, recovery statistics, AUROC, and paired confidence intervals in Appendix B.15;
  • (iii) We added explicit clarification in Fig. 5(b) and the associated search-budget discussion that gradient and random proposals share the CPAP selection rule;
AuthorsOpenReview ↗

Response to Reviewer puJD questions (part 3/3)

Q1: Token Positions Selected in BBH and GSM8K

A: The automatically selected token positions are marked with ⟦ ⟧ in the table below. In BBH, the selected tokens correspond to participant names involved in tracking dance-partner assignments. In GSM8K, they correspond to numerical values or arithmetic operators used in intermediate calculations.

DatasetCase 1Case 2Case 3Pattern
BBH dance partners… Bob is dancing with Patrick, and Claire is dancing with ⟦Sam⟧. …… Then, Claire and ⟦Alice⟧ switch partners. …… Finally, Alice and ⟦Bob⟧ switch partners. …Participant names relevant to tracking partner assignments.
GSM8K… the difference between the final prices … <<540-⟦398⟧=… it will rain … <<4⟦+⟧1=… In a 5 day work week, he responds to … <<⟦64⟧*5=Numerical values and operators involved in intermediate calculations.

Q2: Paired Comparison with Bounded and Unbounded Random Substitution

A: We use GPT-2 Greater-than as a representative example, since localization on other tasks is less sensitive to the choice of replacement tokens. The results show that the downstream residual bound is a key factor in CPAP's effectiveness. Without the bound, random substitution performs substantially worse; with the bound, it achieves comparable localization performance to CPAP.

MetricCPAP (with bound)Random (no bound)Random (with bound)
Reference components recovered10/106/109/10
AUROC1.00000.71520.9958
CPAP − Random AUROC [paired 95% CI]—+0.2848 [0.2164, 0.3112]+0.0042 [0.0003, 0.0306]

Q3: Automatic-Position Circuit Localization and Agreement with Manual Edits

A: 1. Circuit localization with automatically selected positions. Please see our response to W1 for the corresponding circuit localization results.

2. Agreement with handcrafted corruption positions. An input is counted as a hit if the automatically selected positions contain all positions edited by the handcrafted corruption, allowing additional positions to be selected.

GPT-2 taskMiCE-style gradientsInput marginalizationLLM-based selection
Indirect object identification (IOI)48/48 (100%)48/48 (100%)48/48 (100%)
Sentiment90/112 (80.4%)72/112 (64.3%)112/112 (100%)
Greater-than48/48 (100%)48/48 (100%)48/48 (100%)

Existing methods already achieve strong coverage of handcrafted edit positions, particularly on IOI and Greater-than. Yet, we find frontier LLM-based selection performs really well.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Remarks to the Area Chair

Dear Area Chair,

We sincerely thank all reviewers for their constructive feedback and discussion. We are encouraged by the positive feedback on our work (well-motivated problem, intuitive formulation, comprehensive evaluation, and practical significance) and acknowledgements of our efforts during the rebuttal. Below, we briefly summarize the rebuttal.

  • Reviewer puJD: The main concerns were the effectiveness of CPAP's core objective and automatic edit-position selection. We provided additional random-substitution controls and automatic-position experiments, which the reviewer acknowledged as helpful and addressed their main question.

  • Reviewer jMBG: The main concerns involved the motivation for downstream residual preservation and the dependence on manually selected positions. We provided additional theoretical motivation, empirical evidence, and automatic-position experiments, which the reviewer acknowledged as addressing most concerns and increasing both Informativeness and Soundness to 3.

  • Reviewer sows: The reviewer appreciated our rebuttal efforts and increased the Soundness score to 2, while retaining a concern about the precision of our experimental conclusion ("CPAP achieving reliably higher scores compared to other baselines"). We have discussed this concern and clarified our interpretation of the results in our rebuttal (see Response to Reviewer sows (part 2/3)). To further address this concern, we have refined the statement in the main text to: "Compared with existing automatic perturbation methods, CPAP achieves higher AUROC on IOI and Greater-than, while showing comparable but marginally higher AUROC on Sentiment." (Lines 372–374 in the revised manuscript).

We sincerely appreciate the AC's time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.