acceptodds
Under review as a conference paper at ICLR 2027

Parameter Patching: Causal Analysis of Learned Computations in Language Models

Abstract

Large language models increasingly solve complex reasoning and coding tasks, yet learned computations supporting these capabilities remain poorly understood. Activation-based methods, such as activation patching (AP), reveal how internal states contribute to behavior; however, they leave open how these capabilities arise from the model parameters themselves. To study this dependence directly, we introduce **Parameter Patching (PP)**, which probes components through fixed edits to their parameters. As the parameter counterpart of AP, PP modifies the learned transformation applied throughout execution, rather than substituting the activation states it produces. Across tasks of varying complexity, we find that both methods locate components with causal influence on behavior, yet the components they select serve different behavioral functions. On classical interpretability tasks such as IOI and arithmetic, the two methods indeed select heads with distinct characteristics. For instance, on IOI, PP selects heads that more strongly affect the preference between candidate names than those selected by AP. This divergence matters more on complex reasoning and coding tasks, where PP selections generally incur larger accuracy losses under matched knockouts, indicating that they identify components on which these capabilities more strongly depend. Finally, PP selections also translate into practical gains. Fine-tuning PP-selected heads yields larger improvements than fine-tuning AP-selected ones on both mathematical reasoning and agentic coding tasks, and the same heads support effective activation steering. Together, these results show that intervening on learned transformations reveals behavioral dependencies complementary to activation-based analysis and identifies useful targets for improving model capabilities.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Transformers, Language Models, Activation Patching

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_77PkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an alternative to Activation Patching (AP), namely Parameter Patching (PP). Instead of intervening on the resulting activations in a forward pass of the model, PP applies a modification to the learned parameters of the model. The authors compare the two on established tasks like IOI and arithmetic, but also coding and reasoning tasks. They identify locations of AP and PP through how much individual locations "damage" the model, e.g., how much accuracy of model output is reduced if one intervenes on specific weights or activations of those. Even if both methods achieve this, they find different heads to do so. They conclude that both methods are complementary, not replacements, and that PP is better suited for finding locations for improvements in models.

Critical strengths

The idea and motivation of moving from the activation side to the parameter side to learn from and investigate the models is an interesting idea. Evaluations across model families and sizes, together with the different tasks they demonstrate broader applicability. The choice of experiments and their ordering makes sense in light of the story.

Critical weaknesses

Comparisons do not isolate source of benefit for PP

The selection process of PP and AP differs in intervention type, positional coverage, and downstream propagation all at once, but they are not looked at individually and taken together may skew how the two methods perform in comparison if not ablated carefully. Even if it is acknowledged in Appendix F as a limitation, however, a matched AP control is needed to determine which differences explain the PP results. PP acts at every position with full recomputation and AP patches prediction positions while holding later attention outputs fixed. The common knockout evaluation is a standard in circuit discovery evaluation, but does not disentangle these differences. Because of that, it remains unclear which aspect of the procedure explains the different selections. The improvement comparisons introduce another potential advantage for PP, namely, steering strengths and learning rates are chosen using PP and reused for competing selectors, while full fine-tuning receives a fixed learning rate rather than an equivalent reported search. In my opinion, these choices weaken attribution of the PP selection criterion results. An AP control with matched coverage and propagation, alongside equal per-method hyperparameter-search budgets, would strengthen that attribution.

Replacement-induced damage does not establish task-specific computation

The knockout results establish behavioral influence under a specified replacement, but not which computation the selected heads perform. Accuracy loss can reflect disruption of general answer production rather than the task’s underlying computation, a limitation the authors explicitly acknowledge. Here, as with circuit discovery methods, a faithfulness analysis could help validate the found heads. PP’s selection intervention adds ambiguity, namely, averaging other heads’ output-projection weights does not necessarily preserve aligned functions and activation magnitude, so the large damage scores may therefore reflect introducing interference as well as removing a useful contribution. The probability and error analyses provide useful additional evidence, but even the structured operand-copying example does not identify the computation of the head. Alternative controls and hypothesis-driven interventions that predict specific downstream and answer changes would strengthen the mechanistic interpretation.

Questions for authors

Clarifications

  • Why do AP and PP identify different heads? Beyond the coverage and propagation differences above, what is the intuition for their different behavioral effects, e.g., reducing total candidate-name probability versus changing preference between candidates? Which evidence distinguishes a mechanistic explanation from sensitivity to the chosen interventions?

  • How does PP connect to circuit discovery? Could PP be extended from ranking individual heads to identifying and validating their connections across layers? Does it recover components of established circuits, such as IOI, or identify additional components? Could a faithfulness test, checking whether a proposed head preserves task behavior when outside contributions are replaced, help validate this connection?

  • How robust are the improvement margins? Given that random-head training uses one mask per model, how stable are the comparisons across additional masks and training seeds? For SWE-rebench, could you report paired uncertainty estimates for the 66.7% versus 64.7% comparison, accounting for the three generations per test problem, and provide the complete training protocol?

  • What caused the missing steering entries in Tab 3? The caption states that no steering was selected during calibration. What search grid, decision rule, and calibration outcomes led to those decisions? Were no-steering decisions made independently for each selector, or did PP-based calibration determine whether all steering entries were omitted?

11 comments

1/8

Weakness 1a: Sources of PP/AP Differences.

Comparisons do not isolate source of benefit for PP The selection process of PP and AP differs in intervention type, positional coverage, and downstream propagation all at once, but they are not looked at individually and taken together may skew how the two methods perform in comparison if not ablated carefully. Even if it is acknowledged in Appendix F as a limitation, however, a matched AP control is needed to determine which differences explain the PP results. PP acts at every position with full recomputation and AP patches prediction positions while holding later attention outputs fixed. The common knockout evaluation is a standard in circuit discovery evaluation, but does not disentangle these differences. Because of that, it remains unclear which aspect of the procedure explains the different selections.

Response to Weakness 1a:

We thank the reviewer for requesting matched controls, which help separate execution choices from the observed benefits. PP retains stronger knockout effects in the matched comparison and yields practical gains through steering and training. Matched AP patches all token positions with full downstream recomputation.

Our original comparison aims to make explicit the distinction between dependence on input-defined activation states and dependence on shared learned transformations. Paired AP uses controlled input contrasts to ask specific causal questions [1] and has enabled detailed analyses of well-defined tasks such as IOI and arithmetic [2,3]. Our AP instance follows their direct head-to-output analyses. PP is instead designed to align with how models generate responses: shared parameters act repeatedly on changing contexts. A fixed parameter intervention lets us probe dependence on these transformations throughout task execution, without requiring a paired input or activation alignment across executions. This makes PP directly applicable to studying behavioral dependence in complex, free-form generation.

To determine how positional coverage and downstream propagation affect the comparison, we additionally vary these choices in AP. These ablations increase GPT-2 IOI ranking agreement while preserving positive Llama-3-8B GSM8K knockout contrasts:

AP procedureGPT-2 Spearman with PPGPT-2 Top-16 overlapLlama-3-8B PP−AP\mathrm{PP}-\mathrm{AP} damage AUC [95% interval]
Direct propagation, scoring positions.1955/16+0.2305 [0.2070, 0.2528]
Full recomputation, scoring positions.2355/16+0.1886 [0.1652, 0.2114]
Full recomputation, all positions.4907/16+0.1856 [0.1626, 0.2065]

Damage AUC is K0 accuracy minus normalized remaining-accuracy AUC over K={0,1,2,4,8,16}K=\{0,1,2,4,8,16\}; positive contrasts indicate greater PP damage. Each interval compares methods within its own run, rather than estimating how much of the gap each factor causes. Across fourteen GSM8K models, matching also increases median top-16 overlap from 2 to 5 (Appendix F.1).

To test whether these localization results can guide model improvement, we evaluate steering and pinpoint training. With independent validation selection, PP achieves higher accuracy than matched AP in 10 of 14 models for steering and 12 of 14 for training. The following examples illustrate these gains on the same 1,319 GSM8K test questions:

ModelBaseSteering: PP / matched APΔ\Delta (pp)Training: PP / matched APΔ\Delta (pp)
Llama-3.1-8B20.47%25.32% / 21.00%+4.3268.54% / 65.20%+3.34
Llama-3.2-3B12.36%17.21% / 11.90%+5.3142.00% / 39.50%+2.50
Gemma4-E4B36.92%39.20% / 38.74%+0.4558.23% / 56.25%+1.97

Δ\Delta is PP minus matched AP, computed before rounding. Full 14-model results are reported in Appendix F.4 (Table 28 for steering and Table 24 for training).

Together, these findings support PP as a practical interpretability method for complex reasoning: it identifies useful targets for model improvement while retaining an overall advantage after matching positional coverage and downstream propagation.


References:

[1] Towards Best Practices of Activation Patching in Language Models: Metrics and Methods

[2] Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small

[3] Interpreting and Improving Large Language Models in Arithmetic Calculation

2/8

Weakness 1b: Equal Tuning Opportunity.

The improvement comparisons introduce another potential advantage for PP, namely, steering strengths and learning rates are chosen using PP and reused for competing selectors, while full fine-tuning receives a fixed learning rate rather than an equivalent reported search. In my opinion, these choices weaken attribution of the PP selection criterion results. An AP control with matched coverage and propagation, alongside equal per-method hyperparameter-search budgets, would strengthen that attribution.

Response to Weakness 1b:

We thank the reviewer for recommending independent hyperparameter selection, which tests whether the gains depend on PP's shared recipe. After independent tuning, PP achieves higher GSM8K accuracy than original AP on all 14 models in the training comparison. This supports PP as a practical interpretability method for guiding model improvement.

The revised main comparison independently tunes PP, original AP, matched AP, global random and full SFT on validation data. Building on head-targeted training [1], we search {5e-5,1e-4,2e-4,4e-4} for localized training and {2.5e-6,5e-6,1e-5,2e-5} for full SFT. All methods search quarter-, half- and full-epoch checkpoints; localized methods additionally search 1% and 4% head budgets. Steering independently searches the same two head budgets and strengths {0, .001, .005, .01, .025, .05, .1, .25, .5, 1, 1.25, 1.5}.

For ease of reference, we provide the updated Table 3 below. Each method independently selects its configuration on validation data.

ModelBaseSteering RandomSteering APSteering PPTraining RandomTraining APTraining PP
Llama-2-7B2.92.93.04.228.526.530.4
Llama-3-8B19.620.119.330.465.860.162.6
Llama-3.1-8B20.520.722.025.367.357.568.5
Llama-3.2-1B1.71.91.92.010.55.812.8
Llama-3.2-3B12.412.515.017.240.930.242.0
Qwen3-0.6B54.454.355.655.562.762.664.4
Qwen3-1.7B60.060.060.061.978.879.080.4
Qwen3-4B68.371.178.878.189.490.090.3
Qwen3-8B79.079.579.579.691.292.192.6
Qwen3.5-0.8B38.538.138.436.554.347.355.3
Qwen3.5-2B59.558.057.558.174.171.174.1
Qwen3.5-4B82.983.782.483.388.388.290.1
Gemma4-E2B16.216.615.517.726.922.127.7
Gemma4-E4B36.937.538.939.253.756.758.2

Entries are zero-shot GSM8K accuracy (%) on all 1,319 test questions. AP denotes original AP; Random is the mean of three global masks. Bold marks the highest accuracy among Random, AP and PP within each application.

Under the same independent-selection protocol, PP also achieves higher accuracy than matched AP in 10 of 14 models for steering and 12 of 14 for training. Our response to Weakness 1a presents these matched-AP comparisons; full results appear in Appendix F.4 (Tables 28 and 24).

Gemma4-E4B illustrates the additional value of considering retention and update size in this single-seed study:

Gemma4-E4B methodGSM8K accuracyMMLU five-shot change from Base (pp)Trainable parameters
Base36.92%0—
Original AP56.71%−3.3015.07M
Global random, three-mask mean53.68%−3.1110.05M
Full SFT58.07%−14.674.03B
PP58.23%−2.393.28M

PP reaches a similar observed target accuracy to full SFT while retaining more MMLU performance. Parameter counts reflect selected head budgets and layer-specific widths; random values average three masks.

The independently tuned comparison complements our original evaluation under common application settings. Together, these results show that PP's advantage over original AP persists under independent tuning, supporting the value of parameter-based localization for model improvement.


References:

[1] Interpreting and Improving Large Language Models in Arithmetic Calculation

3/8

Weakness 2: Behavioral Dependence and Task Specificity.

Replacement-induced damage does not establish task-specific computation The knockout results establish behavioral influence under a specified replacement, but not which computation the selected heads perform. Accuracy loss can reflect disruption of general answer production rather than the task’s underlying computation...

Response to Weakness 2:

We thank the reviewer for distinguishing behavioral dependence from mechanistic explanation, which sharpens the interpretation of our results. Our evidence establishes that PP provides a simple yet effective way to identify task-relevant components and useful targets for improving model performance on complex tasks.

For complex tasks, useful localization can include shared supporting computations as well as task-specific ones, consistent with circuit reuse [1]. Paired AP asks focused questions through clean/counterfactual contrasts, as in IOI [2], greater-than [3] and arithmetic [4]. PP probes shared learned transformations using a reference from the same checkpoint, without constructing a separate paired input. We examine whether the resulting selections reflect task-relevant dependencies and provide useful targets for model improvement.

First, we test whether the selections depend on the task and whether target-task damage tracks general-text degradation. Holding each model's checkpoint and PP operator fixed, changing the localization task changes the selected heads.

ModelGSM8K / MBPPGSM8K / HumanEvalMBPP / HumanEval
Qwen3-4B7/167/1611/16
Llama-3-8B3/162/169/16

The scores also show little association with two parameter-distance measures. Across Qwen3-4B's 1,152 arithmetic heads, Spearman correlations are 0.041 with distance from the layer mean and 0.053 with the norm-matched replacement displacement. These observations support task-dependent selection rather than a ranking determined simply by distance from the reference.

The off-task control uses the same frozen K16 head sets and activation-mean replacement banks as GSM8K, evaluating 100,000 WikiText-103 tokens per model. Matched AP uses all positions and full recomputation. Positive entries indicate greater target-task damage or less general-text damage relative to matched AP; brackets give paired 95% intervals:

ModelAdditional GSM8K accuracy loss under PP (pp)WikiText NLL reduction relative to matched AP (nats/token)
Qwen3-8B+21.67 [18.89, 24.54]+0.1160 [0.1065, 0.1275]
Llama-3.1-8B+38.42 [34.67, 42.17]−0.3702 [−0.3918, −0.3543]

PP causes greater target-task damage with less general-text degradation on Qwen3-8B, while Llama-3.1-8B shows greater effects on both. Thus, greater target-task damage need not imply greater general-text degradation (Appendix F.2).

Common activation-level knockouts compare the selected sets under the same replacement. We then test whether these selections guide effective steering and pinpoint training on GSM8K, evaluating practical utility independently of localization [5]:

Llama-3-8B evaluationBasePPMatched APPP−AP\mathrm{PP}-\mathrm{AP}
Common-knockout damage AUC, K0–16—0.46350.2779+0.1856 [0.1626, 0.2065]
steering accuracy19.64%30.40%20.55%+9.86 pp
training accuracy19.64%62.62%62.40%+0.23 pp

Rows are separate evaluations; steering and training independently select configurations and use all 1,319 GSM8K test questions (Appendix F.4, Tables 28 and 24). Higher normalized knockout AUC means greater damage; brackets give its 95% interval.

The behavioral analysis in Question 1 distinguishes name preference from name-probability support and identifies a numerical-output dependency through head preservation. Question 2 connects selected heads to established IOI components while separating those findings from the proposed circuit-faithfulness test. These analyses characterize what the selections support without treating damage alone as a mechanistic explanation (Appendices F.1–F.2).

Together, these controls support PP as a complementary approach to locating task-relevant transformations and guiding model improvement, while the behavioral analyses clarify the dependencies supported by the selected components.


References:

[1] Circuit Component Reuse Across Tasks in Transformer Language Models

[2] Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small

[3] How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model

[4] Interpreting and Improving Large Language Models in Arithmetic Calculation

[5] Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models

Reviewer_8JSpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies parameter patching, which is proposed as an alternative to activation patching as a tool for mechanistic interpretability of LLMs. They probe model components through fixed edits to parameters. This allows them to modify the learned transformation expressed through the models parameters space throughout the generation process. They find that the heads found through activation patching and parameter patching serve different functions. Using tasks such as IOI, arithmetic, GSM8K and HumanEval, they find that ablating PP selected heads generally causes larger accuracy losses than ablating AP selected heads, especially on the more complex tasks. They also show that fine-tuning only the PP selected heads improves GSM8K accuracy more than fine-tuning AP selected heads in most models.

Critical strengths
  1. Comprehensive evaluation across models. The work compares PP and AP across 15 models from the GPT-2, Llama, Qwen3, Qwen3.5 and Gemma4 families, on both controlled tasks like IOI and arithmetic and on generation tasks like GSM8K and HumanEval. These are compared against layer matched random controls with paired bootstrap intervals, which is the right approach to conduct such experiments.

  2. More practical localization procedure for free-form generation. Since PP does not need a paired corrupted input or an alignment of activations across executions, it is arguably much easier to apply to multi-token generation and agentic settings, where constructing clean and corrupted pairs is difficult. The training results, where training only the top 16 PP selected output projections outperforms the AP selected and random controls in most models at a fraction of the cost of full SFT, suggest that PP could be a cheap heuristic for choosing which parameters to adapt. I think this is the most promising part of the work.

Critical weaknesses
  1. The comparison between PP and AP is confounded and the evaluation favours PP by design The key claims in the paper are around the impact of intervention and probing on parameters compared to activation. However, in the provided experiments, PP and AP differ along three other aspects.

    • PP edits a head at every position, but AP only substitutes activations at the answer prediction position (or at the continuation positions for generation tasks). This means PP accumulates a head’s effect over every prompt and generated token, so heads which contribute only a little at many positions can appear highly important under PP, but AP might miss heads whose contribution happens earlier in the sequence and favour heads which act directly at the answer.
    • PP recomputes everything downstream of the edit, but AP holds the later attention outputs fixed. I believe that the results change due to this, since it allows PP to capture both the direct and indirect effects of a head, including how later layers amplify or compensate for the change. AP mostly measures a head’s direct effect, so heads whose influence is routed through later layers would be ranked lower by AP.
    • The replacement process itself is quite different. Using norm matched mean of the other heads output matrices for PP changes the kind of information being carried through by the head compared to the corrupt input activation based replacement for AP. . The corrupted input only removes the information which differs between the clean and corrupted prompts, such as the identity of a name, so AP tends to find heads carrying task specific information, while replacing the whole transformation with a layer mean removes everything the head does and can push its output off distribution, so PP may rank heads by how disruptive the replacement is rather than by how relevant they are to the task.

This becomes more of a problem because the common ablation used to compare both selections is a mean activation replacement applied at every prompt and generated position with full recomputation. This means that the heads are evaluated in the same way PP selects them, but not in the way AP selects them. This makes main finding that PP selections "generally incur larger accuracy losses under matched knockouts" is partly expected by design, and does not by itself show that PP finds components the capabilities depend on more strongly.

To support the claims, I believe the work needs a few additional baselines. The most important is AP applied at every position with full recomputation, since this would differ from PP only in replacement while keeping the rest of the intervention process the same. It would also help to rank heads directly by the evaluation ablation itself, which would show how close each method comes to selecting the heads that this evaluation finds most damaging. Zero ablation of each head is another baseline which should be used, since setting W’ = 0 is itself a parameter edit and has been actively used to score head importance (https://arxiv.org/abs/1905.10650).

  1. Larger damage under knockout is taken as capability specific dependence, without controlling for general disruption The PP edit replaces a head's output matrix with the mean of the other heads' matrices in the same layer, which the authors themselves acknowledge "need not represent aligned semantic functions". I would expect this to score heads whose output matrices are furthest from their layer mean as the most important, regardless of the task being studied. I believe several results in the paper suggest that PP edits often break the model in ways which are not specific to the capability at all. On Llama-2-7B arithmetic, knocking out a single PP head gives zero accuracy with 59 of 77 outputs being non-numeric, and on MBPP to HumanEval transfer at K = 16, PP knockouts produce unparseable Python in 52 of 89 Qwen3-4B cases compared with 6 for AP. Without an off-task control, such as perplexity on general text or accuracy on an unrelated task under the same knockouts, I don't think the work can separate dependence on the capability from general disruption. Only the norm matched mean is used as the reference in the main results, so I would also like to see how sensitive the PP rankings are to this choice.

The experiments which use the selected heads to improve the model support the practical claims of the paper but also have issues. The steering strengths and the training learning rates were selected using PP and shared across selectors, which gives PP a big tuning advantage over AP. I think a lot of the gains can be attributed directly to this criteria for training. Steering also only improves over AP and random in a few models, and is the same or negative in several others such as Llama-3.2-3B, Qwen3-8B and Gemma4 models, so I claiming PP "support effective activation steering" is quite a strech. The SWE-rebench result is reported from a single model on only 100 problems with 3 generations each and no intervals, and full SFT itself only improves on the base model by 0.7 points, so I am not sure how much can be concluded from it.

Questions for authors
  • Could you compare PP against AP applied at all positions with full downstream recomputation, so that the two methods only differ in what is being replaced and not in where or how the intervention propagates?
  • Could you include rankings from the evaluation knockout itself (all position mean ablation) and from zero ablation of each head? If the PP selections largely overlap with these, I believe the contribution would be better described as an evaluation of all position head ablation than as a new parameter level method.
  • Could you report an off-task control, such as WikiText perplexity or accuracy on an unrelated task, under the same PP and AP knockouts, to show that the larger damage from PP is specific to the capabilities being studied?
  • Could the steering strengths and learning rates be tuned separately for each selector, or on a criterion that does not favour either method?
  • Please report intervals for the SWE-rebench comparison, and ideally extend it beyond a single model]
  • I would like the work to discuss dynamic weight grafting (https://arxiv.org/abs/2506.20746), which also swaps weights during generation and compares this directly with activation patching, as well as Constructive Circuit Amplification (https://arxiv.org/abs/2512.16914) and Circuit Analysis methods (https://arxiv.org/abs/2502.11812), which already use localized components to choose which parameters to fine-tune for math reasoning. I think the divergence between PP and AP is simply an instance of the known gap between localization and editability instead of evidence of a deeper dependence, and should be discussed accordingly.
11 comments

Part 1/8

Weakness 1a: Propagation and Position Coverage.

The comparison between PP and AP is confounded ..., in the provided experiments, PP and AP differ along three other aspects.

PP edits a head at every position, but AP only substitutes activations at the answer prediction position (or at the continuation positions for generation tasks)...

PP recomputes everything downstream of the edit, but AP holds the later attention outputs fixed...

To support the claims, I believe the work needs a few additional baselines. The most important is AP applied at every position with full recomputation, since this would differ from PP only in replacement while keeping the rest of the intervention process the same.

Question 1: Matched Propagation and Positions.

Could you compare PP against AP applied at all positions with full downstream recomputation, so that the two methods only differ in what is being replaced and not in where or how the intervention propagates?

Response to Weakness 1a & Question 1:

We thank the reviewer for proposing matched coverage and propagation, which helps clarify how execution choices affect the PP/AP comparison. PP retains generally stronger knockout effects and demonstrates practical gains in independently tuned steering and training after matching positional coverage and downstream propagation. Here, matched AP applies patches at all token positions with full downstream recomputation.

Our original comparison aims to make explicit the distinction between dependence on input-defined activation states and dependence on shared learned transformations. Paired AP uses controlled input contrasts to ask specific causal questions [1], as in IOI [2] and arithmetic [3]. Our direct-path instance follows these analyses: later attention outputs are fixed while residual and MLP paths remain live. Under fixed histories and identical replacements at scored positions, these positionwise paths make scoring-position and all-position patching equivalent for the scored outputs. PP is instead designed to align with how models generate responses: shared parameters act repeatedly on changing contexts. A fixed parameter intervention probes these transformations throughout execution without requiring paired inputs or activation alignment, making PP directly applicable to complex, free-form generation.

To determine how positional coverage and downstream propagation affect this comparison, we additionally vary these choices in AP. The ablations increase ranking agreement while retaining positive PP knockout contrasts:

AP procedureGPT-2 Spearman with PPGPT-2 Top-16 overlapLlama-3-8B PP−AP\mathrm{PP}-\mathrm{AP} damage AUC [95% interval]
Direct propagation, scoring positions.1955/16+0.2305 [0.2070, 0.2528]
Full recomputation, scoring positions.2355/16+0.1886 [0.1652, 0.2114]
Full recomputation, all positions.4907/16+0.1856 [0.1626, 0.2065]

Damage AUC is K0 accuracy minus normalized remaining-accuracy AUC over K={0,1,2,4,8,16}K=\{0,1,2,4,8,16\}; higher values mean greater damage. Each interval compares methods within its own run, rather than estimating how much of the gap each factor causes. Across fourteen GSM8K models, matching also increases median top-16 overlap from 2 to 5, while retaining distinct selections with two AP reference-input constructions (Appendix F.1).

To test whether these localization results can guide model improvement, we evaluate steering and pinpoint training with independent validation selection. PP achieves higher accuracy than matched AP in 10 of 14 models for steering and 12 of 14 for training. The following examples illustrate these gains on the same 1,319 GSM8K test questions:

ModelBaseSteering: PP / matched APΔ\Delta (pp)Training: PP / matched APΔ\Delta (pp)
Llama-3.1-8B20.47%25.32% / 21.00%+4.3268.54% / 65.20%+3.34
Llama-3.2-3B12.36%17.21% / 11.90%+5.3142.00% / 39.50%+2.50
Gemma4-E4B36.92%39.20% / 38.74%+0.4558.23% / 56.25%+1.97

Δ\Delta is PP minus matched AP, computed before rounding. Full 14-model results are reported in Appendix F.4 (Table 28 for steering and Table 24 for training).

These results show how studying shared learned transformations can identify useful intervention targets in complex tasks, supporting PP as a simple yet effective interpretability tool for practical model improvement.


References:

[1] Towards Best Practices of Activation Patching in Language Models: Metrics and Methods

[2] Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small

[3] Interpreting and Improving Large Language Models in Arithmetic Calculation

Part 2/8

Weakness 1b: Replacement and Reference Choice.

The replacement process itself is quite different. Using norm matched mean of the other heads output matrices for PP changes the kind of information being carried through by the head compared to the corrupt input activation based replacement for AP...

Zero ablation of each head is another baseline which should be used, since setting W’ = 0 is itself a parameter edit and has been actively used to score head importance (https://arxiv.org/abs/1905.10650).

Only the norm matched mean is used as the reference in the main results, so I would also like to see how sensitive the PP rankings are to this choice.

Question 2: Mean and Zero Baselines.

Could you include rankings from the evaluation knockout itself (all position mean ablation) and from zero ablation of each head? ...

Response to Weakness 1b & Question 2:

We thank the reviewer for requesting zero-ablation and reference controls, which clarify PP's scope. PP probes the shared transformations reused throughout generation to understand task dependencies and identify practical intervention targets. Norm-matched mean replacement is one instance of this approach.

Consider a linear head contribution y=Wxy=Wx, where WW is the output-write matrix and xx its input, with bias omitted. PP supplies RxRx for replacement matrix RR; paired output AP supplies WxrefWx_{\mathrm{ref}} for aligned reference input xrefx_{\mathrm{ref}}. Under matching coverage and downstream execution, with shared bias unchanged, R=0R=0 recovers established zero ablation [1]. At fixed local inputs,

\qquad \Delta y_{\mathrm{PP}}=(R-W)x.$$ An AP contrast with $W(x_{\mathrm{ref}}-x)=0$ preserves a contribution that PP can change when $(R-W)x\ne0$. Nonzero PP replaces the transformation of the current input; activation-mean ablation supplies a fixed discovery-estimated vector $\mu$. Carefully designed AP contrasts isolate focused questions [2]. Task relevance also includes shared supporting computations, consistent with circuit reuse [3]. Our behavioral results illustrate these complementary dependencies: | Evidence | Observation | | --- | --- | | Ten-model IOI panel, original PP/AP, K8 | PP affects relative name preference more in 9/10 models; AP affects total candidate-name probability more in 7/10 | | Llama-2-7B arithmetic, L1.H8 | PP rank 1, original AP 644, matched AP 180. Mean knockout changes 77/77 correct to 0/77, including 59/77 nonnumeric next-token outputs. Preserving this head while removing up to 20 subsequent PP heads retains numeric outputs on all 77 examples | These observations distinguish name selection from probability support and identify a dependency of numerical-answer production. Arithmetic's AP reference preserves operands, so its lower ranking is consistent with preserved numerical context. Independent steering and pinpoint training results (Weakness 1a) demonstrate the practical value of PP-selected targets, while off-task controls (Weakness 2a) assess general disruption. Our default achieves the highest normalized damage AUC among the tested parameter-replacement baselines in both settings. Evaluation uses held-out activation-mean knockout over $K=0,1,\ldots,16$ on baseline-correct examples; higher means greater damage. Random-head and Gaussian references are norm-matched. | Task and model | Norm-matched mean | Raw mean | Zero | Random head | Gaussian | | --- | ---: | ---: | ---: | ---: | ---: | | GPT-2 IOI | **0.4253** | 0.3912 | 0.3884 | 0.4067 | 0.2584 | | Qwen3-4B arithmetic | **0.6113** | 0.5767 | 0.5800 | 0.5174 | 0.5073 | PP's selections overlap with the discovery-only zero and activation-mean rankings to some degree, while retaining distinct head selections in both tasks. For top-16 sets, Jaccard similarity is $|A\cap B|/|A\cup B|$. | Task and model | Jaccard with zero ranking | Jaccard with activation-mean ranking | | --- | ---: | ---: | | GPT-2 IOI | 0.684 | 0.391 | | Qwen3-4B arithmetic | 0.524 | 0.524 | Weakness 1c evaluates these rankings under both knockout operators. Crossed quantization and low-rank controls further show that each edit's own ranking yields the highest observed mean damage under that edit, demonstrating edit-specific sensitivity beyond head ablation (Appendix F.3). Future work could extend PP to more targeted parameter interventions informed by the computational roles of individual components. Together, these controls and the independent application results support **PP as a simple yet effective interpretability tool for understanding task dependencies and guiding practical model improvement.** --- **References:** > [1] Are Sixteen Heads Really Better than One? > > [2] Towards Best Practices of Activation Patching in Language Models: Metrics and Methods > > [3] Circuit Component Reuse Across Tasks in Transformer Language Models

Part 3/8

Weakness 1c: Common-Knockout Evaluation.

This becomes more of a problem because the common ablation used to compare both selections is a mean activation replacement applied at every prompt and generated position with full recomputation. This means that the heads are evaluated in the same way PP selects them, but not in the way AP selects them...

It would also help to rank heads directly by the evaluation ablation itself, which would show how close each method comes to selecting the heads that this evaluation finds most damaging.

Response to Weakness 1c:

We thank the reviewer for questioning evaluator alignment and requesting direct ablation rankings. PP-selected components show behavioral relevance under matched execution and different knockout operators, and yield practical gains in independently tuned steering and training.

Cumulative knockout measures the joint effect of a selected set, following established head and circuit ablations [1,2,3]. It shares PP's coverage and recomputation, but uses a different replacement: PP applies reference matrix RR to current input xx, giving RxRx; mean activation knockout supplies a fixed discovery-estimated vector μ\mu.

First, matching AP's coverage and propagation retains PP's larger knockout effect on Llama-3-8B GSM8K. Both knockout tables report normalized damage AUC over K={0,1,2,4,8,16}K=\{0,1,2,4,8,16\}: K0 accuracy minus normalized remaining-accuracy AUC, with higher values indicating greater damage.

AP localization procedurePP repeat, damage AUC ↑AP damage AUC ↑PP minus AP [95% interval]
Original AP: direct propagation, scoring positions0.46650.2360+0.2305 [0.2070, 0.2528]
Full recomputation, scoring positions0.46500.2764+0.1886 [0.1652, 0.2114]
Matched AP: full recomputation, all positions0.46350.2779+0.1856 [0.1626, 0.2065]

PP repeats keep rankings and head sets fixed, with separately generated outcomes and paired intervals.

Second, we evaluate frozen single-head discovery rankings, including the requested mean- and zero-ablation baselines, on held-out examples under both knockout operators. These cohorts have 100% K0 accuracy; bold marks the largest damage point estimate.

Frozen discovery rankingGPT-2 IOI: mean ↑GPT-2 IOI: zero ↑Qwen3-4B arithmetic: mean ↑Qwen3-4B arithmetic: zero ↑
Norm-matched PP0.40170.80310.54400.7107
Original AP0.57170.56140.46070.4238
Matched AP0.55740.52210.47960.4527
Zero-ablation ranking0.36220.83870.57050.7429
Activation-mean ranking0.52920.54820.57860.5563

GPT-2 favors AP under mean knockout and PP under zero knockout, showing that the common mean evaluator does not uniformly favor PP. On Qwen, PP's zero-knockout contrasts have positive 95% intervals against matched AP [0.172,0.344] and the activation-mean ranking [0.092,0.215]. PP's selected dependencies thus extend beyond the original mean evaluator.

To test whether these selections guide model improvement, we evaluate steering and pinpoint training at fixed 1% and 4% head budgets, with each method independently selecting its configuration on validation. At 1%, PP has higher training accuracy than original AP in 13/14 models and matched AP in 11/14. The Llama-3-8B results below share the same 1,319 GSM8K test questions and Base:

ApplicationHead budgetBaseOriginal APMatched APPP
Steering1% (K10)19.64%20.47%19.18%28.96%
Steering4% (K40)19.64%19.33%20.55%30.40%
Training1% (K10)19.64%51.02%55.57%63.46%
Training4% (K40)19.64%60.12%62.40%62.62%

PP's 1% training gain over matched AP is 7.88 percentage points (95% paired-question interval [5.16,10.69]); their 4% accuracies are close. Appendices F.1 and F.4 provide the controls and full selection protocols.

These results connect task-dependence analysis to practical model improvement, supporting PP as a simple yet effective interpretability tool for reasoning tasks.


References:

[1] Are Sixteen Heads Really Better than One?

[2] Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small

[3] Towards Automated Circuit Discovery for Mechanistic Interpretability

Reviewer_g91wOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a method for identifying task-relevant components in a trained neural network. Specifically, the idea is to ablate a component such as an attention head in a transformer LM by replacing its weights with the mean of all other comparable components, e.g., the mean weights of all other heads in the same layer. With the component mean-ablated, one now measures its impact on task performance, with a larger performance drop being taken as evidence that this component is important for the task. Doing this for every component results in a task-relative ranking of all components. With this ranking, one can now perform analyses similar to what one would do with rankings obtained via activation patching.

Critical strengths
  1. The method enjoys several benefits compared to activation patching and hence could become a standard tool in the interpretability toolbox
Critical weaknesses

none

Questions for authors

Did you experiment with other kinds of parameter patches than same-layer mean ablation? E.g., adding noise, sampling parameters from random heads, mean of heads from all layers etc.

1 comment

part 1/1

Question 1: Parameter Reference Choice and Sensitivity.

Did you experiment with other kinds of parameter patches than same-layer mean ablation? E.g., adding noise, sampling parameters from random heads, mean of heads from all layers etc.

Response to Question 1:

We thank the reviewer for suggesting alternative parameter patches. We tested all the suggested patch types. Several patch choices recover heads also selected by default PP. In the crossed quantization and low-rank tests, each edit's own ranking yields the highest observed mean damage.

To test how the choice of parameter patch affects localization, we compare each variant's top-16 heads with our default norm-matched layer-mean PP on GPT-2 IOI:

MetricRandom headZeroCross-layerGlobal meanNoise 0.1Noise 0.3
Jaccard similarity0.778–0.8820.6840.3910.3910.103–0.1850.143–0.231
  • Jaccard: Intersection over union; ranges span three draws.
  • Random head: Another head from the same layer supplies the reference matrix.
  • Cross-layer mean: Averages same-index heads across other layers.
  • Global mean: Averages all 144 heads, including the target.
  • Reference scale: Random-head, cross-layer and global-mean references are norm-matched.
  • Noise: Additive Gaussian noise with standard deviation 0.1 or 0.3 times the original weight RMS.

Random-head and zero references retain high overlap with default PP, while noise yields more distinct selections. These overlaps show that some head selections persist beyond the default reference. The variation motivates testing whether different rankings capture sensitivity to the specific edit being applied.

To test whether each ranking identifies sensitivity to a specific parameter edit, we apply int4 quantization and rank-32 SVD approximation to independently selected top-eight head sets on the same 454 held-out IOI examples. Rank-32 retains the leading 32 singular components. Columns identify the localization operation; entries are mean decreases in the correct-versus-subject logit margin. Higher values mean greater sensitivity, and negative values indicate margin increases. Random-head and noise results average three separately evaluated selected sets.

Applied parameter editRandom headZeroCross-layerGlobal meanNoise 0.1Noise 0.3Layer meanint4Rank-32
int4 quantization0.8310.7641.3151.381−0.1430.0611.0631.4460.653
Rank-32 approximation1.0251.1500.311−0.075−0.074−0.0260.7950.5381.505

Each edit's own ranking yields the highest observed mean damage in its row. Thus, PP can locate head sets that are especially sensitive to the parameter operation being studied.

These results support PP as a general framework for studying parameter dependence, with norm-matched layer-mean replacement as a simple default. Future work could combine PP with SVD-based parameter analysis [1] and Stochastic Parameter Decomposition [2], using their finer parameter units to test behavioral contributions within a head or weight matrix.

Together, these findings support PP as a simple yet effective practical tool and a general interpretability framework for studying behavioral dependence through targeted parameter interventions. Appendix F.3 reports the complete comparisons. We thank the reviewer again for the constructive suggestions that motivated these experiments and helped clarify PP's broader potential.

References:

[1] SVD as a Fast Interpretability Method for Transformers

[2] Stochastic Parameter Decomposition

Comments on the paper

1 comment

Dear Reviewers, Area Chairs, and Program Chairs,

We thank all three reviewers for helping us clarify what makes parameter patching (PP) an effective patching framework. PP ranks heads by perturbing their weights. This approach complements widely studied activation patching (AP) on controlled tasks, guides steering and fine-tuning on complex tasks, and expands the range of interventions we can study.

The reviewers recognized the motivation for PP, its practical potential, and the breadth of our evaluation.

[Scientific motivation and potential impact]

  • Reviewer 77Pk: “... moving from the activation side to the parameter side to learn from and investigate the models is an interesting idea.”
  • Reviewer g91w: “The method enjoys several benefits compared to activation patching and hence could become a standard tool in the interpretability toolbox”

[Breadth and practical relevance]

  • Reviewer 8JSp: “Comprehensive evaluation across models.”
  • Reviewer 77Pk: “The choice of experiments and their ordering makes sense in light of the story.”
  • Reviewer 8JSp: “More practical localization procedure for free-form generation.”

During the response period, we addressed the reviewers’ questions through additional experiments and analyses.

[Downstream Validation on Complex Tasks]

  • We confirmed PP’s consistent GSM8K knockout advantage over original AP on 13/14 models and over fully matched AP on Llama-3-8B (damage AUC +0.1856).
  • We confirmed PP’s consistent superiority on complex downstream tasks. In independently tuned GSM8K training experiments, PP outperformed AP on 12/14 models and random selection on 13/14. In independently tuned GSM8K steering experiments, PP outperformed AP on 10/14 models and random selection on 12/14.
  • We showed that PP does not always disrupt general capabilities more than AP. On Qwen3-8B, for example, knocking out PP-selected heads caused greater GSM8K damage but less WikiText degradation.
  • We evaluated the retention of general capabilities after training and showed that PP achieved the best trade-off between improving target performance and retaining general capabilities.
  • We extended our experiments to SWE-bench Verified on Qwen3.8-27B to provide preliminary validation of PP’s superior performance on real-world tasks.

[Additional Interpretability Insights on Controlled Tasks]

  • We tested alternative parameter references, including cross-layer means, the global mean, and additive noise, identifying both shared and reference-specific heads.
  • We tested quantization and low-rank reconstruction directly in weight space, extending beyond standard AP and broadening the design space for future interpretability methods.
  • We showed that PP and AP identified heads with different characteristics on the controlled IOI task, with PP-selected heads more strongly affecting relative name preference and AP-selected heads more strongly reducing total name probability.
  • We manually annotated LogitLens readouts for each method’s top 16 heads across three tasks, identifying additional PP heads that exposed input operands in arithmetic and equation structure in GSM8K.
  • We distinguished default PP from zero ablation (all-head ablation) using the known IOI induction head L5.H5, which ranked 9th with default PP but 65th with Zero.
  • We recovered all four established IOI S-Inhibition heads and proposed path patching to test cross-layer connections and circuit faithfulness.

[Further Clarifications]

  • We clarified how one PP edit changes a shared transformation wherever it is used, mirroring how model weights operate during training and inference.
  • We explained why task relevance does not require task exclusivity and how selecting shared components with PP can improve the balance between target gains and retained capabilities.
  • We explained how established mean and zero knockouts can reveal different IOI dependencies and confirmed PP’s GSM8K advantage after matching AP’s coverage and propagation.
  • We tested random sparse training, a strong baseline in prior work, and showed that PP outperformed every sampled four-head mask at all three seeds on Qwen3-8B. The variability across random masks highlights the practical value of using PP for head selection.
  • We detailed the shared training protocol and problem-level evaluation for SWE-bench Verified and SWE-rebench V2.
  • We compared PP with Dynamic Weight Grafting, Constructive Circuit Amplification, and CircuitLoRA and explained how PP identifies promising targets for downstream applications.

All of the above changes have been incorporated into the revised manuscript. We thank the reviewers for helping us sharpen both the interpretation of PP and the tests of its practical value. We also thank the Area Chair for overseeing the discussion.

Best regards and thanks,

Authors of #51765

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.