- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes an alternative to Activation Patching (AP), namely Parameter Patching (PP). Instead of intervening on the resulting activations in a forward pass of the model, PP applies a modification to the learned parameters of the model. The authors compare the two on established tasks like IOI and arithmetic, but also coding and reasoning tasks. They identify locations of AP and PP through how much individual locations "damage" the model, e.g., how much accuracy of model output is reduced if one intervenes on specific weights or activations of those. Even if both methods achieve this, they find different heads to do so. They conclude that both methods are complementary, not replacements, and that PP is better suited for finding locations for improvements in models.
Critical strengths
The idea and motivation of moving from the activation side to the parameter side to learn from and investigate the models is an interesting idea. Evaluations across model families and sizes, together with the different tasks they demonstrate broader applicability. The choice of experiments and their ordering makes sense in light of the story.
Critical weaknesses
Comparisons do not isolate source of benefit for PP
The selection process of PP and AP differs in intervention type, positional coverage, and downstream propagation all at once, but they are not looked at individually and taken together may skew how the two methods perform in comparison if not ablated carefully. Even if it is acknowledged in Appendix F as a limitation, however, a matched AP control is needed to determine which differences explain the PP results. PP acts at every position with full recomputation and AP patches prediction positions while holding later attention outputs fixed. The common knockout evaluation is a standard in circuit discovery evaluation, but does not disentangle these differences. Because of that, it remains unclear which aspect of the procedure explains the different selections. The improvement comparisons introduce another potential advantage for PP, namely, steering strengths and learning rates are chosen using PP and reused for competing selectors, while full fine-tuning receives a fixed learning rate rather than an equivalent reported search. In my opinion, these choices weaken attribution of the PP selection criterion results. An AP control with matched coverage and propagation, alongside equal per-method hyperparameter-search budgets, would strengthen that attribution.
Replacement-induced damage does not establish task-specific computation
The knockout results establish behavioral influence under a specified replacement, but not which computation the selected heads perform. Accuracy loss can reflect disruption of general answer production rather than the task’s underlying computation, a limitation the authors explicitly acknowledge. Here, as with circuit discovery methods, a faithfulness analysis could help validate the found heads. PP’s selection intervention adds ambiguity, namely, averaging other heads’ output-projection weights does not necessarily preserve aligned functions and activation magnitude, so the large damage scores may therefore reflect introducing interference as well as removing a useful contribution. The probability and error analyses provide useful additional evidence, but even the structured operand-copying example does not identify the computation of the head. Alternative controls and hypothesis-driven interventions that predict specific downstream and answer changes would strengthen the mechanistic interpretation.
Questions for authors
Clarifications
-
Why do AP and PP identify different heads? Beyond the coverage and propagation differences above, what is the intuition for their different behavioral effects, e.g., reducing total candidate-name probability versus changing preference between candidates? Which evidence distinguishes a mechanistic explanation from sensitivity to the chosen interventions?
-
How does PP connect to circuit discovery? Could PP be extended from ranking individual heads to identifying and validating their connections across layers? Does it recover components of established circuits, such as IOI, or identify additional components? Could a faithfulness test, checking whether a proposed head preserves task behavior when outside contributions are replaced, help validate this connection?
-
How robust are the improvement margins? Given that random-head training uses one mask per model, how stable are the comparisons across additional masks and training seeds? For SWE-rebench, could you report paired uncertainty estimates for the 66.7% versus 64.7% comparison, accounting for the three generations per test problem, and provide the complete training protocol?
-
What caused the missing steering entries in Tab 3? The caption states that no steering was selected during calibration. What search grid, decision rule, and calibration outcomes led to those decisions? Were no-steering decisions made independently for each selector, or did PP-based calibration determine whether all steering entries were omitted?
11 comments
2/8
Weakness 1b: Equal Tuning Opportunity.
The improvement comparisons introduce another potential advantage for PP, namely, steering strengths and learning rates are chosen using PP and reused for competing selectors, while full fine-tuning receives a fixed learning rate rather than an equivalent reported search. In my opinion, these choices weaken attribution of the PP selection criterion results. An AP control with matched coverage and propagation, alongside equal per-method hyperparameter-search budgets, would strengthen that attribution.
Response to Weakness 1b:
We thank the reviewer for recommending independent hyperparameter selection, which tests whether the gains depend on PP's shared recipe. After independent tuning, PP achieves higher GSM8K accuracy than original AP on all 14 models in the training comparison. This supports PP as a practical interpretability method for guiding model improvement.
The revised main comparison independently tunes PP, original AP, matched AP, global random and full SFT on validation data. Building on head-targeted training [1], we search {5e-5,1e-4,2e-4,4e-4} for localized training and {2.5e-6,5e-6,1e-5,2e-5} for full SFT. All methods search quarter-, half- and full-epoch checkpoints; localized methods additionally search 1% and 4% head budgets. Steering independently searches the same two head budgets and strengths {0, .001, .005, .01, .025, .05, .1, .25, .5, 1, 1.25, 1.5}.
For ease of reference, we provide the updated Table 3 below. Each method independently selects its configuration on validation data.
| Model | Base | Steering Random | Steering AP | Steering PP | Training Random | Training AP | Training PP |
|---|---|---|---|---|---|---|---|
| Llama-2-7B | 2.9 | 2.9 | 3.0 | 4.2 | 28.5 | 26.5 | 30.4 |
| Llama-3-8B | 19.6 | 20.1 | 19.3 | 30.4 | 65.8 | 60.1 | 62.6 |
| Llama-3.1-8B | 20.5 | 20.7 | 22.0 | 25.3 | 67.3 | 57.5 | 68.5 |
| Llama-3.2-1B | 1.7 | 1.9 | 1.9 | 2.0 | 10.5 | 5.8 | 12.8 |
| Llama-3.2-3B | 12.4 | 12.5 | 15.0 | 17.2 | 40.9 | 30.2 | 42.0 |
| Qwen3-0.6B | 54.4 | 54.3 | 55.6 | 55.5 | 62.7 | 62.6 | 64.4 |
| Qwen3-1.7B | 60.0 | 60.0 | 60.0 | 61.9 | 78.8 | 79.0 | 80.4 |
| Qwen3-4B | 68.3 | 71.1 | 78.8 | 78.1 | 89.4 | 90.0 | 90.3 |
| Qwen3-8B | 79.0 | 79.5 | 79.5 | 79.6 | 91.2 | 92.1 | 92.6 |
| Qwen3.5-0.8B | 38.5 | 38.1 | 38.4 | 36.5 | 54.3 | 47.3 | 55.3 |
| Qwen3.5-2B | 59.5 | 58.0 | 57.5 | 58.1 | 74.1 | 71.1 | 74.1 |
| Qwen3.5-4B | 82.9 | 83.7 | 82.4 | 83.3 | 88.3 | 88.2 | 90.1 |
| Gemma4-E2B | 16.2 | 16.6 | 15.5 | 17.7 | 26.9 | 22.1 | 27.7 |
| Gemma4-E4B | 36.9 | 37.5 | 38.9 | 39.2 | 53.7 | 56.7 | 58.2 |
Entries are zero-shot GSM8K accuracy (%) on all 1,319 test questions. AP denotes original AP; Random is the mean of three global masks. Bold marks the highest accuracy among Random, AP and PP within each application.
Under the same independent-selection protocol, PP also achieves higher accuracy than matched AP in 10 of 14 models for steering and 12 of 14 for training. Our response to Weakness 1a presents these matched-AP comparisons; full results appear in Appendix F.4 (Tables 28 and 24).
Gemma4-E4B illustrates the additional value of considering retention and update size in this single-seed study:
| Gemma4-E4B method | GSM8K accuracy | MMLU five-shot change from Base (pp) | Trainable parameters |
|---|---|---|---|
| Base | 36.92% | 0 | — |
| Original AP | 56.71% | −3.30 | 15.07M |
| Global random, three-mask mean | 53.68% | −3.11 | 10.05M |
| Full SFT | 58.07% | −14.67 | 4.03B |
| PP | 58.23% | −2.39 | 3.28M |
PP reaches a similar observed target accuracy to full SFT while retaining more MMLU performance. Parameter counts reflect selected head budgets and layer-specific widths; random values average three masks.
The independently tuned comparison complements our original evaluation under common application settings. Together, these results show that PP's advantage over original AP persists under independent tuning, supporting the value of parameter-based localization for model improvement.
References:
[1] Interpreting and Improving Large Language Models in Arithmetic Calculation
3/8
Weakness 2: Behavioral Dependence and Task Specificity.
Replacement-induced damage does not establish task-specific computation The knockout results establish behavioral influence under a specified replacement, but not which computation the selected heads perform. Accuracy loss can reflect disruption of general answer production rather than the task’s underlying computation...
Response to Weakness 2:
We thank the reviewer for distinguishing behavioral dependence from mechanistic explanation, which sharpens the interpretation of our results. Our evidence establishes that PP provides a simple yet effective way to identify task-relevant components and useful targets for improving model performance on complex tasks.
For complex tasks, useful localization can include shared supporting computations as well as task-specific ones, consistent with circuit reuse [1]. Paired AP asks focused questions through clean/counterfactual contrasts, as in IOI [2], greater-than [3] and arithmetic [4]. PP probes shared learned transformations using a reference from the same checkpoint, without constructing a separate paired input. We examine whether the resulting selections reflect task-relevant dependencies and provide useful targets for model improvement.
First, we test whether the selections depend on the task and whether target-task damage tracks general-text degradation. Holding each model's checkpoint and PP operator fixed, changing the localization task changes the selected heads.
| Model | GSM8K / MBPP | GSM8K / HumanEval | MBPP / HumanEval |
|---|---|---|---|
| Qwen3-4B | 7/16 | 7/16 | 11/16 |
| Llama-3-8B | 3/16 | 2/16 | 9/16 |
The scores also show little association with two parameter-distance measures. Across Qwen3-4B's 1,152 arithmetic heads, Spearman correlations are 0.041 with distance from the layer mean and 0.053 with the norm-matched replacement displacement. These observations support task-dependent selection rather than a ranking determined simply by distance from the reference.
The off-task control uses the same frozen K16 head sets and activation-mean replacement banks as GSM8K, evaluating 100,000 WikiText-103 tokens per model. Matched AP uses all positions and full recomputation. Positive entries indicate greater target-task damage or less general-text damage relative to matched AP; brackets give paired 95% intervals:
| Model | Additional GSM8K accuracy loss under PP (pp) | WikiText NLL reduction relative to matched AP (nats/token) |
|---|---|---|
| Qwen3-8B | +21.67 [18.89, 24.54] | +0.1160 [0.1065, 0.1275] |
| Llama-3.1-8B | +38.42 [34.67, 42.17] | −0.3702 [−0.3918, −0.3543] |
PP causes greater target-task damage with less general-text degradation on Qwen3-8B, while Llama-3.1-8B shows greater effects on both. Thus, greater target-task damage need not imply greater general-text degradation (Appendix F.2).
Common activation-level knockouts compare the selected sets under the same replacement. We then test whether these selections guide effective steering and pinpoint training on GSM8K, evaluating practical utility independently of localization [5]:
| Llama-3-8B evaluation | Base | PP | Matched AP | |
|---|---|---|---|---|
| Common-knockout damage AUC, K0–16 | — | 0.4635 | 0.2779 | +0.1856 [0.1626, 0.2065] |
| steering accuracy | 19.64% | 30.40% | 20.55% | +9.86 pp |
| training accuracy | 19.64% | 62.62% | 62.40% | +0.23 pp |
Rows are separate evaluations; steering and training independently select configurations and use all 1,319 GSM8K test questions (Appendix F.4, Tables 28 and 24). Higher normalized knockout AUC means greater damage; brackets give its 95% interval.
The behavioral analysis in Question 1 distinguishes name preference from name-probability support and identifies a numerical-output dependency through head preservation. Question 2 connects selected heads to established IOI components while separating those findings from the proposed circuit-faithfulness test. These analyses characterize what the selections support without treating damage alone as a mechanistic explanation (Appendices F.1–F.2).
Together, these controls support PP as a complementary approach to locating task-relevant transformations and guiding model improvement, while the behavioral analyses clarify the dependencies supported by the selected components.
References:
[1] Circuit Component Reuse Across Tasks in Transformer Language Models
[2] Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small
[3] How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model
[4] Interpreting and Improving Large Language Models in Arithmetic Calculation
[5] Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models
1/8Weakness 1a: Sources of PP/AP Differences.
Response to Weakness 1a:
We thank the reviewer for requesting matched controls, which help separate execution choices from the observed benefits. PP retains stronger knockout effects in the matched comparison and yields practical gains through steering and training. Matched AP patches all token positions with full downstream recomputation.
Our original comparison aims to make explicit the distinction between dependence on input-defined activation states and dependence on shared learned transformations. Paired AP uses controlled input contrasts to ask specific causal questions [1] and has enabled detailed analyses of well-defined tasks such as IOI and arithmetic [2,3]. Our AP instance follows their direct head-to-output analyses. PP is instead designed to align with how models generate responses: shared parameters act repeatedly on changing contexts. A fixed parameter intervention lets us probe dependence on these transformations throughout task execution, without requiring a paired input or activation alignment across executions. This makes PP directly applicable to studying behavioral dependence in complex, free-form generation.
To determine how positional coverage and downstream propagation affect the comparison, we additionally vary these choices in AP. These ablations increase GPT-2 IOI ranking agreement while preserving positive Llama-3-8B GSM8K knockout contrasts:
Damage AUC is K0 accuracy minus normalized remaining-accuracy AUC over K={0,1,2,4,8,16}; positive contrasts indicate greater PP damage. Each interval compares methods within its own run, rather than estimating how much of the gap each factor causes. Across fourteen GSM8K models, matching also increases median top-16 overlap from 2 to 5 (Appendix F.1).
To test whether these localization results can guide model improvement, we evaluate steering and pinpoint training. With independent validation selection, PP achieves higher accuracy than matched AP in 10 of 14 models for steering and 12 of 14 for training. The following examples illustrate these gains on the same 1,319 GSM8K test questions:
Δ is PP minus matched AP, computed before rounding. Full 14-model results are reported in Appendix F.4 (Table 28 for steering and Table 24 for training).
Together, these findings support PP as a practical interpretability method for complex reasoning: it identifies useful targets for model improvement while retaining an overall advantage after matching positional coverage and downstream propagation.
References: