- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper investigates Capability Enhancement Localization (CEL), proposing that ablating a sparse set of attention heads can improve task-specific performance without fine-tuning. The paper’s authors support this proposal using both an empirical and a theoretical study:
-
Empirical Study: The authors conduct an experiment in which they take a selection of models (SmolLM3, Qwen2.5, Llama-3.2) and tasks (math capabilities: Arithmetic; code generation: MBPP; language skills: HellaSwag, BoolQ). For each model-task pair, they search, using their proposed algorithm, for a sparse set of heads to prune for better model results on the given task. Finally, they compare the pruned and intact models on test data.
-
Theoretical Explanation: The authors construct a 3-head softmax attention model with diagonal parameter matrices and 2 conflicting linear tasks over . This architecture is used to theoretically prove a proposed bias of transformers, where each feature is concentrated in only one head. Hence, pruning the head related to the conflicting feature can help one task while harming the other, if said head doesn’t process other features relevant to the first task.
Critical strengths
-
The concept of head ablation for task improvement is interesting and potentially useful. The empirical study across models and tasks, along with the use of a disjoint sample for discovery and support, strengthens the claim of improvement.
-
Rigorous mathematical derivation of feature isolation: For the proposed 3-head diagonal attention model and with small initialization, the paper proves that gradient descent forces feature localization, thereby showing theoretically cases in which test-time head ablations can improve performance on desired tasks.
Critical weaknesses
-
Skewed data splits and sample scarcity: The empirical evaluation relies on small discovery and test samples and skewed data splits. Specifically, the MBPP evaluation uses discovery and test sets with only 374 and 500 samples, respectively, resulting in unstable test generalization where performance gains fluctuate dramatically (from +0.20 to +1.80) by changing the discovery-test split. Furthermore, the Arithmetic task utilizes a highly unconventional 2:1 discovery-test ratio, allocating 2,000 and 1,000 samples for discovery and test sets, respectively. This methodological brittleness undermines the reliability of the empirical claims and weakens statistical confidence. It makes it difficult to trust the observed CEL as a robust, generalizable phenomenon rather than a result of the specific dataset construction.
-
Simplified transformer architecture in theoretical study: The authors assume a highly restrictive transformer architecture where: (1) the query, key, value, and output projection matrices are diagonal and (2) cross-attention against a ”fixed memory” () is used instead of the usual sequence self-attention. These assumptions, as the authors acknowledge, eliminate cross-coordinate interactions and thus substantially simplify the proof. On the other hand, they separate the task features in isolated dimensions, making the optimization blind to any inter-task conflicts. Consequently, the network does not intelligibly learn to ”route” conflicting capabilities to different heads to resolve interference -- the localization results directly from the random Gaussian initialization.
-
Algorithmic devolution compared to prior baselines: The head ablation algorithm is substantially less efficient than the compressed sensing (LASSO) framework from Bair et al. (2026), which the authors cite. Furthermore, the proposed algorithm is blind to synergistic head interactions, prematurely discarding head combinations that are only beneficial when ablated jointly (heads from consequent layers could behave as such). Without justifying why this search method was adopted over existing techniques, the algorithm’s contribution remains methodologically fragile.
-
Absence of general capability evaluation: The manuscript does not evaluate the ablated models on standard general language benchmarks. Hence, we cannot determine whether the observed gains trade off improved performance on a desired task with overall loss of language capabilities. This could undermine the claimed utility and safety of CEL.
Questions for authors
-
The empirical results currently rely on a small sample pool for MBPP (874 samples in total) and unconventional discovery-to-test ratios (e.g., the 2:1 ratio for Arithmetic). Could you provide additional evaluation results utilizing larger sample pools and balanced data splits to confirm that the observed CEL is a robust phenomenon rather than an artifact of sample selection?
-
Given that the manuscript cites Bair et al., which demonstrated that LASSO recovers sparse, task-critical head sets in as few as 100 to 200 evaluations while avoiding interaction blindness, why was this algorithm not adopted?
-
When the sparse head subsets are ablated to improve a target task, what is the impact on general language tasks? Do they perform comparably to the original intact model?
5 comments
Official Rebuttal to Reviewer rGLk by the authors [2/n]
B. On the Experiments (continue)
-
Q: Why use small sample pool for MBPP? Why ratio 2:1 ratio for Arithmetic.
A: We will address the points above as follows:
(1) Split sizes: for MBPP we do NOT subsample. D_{dist} is its entire official training partition (374 examples), and D_{test} is the entire official test partition (500 problems). In this case, no larger or different split is available.
For Arithmetic, the benchmark is a single pool of 20,000 examples with NO predefined split; we simply choose to match HellaSwag and BoolQ, and choose a disjoint set of 1,000 examples.
(2) Discovery-test ratio: we note that is NOT a training set, as it is only use to select which heads to ablate. The ratio therefore does not affect validity. What matters most is that the two sets are DISJOINT (as in our settings) and that is large enough for a good estimate of the true task distribution.
To remove the reviewer’s concern that CEL maybe an artifact of sample selection, we provide the results accross multiple runs, as shown in the below.
-
Q: Additional evaluation results to confirm that CEL is a robust phenomenon rather than artifact of sample selections?
A: To address this issue, we provide the mean and CIs of different models across different tasks (Arithmetic, BoolQ, and Hellaswag) in 5 runs, the mean performance with random 3-head ablations (20 runs), and the mean of best single-head (on discovery set) performance on the test set.
For MBPP, we note that the dataset has a training partition of 374 examples and a test partition of 500 examples. We take the whole training partition as and the whole test partition as , so it is hard to demonstrate the CEL results across runs. See Appendix A.2 for detailed discussion. In the below, we present the mean of random ablation (across 20 runs) and compare it to the performance gain from CEL in our single run, though.
| Task | Model | Selected subset: mean [95% CI] | SD | Random subsets | Best single head | |
|---|---|---|---|---|---|---|
| Arithmetic | Qwen2.5-3B | +2.72 [+1.91, +3.53] | 0.65 | -3.81 | +1.40 | |
| Llama-3.2-3B | +8.40 [+7.41, +9.39] | 0.79 | +0.05 | +3.66 | ||
| SmolLM3-3B | +12.82 [+12.25, +13.39] | 0.46 | -4.24 | +5.86 | ||
| BoolQ | Qwen2.5-3B | +2.07 [+1.58, +2.56] | 0.40 | -0.24 | +0.70 | |
| Llama-3.2-3B | +2.06 [+1.90, +2.22] | 0.13 | +0.15 | +1.19 | ||
| SmolLM3-3B | +2.35 [+2.06, +2.64] | 0.23 | -0.47 | +2.04 | ||
| HellaSwag | Qwen2.5-3B | +1.27 [+1.01, +1.53] | 0.21 | -0.29 | +0.32 | |
| Llama-3.2-3B | +1.65 [+1.28, +2.01] | 0.29 | -0.20 | +0.76 | ||
| SmolLM3-3B | +1.21 [+1.02, +1.39] | 0.15 | -0.29 | +0.44 | ||
| Model | Selected subset (Table 1) | Random subsets |
|---|---|---|
| Qwen2.5-3B | +1.80 | -0.78 |
| Llama-3.2-3B | +5.20 | +0.51 |
| SmolLM3-3B | +1.60 | -0.03 |
-
Q: Effect of ablating a beneficial combination of a task on other tasks
A. For this question, we run it on Arithmetic Intervention. For each model, we ablate the head combination selected for Arithmetic and evaluate the same ablated model, with no further selection, on three other benchmarks. Entries are changes in performance (pp) relative to the intact model.
| Model | Ablated heads | Arithmetic (target) | BoolQ | HellaSwag | MBPP |
|---|---|---|---|---|---|
| Qwen2.5-3B | L0H7, L0H9, L10H12 | +3.50 | +1.16 | -0.01 | -0.20 |
| Llama-3.2-3B | L8H16, L9H10, L10H11 | +9.30 | -2.05 | +0.62 | +0.20 |
| SmolLM3 | L5H13, L7H0, L13H10 | +13.10 | -0.67 | +0.10 | +0.20 |
This is what we expect. The heads are selected for one task only, so their removal can improve, degrade, or leave another task unchanged, depending on how that task relates to the removed component. Our toy model shows the same picture: tasks A and B of Theorem 4.5 are the two extreme cases (gain t^2 and damage 3t^2), a task that agrees with A on the shared feature also improves, and a task that does not use this feature is unaffected. The intervention is therefore task-specific rather than universally beneficial or harmful.
Official Rebuttal to Reviewer rGLk by the authors [3/n]
C. Additional results and experiments
Besides the questions raised by the reviewer, we also provide some additional results for the completeness of the draft.
-
In the revised draft, we restate our results for the general case of H heads: this is a generalization of the original case with three heads. All the related materials are updated accordingly.
-
We provide additional experiments validating our theoretical prediction in Section 4.5 (marked in RED).
Final Comments
Again, we thank the reviewer for thoughtful and constructive feedback. We hope that our answers address the reviewer’s concerns, and we are happy to address further questions, if any. We respectfully request that the reviewer re-evaluate the work in light of our rebuttal. Thank you, and have a good day!
References
[1] Boix-Adserà et al. Transformers learn through gradual rank increase. NeurIPS’23.
[2] Vladymyrov et al., Linear Transformers are Versatile In-Context Learners, NeurIPS’24
[3] Rosu et al., From Softmax to Score: Transformers Can Effectively Implement In-Context Denoising Steps, NeurIPS’25
[4] Pesme and Flammarion, Saddle-to-Saddle Dynamics in Diagonal Linear Networks, NeurIPS’23
[5] Ramsauer et al, Hopfield networks is all you need, ICLR’21
[6] Bair et al. Compressed Sensing for Capability Localization in Large Language Models, ICML’26
[7] Held and Yang, Shapley head pruning: Identifying and removing interference in multilingual transformers, EACL’23
[8] Jin et al. Cutting off the head ends the conflict: A mechanism for interpreting and mitigating knowledge conflicts in language models, ACL’24 Finding
Official Rebuttal to Reviewer rGLk by the authors [1/n]
We thank the reviewer for their time in reviewing our draft and for the constructive feedback. We are glad that the reviewer values our theoretical contribution. However, we are sorry that there is some misunderstanding and concern on the reviewer about your theoretical settings and the experimental designs, which we will address as follows.
A. On the problem settings
Q: on the choice of diagonal weight matrix and fixed memory banks:
A: We note that the setting of diagonal weight matrices and fixed memory banks (but importantly with learned key and value projections) legitimate theoretical simplifications and has been adopted in many prior theoretical works (see e.g., [1, 2, 3, 4, 5]). Since our work is the first one trying to provide a theoretical explanation for CEL (to the best of our knowledge), we believe that this setting is a reasonable starting point and should not be considered a critical weakness. We already made this point in the preface to Section 4. But again, we agree with the reviewer that extending the analysis to more general settings is an interesting and important question for future work.
Q: “due the the problem settings…. the localization results directly from the random Gaussian initialization.”
A: We respectfully push back on this point. Actually, our problem setting is designed to isolate the following specific question: given an available representation, how does joint training allocate its contribution across attention heads, and when does that allocation enable ablation?
By design, at initialization, every head has access to every feature, and there is no localization at this stage. We can then show that at initialization, there is no CEL: ablating any head yields minimal effect on the performance of the considered tasks. We formalized this claim in the revised draft (Section 4.2.1, Proposition 4.2 and Corollary 4.3, marked in RED). To provide more evidence for this, we provide an additional experiment (see e.g., Figure 1.a, marked in RED) showing that at initialization, there is no CEL. The above provides concrete evidence that CEL cannot result merely from Gaussian initialization, and has to emerge throughout training dynamics.
B. On the Experiments
Q: “Why not using LASSO as in [6]”:
A: That’s a good question. Actually, we did try this method for our setting, but (1) the LASSO [6] is NOT necessarily much cheaper than singleton scan for this particular problem, and (2) the results given by LASSO are unstable, and many times the suggested combinations (using the discovery set) do NOT yield improvements on the test set.
To see (1), note that in [6], given a model and a task, the objective is to find critical heads such that: (i) they are sparse, (ii) they have strong signal: ablating one head might cause a significant drop in performance, and (iii) they are persistent: if a critical head appears in an ablated combination, it is likely damaging the performance due to its strong effect. However, those premises do not hold for CEL as the enhancement heads’ effect is not as significant as damaging effect of critical heads, and as a result, the enhancement head’s effect will be “blurred” when appear in an ablated combination. That’s why, when you try to gather the performance changes (in the discovery set) of ablated combinations and feed them into LASSO models, the suggested enhancement heads by LASSO are not so good. For example, in our old run with Llama 3.2 1B, the LASSO-suggested combination gives -0.06/-1.10/-3.40pp in BoolQ/HellaSwag/MBPP, respectively.
For the point (2), when running LASSO [6] for CEL, you typically have to runs hundred evaluations for hundreds of random head combinations due to the weak signals, and then use the top suggested heads to move to the search phase. The first phase is not that much of a saving compared to singleton search, and the second phase is exactly the same. That’s why we tried but chose not to go with it.
Besides, as we emphasized throughout the draft, the key point of the empirical results (Section 3) in our draft is NOT trying to come up with an efficient algorithm for finding CEL. Instead, it is to show that CEL effects are prevalent across models and tasks, compared to ad-hoc explanations/scenarios like prior works [7, 8], and which serves as the motivation for our theoretical explanation for CEL. We believe that the simple method we used achieves this goal well. We already noted that the task of developing a more efficient search procedure is an open question for future work.