acceptodds
Under review as a conference paper at ICLR 2027

Can Capability Enhancement Be Localized?

Abstract

Transformer-based language models have shown strong capabilities across many tasks and domains. However, prior work shows that, for a given task, not all attention heads matter equally. Specifically, they suggest that many redundant heads can be safely removed with little-to-no effect, while a small set of important heads, when ablated, severely damages the model's performance on the given task. In this work, we investigate a complementary aspect, called Capability Enhancement Localization (CEL): given a model and a task, does there exist a small set of heads whose removal improves the task performance?} We first investigate this question empirically across models from different families. Using a simple two-stage routine that screens candidate heads and evaluates their joint ablations, we identify sparse head combinations whose removal improves task-specific performance without further fine-tuning. Motivated by these observations, we ask theoretically whether such a phenomenon can naturally arise from training dynamics, and under what conditions. To this end, we construct a simplified multi-head softmax attention model and two tasks with conflicting features, and show that gradient descent from small random initialization provably concentrates the conflicting features into particular heads. We further show that ablating such heads improves performance on one task while harming the other, even though training converges to a global optimum of the joint objective. This establishes a possible training-induced theoretical explanation for CEL.

Primary area:
interpretability and explainable AI
Keywords:
localization, attention heads, capability localization, task localization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rGLkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates Capability Enhancement Localization (CEL), proposing that ablating a sparse set of attention heads can improve task-specific performance without fine-tuning. The paper’s authors support this proposal using both an empirical and a theoretical study:

  • Empirical Study: The authors conduct an experiment in which they take a selection of models (SmolLM3, Qwen2.5, Llama-3.2) and tasks (math capabilities: Arithmetic; code generation: MBPP; language skills: HellaSwag, BoolQ). For each model-task pair, they search, using their proposed algorithm, for a sparse set of heads to prune for better model results on the given task. Finally, they compare the pruned and intact models on test data.

  • Theoretical Explanation: The authors construct a 3-head softmax attention model with diagonal parameter matrices and 2 conflicting linear tasks over R3R^3. This architecture is used to theoretically prove a proposed bias of transformers, where each feature is concentrated in only one head. Hence, pruning the head related to the conflicting feature can help one task while harming the other, if said head doesn’t process other features relevant to the first task.

Critical strengths
  • The concept of head ablation for task improvement is interesting and potentially useful. The empirical study across models and tasks, along with the use of a disjoint sample for discovery and support, strengthens the claim of improvement.

  • Rigorous mathematical derivation of feature isolation: For the proposed 3-head diagonal attention model and with small initialization, the paper proves that gradient descent forces feature localization, thereby showing theoretically cases in which test-time head ablations can improve performance on desired tasks.

Critical weaknesses
  • Skewed data splits and sample scarcity: The empirical evaluation relies on small discovery and test samples and skewed data splits. Specifically, the MBPP evaluation uses discovery and test sets with only 374 and 500 samples, respectively, resulting in unstable test generalization where performance gains fluctuate dramatically (from +0.20 to +1.80) by changing the discovery-test split. Furthermore, the Arithmetic task utilizes a highly unconventional 2:1 discovery-test ratio, allocating 2,000 and 1,000 samples for discovery and test sets, respectively. This methodological brittleness undermines the reliability of the empirical claims and weakens statistical confidence. It makes it difficult to trust the observed CEL as a robust, generalizable phenomenon rather than a result of the specific dataset construction.

  • Simplified transformer architecture in theoretical study: The authors assume a highly restrictive transformer architecture where: (1) the query, key, value, and output projection matrices are diagonal and (2) cross-attention against a ”fixed memory” (±ej\pm e_j) is used instead of the usual sequence self-attention. These assumptions, as the authors acknowledge, eliminate cross-coordinate interactions and thus substantially simplify the proof. On the other hand, they separate the task features in isolated dimensions, making the optimization blind to any inter-task conflicts. Consequently, the network does not intelligibly learn to ”route” conflicting capabilities to different heads to resolve interference -- the localization results directly from the random Gaussian initialization.

  • Algorithmic devolution compared to prior baselines: The head ablation algorithm is substantially less efficient than the compressed sensing (LASSO) framework from Bair et al. (2026), which the authors cite. Furthermore, the proposed algorithm is blind to synergistic head interactions, prematurely discarding head combinations that are only beneficial when ablated jointly (heads from consequent layers could behave as such). Without justifying why this search method was adopted over existing techniques, the algorithm’s contribution remains methodologically fragile.

  • Absence of general capability evaluation: The manuscript does not evaluate the ablated models on standard general language benchmarks. Hence, we cannot determine whether the observed gains trade off improved performance on a desired task with overall loss of language capabilities. This could undermine the claimed utility and safety of CEL.

Questions for authors
  • The empirical results currently rely on a small sample pool for MBPP (874 samples in total) and unconventional discovery-to-test ratios (e.g., the 2:1 ratio for Arithmetic). Could you provide additional evaluation results utilizing larger sample pools and balanced data splits to confirm that the observed CEL is a robust phenomenon rather than an artifact of sample selection?

  • Given that the manuscript cites Bair et al., which demonstrated that LASSO recovers sparse, task-critical head sets in as few as 100 to 200 evaluations while avoiding interaction blindness, why was this algorithm not adopted?

  • When the sparse head subsets are ablated to improve a target task, what is the impact on general language tasks? Do they perform comparably to the original intact model?

5 comments
AuthorsOpenReview ↗

Official Rebuttal to Reviewer rGLk by the authors [1/n]

We thank the reviewer for their time in reviewing our draft and for the constructive feedback. We are glad that the reviewer values our theoretical contribution. However, we are sorry that there is some misunderstanding and concern on the reviewer about your theoretical settings and the experimental designs, which we will address as follows.

A. On the problem settings

  1. Q: on the choice of diagonal weight matrix and fixed memory banks:

    A: We note that the setting of diagonal weight matrices and fixed memory banks (but importantly with learned key and value projections) legitimate theoretical simplifications and has been adopted in many prior theoretical works (see e.g., [1, 2, 3, 4, 5]). Since our work is the first one trying to provide a theoretical explanation for CEL (to the best of our knowledge), we believe that this setting is a reasonable starting point and should not be considered a critical weakness. We already made this point in the preface to Section 4. But again, we agree with the reviewer that extending the analysis to more general settings is an interesting and important question for future work.

  2. Q: “due the the problem settings…. the localization results directly from the random Gaussian initialization.”

    A: We respectfully push back on this point. Actually, our problem setting is designed to isolate the following specific question: given an available representation, how does joint training allocate its contribution across attention heads, and when does that allocation enable ablation?

    By design, at initialization, every head has access to every feature, and there is no localization at this stage. We can then show that at initialization, there is no CEL: ablating any head yields minimal effect on the performance of the considered tasks. We formalized this claim in the revised draft (Section 4.2.1, Proposition 4.2 and Corollary 4.3, marked in RED). To provide more evidence for this, we provide an additional experiment (see e.g., Figure 1.a, marked in RED) showing that at initialization, there is no CEL. The above provides concrete evidence that CEL cannot result merely from Gaussian initialization, and has to emerge throughout training dynamics.

B. On the Experiments

  1. Q: “Why not using LASSO as in [6]”:

    A: That’s a good question. Actually, we did try this method for our setting, but (1) the LASSO [6] is NOT necessarily much cheaper than singleton scan for this particular problem, and (2) the results given by LASSO are unstable, and many times the suggested combinations (using the discovery set) do NOT yield improvements on the test set.

    To see (1), note that in [6], given a model and a task, the objective is to find critical heads such that: (i) they are sparse, (ii) they have strong signal: ablating one head might cause a significant drop in performance, and (iii) they are persistent: if a critical head appears in an ablated combination, it is likely damaging the performance due to its strong effect. However, those premises do not hold for CEL as the enhancement heads’ effect is not as significant as damaging effect of critical heads, and as a result, the enhancement head’s effect will be “blurred” when appear in an ablated combination. That’s why, when you try to gather the performance changes (in the discovery set) of ablated combinations and feed them into LASSO models, the suggested enhancement heads by LASSO are not so good. For example, in our old run with Llama 3.2 1B, the LASSO-suggested combination gives -0.06/-1.10/-3.40pp in BoolQ/HellaSwag/MBPP, respectively.

    For the point (2), when running LASSO [6] for CEL, you typically have to runs hundred evaluations for hundreds of random head combinations due to the weak signals, and then use the top suggested heads to move to the search phase. The first phase is not that much of a saving compared to singleton search, and the second phase is exactly the same. That’s why we tried but chose not to go with it.

    Besides, as we emphasized throughout the draft, the key point of the empirical results (Section 3) in our draft is NOT trying to come up with an efficient algorithm for finding CEL. Instead, it is to show that CEL effects are prevalent across models and tasks, compared to ad-hoc explanations/scenarios like prior works [7, 8], and which serves as the motivation for our theoretical explanation for CEL. We believe that the simple method we used achieves this goal well. We already noted that the task of developing a more efficient search procedure is an open question for future work.

AuthorsOpenReview ↗

Official Rebuttal to Reviewer rGLk by the authors [2/n]

B. On the Experiments (continue)

  1. Q: Why use small sample pool for MBPP? Why ratio 2:1 ratio for Arithmetic.

    A: We will address the points above as follows:

    (1) Split sizes: for MBPP we do NOT subsample. D_{dist} is its entire official training partition (374 examples), and D_{test} is the entire official test partition (500 problems). In this case, no larger or different split is available.

    For Arithmetic, the benchmark is a single pool of 20,000 examples with NO predefined split; we simply choose ∣Ddist∣=2,000|D_{dist}| = 2,000 to match HellaSwag and BoolQ, and choose a disjoint set DtestD_{test} of 1,000 examples.

    (2) Discovery-test ratio: we note that DdistD_{dist} is NOT a training set, as it is only use to select which heads to ablate. The ratio ∣Ddist∣:∣Dtest∣|D_{dist}|: |D_{test} | therefore does not affect validity. What matters most is that the two sets are DISJOINT (as in our settings) and that DdistD_{dist} is large enough for a good estimate of the true task distribution.

    To remove the reviewer’s concern that CEL maybe an artifact of sample selection, we provide the results accross multiple runs, as shown in the below.

  2. Q: Additional evaluation results to confirm that CEL is a robust phenomenon rather than artifact of sample selections?

    A: To address this issue, we provide the mean and CIs of different models across different tasks (Arithmetic, BoolQ, and Hellaswag) in 5 runs, the mean performance with random 3-head ablations (20 runs), and the mean of best single-head (on discovery set) performance on the test set.

    For MBPP, we note that the dataset has a training partition of 374 examples and a test partition of 500 examples. We take the whole training partition as DdistD_{dist} and the whole test partition as DtestD_{test}, so it is hard to demonstrate the CEL results across runs. See Appendix A.2 for detailed discussion. In the below, we present the mean of random ablation (across 20 runs) and compare it to the performance gain from CEL in our single run, though.

TaskModelSelected subset: mean [95% CI]SDRandom subsetsBest single head
ArithmeticQwen2.5-3B+2.72 [+1.91, +3.53]0.65-3.81+1.40
Llama-3.2-3B+8.40 [+7.41, +9.39]0.79+0.05+3.66
SmolLM3-3B+12.82 [+12.25, +13.39]0.46-4.24+5.86
BoolQQwen2.5-3B+2.07 [+1.58, +2.56]0.40-0.24+0.70
Llama-3.2-3B+2.06 [+1.90, +2.22]0.13+0.15+1.19
SmolLM3-3B+2.35 [+2.06, +2.64]0.23-0.47+2.04
HellaSwagQwen2.5-3B+1.27 [+1.01, +1.53]0.21-0.29+0.32
Llama-3.2-3B+1.65 [+1.28, +2.01]0.29-0.20+0.76
SmolLM3-3B+1.21 [+1.02, +1.39]0.15-0.29+0.44
ModelSelected subset (Table 1)Random subsets
Qwen2.5-3B+1.80-0.78
Llama-3.2-3B+5.20+0.51
SmolLM3-3B+1.60-0.03
  1. Q: Effect of ablating a beneficial combination of a task on other tasks

    A. For this question, we run it on Arithmetic Intervention. For each model, we ablate the head combination selected for Arithmetic and evaluate the same ablated model, with no further selection, on three other benchmarks. Entries are changes in performance (pp) relative to the intact model.

ModelAblated headsArithmetic (target)BoolQHellaSwagMBPP
Qwen2.5-3BL0H7, L0H9, L10H12+3.50+1.16-0.01-0.20
Llama-3.2-3BL8H16, L9H10, L10H11+9.30-2.05+0.62+0.20
SmolLM3L5H13, L7H0, L13H10+13.10-0.67+0.10+0.20

This is what we expect. The heads are selected for one task only, so their removal can improve, degrade, or leave another task unchanged, depending on how that task relates to the removed component. Our toy model shows the same picture: tasks A and B of Theorem 4.5 are the two extreme cases (gain t^2 and damage 3t^2), a task that agrees with A on the shared feature also improves, and a task that does not use this feature is unaffected. The intervention is therefore task-specific rather than universally beneficial or harmful.

AuthorsOpenReview ↗

Official Rebuttal to Reviewer rGLk by the authors [3/n]

C. Additional results and experiments

Besides the questions raised by the reviewer, we also provide some additional results for the completeness of the draft.

  1. In the revised draft, we restate our results for the general case of H heads: this is a generalization of the original case with three heads. All the related materials are updated accordingly.

  2. We provide additional experiments validating our theoretical prediction in Section 4.5 (marked in RED).

Final Comments

Again, we thank the reviewer for thoughtful and constructive feedback. We hope that our answers address the reviewer’s concerns, and we are happy to address further questions, if any. We respectfully request that the reviewer re-evaluate the work in light of our rebuttal. Thank you, and have a good day!

References

[1] Boix-Adserà et al. Transformers learn through gradual rank increase. NeurIPS’23.

[2] Vladymyrov et al., Linear Transformers are Versatile In-Context Learners, NeurIPS’24

[3] Rosu et al., From Softmax to Score: Transformers Can Effectively Implement In-Context Denoising Steps, NeurIPS’25

[4] Pesme and Flammarion, Saddle-to-Saddle Dynamics in Diagonal Linear Networks, NeurIPS’23

[5] Ramsauer et al, Hopfield networks is all you need, ICLR’21

[6] Bair et al. Compressed Sensing for Capability Localization in Large Language Models, ICML’26

[7] Held and Yang, Shapley head pruning: Identifying and removing interference in multilingual transformers, EACL’23

[8] Jin et al. Cutting off the head ends the conflict: A mechanism for interpreting and mitigating knowledge conflicts in language models, ACL’24 Finding

Reviewer_Dy2aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks whether removing a few attention heads can improve a language model on a given task. It selects up to three heads on discovery data and tests the fixed subset on separate examples. Experiments cover three 3B models and four tasks. A simple three-head model explains how joint training can leave a head that hurts one task.

Critical strengths
  1. Head selection and testing use separate data. Table 1 shows large arithmetic gains, including 13.1 percentage points for SmolLM3 and 6.3 for Llama. These results go beyond fitting the examples used to select the heads.
  2. Appendix B proves that gradient descent can place each feature in a single head under small random initialization. The benefit of head removal follows from this learned structure, rather than assuming it from the start.
Critical weaknesses
  1. Beneficial head removal is already known [1] [2]. Table 1 lacks test results for random subsets and the best single head selected on discovery data. It also lacks confidence intervals and repeated discovery splits. Several gains are only 1–2 points, so their stability and the value of searching combinations remain unclear.
  2. The theory uses fixed feature directions and diagonal projections. These choices make the optimization much simpler than in a language model. The paper does not test whether the same feature allocation or tradeoff between tasks appears in the real models. The theory offers a possible explanation, but the experimental link is still weak.
Questions for authors
  1. Can you report confidence intervals and compare against random subsets and the best single head across different discovery splits?
  2. Which part of the training-dynamics proof is new compared with [3]?

References

[1] Held and Yang. Shapley Head Pruning: Identifying and Removing Interference in Multilingual Transformers. 2023.

[2] Nguyen et al. Structured Pruning for Diverse Best-of-N Reasoning Optimization. 2025.

[3] Boix-Adserà et al. Transformers learn through gradual rank increase. 2023.

7 comments
AuthorsOpenReview ↗

Official Rebuttal to Reviewer Dy2a by the authors [1/n]

We thank the reviewer for his time and constructive feedback. We will address the reviewer’s concerns as follows.

A. On the theoretical setting concerns and proof technique comparison

  1. Q: Concerns about the diagonal weight matrix settings

    A: We note that the setting of diagonal weight matrices is a legitimate theoretical simplification and widely adopted in prior theoretical works (e.g., [4, 5, 6], including your referenced paper [3]). Since our work is the first one trying to provide a theoretical explanation for CEL (to the best of our knowledge), we believe that this setting is a reasonable starting point and should not be considered a critical weakness. We already made this point in the preface of Section 4. But again, we agree with the reviewer that extending the analysis to more general settings is an interesting and important question for future work.

  2. Q: Comparison to the training dynamic proof technique in [3]

    A: That’s a good question. Since [3] and our draft share some similarities in the settings (diagonal weight matrices, small initialization scale, population loss objective, etc.), one might naturally think there must be some overlaps in the proof technique. However, the goals of the two drafts are different, and the overlap is only somewhat limited to one step: both use a conservation law to reduce the leading-order dynamics to a scalar equation. But other than that, our proof is different in the following ways:

    i. Different dynamical regimes: concretely, in [3], the coefficient is a product of two factors, so the leading-order dynamics are linear and weights grow exponentially on a log⁡(1/α)\log(1/\alpha) time scale. In our case, the coefficient ovtanh⁡(qk)ov \tanh(qk) has four factors, and the leading-order dynamics are cubic, and escape occurs by finite-time blow-up (with the escape time demonstrated in Lemma B.1).

    ii. Different selection mechanism: in [3], coordinates are separated by their distinct grad signals, and the uniqueness of the winning coordinate is assumed. In our case, all heads competing for a feature receive the same grad signal, and we prove that the escape times are almost surely distinct (over the draw of random init), and the winning head is uniform over heads and independent across features (Lemma B.1).

    iii. Discrete GD: in [3], they analyze grad flow. In our draft, analyzing grad flow is the first phase, and we then convert the claims to a (small) fixed step size (Lemma B.2).

    iv. Unconditional, global-in-time control: In [3], they assume strict local minimality and noise-robust convergence of the nonlinear phase. In our draft, we prove the corresponding statements in our model (Lemmas B.3-B.5).

    But again, the individual tools are standard (in the mathematical sense) in both papers (e.g., conservation laws, ODE comparison, Gronwall-type bounds, …). What is new is the analysis of our CEL setting and the head-level localization.

AuthorsOpenReview ↗

Official Rebuttal to Reviewer Dy2a by the authors [2/n]

B. Changes in Revised Draft and Additional Experimental Results

  1. In the revised draft, we restate our results for the general case of HH heads: this is a simple generalization of the original case with three heads. All the related materials are updated accordingly.

  2. Q: CIs, and compare against random subsets and the best single head.

    A: We first clarify that our original idea for Section 3 is to demonstrate that the CEL phenomenon happens across model families and tasks, compared to ad hoc explanations/scenarios (like resolving internal and external memory conflicts [7], or removing language conflicts [1]) as in prior works. That’s why we try to present the empirical results minimally.

    As the reviewer requested, here are the means and CIs of different models across different tasks (Arithmetic, BoolQ, and Hellaswag) in 5 runs, the mean of performance with random 3-head ablations (20 runs), and the mean of best single-head (on discovery set) performance on the test set.

    For MBPP, we note that the dataset has a training partition of 374 examples and a test partition of 500 examples. We take the whole training partition as DdistD_{dist} and the whole test partition as DtestD_{test}, so it is hard to demonstrate the CEL results across runs. See Appendix A.2 for detailed discussion. In the below, we present the mean of random ablation and compare it to the performance gain from CEL in our single run, though.

TaskModelSelected subset: mean [95% CI]SDRandom subsetsBest single head
ArithmeticQwen2.5-3B+2.72 [+1.91, +3.53]0.65-3.81+1.40
Llama-3.2-3B+8.40 [+7.41, +9.39]0.79+0.05+3.66
SmolLM3-3B+12.82 [+12.25, +13.39]0.46-4.24+5.86
BoolQQwen2.5-3B+2.07 [+1.58, +2.56]0.40-0.24+0.70
Llama-3.2-3B+2.06 [+1.90, +2.22]0.13+0.15+1.19
SmolLM3-3B+2.35 [+2.06, +2.64]0.23-0.47+2.04
HellaSwagQwen2.5-3B+1.27 [+1.01, +1.53]0.21-0.29+0.32
Llama-3.2-3B+1.65 [+1.28, +2.01]0.29-0.20+0.76
SmolLM3-3B+1.21 [+1.02, +1.39]0.15-0.29+0.44
ModelSelected subset (Table 1)Random subsets
Qwen2.5-3B+1.80-0.78
Llama-3.2-3B+5.20+0.51
SmolLM3-3B+1.60-0.03
  1. Q: “Link between theory and experiments”:

    A: To verify our theory, we provide some validating experiments. Concretely, we consider the one-layer attention model with diagonal weight matrices, H∈{4,8,16,32,64}H \in \{4, 8, 16, 32, 64\} heads, and use 10,000 Gaussian initializations θ0=αG\theta^0 = \alpha G per setting. First, we investigate the effect of ablating heads at initialization for the model with H=4H = 4. To do that, we ablate each head at different initialization scales \alpha varying from 10−410^{-4} to 10−210^{-2}. Then, we train the model from α=10−2\alpha = 10^{-2} using full-batch GD on the population loss, with learning rate η=0.1\eta = 0.1 until convergence, and then ablate each head of the trained model.

    The results are provided in Section 4.5 and Figure 1 in the revised draft (marked in RED). Concretely, at initialization, ablating heads give negligible effects: the largest gain on task A scales as α4\alpha^{4}, as predicted by Corollary 4.3, and is far below the effect of beneficial head ablation after training. After training, a beneficial head exists in approximately 75%75\%, 87.5%87.5\%, 93.3%93.3\%, 96.9%96.9\%, and 98.4%98.4\% of the runs for H=4,8,16,32,64H = 4, 8, 16, 32, 64. This is within 0.50.5pp of the bound 1−1H1 - \frac{1}{H} predicted by Theorem 4.5. Therefore, we can see that CEL is absent at initialization and emerges from the training dynamics, as the theory predicted.

Final Comments

Again, we thank the reviewer for thoughtful and constructive feedback. We hope that our answers address the reviewer’s concerns, and we are happy to address further questions, if any. We respectfully request that the reviewer re-evaluate the work in light of our rebuttal. Thank you, and have a good day!

References

[1] Held and Yang. Shapley Head Pruning: Identifying and Removing Interference in Multilingual Transformers. EACL’23.

[2] Nguyen et al. Structured Pruning for Diverse Best-of-N Reasoning Optimization. ACL’25 Findings.

[3] Boix-Adserà et al. Transformers learn through gradual rank increase. NeurIPS’23.

[4] Vladymyrov et al., Linear Transformers are Versatile In-Context Learners, NeurIPS’24

[5] Rosu et al., From Softmax to Score: Transformers Can Effectively Implement In-Context Denoising Steps, NeurIPS’25

[6] Pesme and Flammarion Saddle-to-Saddle Dynamics in Diagonal Linear Networks, NeurIPS’23

[7] Jin et al., Cutting off the head ends the conflict: A mechanism for interpreting and mitigating knowledge conflicts in language models, ACL’24 Findings.

Thank you for the rebuttal. The new baselines and the comparison with [3] answer my questions on empirical stability and the proof technique. However, I still have two concerns:

  1. The theoretical model still has a weak link to the real LLM experiments. The new simulations in Section 4.5 validate the theoretical predictions within the assumed setting. However, neither these simulations nor the mixed cross-task results reported to Reviewer rGLk establish that the proposed feature-conflict mechanism explains the gains in the 3B models. Although I appreciate the clarified scope, the explanatory connection between the theory and the empirical findings remains limited.

  2. There is an error in the revised Theorem 4.5. The event E=h0≠h1E={h_0\ne h_1} allows h0=h2h_0=h_2, but the appendix proof assumes the removed head carries neither feature 1 nor feature 2. Also, Line 455 in the main text writes 1h0≠h1\mathbf{1}{h_0\ne h_1} for the c2c^2 term, which is always 1 under EE. From Theorem 4.4, the damage on task B under EE should be DBn=3t2+c21h0=h2+OG(α4)D_B^n = 3t^2 + c^2\mathbf{1}{h_0=h_2} + \mathcal{O}_G(\alpha^4). Please also clarify why the main text and appendix state different probability lower bounds.

I therefore maintain my rating of 2 at this stage.

Reviewer_tKadOpenReview ↗
Rating
1
Confidence
3(−1)
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies a set of attention heads whose removal will improve LLM's performance on a given task. The paper names it as Capability Enhancement Localization (CEL): the output of a head is zeroed before the attention output projection, and a subset of heads counts as "beneficial" when the resulting accuracy gain is greater than zero. The authors search for CEL heads on three models at the 3B scale and four relatively simple tasks, and the experiments show that all twelve model-task combinations obtain a positive gain. In addition, the authors construct a one-layer, three-head softmax attention model with two tasks whose features conflict, and show theoretically that gradient descent from a small random initialization concentrates each feature into a single head. When the conflicting feature occupies a head of its own, removing that head improves one task while harming the other, even though the joint objective has already converged to a global optimum.

Critical strengths
  1. The topic that "pruning can actually improve model performance" is valuable, and the authors propose corresponding experimental and validation procedures for it.
Critical weaknesses
  1. The procedure the paper uses to locate attention heads is not reasonable.

    (a) According to Algorithm 1, CEL heads are selected by enumerating the individual attention heads and then enumerating combinations of them, and every enumeration and every combination requires modifying the model once and running one downstream evaluation. First, this makes the resulting CEL heads strongly dependent on the discovery set (Line 215). Second, for a model with NN heads, the total number of modifications and evaluations is N+∑k=2∣S∣CpkN + \sum_{k=2}^{|S|} C_p^k. The models used here are at the 3B scale, with 576 attention heads (Qwen2.5 / SmolLM3) and 672 (Llama-3.2); with p=10p=10, locating the CEL heads of a single model already costs nearly 800 evaluations. This overhead is very large, and it grows worse as models become larger, since larger models are generally slower to evaluate and would call for larger values of pp and ∣S∣|S|. I therefore do not think this is a method that transfers well to most LLMs.

    (b) Even with such a heavy search, CEL still does not guarantee that the ablated heads are optimal, as the authors themselves state "individual-head screening can exclude combinations whose benefits emerge only through joint ablation" (Line 456).

  2. The theoretical part of the paper is seriously incomplete. By the definition in Section 4.1, the two tasks can be told apart simply by whether the input has v=0 or w=0, which would allow a predictor to handle both tasks without any conflict; yet the setup forces the attention predictor to be linear, which is entirely unlike the situation in a real LLM. What the paper calls CEL is therefore only conflict detection and resolution under such a restriction, and a toy model of this kind cannot explain what happens in real LLMs. Worse still, the theoretical section as a whole has no numerical validation whatsoever.

  3. The writing has severe problems.

    (a) The paper repeatedly refers to things that do not exist: Table 2 (Line 186) and Table 3 (Line 201) are mentioned in the text, yet the paper contains only a single Table 1; the main text refers to "Appendix B" three times, but the paper has no appendix at all. [Update] Sorry for my careless and arbitrary, now I see these contents in Supplementary Material.

    (b) The paper contains no figure, and a single table carries the whole experimental content. [Update] Revised paper added two figures about validations.

    (c) VALIDATION.md in the authors' public anonymous repository states that "No full 3B GPU evaluation or twelve-cell sweep was run for this release." Since even the experimental environment actually used by the authors cannot be verified, this paper is very difficult to reproduce.

Questions for authors
  1. Does the head ablation performed for one task in Section 3 degrade the performance on the other tasks?
5 comments
AuthorsOpenReview ↗

Official Rebuttal to Reviewer tKad by the authors [1/n]

We thank the reviewer for the careful and constructive feedback. We are glad that the reviewer went through the draft and the released code thoroughly. We are sorry that the reviewers have some misunderstandings about our release and writing, as well as concerns about the problem setup. We will address the reviewer's concerns as follows.

A. Misunderstanding about the draft's format and code release

  1. Q: “Where is the Appendix?”

    A: Actually, when you click on the “Supplementary Material” (the one right below the abstract), you will be forwarded to the 25-page (now 28-page in the revised draft) version that contains BOTH the main text and the Appendix. If you only click on the main submission (the PDF at the top of the page), it will only contain the main body, due to ICLR’s policy. Sorry for the confusion!

  2. Q: “Where is Table 2 (Line 186) and Table 3 (Line 201)” in the main text?

    A: After navigating the 25-page version (in the revised draft, it is 28 pages), the reviewer can see Tables 2 and 3 right in Appendix A.

  3. Q: “Where is the experimental environment in the anonymous repo?”

    A: Following the normal convention, the environment (required packages) for the experiment can easily be found in “requirements.txt” in our anonymous repo. Moreover, we did incorporate how to install the requirements in the “Installation” section, right in the README.md. However, although we also already discussed the hardware requirement in the main text, we agree with the reviewers that we should also discuss the software requirement (and potentially reference the anonymous repo) to avoid confusion.

  4. Q: “Why the text “No full 3B GPU evaluation or twelve-cell sweep was run for this release.”

    A: Sorry for the confusion. When preparing the anonymous repo (using our original repo), we explicitly included this term in the agent’s prompt. If not, the agent will overengineer by rerunning the FULL experiments to verify the runnability of the released code. This is redundant and very costly, as we can verify the runnability by (partially) doing a few inference passes. This is why we had that text on VALIADATION.md.

    Actually, the VALIDATION.md is our progress file to check the runnability of the released anonymous repo, as the reviewer can see based on its content, not related to the experiments themselves. We can update the anonymous repo to remove confusing texts if the reviewer requests, though we will leave it to the original state to the reviewer for easy comparison.

B. On the concerns of our procedure

  1. Clarifying the scope of Section 3 and the role of Algorithm 1: We would like to clarify that the search procedure is NOT the key contribution of the paper, and we do NOT propose it as a method to be deployed on arbitrary LLMs. Actually, its role in Section 3 is to establish that beneficial sparse head ablations exist across several tasks, rather than in specific/ad-hoc conflict settings as explained in prior works (e.g., interference between languages [6], or between external and internal knowledge), which do not apply to tasks such as Arithmetic. This then serves as the motivation for our theoretical analysis, which, to the best of our knowledge, is the first to explain how such heads can emerge from training. As stated in the paper, we deliberately keep the search procedure simple.

  2. On the optimality: We agree that our procedure is not optimal, and we stated this already in the paper. However, there are two points that we want to make clear here. First, for an existence claim (the scope of Section 3 above), optimality is NOT required: one beneficial subset suffices, and a stronger search could only find better gains, so our reported gains are conservative. Second, and most importantly, no practical procedure can offer optimality. This is because selecting the best subset of at most kk heads is a combinatorial optimization problem (NP-Hard in general). Besides, even if you have an oracle for selecting the best combinations for the discovery set DdistD_{dist}, that set is still not optimal for the true task distribution DtaskD_{task}, since DdistD_{dist} is just an empirical distribution approximating DtaskD_{task}. Therefore, demanding an optimal procedure here is unnecessary (for the purpose of Section 3) and impractical.

  3. On the dependence on the discovery set DdistD_{dist}: We note that any data-driven selection method depends on its data. Therefore, the right question here to ask is whether the held-out gain is stable. To address this concern, we repeatedly conduct the full procedure with independent draws of the discovery set DdistD_{dist}. See below for the details.

AuthorsOpenReview ↗

Official Rebuttal to Reviewer tKad by the authors [2/n]

C. On the problem setting of the draft, the contribution scope

  1. Q: “The theoretical part of the paper is seriously incomplete. By the definition in Section 4.1, the two tasks can be told apart simply by whether the input has v=0 or w=0, which would allow a predictor to handle both tasks without any conflict; yet the setup forces the attention predictor to be linear, which is entirely unlike the situation in a real LLM.

    A: We respectfully push back on this point. It is true that a predictor from an UNRESTRICTED hypothesis class could indeed separate two tasks and fit both exactly. However, this is not the case in general (e.g., misspecification/non realizability/agnostic learning), and our case concerns a restricted hypothesis class: one-layer attention models with diagonal weights, whose coefficient on the shared feature UU cannot depend on the task (Prop 4.1). The optimal predictor that the reviewer mentioned clearly lies outside this class, so the learning problem is agnostic, and the best model can do is a compromise.

    The existence of a better predictor outside the considered hypothesis class does NOT make an analysis of that class incomplete. By the same argument, any analysis of logistic regression on data that is not linearly separable would be incomplete because of the (optimal) Bayes classifier, although the logistic model cannot represent it. Analyzing restricted and misspecified model classes is very standard in learning theory, including analysis of transformers. For example, the literature on in-context linear regression studying one-layer linear attn are many [1, 2, 3, 4]. Therefore, we believe that saying that our theory is “seriously incomplete” merely based on the argument above is very unfair and harsh.

    Another point is that the restriction is also what connects the model to CEL. If a trained network were optimal for every task separately, removing a component could never improve a task; the gains in Section 3 show that real models are not in that regime. Our model isolates one mechanism that produces this: a component shared by tasks with conflicting requirements is trained to a compromise, and gradient descent concentrates that compromise in a single head, so that removing the head helps one task. We do not claim this is the only source of beneficial ablations in LLMs. Consistent with this, the heads selected for Arithmetic also change accuracy on other tasks, in both directions.

  2. Q: ” The theoretical section as a whole has no numerical validation whatsoever.”

    A: We thank the reviewer for the suggestion and provide validating experiments in the revised draft. Concretely, we consider the one-layer attention model with diagonal weight matrices, H∈{4,8,16,32,64}H \in \{4, 8, 16, 32, 64\} heads, and use 10,000 Gaussian initializations θ0=αG\theta^0 = \alpha G per setting. First, we investigate the effect of ablating heads at initialization for the model with H=4H = 4. To do that, we ablate each head at different initialization scales α\alpha varying from 10−410^{-4} to 10−210^{-2}. Then, we train the model from α=10−2\alpha = 10^{-2} using full-batch GD on the population loss, with learning rate η=0.1\eta = 0.1 until convergence, and then ablate each head of the trained model.

    The results are provided in Section 4.5 and Figure 1 in the revised draft (marked in RED). Concretely, at initialization, ablating heads give negligible effects: the largest gain on task A scales as α4\alpha^{4}, as predicted by Corollary 4.3, and is far below the effect of beneficial head ablation after training. After training, a beneficial head exists in approximately 75%75\%, 87.5%87.5\%, 93.3%93.3\%, 96.9%96.9\%, and 98.4%98.4\% of the runs for H=4,8,16,32,64H = 4, 8, 16, 32, 64. This is within 0.50.5pp of the bound 1−1H1 - \frac{1}{H} predicted by Theorem 4.5. Therefore, we can see that CEL is absent at initialization and emerges from the training dynamics, as the theory predicted.

AuthorsOpenReview ↗

Official Rebuttal to Reviewer tKad by the authors [3/n]

D. Additional Results

  1. In the revised draft, we restate our results for the general case of H heads: this is a generalization of the original case with three heads. All the related materials are updated accordingly.

  2. Stability of CEL: Here are the mean and CIs of different models across different tasks (Arithmetic, BoolQ, and Hellaswag) in 5 runs, and the mean of performance with random 3-head ablations (20 runs), and the mean of best single-head (on discovery set) performance on the test set.

    For MBPP, we note that the dataset has a training partition of 374 examples and a test partition of 500 examples. We take the whole training partition as DdistD_{dist} and the whole test partition as DtestD_{test}, so it is hard to demonstrate the CEL results across runs. See Appendix A.2 for detailed discussion. In the below, we present the mean of random ablation and compare it to the performance gain from CEL in our single run, though.

TaskModelSelected subset: mean [95% CI]SDRandom subsetsBest single head
ArithmeticQwen2.5-3B+2.72 [+1.91, +3.53]0.65-3.81+1.40
Llama-3.2-3B+8.40 [+7.41, +9.39]0.79+0.05+3.66
SmolLM3-3B+12.82 [+12.25, +13.39]0.46-4.24+5.86
BoolQQwen2.5-3B+2.07 [+1.58, +2.56]0.40-0.24+0.70
Llama-3.2-3B+2.06 [+1.90, +2.22]0.13+0.15+1.19
SmolLM3-3B+2.35 [+2.06, +2.64]0.23-0.47+2.04
HellaSwagQwen2.5-3B+1.27 [+1.01, +1.53]0.21-0.29+0.32
Llama-3.2-3B+1.65 [+1.28, +2.01]0.29-0.20+0.76
SmolLM3-3B+1.21 [+1.02, +1.39]0.15-0.29+0.44
ModelSelected subset (Table 1)Random subsets
Qwen2.5-3B+1.80-0.78
Llama-3.2-3B+5.20+0.51
SmolLM3-3B+1.60-0.03

Final Comments

We again thank the reviewer for the careful reading and constructive feedback. We hope that our answers above address the reviewer’s concerns, and we are more than happy to address further issues raised by the reviewer. However, we believe that the original evaluation of the reviewer is very harsh. We respectfully request that the reviewer reevaluate our work in light of our rebuttal. Thank you and have a good day!

References

[1] Oswald et al., Transformers Learn In-Context by Gradient Descent, ICML’23

[2] Ahn et al., Transformers Learn to Implement Preconditioned Gradient Descent for In-Context Learning, NeurIPS’23

[3] Zhang et al., Trained Transformers Learn Linear Models In-Context, JLMR’24

[4] Mahankali et al., One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention, ICLR’24

[5] Boix-Adsera et al., Transformers learn through gradual rank increase, NeurIPS’23

[6] Held and Yang, Shapley head pruning: Identifying and removing interference in multilingual transformers, ECL’23

[7] Jin et al. Cutting off the head ends the conflict: A mechanism for interpreting and mitigating knowledge conflicts in language models, ACL’24 Finding

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.