acceptodds
Under review as a conference paper at ICLR 2027

Ranking and Partitioning of Attention Heads for Sparse and Continual Reinforcement Learning Fine-tuning

Abstract

We study sparse and continual reinforcement learning fine-tuning of language models through attention-head ranking and partitioning. For sparse learning, we rank attention heads by gradient norm to identify a small subset of heads that can be trained effectively. For continual learning, we partition heads across tasks to reduce cross-task interference and preserve previously learned capabilities. Theoretically, we relate gradient norm to training effectiveness, and further characterize why sparse, separated head updates can remain stable. Empirically, across math, instruction-following, and coding tasks, tuning higher-ranked heads consistently outperforms tuning lower-ranked while often matching denser baselines, and head partitioning improves retention while preserving adaptation to new tasks.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
sparse training, multi-task learning, reinforcement learning, continual learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tXppOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to finetune language models with reinforcement learning (RL) by updating only a small number of attention heads. The authors rank the attention heads using their gradient norms at the beginning of training and select some of them for updating. For CL, they assign different groups of heads to different tasks, aiming to reduce interference and forgetting between tasks. The paper also provides theoretical analysis to explain why gradient norms may related to training progress. What's more, why separating attention heads may help reduce task interference. Experiment on mathematical reasoning, instruction following, and coding show that most updating ranked heads will performs better than updating low ranked heads. The results also show that head parttioning creates a trade-off between learning new tasks and maintaining performance on previous tasks.

Critical strengths
  1. The paper presents a simple approach to selective RL fine-tuning.

The authors use initial gradient norms to select the attention heads to update. The experiments show that high-ranked heads generally perform better than low-ranked heads across the tested models and tasks. In some settings, selective tuning also gives results comparable to denser tuning methods. These results provide initial support for using gradient norms as a head-selection criterion.

  1. The paper examines whether head partitioning can reduce interference between tasks.

In several sequential-training settings, head partitioning retains previous-task performance better than full-attention tuning. The head-reset experiments also suggest that some task-specific changes are localized within the assigned head subsets. Overall, these results provide preliminary evidence that head partitioning may help balance learning new tasks and retaining earlier capabilities.

Critical weaknesses
  1. The connection between the theoretical analysis and practical language-model fine-tuning could be clearer.

The theoretical results rely on several assumptions, and the current experiments provide limited evidence about whether the same conclusions hold for larger and more widely used language models. Additional discussion or experiments would help clarify the practical scope of the theoretical findings.

  1. The reliability of initial gradient norms as a head-selection criterion needs further analysis.

Gradients in reinforcement learning can vary during training and across tasks. It would be helpful to study how stable the head rankings are and to compare this criterion with other simple selection methods. Such analysis would provide stronger support for this important design choice.

Questions for authors

1, See Critical Weaknesses 2, See Critical Weaknesses

3, The paper’s motivation relies partly on the idea that task adaptation is concentrated in task-relevant attention heads and that gradient information can identify such heads. Related observations have previously been reported by [1].

4, Could the authors specify exactly which parameters are treated as belonging to an individual attention head in both the theory and implementation? In particular, does a head block include the corresponding slices of the Q/K/V/O projections, and is the gradient norm normalized by the number or scale of parameters? I think the experiments should add more ablation study to compare with other related methods to prove that your method is better.

[1] Aristeidis Panos, Rahaf Aljundi, Daniel Olmeda Reino, and Richard E. Turner. “Efficient Few-Shot Continual Learning in Vision-Language Models.” Transactions on Machine Learning Research, 2025.

4 comments
AuthorsOpenReview ↗

Response to Reviewer tXpp (1/2)

W1

The connection between the theoretical analysis and practical language-model fine-tuning could be clearer. The theoretical results rely on several assumptions, and the current experiments provide limited evidence about whether the same conclusions hold for larger and more widely used language models. Additional discussion or experiments would help clarify the practical scope of the theoretical findings.

A: Please see "Response to all reviewers: Empirical validation of theory". We have added empirical values of β,βA,βB\beta,\beta_A,\beta_B appearing in Props 4,9. We also clarified the validity of the obtained theoretical bounds.

W2

The reliability of initial gradient norms as a head-selection criterion needs further analysis. Gradients in reinforcement learning can vary during training and across tasks. It would be helpful to study how stable the head rankings are and to compare this criterion with other simple selection methods. Such analysis would provide stronger support for this important design choice.

A: Please see "Response to all reviewers: Empirical validation of theory". We reported the stable ranking consistency of gradient norms during RL training.

AuthorsOpenReview ↗

Response to Reviewer tXpp (2/2)

Q3

The paper’s motivation relies partly on the idea that task adaptation is concentrated in task-relevant attention heads and that gradient information can identify such heads. Related observations have previously been reported by [1].

A: Thank you for pointing out this related work! We've included the discussion of [1] in the revised Related work section. This recent empirical work [1] uses gradient-based attention-head selection in the vision encoder of VLMs for class-incremental few-shot learning.

We acknowledge that our implemented algorithm is indeed based on the idea that task adaptation is concentrated in task-relevant attention heads, which is also discussed in [2,3], or other structural units, such as layers [4] and neurons [5]. And they're already discussed in our Related work section.

We wish to clarify that, rather than simply transferring this idea to another setting, our main focus is to provide a systematic understanding of attention-head sparsity in LLM RL fine-tuning, including optimization dynamics (Section 4, 5) and second-order analysis (Section 6). We also investigate how to allocate attention heads across complex and different RL tasks to mitigate forgetting, whereas [1] mainly considers continual learning across different object categories in visual adaptation.

Furthermore, while [1] finds that few attention heads (two attention heads per layer) are sufficient for visual adaptation, our sparsity–performance trade-off experiments suggest that RL fine-tuning benefits from a relatively larger proportion of attention heads (e.g., 30–50%).

[1] Efficient Few-Shot Continual Learning in Vision-Language Models. Panos et al. TMLR 2025.

[2] Compressed sensing for capability localization in large language models. Bair et al. ICML 2026.

[3] Specialization of softmax attention heads: insights ´ from the high-dimensional single-location model. Sagitova et al.

[4] Is one layer enough? training a single transformer layer can match full-parameter rl training. Zhang et al.

[5] LLM-Pruner: On the Structural Pruning of Large Language Models. Ma et al. NeurIPS 2023.

Q4

(i) Could the authors specify exactly which parameters are treated as belonging to an individual attention head in both the theory and implementation? In particular, does a head block include the corresponding slices of the Q/K/V/O projections, and is the gradient norm normalized by the number or scale of parameters? (ii) I think the experiments should add more ablation study to compare with other related methods to prove that your method is better.

A: (i) Yes. In both the theory and implementation, the gradient norm is calculated using the corresponding slices of Q/K/V projections (we exclude O-projection gradients from the ranking score for simplicity). Specifically, query heads sharing one key/value pair are counted as in the same block. We didn't normlize the norm using either number of scale, as the attention blocks within a model have the same parameter size.

(ii) We've conducted several ablation studies during the rebuttal. Please see "Response to All Reviewers: Ablation studies" for details.

  • We conducted a controlled layer-level budget-fixed ablation, selecting either the top or bottom 50% of heads within each layer, thus ensuring identical layer-wise training budgets. The results show consistent advantages of per-layer Top on instruction following.
  • We conducted a same-budget ablation comparing head partitioning with reusing the stage-I heads for task B in continual learning. The results show that partitioning effectively improves old-task retention.

As we focus on systematically understanding the mechanisms of attention-head sparse fine-tuning, comparisons with sparse fine-tuning methods based on other parameter units are beyond the scope.

As the end of the author-reviewer discussion is approaching, we kindly ask for your review of our revised paper and response. Should you have any remaining questions or require further clarification, please don't hesitate to let us know. We would be more than happy to answer any further questions.

Thanks for the clarifications and additional experiments. The additional analysis and ablations partially strengthen the empirical support.

Although the response clarifies the differences from prior work, I remain unconvinced that the insights could represent sufficiently contribution. I will keep my original score.

Reviewer_tQCdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates sparse and continual reinforcement learning (RL) fine-tuning of language models through the ranking and partitioning of attention heads. For sparse fine-tuning, the authors propose ranking attention heads based on their gradient norms and updating only those that rank the highest. Regarding continual learning, the authors explore assigning non-overlapping or partially shared subsets of attention heads to different tasks, aiming to mitigate the problem of "catastrophic forgetting."

The paper provides a theoretical analysis that elucidates the relationship between gradient norms, the optimization process, and cross-task interference, while also conducting a second-order analysis of the attention-head structure. Experiments performed on Qwen3-1.7B, Qwen3-4B, and Mistral-7B models, demonstrate that top-ranked attention heads generally outperform lower-ranked ones and can achieve performance comparable to denser fine-tuning approaches. Experiments in continual learning further show that partitioning attention heads enables the model to better retain knowledge from previously learned tasks.

Critical strengths

1. The gradient-based attention-head selection mechanism is supported by robust empirical evidence.

Experiments demonstrate that ranking attention heads by their gradient norms is an effective criterion for selecting heads for training. Across various models and tasks, updating the top-ranked attention heads consistently outperforms updating lower-ranked ones, often achieving performance comparable to "full-attention" training or full-parameter fine-tuning. Further analysis regarding sparsity levels, ranking consistency, and sensitivity to learning rates reinforces this observation.

2. It provides a unified perspective on sparse fine-tuning and continual learning.

This paper bridges the gap between "selecting attention heads for efficient optimization" and "partitioning attention heads to mitigate cross-task interference." Theoretical analysis offers valuable intuition for both scenarios, while "head-reset" experiments provide further evidence for the existence of "functional localization."

Critical weaknesses
  1. The methodological innovation is limited, and the justification for the specific attention-head selection strategy is insufficient. The core methodology relies on two relatively well-established concepts: using gradient magnitudes to identify important parameters, and isolating subsets of parameters to reduce interference in continual learning. While applying these concepts at the level of "attention heads" is logical, the specific ranking and allocation strategy proposed is overly simplistic. Furthermore, much of the theoretical argumentation is based on general gradient-flow identities and conditional smoothness bounds. For instance, the conclusion that "selecting attention heads with the largest gradient norms maximizes the initial loss reduction" follows directly from standard gradient-flow dynamics. Although the paper’s second-order analysis of the attention mechanism is more novel, the link between this analysis and the empirical superiority of the proposed selection strategy remains somewhat indirect.

  2. A lack of critical experimental comparisons makes it difficult to clearly define the advantages of this method over other existing approaches. Regarding sparse fine-tuning, this paper employs a global selection strategy, whereas the "random selection" baseline selects a fixed proportion of attention heads from each layer. Since the paper notes that high-gradient attention heads are not uniformly distributed across layers, the observed performance gains might stem, in part, from a more favorable selection of layers rather than the inherent effectiveness of the "attention-head ranking" method itself. To isolate these factors, a "layer-matched" random baseline is required. In the context of continual learning, the study primarily compares its approach against "full-attention" and "full-parameter fine-tuning," yet it lacks a robust baseline tailored for continual learning tasks under equivalent parameter budgets. Furthermore, improved retention of prior knowledge sometimes comes at the expense of adaptability to new tasks. For instance, in the "Qwen3-4B: Coding → Instruction Following (IF)" experiment (see Table 6), the proposed partitioning method achieves a performance of 53.1% on the new task, whereas continual fine-tuning using a full-attention mechanism reaches 62.2%.

Questions for authors

Same as weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer tQCd

W1

(i) The methodological innovation is limited, and the justification for the specific attention-head selection strategy is insufficient. The core methodology relies on two relatively well-established concepts: using gradient magnitudes to identify important parameters, and isolating subsets of parameters to reduce interference in continual learning. While applying these concepts at the level of "attention heads" is logical, the specific ranking and allocation strategy proposed is overly simplistic. (ii) Furthermore, much of the theoretical argumentation is based on general gradient-flow identities and conditional smoothness bounds. For instance, the conclusion that "selecting attention heads with the largest gradient norms maximizes the initial loss reduction" follows directly from standard gradient-flow dynamics. Although the paper’s second-order analysis of the attention mechanism is more novel, the link between this analysis and the empirical superiority of the proposed selection strategy remains somewhat indirect.

A: (i) We wish to clarify that, our main goal is not to propose a sophisticated new algorithm, but to provide a systematic empirical and theoretical understanding of head-level sparse RL fine-tuning, including optimization dynamics, second-order properties, and cross-task interference.

(ii) To strengthen the connection between theory and practice, we have added empirical measurements of directional second-order quantities and head-ranking consistency throughout training. Please see "Response to All Reviewers: Empirical Validation of Theory" for the detailed results.

W2

(i) A lack of critical experimental comparisons makes it difficult to clearly define the advantages of this method over other existing approaches. Regarding sparse fine-tuning, this paper employs a global selection strategy, whereas the "random selection" baseline selects a fixed proportion of attention heads from each layer. Since the paper notes that high-gradient attention heads are not uniformly distributed across layers, the observed performance gains might stem, in part, from a more favorable selection of layers rather than the inherent effectiveness of the "attention-head ranking" method itself. To isolate these factors, a "layer-matched" random baseline is required. (ii) In the context of continual learning, the study primarily compares its approach against "full-attention" and "full-parameter fine-tuning," yet it lacks a robust baseline tailored for continual learning tasks under equivalent parameter budgets. (iii) Furthermore, improved retention of prior knowledge sometimes comes at the expense of adaptability to new tasks. For instance, in the "Qwen3-4B: Coding → Instruction Following (IF)" experiment (see Table 6), the proposed partitioning method achieves a performance of 53.1% on the new task, whereas continual fine-tuning using a full-attention mechanism reaches 62.2%.

A:

(i) We agree with the reviewer's concern that the performance gains may arise from both (1) layer allocation and (2) within-layer head selection. To disentangle these factors more clearly, we conducted a controlled layer-level budget-fixed ablation, selecting either the top or bottom 50% of heads within each layer, thus ensuring identical layer-wise training budgets. The results show consistent advantages of per-layer Top on instruction following. Please see "Response to All Reviewers: Ablation studies" for details.

(ii) We conducted a same-budget ablation comparing head partitioning with reusing the stage-I heads for task B in continual learning. The results show that partitioning effectively improves old-task retention. Please see "Response to All Reviewers: Ablation studies" for details.

(iii) We wish to emphasize that the stability–plasticity trade-off is one of the biggest challenges in continual learning. While full-attention tuning achieves higher performance on the new task, it also suffers from substantially more forgetting on the previous task. Our method thus achieves much better retention while still adapting to the new task. In fact, our method achieves a Pareto-optimal trade-off between old-task retention and new-task performance in most settings. Our goal is to systematically investigate such trade-offs rather than maximize new-task performance alone.

As the end of the author-reviewer discussion is approaching, we kindly ask for your review of our revised paper and response. Should you have any remaining questions or require further clarification, please don't hesitate to let us know. We would be more than happy to answer any further questions.

Reviewer_biZnOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks which attention heads to train during RL fine-tuning of language models. Heads are ranked by their gradient norm at the base model, and only the top half is trained. For continual learning, heads are split across tasks (top-bottom, round-robin, or with a shared subset) to limit interference. The theory combines a gradient-flow argument, a group-lasso view of Adam, and a Hessian analysis of multi-head attention. On several models of size under 8B over math, instruction following and coding, top-ranked heads beat bottom-ranked ones and often match full tuning. Partitioning reduces forgetting compared with continual full-attention tuning. Evaluation is greedy pass@1, mostly from single runs

Critical strengths

The method is quite intersting. Ranking needs one gradient computation at base model, and it seems to me that it is stable across data subsets.

Critical weaknesses

Prior to giving the weaknesses, I couldnt justify why this paper is listed in "other topics in machine learning". There are plenty of other options for the authors to select, which could be more suitable. The paper is very hard to understand, with loads of Propositions and Remarks, and a lot of them contains assumptions, which make me believe that the method barely work in practice. Following the instruction of the conference, I list 2 main concerns below. But the authors should be well noted that there are plenty of other problems reagarding to the propositions in the paper.

  • I found the theory behind the ranking less convincing than the paper suggests. Corollary 3 is actually an equality as both sides reduce to t Gˉ(Sk;t)2t\,\bar{G}(S_k;t)^2, so it tells us nothing about the selected heads beyond Proposition 1. Proposition 4 only keeps the ordering as long as the gradient-norm ratio between two head sets is larger than eβte^{\beta t}. Since beta is never estimated, I don’t think the “hundreds of steps” claim can be drawn from it. The bigger issue for me is the optimizer. Training uses Adam, and under Adam’s own quadratic model the best k heads are the ones that maximize ∑imi2/(vi+ϵ)\sum_i m_i^2/(\sqrt{v_i}+\epsilon). At the first step this is roughly the l1 norm of the head gradient, not the l2 norm. Proposition 5 only lands on the l2 ranking because the group penalty is put in the Euclidean metric instead of Adam’s. The constant in Proposition 6 is the ratio of the smallest to the largest vi\sqrt{v_i}. That ratio should be close to zero for heads with millions of parameters, so the bound says very little

  • The isolation and partitioning arguments don’t really explain why heads, or why disjoint sets. Proposition 10 holds for any blocks that add up in the output, MLP included, and it only covers HfH_f. For continual learning, Proposition 7 bounds forgetting by the task-A gradient on SBS_B. But training on A pushes that gradient down on SAS_A itself, so near a task-A optimum the first-order bound would favor reusing SAS_A.

Questions for authors
  • Corollary 3 seems to hold with equality. What does it add beyond Proposition 1?

  • Why rank by the l2 gradient norm when training with Adam, whose local model is minimized by ∑imi2/(vi+ϵ)\sum_i m_i^2/(\sqrt{v_i}+\epsilon)?

  • What are typical values of beta (Propositions 4 and 9) and of the constant in Proposition 6?

  • How does partitioning compare with retraining the same top heads for task B at the same budget?

  • Are the top versus random gaps in Table 1 significant? Most runs use one seed and the std seems large

6 comments
AuthorsOpenReview ↗

Response to Reviewer biZn (1/n)

Comment 1

Prior to giving the weaknesses, I couldnt justify why this paper is listed in "other topics in machine learning". There are plenty of other options for the authors to select, which could be more suitable.

A: Regarding the primary area, our paper spans several topics, including reinforcement learning, optimization, foundation model, and interpretability, but does not fit neatly into any single one, which motivated our choice. We are open to changing it to a more suitable category if the paper is accepted.

Comment 2

The paper is very hard to understand, with loads of Propositions and Remarks, and a lot of them contains assumptions, which make me believe that the method barely work in practice.

A: Our propositions are relatively simple and are accompanied by remarks intended to help readers understand their implications. In the revised version, we have explicitly stated the assumptions used in Sections 4 and 5 to improve clarity.

We respectfully disagree with the reviewer that "a lot of them contains assumptions, which make me believe that the method barely work in practice". We wish to clarify that, (i) apart from Section 6, we only use two kinds of assumptions: the gradient-flow assumption and directional second-order bounds; (ii) Section 6 specifically analyzes the second-order bounds and explaining the intuition behind them, where the assumptions involved are standard regularity conditions. This discussion is also included in Remark 2.

Regarding practical effectiveness, we have conducted extensive and systematic experiments to investigate the mechanisms behind attention heads sparse-tuning. And now we provide additional empirical evidence supporting the assumptions' validity in practice (see "Response to all reviewers: Empirical validation of theory"). We hope the reviewer will take these empirical contributions into account when evaluating our work.

Comment 3

But the authors should be well noted that there are plenty of other problems reagarding to the propositions in the paper.

A: We wish the reviewer can state those issues explicitly, and we are more than happy and grateful to address them.

AuthorsOpenReview ↗

Response to Reviewer biZn (2/n)

W1

I found the theory behind the ranking less convincing than the paper suggests. (i) Corollary 3 is actually an equality as both sides reduce to t,Gˉ(Sk;t)2t,\bar G(S_k;t)^2, so it tells us nothing about the selected heads beyond Proposition 1. (ii) Proposition 4 only keeps the ordering as long as the gradient-norm ratio between two head sets is larger than eβte^{\beta t}. Since beta is never estimated, I don’t think the “hundreds of steps” claim can be drawn from it.
(iii) The bigger issue for me is the optimizer. Training uses Adam, and under Adam’s own quadratic model the best k heads are the ones that maximize ∑imi2/(vi+ϵ)\sum_i m_i^2/(\sqrt{v_i}+\epsilon). At the first step this is roughly the l1 norm of the head gradient, not the l2 norm. Proposition 5 only lands on the l2 ranking because the group penalty is put in the Euclidean metric instead of Adam’s. The constant in Proposition 6 is the ratio of the smallest to the largest vi\sqrt{v_i}. That ratio should be close to zero for heads with millions of parameters, so the bound says very little.

A: (i) We thank the reviewer for pointing this out. Yes, Corollary 3 holds with equality and we have revised it accordingly. However, we respectfully disagree with the point that "so it tells us nothing". Our intention was to provide an integral reformulation of Proposition 1, illustrating that if the selected heads maintain their advantage in time-averaged gradient norms, to provide optimality guarantee for finite-time loss reduction. This reformulation is intended to help readers better understand why we subsequently bounded the time-averaged gradient norms using the second-order bound assumption.

(ii) We would like to clarify a misunderstanding regarding the time scale tt in gradient flow. It's a standard result (see, e.g., Section 2 of [3]) that, NN gradient descent steps with learning rate η\eta correspond to a gradient-flow time horizon of t=Nηt=N\eta. Therefore, hundreds of gradient descent steps do not correspond to a large tt. In our experiments, the learning rate is typically 3×10−63\times10^{-6}, so 100 steps correspond to t=3×10−4t=3\times10^{-4}, and several hundred steps correspond to tt on the order of 10−310^{-3}, which is small. Even with β\beta on the order of 10310^{3}, the bound is still acceptable. In "Response to All Reviewers: Empirical Validation of Theory", we report empirical estimates of β,βA,βB\beta,\beta_A,\beta_B, which are typically of order 10110^1--10210^2, with the largest values remaining on the order of 10310^3, and that the initially selected top-50% heads generally maintain their gradient-norm advantage throughout training.

(iii) We adopted the most standard group lasso formulation [2] where a group-level ℓ2\ell_2 norm penalty is used. Our intention in Prop 5 is to provide an interpretable explanation of our selection criterion from an optimizer perspective, showing that the ℓ2\ell_2 ranking naturally arises from the standard group-lasso formulation. This criterion is also supported by the gradient-flow analysis and the empirical stability of gradient rankings (see "Response to all reviewers: Empirical validation of theory"). As for Prop 6, we agree with that the optimality bound is a bit loose, and have revised it from coordinate-wise to head-wise and moved it to appendix.

The reviewer’s proposed criterion is optimal yet less suitable for our fixed head selection: it's no longer equivalent to the ℓ1\ell_1 norm beyond the first step, and its dependence on Adam's moment estimates makes it harder to characterize using initial gradients alone.

To further support our choice, we report empirical approximation ratios comparing ℓ2\ell_2 selection with Adam-optimal selection under the same head budget. The ratios remain close to 11 (please see our response to Q3 for details), indicating that ours is simple and good enough.

AuthorsOpenReview ↗

Response to Reviewer biZn (3/n)

W2

The isolation and partitioning arguments don’t really explain why heads, or why disjoint sets. (i) Proposition 10 holds for any blocks that add up in the output, MLP included, and it only covers HfH_f. (ii) For continual learning, Proposition 7 bounds forgetting by the task-A gradient on SBS_B. But training on A pushes that gradient down on SAS_A itself, so near a task-A optimum the first-order bound would favor reusing SAS_A.

A: (i) We would like to clarify that Prop 12 specifically studies HoH_o (empirically supported by [1]), complementing Prop 10's analysis of HfH_f. Moreover, Prop 11 directly exploits the analytical structure of attention heads to establish an attention-specific second-order bound.

We agree that Prop 10 also applies to general additive blocks, but this generality shouldn't be viewed as a weakness. Our main goal is to provide a systematic empirical and theoretical understanding of attention-head-level sparse fine-tuning in LLM RL, including its optimization dynamics, second-order properties, and cross-task interference. Extending our analysis and sparse fine-tuning framework to other structures, such as MLP neurons, is an interesting direction that we leave for future work, as stated in Section 7.

(ii) Parameter isolation is a well-established approach to mitigating catastrophic forgetting. For example, it's already well discussed in [4,5]. We also provide intuitions in Section 5.1, "Updating the same parameters for task BB can overwrite the parameter changes induced by task AA. This concern is particularly pronounced when the trainable parameters are restricted to a sparse subset, motivating the use of disjoint head sets across tasks."

Furthermore, we wish to clarify one misunderstanding here: being near a task-A optimum doesn't imply that the gradient norm of SAS_A is smaller than the gradient norm of SAcS_A^c. Please see "Response to all reviewers: Empirical validation of theory" where we provide experimental evidence showing that the gradient norms on SAS_A generally remain larger than those on SAcS_A^c throughout training. Also, please see "Response to all reviewers: Ablation studies", where we show that partition is empirically better than reusing in mitigating forgetting.

[1] Fine-tuning attention modules only: Enhancing weight disentanglement in task arithmetic. Jin et al. ICLR 2025.

[2] Model Selection and Estimation in Regression with Grouped Variables. Yuan et al. Journal of the Royal Statistical Society Series B: Statistical Methodology.

[3] Continuous vs. Discrete Optimization of Deep Neural Networks. Elkabetz et al. NeurIPS 2021.

[4] Overcoming Catastrophic Forgetting with Hard Attention to the Task. Serra et al. ICML 2018.

[5] Towards guarantees for parameter isolation in continual learning. Lanzillotta et al.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to all reviewers: Empirical validation of theory

As multiple reviewers have asked about our empirical validation of the assumptions and theories, we address this concern here. These results and discussions are now included in the revised Appendix B.3 and Remark 2.

Table: Consistency of top-50% attention-head rankings throughout sparse RL fine-tuning. We report the overlap ratio (%) between the top-50% heads identified at each training step and those at initialization, across different models and tasks. The initially selected top-50% heads generally maintain their gradient-norm advantage throughout training.

ModelStepMathIFCoding
Qwen3-1.7B0100.0100.0100.0
20092.986.688.4
40094.672.387.5
60091.188.4
Qwen3-4B0100.0100.0100.0
20088.293.891.0
40084.793.192.4
60091.791.7

Table: Directional second-order quantities measured at different training steps throughout sparse RL fine-tuning. We report ∣(∇θSL(θ(t;S)))⊤∇θS2L(θ(t;S))∇θSL(θ(t;S))∣∥∇θSL(θ(t;S))∥22\frac{\left|(\nabla_{\theta_S}\mathcal L(\theta(t;S)))^\top\nabla_{\theta_S}^2\mathcal L(\theta(t;S))\nabla_{\theta_S}\mathcal L(\theta(t;S))\right|}{\Vert\nabla_{\theta_S}\mathcal L(\theta(t;S))\Vert_2^2}.

ModelStepMathIFCoding
Qwen3-1.7B07.97×10−27.97\times 10^{-2}2.35×1032.35\times 10^{3}3.22×1013.22\times 10^{1}
2001.02×10−21.02\times 10^{-2}8.18×1008.18\times 10^{0}7.11×1007.11\times 10^{0}
4009.16×10−29.16\times 10^{-2}1.48×1031.48\times 10^{3}1.37×1011.37\times 10^{1}
6008.56×1018.56\times 10^{1}4.80×1004.80\times 10^{0}
Qwen3-4B01.45×1021.45\times 10^{2}8.51×1018.51\times 10^{1}1.09×1021.09\times 10^{2}
2005.13×1025.13\times 10^{2}2.71×1022.71\times 10^{2}1.62×1011.62\times 10^{1}
4004.53×1024.53\times 10^{2}5.45×1035.45\times 10^{3}4.10×1014.10\times 10^{1}
6001.55×1031.55\times 10^{3}2.57×1012.57\times 10^{1}

Table: Directional second-order quantities measured at different training steps throughout stage-II of continual RL fine-tuning on tasks A→BA\rightarrow B, using the top-50% →\rightarrow bottom-50% head allocation strategy. The old-task and new-task terms are

∣∇θSBLA(θ(t))⊤∇θSB2LA(θ(t))∇θSBLB(θ(t))∣∣∇θSBLA(θ(t))∣2∣∇θSBLB(θ(t))∣2\frac{\vert \nabla_{\theta_{S_B}}\mathcal L_A(\theta(t))^\top\nabla_{\theta_{S_B}}^2\mathcal L_A(\theta(t))\nabla_{\theta_{S_B}}\mathcal L_B(\theta(t))\vert}{\vert\nabla_{\theta_{S_B}}\mathcal L_A(\theta(t))\vert_2 \vert\nabla_{\theta_{S_B}}\mathcal L_B(\theta(t))\vert_2} and ∣∇θSBLB(θ(t))⊤∇θSB2LB(θ(t))∇θSBLB(θ(t))∣∥∇θSBLB(θ(t))∥22\frac{\vert \nabla_{\theta_{S_B}}\mathcal L_B(\theta(t))^\top\nabla_{\theta_{S_B}}^2\mathcal L_B(\theta(t))\nabla_{\theta_{S_B}}\mathcal L_B(\theta(t))\vert}{\|\nabla_{\theta_{S_B}}\mathcal L_B(\theta(t))\|_2^2}, respectively. Step 0 denotes the stage-I final checkpoint before training task BB.

ModelTask sequenceStage-II stepOld taskNew task
Qwen3-1.7BMath →\rightarrow Coding05.90×10−25.90\times 10^{-2}1.97×1011.97\times 10^{1}
2004.38×10−34.38\times 10^{-3}2.39×10−12.39\times 10^{-1}
4006.95×10−26.95\times 10^{-2}1.26×1001.26\times 10^{0}
6004.98×10−24.98\times 10^{-2}3.39×1003.39\times 10^{0}
Qwen3-4BMath →\rightarrow IF08.99×1008.99\times 10^{0}6.42×1016.42\times 10^{1}
2002.74×1002.74\times 10^{0}8.26×1018.26\times 10^{1}
4008.24×1018.24\times 10^{1}5.55×1005.55\times 10^{0}
6001.49×1011.49\times 10^{1}9.72×1019.72\times 10^{1}
Coding →\rightarrow IF09.34×10−29.34\times 10^{-2}1.87×1021.87\times 10^{2}
2003.68×10−13.68\times 10^{-1}2.47×1012.47\times 10^{1}
4005.85×10−15.85\times 10^{-1}6.73×1016.73\times 10^{1}
6007.65×10−47.65\times 10^{-4}2.91×1012.91\times 10^{1}

Clarification on the 1−e−2βt2βt,e2βt−12βt\sqrt{\frac{1-e^{-2\beta t}}{2\beta t}},\sqrt{\frac{e^{2\beta t}-1}{2\beta t}} appearing in Prop 4 and 9

NN gradient descent steps with learning rate η\eta correspond to a gradient-flow time horizon of T=NηT=N\eta (see, e.g., Section 2 of https://arxiv.org/abs/2107.06608). In our experiments, the learning rate is typically 3×10−63\times10^{-6}, so 100 steps correspond to t=3×10−4t=3\times10^{-4}, and several hundred steps correspond to tt on the order of 10−310^{-3}, which is small. We have e2βt−12βt≤3\sqrt{\frac{e^{2\beta t}-1}{2\beta t}}\le 3 and 1−e−2βt2βt≥0.5\sqrt{\frac{1-e^{-2\beta t}}{2\beta t}}\ge 0.5 in most cases, which is still acceptable.

AuthorsOpenReview ↗

Response to all reviewers: Ablation studies

As multiple reviewers have asked about our ablation studies, we address these concerns here. These discussions are now included in the revised Appendix B.4.

Ablation study: partition v.s. reuse the heads

Table: Comparison of old-task retention after sequential fine-tuning on tasks A →\rightarrow B. Both methods start from the same stage-I checkpoint and train 50% of attention heads per stage. In stage II, Partition trains the complementary heads, while Reuse continues training the stage-I heads. We report old-task accuracy (%) on in-distribution and out-of-distribution evaluations, with changes from stage I. Higher stage-II accuracy is highlighted in bold. Stage-I performance is reported as the baseline before stage-II training.

ModelTask sequenceStage-I headsAfter Stage I (In-dist)After Stage I (Out-dist)Partition (In-dist)Partition (Out-dist)Reuse (In-dist)Reuse (Out-dist)
Qwen3-1.7BMath →\rightarrow CodingTop 50%37.5075.6035.80 (↓ 1.70)75.20 (↓ 0.40)28.90 (↓ 8.60)67.60 (↓ 8.00)
Qwen3-4BMath →\rightarrow IFTop 50%59.3088.4060.00 (↑ 0.70)87.20 (↓ 1.20)55.50 (↓ 3.80)84.40 (↓ 4.00)
Qwen3-4BCoding →\rightarrow IFTop 50%69.0051.4068.30 (↓ 0.70)52.60 (↑ 1.20)67.72 (↓ 1.28)46.80 (↓ 4.60)

Ablation study: layer-level budget-fixed head selection

Table: Comparison of layer-level budget-fixed head selection on instruction following. We select either the top or bottom 50% of attention heads within each layer, ensuring identical layer-wise training budgets. We report in-distribution and out-of-distribution accuracy (%). Higher accuracy is highlighted in bold.

ModelBase (In-dist)Base (Out-dist)Per-layer Top 50% (In-dist)Per-layer Top 50% (Out-dist)Per-layer Bottom 50% (In-dist)Per-layer Bottom 50% (Out-dist)
Qwen3-1.7B41.7017.7052.6020.0048.8019.67
Qwen3-4B50.5021.0057.8026.6755.5024.33
Mistral-7B27.4018.3084.6033.3361.5024.33

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.