- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to finetune language models with reinforcement learning (RL) by updating only a small number of attention heads. The authors rank the attention heads using their gradient norms at the beginning of training and select some of them for updating. For CL, they assign different groups of heads to different tasks, aiming to reduce interference and forgetting between tasks. The paper also provides theoretical analysis to explain why gradient norms may related to training progress. What's more, why separating attention heads may help reduce task interference. Experiment on mathematical reasoning, instruction following, and coding show that most updating ranked heads will performs better than updating low ranked heads. The results also show that head parttioning creates a trade-off between learning new tasks and maintaining performance on previous tasks.
Critical strengths
- The paper presents a simple approach to selective RL fine-tuning.
The authors use initial gradient norms to select the attention heads to update. The experiments show that high-ranked heads generally perform better than low-ranked heads across the tested models and tasks. In some settings, selective tuning also gives results comparable to denser tuning methods. These results provide initial support for using gradient norms as a head-selection criterion.
- The paper examines whether head partitioning can reduce interference between tasks.
In several sequential-training settings, head partitioning retains previous-task performance better than full-attention tuning. The head-reset experiments also suggest that some task-specific changes are localized within the assigned head subsets. Overall, these results provide preliminary evidence that head partitioning may help balance learning new tasks and retaining earlier capabilities.
Critical weaknesses
- The connection between the theoretical analysis and practical language-model fine-tuning could be clearer.
The theoretical results rely on several assumptions, and the current experiments provide limited evidence about whether the same conclusions hold for larger and more widely used language models. Additional discussion or experiments would help clarify the practical scope of the theoretical findings.
- The reliability of initial gradient norms as a head-selection criterion needs further analysis.
Gradients in reinforcement learning can vary during training and across tasks. It would be helpful to study how stable the head rankings are and to compare this criterion with other simple selection methods. Such analysis would provide stronger support for this important design choice.
Questions for authors
1, See Critical Weaknesses 2, See Critical Weaknesses
3, The paper’s motivation relies partly on the idea that task adaptation is concentrated in task-relevant attention heads and that gradient information can identify such heads. Related observations have previously been reported by [1].
4, Could the authors specify exactly which parameters are treated as belonging to an individual attention head in both the theory and implementation? In particular, does a head block include the corresponding slices of the Q/K/V/O projections, and is the gradient norm normalized by the number or scale of parameters? I think the experiments should add more ablation study to compare with other related methods to prove that your method is better.
[1] Aristeidis Panos, Rahaf Aljundi, Daniel Olmeda Reino, and Richard E. Turner. “Efficient Few-Shot Continual Learning in Vision-Language Models.” Transactions on Machine Learning Research, 2025.
4 comments
Response to Reviewer tXpp (2/2)
Q3
The paper’s motivation relies partly on the idea that task adaptation is concentrated in task-relevant attention heads and that gradient information can identify such heads. Related observations have previously been reported by [1].
A: Thank you for pointing out this related work! We've included the discussion of [1] in the revised Related work section. This recent empirical work [1] uses gradient-based attention-head selection in the vision encoder of VLMs for class-incremental few-shot learning.
We acknowledge that our implemented algorithm is indeed based on the idea that task adaptation is concentrated in task-relevant attention heads, which is also discussed in [2,3], or other structural units, such as layers [4] and neurons [5]. And they're already discussed in our Related work section.
We wish to clarify that, rather than simply transferring this idea to another setting, our main focus is to provide a systematic understanding of attention-head sparsity in LLM RL fine-tuning, including optimization dynamics (Section 4, 5) and second-order analysis (Section 6). We also investigate how to allocate attention heads across complex and different RL tasks to mitigate forgetting, whereas [1] mainly considers continual learning across different object categories in visual adaptation.
Furthermore, while [1] finds that few attention heads (two attention heads per layer) are sufficient for visual adaptation, our sparsity–performance trade-off experiments suggest that RL fine-tuning benefits from a relatively larger proportion of attention heads (e.g., 30–50%).
[1] Efficient Few-Shot Continual Learning in Vision-Language Models. Panos et al. TMLR 2025.
[2] Compressed sensing for capability localization in large language models. Bair et al. ICML 2026.
[3] Specialization of softmax attention heads: insights ´ from the high-dimensional single-location model. Sagitova et al.
[4] Is one layer enough? training a single transformer layer can match full-parameter rl training. Zhang et al.
[5] LLM-Pruner: On the Structural Pruning of Large Language Models. Ma et al. NeurIPS 2023.
Q4
(i) Could the authors specify exactly which parameters are treated as belonging to an individual attention head in both the theory and implementation? In particular, does a head block include the corresponding slices of the Q/K/V/O projections, and is the gradient norm normalized by the number or scale of parameters? (ii) I think the experiments should add more ablation study to compare with other related methods to prove that your method is better.
A: (i) Yes. In both the theory and implementation, the gradient norm is calculated using the corresponding slices of Q/K/V projections (we exclude O-projection gradients from the ranking score for simplicity). Specifically, query heads sharing one key/value pair are counted as in the same block. We didn't normlize the norm using either number of scale, as the attention blocks within a model have the same parameter size.
(ii) We've conducted several ablation studies during the rebuttal. Please see "Response to All Reviewers: Ablation studies" for details.
- We conducted a controlled layer-level budget-fixed ablation, selecting either the top or bottom 50% of heads within each layer, thus ensuring identical layer-wise training budgets. The results show consistent advantages of per-layer Top on instruction following.
- We conducted a same-budget ablation comparing head partitioning with reusing the stage-I heads for task B in continual learning. The results show that partitioning effectively improves old-task retention.
As we focus on systematically understanding the mechanisms of attention-head sparse fine-tuning, comparisons with sparse fine-tuning methods based on other parameter units are beyond the scope.
As the end of the author-reviewer discussion is approaching, we kindly ask for your review of our revised paper and response. Should you have any remaining questions or require further clarification, please don't hesitate to let us know. We would be more than happy to answer any further questions.
Thanks for the clarifications and additional experiments. The additional analysis and ablations partially strengthen the empirical support.
Although the response clarifies the differences from prior work, I remain unconvinced that the insights could represent sufficiently contribution. I will keep my original score.
Response to Reviewer tXpp (1/2)
W1
The connection between the theoretical analysis and practical language-model fine-tuning could be clearer. The theoretical results rely on several assumptions, and the current experiments provide limited evidence about whether the same conclusions hold for larger and more widely used language models. Additional discussion or experiments would help clarify the practical scope of the theoretical findings.
A: Please see "Response to all reviewers: Empirical validation of theory". We have added empirical values of β,βA,βB appearing in Props 4,9. We also clarified the validity of the obtained theoretical bounds.
W2
The reliability of initial gradient norms as a head-selection criterion needs further analysis. Gradients in reinforcement learning can vary during training and across tasks. It would be helpful to study how stable the head rankings are and to compare this criterion with other simple selection methods. Such analysis would provide stronger support for this important design choice.
A: Please see "Response to all reviewers: Empirical validation of theory". We reported the stable ranking consistency of gradient norms during RL training.