Not All Attention Heads Help: Spectral Attention Head Selection for Low-Rank Adaptation

Low-Rank Adaptation (LoRA) enables efficient adaptation of pretrained large language models by parameterizing a weight matrix as $W=W_0+BA$, where only the low-rank factors $A$ and $B$ are optimized. Existing LoRA variants mainly improve the initialization, or optimization process of these low-rank factors, but rarely examine how LoRA is optimized when applied to multi-head attention. An attention projection $W_0$ contains head-specific blocks, while LoRA applies one low-rank update to the entire projection. The factor $B$ can be divided into head-specific blocks, whereas $A$ is shared and updated by the sum of gradient contributions from all heads. The squared norm of this aggregated gradient contains cross-head terms, so the effect of each head depends on how its gradient combines with the others. This cross-head coupling motivates us to investigate selective aggregation for updating $A$. In this paper, we propose a spectral head selection method that selects which heads contribute to the update of $A$, while keeping all $B$ trainable. Specifically, we first derive a reference subspace that preserves the maximum dense gradient energy across heads, and use spectral clustering to select an initial head set based on how much of its gradient lies in this subspace. To maintain effective selection throughout training, we further develop a lightweight adaptive mechanism that refines the selected head set without repeatedly performing costly spectral clustering. Our method introduces no additional parameters or inference overhead and can be integrated with various LoRA variants. Experiments on NLU and NLG tasks across multiple models, together with ablation studies, consistently demonstrate the effectiveness of our method.