acceptodds
Under review as a conference paper at ICLR 2027

Specialization of softmax attention heads: insights from the single-location model

Abstract

Multi-head attention enables transformer models to represent multiple attention patterns simultaneously. Empirically, head specialization emerges in distinct stages during training, while many heads remain redundant and learn similar representations. We propose a theoretical model capturing this phenomenon, based on the multi-index and single-location regression frameworks. In the first part, we analyze the training dynamics of multi-head softmax attention under SGD, revealing an initial unspecialized phase followed by a multi-stage specialization phase in which different heads sequentially align with latent signal directions. In the second part, we study the impact of attention activation functions on performance. We introduce the Bayes-softmax attention, which achieves optimal prediction performance in this setting.

Primary area:
learning theory
Keywords:
multi-head attention, softmax attention, sparse information retrieval, high-dimensional limit, multi-index models, high-dimensional statistics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rXHzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The draft studies the training behavior of multi-head attention. The authors study, under a certain setting, the dynamics of multi-heads, including the redundancy and emerging behaviors. There are two stages of behaviors in their setting: the first is when all the heads tend to learn the easiest representation, i.e., the average, then after certain steps, the heads start to specialize. Based on these observations, the authors propose a closed system of equations tracking the full evolution of head alignments along the SGD trajectory and introduce the Bayes-softmax attention, which, under this specific setting, prescribes the optimal number of heads and the way to normalize them, while standard softmax is generally suboptimal because redundant or poorly specialized heads can introduce persistent noise.

Critical strengths
  1. The authors propose a tractable framework of a low-dim system of head-signal alignment and orthogonal overleaps, connecting the head specialization and training dynamics. In this framework, they are able to connect the early collective learning and later specialization behaviors.
  2. Also within the framework, they deserve the Bayes predictor and show how joint normalization across heads and tokens can represent it for a finite support latent distribution, which gives Bays-softmax.
Critical weaknesses
  1. Regarding Proposition 4.3: For L=5L=5, H=3H=3, and signal k^=±Bk∗\hat{k}=\pm Bk^{\ast}, softmax-v keys (B−1/2k∗,−B−1/2k∗,0)(B^{-1/2}k^{\ast},-B^{-1/2}k^{\ast},0) with fixed weights (4,4,−5)(4,4,-5) under Eq. (3) yield vanishing risk as B→∞B\to\infty after D→∞D\to\infty, through cancellation. This challenges the strong-signal interpretation of Proposition 4.3, though not fixed-signal positivity; the readout constraints and lower-bound quantifiers need clarification.
  2. And about Lemma A.8: Orthogonality makes the Gaussian noise projections independent, but does not preserve independence after marginalizing over θ\theta. For example, θ∼Unif(0,1),(2,−1)\theta\sim\mathrm{Unif}\\{(0,1),(2,-1)\\} has mean (1,0)(1,0) but cross-covariance −1-1. The expectation factorization is therefore unjustified for general PθP_{\theta}, leaving the mean-direction invariance underlying Proposition 3.3 unsupported without additional assumptions.

correct me if I'm wrong for both questions.

Questions for authors

They are mostly about Proposition 4.3 and Lemma A.8. Please see above.

1 comment

We thank the reviewer for the detailed comments and remarks.

W1, Q: the ill-definiteness of the softmax-v holds for L=2L=2 tokens. For larger LL, as examplified by the reviewer, it may not hold. Still, the softmax-v is sub-optimal because it only depends on the difference between its inputs, while the Bayes estimator does not. Being able to dynamically deactivate heads when input scores are low is needed. Notice that the addition of the constant bias in softmax-1 precisely allows to break the invariance of softmax w.r.t. a constant shift, while turning off heads. We added this explanation to the revised article.

More precisely, we show that at L=2L=2 for symmetric data distribution the v-softmax cannot recover anything; and for L≥2L\ge 2, for any finite signal there is a strictly positive data-dependent bound between the softmax-v and the Bayes risk; and this bound does not depend on the number of heads or the parameters of the softmax-v. We corrected Proposition 4.3 and the associated appendices in the revised version of the article.

W2, Q: indeed Lemma A8 requires an additional assumption on PθP_\theta, that the mean direction is an eigendirection of the covariance. It holds for the flipping spike distribution we consider. In the general case when this assumption does not hold, there is a minor change to Proposition 3.3 of the main: the invariant manifold always holds, and at initialization the gradient points towards the mean signal; consequently the heads stay on the invariant manifold and initially move towards the mean signal. We corrected Prop. 3.3 and Lemma A.8 in the revised article. Edit: following discussion with reviewer 76kH we made the 2nd part of Lemma A8 more solid with a rotational-invariance assumption on PθP_\theta. The mean-direction invariance directly follows then.

We hope to have addressed all the reviewer's concerns about the article. Overall, we would like to emphasize that our main results and contributions remain unchanged (model, closed-form description of SGD, fast unspecialized and slow specialization phases, Bayes risk and importance of head normalization).

Reviewer_HdQzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies head specialization in a high-dimensional single-location regression model. It builds a theoretical framework to characterize the multi-head training dynamics and describes an initial unspecialized phase followed by sequential specialization along signal directions. It also compares attention normalizations and introduces Bayes-softmax, which can represent the Bayes estimator for finitely supported signal distributions. Synthetic experiments and an MNIST-based detection task illustrate the predicted behaviors.

Critical strengths
  1. The order-parameter framework provides a tractable setting for understanding specialization in softmax attention. The local curvature analysis connects symmetry breaking to the signal covariance, and the predicted training dynamics agree well with finite-dimensional SGD simulations.

  2. The explicit Bayes estimator provides a useful benchmark for comparing attention architectures. Its representation by Bayes-softmax connects the normalization rule to the statistical structure of the task.

Critical weaknesses
  1. The expressivity comparison in Proposition 4.3 needs correction. The statement should clarify whether its error lower bound can approach zero as the signal becomes stronger, or must remain above a positive constant independent of signal strength. For softmax-v, signed coefficients can cancel misaligned heads. For example, take L=H=3L=H=3, F=1F=1, θ=±B\theta=\pm B with equal probability, and $ k_1=B^{-1/2}k^{\ast},\ k_2=-B^{-1/2}k^{\ast},\ k_3=0,\ v=(6,6,-9).$ As D→∞D\to\infty followed by B→∞B\to\infty, the attention maps approach eϵe_\epsilon, (1−eϵ)/2(\mathbf{1}-e_\epsilon)/2, and 1/3\mathbf{1}/3, up to swapping the first two. Accounting for 1/H1/H aggregation, their combination is$ 2e_\epsilon+(\mathbf{1}-e_\epsilon)-\mathbf{1}=e_\epsilon.$ Thus, the risk tends to zero. This challenges the claimed strong-signal obstruction for softmax-v and the argument that input-dependent head deactivation is necessary.

  2. The proof of the specialization time in Proposition 3.4 is incomplete. With m~=Dζm\tilde m=D^\zeta m and γ/Nb=clrD−1−2ζ\gamma/N_b=c_{\mathrm{lr}}D^{-1-2\zeta}, the projected single-sample gradient has O(1)O(1) variance on bounded parameter sets. The diffusion covariance therefore appears to scale as O((γ/Nb)D2ζ)=O(D−1),O\left((\gamma/N_b)D^{2\zeta}\right)=O(D^{-1}), rather than approaching the nonzero limit claimed in Lemma A.5. Lemma A.6 uses a convergence result that applies to time intervals whose length stays fixed as DD grows. However, the proof then uses this result at times of order log⁡D\log D, which increase with DD. Accuracy over a fixed time interval does not automatically imply accuracy over these longer intervals, since approximation errors may accumulate or be amplified during training. So the authors need an additional argument showing that these errors remain small enough until specialization occurs. Without this control, the proof does not establish the claimed Θ(log⁡D)\Theta(\log D) specialization time under the stated assumptions.

Questions for authors
  1. Are the softmax-v weights constrained to be nonnegative? If signed weights are allowed, how does Proposition 4.3 account for the construction above? Please also specify whether the claimed positive lower bound is independent of signal strength and applies to all model parameters, including those chosen separately for each signal strength.

  2. Could the authors explicitly derive the diffusion covariance in Lemma A.5 and justify the approximation over O(log⁡D)O(\log D) times?

  3. Which mechanisms are new relative to Zhang et al. [1], who already show sequential eigenvector learning and sharp transitions at small initialization? Please also compare with He et al. [2] on learned head patterns, superposition, and zero-sum output-value weights, and Zucchet et al. [3] on plateau lengths in single-location regression. Finally, please discuss the relationship between Bayes-softmax and the mixture-of-keys attention of Nguyen et al. [4], whose normalization also sums jointly over token positions and mixture components, and clarify the differences in parameterization and optimality guarantees.

[1] Yedi Zhang, Aaditya K. Singh, Peter E. Latham, and Andrew M. Saxe. Training Dynamics of In-Context Learning in Linear Attention. ICML, 2025.

[2] Jianliang He, Xintian Pan, Siyu Chen, and Zhuoran Yang. In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention. ICML, 2025.

[3] Nicolas Zucchet, Francesco d'Angelo, Andrew K. Lampinen, and Stephanie C. Y. Chan. The Emergence of Sparse Attention: Impact of Data Distribution and Benefits of Repetition. NeurIPS, 2025.

[4] Tam Minh Nguyen, Tan Minh Nguyen, Dung D. D. Le, Duy Khuong Nguyen, Viet-Anh Tran, Richard Baraniuk, Nhat Ho, and Stanley Osher. Improving Transformers with Probabilistic Attention Keys. ICML, 2022.

3 comments

We thank the reviewer for the detailed comments and remarks.

W1, Q1: for the lower-bound on softmax, it is data-dependent and parameter-independent. We made this point more precise in the revised article.

The ill-definiteness of the softmax-v holds for L=2L=2 tokens. For larger LL, as examplified by the reviewer, it may not hold. Still, the softmax-v is sub-optimal because it only depends on the difference between its inputs, while the Bayes estimator does not. Being able to dynamically deactivate heads when input scores are low is needed. Notice that the addition of the constant bias in softmax-1 precisely allows to break the invariance of softmax w.r.t. a constant shift, while turning off heads. We added this explanation to the revised article.

More precisely, we show that at L=2L=2 for symmetric data distribution the v-softmax cannot recover anything; and for L≥2L\ge 2, for any finite signal there is a strictly positive data-dependent bound between the softmax-v and the Bayes risk; and this bound does not depend on the number of heads or the parameters of the softmax-v. We corrected Proposition 4.3 and the associated appendices in the revised version of the article.

W2, Q2: the reviewer is right, the proof of proposition 3.4. was incomplete. In the revised version we fix the sufficient statistics scaling m~=D1/2m\tilde{m} = D^{1/2}m and keep the learning rate scaling to make sure that the diffusion covariance doesn't dissappear. We also fix the proof of lemma A.6, first obtaining overlap of order D−1/2D^{-1/2} in constant time and then controlling the full high dimensional SGD trajectory to show that escape time is O(log⁡D)O(\log{D}) by controlling the martingale noise term, the drift and the residual separately similar to the technique of Ben Arous et al. [5]. It leads to the specialization in Prop 3.4 being achieved with a constant arbitrary large probability.

Q3: In Zhang et al. [1] the problem is in-context linear regression learned with linear attention model. The mechanism they describe is that the heads learn eigenvectors of the input token covariance one by one, with unspecialized heads being exactly 0. In our model, the attention is non-linear and the dynamics exhibit the phase, where the unspecialized heads learn the mean signal direction instead of staying 0, and when heads start specializing, they don't learn different eigenvectors efe_f in different heads, but instead they split between heads that learn efe_f and the heads that learn −ef-e_f, in the end learning all possible signed sum combinations of eigenvectors. The common point between Zhang et al. [1] and our work is that the spectrum of the input covariance determines the order of the learned eigendirections.

He et al. [2] consider KQKQ matrices that are not restricted to be low-rank, so we cannot do direct comparison between the learned patterns of key vectors or the superposition behavior. At the same time zero-sum output-value weights mechanism gives intuition of how the negative head weights in softmax-v model can be learned (and considering this is crucial to understanding noise canceling of the redundant heads).

Zucchet et al. [3] model attention scores completely independent from the input sequence, therefore the comparison cannot be direct, but they derive that the escape time depends on the dimension TT of the learned attention parameter as C1Tln⁡C2TC_1 T\ln{C_2 T}, which is similar to our Dlog⁡DD\log{D} scaling when the model starts from a saddle point.

Mixture-of-keys attention proposed by Nguyen et al. [4] would be an intuitive generalization of our Bayes-softmax model to the case of self-attention with learned key and query matrices. In our case we are able to derive B-softmax and demonstrate optimality on our specific task because of intentional simplifications in the model model, while Nguyen et al. [4] study the standard self-attention variant and adopt a probabilistic view on the query vectors as samples from the Gaussian mixtures and motivating their design choice by allowing a more expressive probability distribution class for the query vectors. They don't make assumptions on the inputs distribution, so they don't give optimality guarantees, but they demonstrate empirically the advantage of their model. We add this reference to the introduction in the revised version.

We hope to have addressed all the reviewer's concerns about the article. Overall, we would like to emphasize that our main results and contributions remain unchanged (model, closed-form description of SGD, fast unspecialized and slow specialization phases, Bayes risk and importance of head normalization).

[5] Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. Online stochastic gradient descent on non-convex losses from high-dimensional inference

Thank you for the detailed response and revision. The corrected Proposition 4.3 removes the incorrect claim about softmax-v, and I appreciate the new two-stage proof of Proposition 3.4. But I still have some remaining concerns that under the new scaling γ/Nb=clrD−1\gamma/N_b=c_{\mathrm{lr}}D^{-1}, the quadratic term in the SGD update of rhhr_{hh} gives a Θ(1)\Theta(1) drift that is missing from Eqs. (72) and (83), which may change the fixed point used in Lemma A.6. And the revised Proposition 4.3 separates softmax-v from softmax-1 only for L=2L=2, while the introduction, Section 4 and the discussion of Fig. 4 (L=5L=5) still present input-adaptive normalization as necessary. I am therefore maintaining my rating of 2.

We thank the reviewer for noticing the scaling inconsistency in Prop. 3.4. Indeed, we should consider γNb−1=clrD−1−2ζ\gamma N_b^{-1} = c_\text{lr}D^{-1 -2 \zeta} and mhf→m~hf=D1/2+ζmhfm_{hf} \rightarrow \tilde{m}_{hf} = D^{1/2 + \zeta}m_{hf} for some arbitrarily small constant ζ>0\zeta>0, which is consistent with previous assumptions. Notice, that it doesn't affect the escape time order in Lemma A.6. We correct it in the section A.3.2 of appendix.

We understand that current phrasing of section 4 might not discuss the case where softmax-v from softmax-1 both succeed in enough detail compared to the case where softmax-v is limited. Prop. 4.3 provides an intuition and a particular case where the two activations can be separated, and Fig. 4 (right panel, Isotropic Gaussian distribution) provides numerical evidence for the benefit of the later under some assumptions on the spike distribution. We will add the softmax-v success example and highlight that assumptions on the inputs affect the conclusions about activation functions in the revised version.

Reviewer_76kHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how attention heads specialize in a high-dimensional single-location regression task. One token contains a structured latent signal, while the remaining tokens contain noise. The authors derive a low-dimensional description of training dynamics and analyze an initial phase of shared learning followed by head specialization. They also investigate sequential feature learning and compare different attention normalization schemes. Their proposed Bayes-softmax jointly normalizes scores across heads and tokens and can represent the Bayes estimator for finitely supported signal distributions in the high-dimensional limit. Experiments on synthetic data and an MNIST-based detection task support the analysis.

Critical strengths
  1. The paper provides a useful account of how head specialization develops during training. The order parameters make the evolution of feature alignment easy to interpret. The agreement between the theoretical predictions and finite-dimensional SGD in Figure 1 supports this description. In particular, the distinction between early shared learning and later specialization offers a concrete explanation for staged head emergence.

  2. The connection between attention normalization and Bayesian inference is well motivated. Propositions 4.1–4.2 show how joint normalization across heads and tokens can represent the Bayes estimator in the stated setting. This gives a principled explanation for allowing head contributions to depend on the input, rather than requiring every head to contribute equally.

Critical weaknesses
  1. The proof of the specialization timescale needs further clarification. In Appendix A.3, the authors use a limiting SDE and a local approximation near an unstable stationary point to derive escape on an O(log⁡D)O(\log D) time horizon. The proof does not clearly explain how the approximation remains valid over this growing time horizon and until the process leaves the local neighborhood. There is also an apparent scaling inconsistency: Lemma A.5 defines m~=Dζm\widetilde m=D^\zeta m, while Lemma A.6 later uses a threshold proportional to D2ζD^{2\zeta}. These points matter because the specialization timescale is a central theoretical result.

  2. The evidence for the normalization advantage remains tied to the restricted architecture. The main model uses identity value transformations and uniformly averages head outputs. The softmax-v baseline adds learned scalar head weights, but does not cover general learned value and output projections. The MNIST experiment changes the signal distribution while retaining the same basic architecture. The results therefore support the proposed mechanism within this model, but provide limited evidence for how it carries over to richer attention models. A controlled experiment with learned projections, or a more explicit restriction of the broader claims, would strengthen the paper.

Questions for authors
  1. Could you clarify the rescaling and escape threshold in Lemmas A.5–A.6? What error bound ensures that the limiting-process approximation remains valid over an$O(\log D)$ time horizon and up to the onset of specialization?

  2. The proof of Lemma A.8 uses independence between orthogonal projections of χ∗\chi^*. These projections are conditionally Gaussian given the latent signal, but need not be jointly Gaussian after averaging over a general PθP_\theta. What assumptions on PθP_\theta are needed for this step, and should the scope of Proposition 3.3 be restricted?

  3. Have the authors tested whether the normalization advantage persists with learned value and output projections? If such results are unavailable, could they clarify which conclusions apply specifically to the restricted architecture?

3 comments

We thank the reviewer for the detailed comments and remarks.

W1, Q1: the reviewer is right that the dynamics convergence on the timescale log⁡D\log{D} is not given. We change the sufficient statistics scaling to m~=D1/2m\tilde{m} = D^{1/2}m and fix the proof of Lemma A.6 using the two stages: obtaining overlap of order D−1/2D^{-1/2} in constant time and considering the full SGD dynamics initialized at the end of the previous step. The second step requires controlling the drift, residual and martingale terms, which is similar to the technique of Ben Arous et al. [1]. It leads to the specialization in Prop 3.4 being achieved with a constant arbitrary large probability.

Q2: indeed Lemma A8 requires an additional assumption on PθP_\theta, that the mean direction is an eigendirection of the covariance. It holds for the flipping spike distribution we consider. In the general case when this assumption does not hold, there is a minor change to Proposition 3.3 of the main: the invariant manifold always holds, and at initialization the gradient points towards the mean signal; consequently the heads stay on the invariant manifold and initially move towards the mean signal. We corrected Prop. 3.3 and Lemma A.8 in the revised article.

W2, Q3: learned value and output projections introduce a new way to suppress redundant heads and can play more important role especially if the signal and noise subspaces do not coincide (if the value matrix projects on the signal subspace, it automatically suppresses the noise). We agree that it is an interesting direction to study, and notice that the benefits of softmax-1 were empirically studied in realistic experiments for example by Kaul et al. [2], and, as was pointed out by reviewer HdQz, Nguyen et al. [3] demonstrate the advantage of an attention variant similar to B-softmax. In our setting, value and output matrices would not change anything because of the isotropy of the tokens.

We hope to have addressed all the reviewer's concerns about the article. Overall, we would like to emphasize that our main results and contributions remain unchanged (model, closed-form description of SGD, fast unspecialized and slow specialization phases, Bayes risk and importance of head normalization).

[1] Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. Online stochastic gradient descent on non-convex losses from high-dimensional inference

[2] Prannay Kaul, Chengcheng Ma, Ismail Elezi, and Jiankang Deng. From attention to activation: Unraveling the enigmas of large language models

[3] Tam Minh Nguyen, Tan Minh Nguyen, Dung D. D. Le, Duy Khuong Nguyen, Viet-Anh Tran, Richard Baraniuk, Nhat Ho, and Stanley Osher. Improving Transformers with Probabilistic Attention Keys

Thanks for the authors' detailed response. For Lemma A.6 I think the change of timescale is acceptable. For Lemma A.8 I still have some concerns on the independence and so as Proposition 3.3. I think this is a vaulable work but still need to be more solid, so I will keep my score for now.

We thank the reviewer for answering to our rebuttal, for all the work and for the positive assessment. To make the 2nd part of Lemma A.8 more solid we revised the manuscript with a stronger rotational-invariance assumption on PθP_\theta, directly insuring the gradient stays colinear to Eθ\mathbb{E}\theta. This assumption holds for the flipping spike distribution we consider. We would like to stress that all our main results remain unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.