acceptodds
Under review as a conference paper at ICLR 2027

Mean–Covariance Competition Governs Collapse and Multi-Cluster States in Causal Attention Dynamics

Abstract

Causal self-attention can exhibit two distinct regimes: token representations may collapse toward a single dominant direction, forming attention sinks, or they may remain organized into multiple clusters. Despite extensive empirical observations, the mechanism governing the transition between these regimes remains poorly understood. In this work, we develop a dynamical and mean-field framework to explain this transition. We first use a first-order expansion of the causal softmax interaction to decompose the dynamics into two competing terms: a mean-driven alignment term, which depends on the prefix mean of previous tokens, and a covariance-driven routing term, which depends on the prefix covariance and the query–key interaction. We then show that, near an eigendirection of the value matrix, a spectral gap suppresses transverse perturbations and drives collapse toward a single dominant cluster, providing a mechanism for attention-sink formation. Conversely, when the cluster geometry keeps the prefix mean close to zero while maintaining nontrivial covariance, the alignment effect is weakened and covariance-driven routing remains active, allowing multi-cluster structures to persist. A key novelty of our approach is the use of Fourier analysis in the depth variable, which reveals the frequency-dependent structure of causal attention. These results identify causal attention dynamics as a competition between mean-driven spectral alignment and covariance-driven routing. These mechanisms are then examined by simulations and layerwise analyses of pretrained Llama-3.1-8B model.

Primary area:
learning theory
Keywords:
token dynamic, token clustering, causal attention, mean-field dynamic

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7n7sOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work tries to figure out why the token collapse problem (whether tokens converge to a single cluster or multiple clusters) occurs under causal attention dynamics. To solve it, they decompose a special case of causal attention (in which every token is normalized (e.g., x+=x+/∥x+∥x^+ = {x^+}/{\|x^+\|}), so that the updated representation lies on the unit hypersphere) into two components: mean-driven alignment and covariance-driven routing.

Then, for each case (when the tokens have already collapsed into either the single-cluster or multi-cluster regime), they analyze the local stability (using Fourier analysis or a first-order Taylor expansion) to show each component's effects on each regime (e.g., mean →\to single and covariance →\to multi).

Finally, by calculating the evolution of the trace and mean values, they try to show that such analysis can also be supported by real experiments.

Critical strengths

I think the overall problem setup and the intuition behind explaining the difference between single- and multi-cluster structures through the mean-covariance relationship are intuitively understandable (I like the characterization that the prefix mean handles alignment while the prefix covariance handles the routing term).

Critical weaknesses

I just wonder about the significance of this work.

  1. From Eq. (5), x˙i=Pxi(Vxˉi)⏞O(1)+βPxi(VΣiT⊤xi)⏞O(β)\dot x_i = \overbrace{P_{x_i} (V\bar x_i)}^{\mathcal O(1)} + \overbrace{\beta P_{x_i} (V\Sigma_i T^\top x_i)}^{\mathcal O(\beta)} under the Taylor expansion as β→0\beta\to 0. When we imagine a scenario like xˉi≈0\bar x_i\approx 0 while Σi≠0\Sigma_i\ne 0, I think this is mathematically valid. However, I cannot be certain whether an analysis under such conditions could have further potential beyond this work. Plus, I think the normalization condition is also quite a strong assumption (although such a condition makes the update representable as a simple tangential term), so, in short, could the authors clarify my concerns about the significance of these conditions?

  2. In Section 4.3, a stationary state exists under m(s,t)≈∑kαkvk=0m(s,t) \approx \sum _k \alpha_k v_k = 0 and Σ(s,t)≈∑kαkvkvk⊤=(1/d)⋅Id\Sigma(s,t) \approx \sum _k \alpha_k v_k v_k^\top = (1/d) \cdot I_d. Thus, Px(Vm)=0P_x(Vm) = 0 and Px(VΣT⊤x)=Px(x/d)=0(∵VT⊤=I)P_x(V\Sigma T^\top x) = P_x(x/d) = 0 (\because VT^\top = I ). Regarding lines 25-26, which state that covariance-driven routing remains active, allowing multi-cluster structures to persist, I just expect/hope that covariance dynamics makes multi-cluster structures persist (i.e., some non-zero dynamics keep each vector in such a system, for example, bmean≠0b^{\rm mean} \ne 0 & bcov.≠0b^{\rm cov.} \ne 0 but bmean+bcov.=0b^{\rm mean} + b^{\rm cov.}= 0). However, at stationarity, both terms are just zero, so it is hard to understand why this can be interpreted as stated in lines 25-26 (including the conclusion or other strong statements). I personally think that to state it as such, one should at least imagine a perturbed scenario like xi=vk(i)+εyi+O(ε2)x _ i = v _{k(i)} + \varepsilon y_i + \mathcal O(\varepsilon^2) and analyze the dynamics of y˙=Jy\dot y = Jy.

  3. Continuing from (2), do we actually need the covariance-driven routing under the authors' assumptions? Because the authors consider m=0m=0, a stationary state exists even when the covariance term does not exist (e.g., β=0,m=0  ⟹  x˙=Px(Vm)=0\beta=0,m=0\implies \dot x = P_x (Vm) = 0). i.e., my question is: do we really need an analysis of the covariance-driven routing term to validate the existence of a multi-cluster equilibrium?

  4. For the Fourier analysis, the authors intended to show that, in a mean-dominant scenario (i.e., single-cluster cases), every harmonic of a small perturbation signal eventually converges under the V=V⊤V=V^\top constraint (w/ the additional spectral constraint λlargest>0\lambda_{\rm largest} > 0). However, these results are also established only when the vectors are already sufficiently aligned w/ the leading vector eleadinge_{\rm leading}. Regarding lines 56-57, I think this kind of analysis is not sufficient to establish causal effects (e.g., What mechanism determines whether causal attention collapses ...). It cannot explain 'how the tokens eventually collapse from their initial state', and it rather explains simple local stability, again w/ the strong assumption V=V⊤V=V^\top. Additionally, in the case of V=V⊤V=V^\top, shouldn't we consider its effects on the dynamics? Let's think about a conventional Hopfield-style network and the single vector's dynamics x˙=Px(Vx)\dot x = P_x(Vx). Then, it is trivial that V's leading eigenvector becomes a minimum of a conventional energy function (like Φ≜−12x⊤Vx\Phi\triangleq -\frac12 x^\top Vx), and thus, when the spectral gap is positive, the reduction of small perturbations is already known.

Overall, I feel the bunch of theoretical findings cannot fully support the paper's main claim and title.

Questions for authors

Please focus on clarifying the weaknesses above. I acknowledge weaknesses could be from my misleading so if author sufficiently clarify or correct my concerns, I will happy to change my score from reject to accept.

5 comments
AuthorsOpenReview ↗

Reply [1/2]

Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.


W1. From Eq. (5), x˙i=Pxi(Vxˉi)⏞O(1)+βPxi(VΣiT⊤xi)⏞O(β)\dot{x}_i = \overbrace{P_{x_i}(V \bar{x}_i)}^{\mathcal{O}(1)} + \overbrace{\beta P_{x_i}(V \Sigma_i T^\top x_i)}^{\mathcal{O}(\beta)} under the Taylor expansion as β→0\beta \to 0. When we imagine a scenario like xˉi≈0\bar{x}_i \approx 0 while Σi≠0\Sigma_i \neq 0, I think this is mathematically valid. However, I cannot be certain whether an analysis under such conditions could have further potential beyond this work. Plus, I think the normalization condition is also quite a strong assumption (although such a condition makes the update representable as a simple tangential term), so, in short, could the authors clarify my concerns about the significance of these conditions?

Answer. Regarding the potential beyond this work, the small-prefix-mean regime gives us a starting point for studying the stability of multi-cluster states. We establish that certain balanced configurations are stationary. When the projected mean term is of order β\beta, even a small imbalance can make it comparable to the covariance correction. Perturbing the balanced equilibria would let us determine which imbalances decay and which grow, and investigate when that growth leads to clusters merging.

Regarding normalization, pretrained Transformers vary both in the rule they use, such as LayerNorm or RMSNorm, and in where they apply it, such as pre-norm or post-norm. Following the spherical post-norm model discussed by Rigollet [A], we use an idealized form of RMSNorm that rescales each token to unit norm after the attention residual update. This gives a closed equation for token directions on the sphere and makes the analysis tractable. Rigollet [A] shows that several idealized normalization schemes share the same projected attention field. This is why we expect the same competition to remain relevant under other normalization schemes of this form.

[A] Rigollet, The Mean-Field Dynamics of Transformers, Proceedings of the International Congress of Mathematicians 2026 - Volume 7.

W2. In Section 4.3, a stationary state exists under m(s,t)≈∑kαkvk=0m(s, t) \approx \sum_k \alpha_k v_k = 0 and Σ(s,t)≈∑kαkvkvk⊤=(1/d)⋅Id\Sigma(s, t) \approx \sum_k \alpha_k v_k v_k^\top = (1/d) \cdot I_d. Thus, Px(Vm)=0P_x(Vm) = 0 and Px(VΣT⊤x)=Px(x/d)=0 (∵VT⊤=I)P_x(V \Sigma T^\top x) = P_x(x/d) = 0 \ (\because VT^\top = I). Regarding lines 25-26, which state that covariance-driven routing remains active, allowing multi-cluster structures to persist, I just expect/hope that covariance dynamics makes multi-cluster structures persist (i.e., some non-zero dynamics keep each vector in such a system, for example, bmean≠0b^{\text{mean}} \neq 0 & bcov≠0b^{\text{cov}} \neq 0 but bmean+bcov=0b^{\text{mean}} + b^{\text{cov}} = 0). However, at stationarity, both terms are just zero, so it is hard to understand why this can be interpreted as stated in lines 25-26 (including the conclusion or other strong statements). I personally think that to state it as such, one should at least imagine a perturbed scenario like xi=vk(i)+ϵyi+O(ϵ2)x_i = v_{k(i)} + \epsilon y_i + \mathcal{O}(\epsilon^2) and analyze the dynamics of y˙=Jy\dot{y} = Jy.

Answer. Our purpose in Section 4.3 is to use the mean–covariance decomposition to identify noncollapsed stationary configurations and illustrate the richness of the equilibrium structures admitted by the truncated dynamics. We did not intend to claim that covariance actively stabilizes these configurations. We agree that the phrase of our observation “routing remains active, allowing multi-cluster structures to persist” give the wrong impression what the stationary constructions can say, and we will delete this wording and the corresponding statements in the conclusion. What we meant is that, when tokens form multiple clusters in a nonsymmetric configuration, the covariance may contribute nontrivially to the routing dynamics, even at equilibrium. The reviewer's suggested perturbation analysis would address the further question of stability. This is an interesting and nontrivial problem that lies beyond the scope of our present equilibrium constructions.

AuthorsOpenReview ↗

Reply [2/2]

W3. Continuing from (2), do we actually need the covariance-driven routing under the authors' assumptions? Because the authors consider m=0m = 0, a stationary state exists even when the covariance term does not exist (e.g., β=0,m=0  ⟹  x˙=Px(Vm)=0\beta = 0, m = 0 \implies \dot{x} = P_x(Vm) = 0). i.e., my question is: do we really need an analysis of the covariance-driven routing term to validate the existence of a multi-cluster equilibrium?

Answer. The reviewer is correct that, when β=0\beta=0, a vanishing prefix mean m=0m=0 already gives a stationary state. Our contribution is to identify which multi-cluster configurations remain stationary when the covariance correction is included. For β>0\beta>0, zero prefix mean alone is generally insufficient, since the projected covariance contribution must also vanish at each support point.

When token representations are dispersed, covariance captures their second-order directional spread, which the prefix mean alone cannot describe. Through its interaction with the query–key matrix, it also captures how token deviations correlate with query-dependent similarity scores. Thus, configurations with the same prefix mean can exhibit different covariance-driven dynamics. Our stationary constructions identify geometric conditions under which this additional contribution is compatible with multi-cluster equilibria.

More broadly, our results concern linear response near alignment. Our strength is that, by treating token position as a continuum, Fourier analysis reveals already at the linear level why the spectral properties emphasized in [KPR24] and [NPDNV26] matter for collective alignment. The roles of the spectral gap and the leading eigenvalue become explicit in a single calculation. Our work therefore complements those nonlinear studies: we trade global nonlinear guarantees for a transparent explanation of the spectral mechanism and the conditions behind local contraction.

[KPR24] Nikita Karagodin, Yury Polyanskiy, and Philippe Rigollet. Clustering in Causal Attention Masking. Advances in Neural Information Processing Systems (NeurIPS), 2024.

[NPDNV26] Trinh Tien Nguyen, Duy-Tung Pham, Hoang-Son Do, Tan M. Nguyen, and Thieu Vo. Two-Clustering Regime of Token Dynamics in Causal Attention. Advances in Neural Information Processing Systems (NeurIPS), 2026.

W5. Overall, I feel the bunch of theoretical findings cannot fully support the paper's main claim and title.

Answer. We believe that our theoretical findings support the paper’s main claim and title. Equation (5) identifies two contributions to the same causal attention update, and our results show how their roles depend on the geometry of the token representations. Near an aligned state, the covariance contribution is second order in the perturbation. Under the stated spectral assumptions, the mean term controls local contraction through the spectrum of the value matrix. This linearization also holds for exact softmax at every fixed finite β\beta. In a balanced configuration, distinct clusters can cancel in the prefix mean while retaining nonzero covariance. Section 4.3 establishes that some such configurations are stationary.

We have revised the scope of the claim explicit in the conclusion. The theory identifies mechanisms that control local alignment and can change the stability of clustered states. A global classification of which initial configurations converge to each regime remains open.


We hope we have cleared your concerns about our work. We have also revised our manuscript according to your comments, and we would appreciate it if we can get your further feedback at your earliest convenience.

Sorry for my late response and Thank you for the author's detailed responses and revisions.

I acknowledge that I've misclassified such simplified settings (e.g., spherical normalization, structural assumptions) are common in theoretical studies of attention dynamics.


However, my central concern remains whether the theoretical findings sufficiently support the paper's main claim and title.

As I understand it, the main theoretical results establish:

  • Local contraction near an already aligned state, characterized through Fourier analysis and spectral conditions.
  • The existence of certain stationary multi-cluster configurations under specific algebraic conditions.

I acknowledge the technical contribution of the Fourier analysis.

However, the first result does not explain how tokens reach the aligned regime from their initial configurations.
For the second result, the authors explicitly agree that their stationary constructions do not establish stability or attraction.
They also acknowledge that a stationary state can already exist without the covariance term when β=0\beta=0 and the prefix mean vanishes.

I appreciate the additional simulations and the perturbation toy example provided in response to other reviewers. Nevertheless, I remain unconvinced that these results establish how mean–covariance competition governs the formation or persistence of the two regimes.

The authors explicitly state:

"We believe that our theoretical findings support the paper’s main claim and title."

However, the authors also acknowledge that their stationary constructions do not establish stability or attraction, and that the suggested perturbation analysis would address these unresolved questions. Nevertheless, they maintain that the current theoretical results are sufficient to support the paper's main claim, how?

I would welcome the AC's (or other reviewer's) assessment of whether the current theoretical evidence sufficiently substantiates the paper's central claim, particularly the use of "governs" in the title.

For these reasons, if author cannot clarify these points, I lean toward to maintain my original rating.

Reviewer_TTVSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper decomposes causal self-attention dynamics into a mean-driven alignment term and a covariance-driven routing term via a first-order expansion in inverse temperature β\beta. In a position-aware mean-field limit followed by monokinetic reduction, a log-depth change of variables r=−log⁡sr = -\log s turns the causal prefix average into an exponential convolution with Fourier multiplier 1/(1−iξ)1/(1 - i\xi). Linearization near a constant eigenprofile yields explicit decay rates in terms of the spectrum of VV: spectral gap at low frequency, leading eigenvalue at high frequency, polynomial transients from Jordan blocks. The same linearization holds for exact softmax at any finite β\beta. Stationary multi-cluster measures are constructed, and the three predicted regimes are illustrated on S1S^1 while Assumption 3.1 is validated on a replicated-layer-20 Llama-3.1-8B stack.

Critical strengths
  • Clean mean vs covariance decomposition is a useful organizing principle.
  • Log-depth transformation is a genuinely clever move that exposes VV's spectrum via a simple Fourier multiplier.
  • Rate formulas are explicit and interpretable: the symmetric case gives decay rate λ1−λ2/(1+ξ2)\lambda_1 - \lambda_2/(1 + \xi^2), interpolating cleanly between the spectral gap at ξ=0\xi = 0 and the leading eigenvalue as ∣ξ∣→∞|\xi| \to \infty.
  • The exact-softmax linearization extension (Proposition B.8) significantly broadens applicability beyond the small-β\beta truncation.
  • Explicit stationary clustered constructions (Theorems B.13 and B.14) give a mathematically clean account of persistent multi-cluster states, with nonzero covariance but vanishing projected drift.
  • Provides a ractical diagnostic is computable on real networks.
Critical weaknesses
  • Single-head, no MLP with a strong spherical normalizatio a substantial gap to real transformers.
  • Small-β\beta expansion is really controlled by β∥T∥\beta \|T\|, which is not small in trained models; this is not quantified anywhere.
  • All Fourier-mode results are linear response near a constant eigenprofile; no nonlinear or global collapse results.
  • Stationary multi-cluster measures are existence only; no stability or attraction.
  • Algebraic conditions for the clustered equilibria (VT⊤=IV T^\top = I, shared eigenbases) are unlikely in trained networks, and the paper does not discuss approximate or perturbation versions.
  • The Llama-3.1-8B experiment validates only Assumption 3.1, not any dynamical prediction of the theory. The replicated-layer-20 construction further distances the setup from real inference.
  • Equating representation collapse with "attention sink" is loose; the paper is really about representation dynamics, and Appendix A hedges on this without resolving it in the main text.
  • S1S^1 experiments are hand-picked demos rather than stringent tests.
Questions for authors
  • What is the empirical size of β∥T∥\beta \|T\| in Llama's layer 20? If it is not small, how should we read the controlled nature of the expansion?
  • Any trajectory-level comparison of exact (Equation 2) vs truncated (Equation 5) dynamics away from the aligned state?
  • How robust is the stationarity in Theorem B.13 to small perturbations of VT⊤V T^\top away from IdI_d? Is there a continuity statement?
  • What changes if VV has nontrivial complex eigenvalues rather than the real-eigenvalue cases covered?
6 comments
AuthorsOpenReview ↗

Reply [1/5]

Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.


W1. Single-head, no MLP with a strong spherical normalization a substantial gap to real transformers.

W5. Algebraic conditions for the clustered equilibria (VT⊤=IVT^\top = I, shared eigenbases) are unlikely in trained networks, and the paper does not discuss approximate or perturbation versions.

Answer to W1-W5. We restrict our study to single-head attention without an MLP and impose assumptions on QQ, KK, and VV that make the analysis tractable and allow us to prove our main results. We agree that our theory does not yet cover the most general setting of Transformer and a simplified version of Transformer is used for the theoretical study. At the same time, we would stress that this kind of restriction is standard in theoretical studies of attention dynamics. It is common to work in a simplified but still expressive regime that makes rigorous analysis feasible while preserving the key qualitative mechanisms of interest. Several related works in this area follow the same philosophy, for example:

[A] Rigollet, The Mean-Field Dynamics of Transformers, Proceedings of the International Congress of Mathematicians 2026 - Volume 7. (The assumption Q(t)=K(t)=V(t)=IdQ(t)=K(t)=V(t)=Id was used.)

[B] Geshkovski et al. The Emergence of Clusters in Self-Attention Dynamics. NeurIPS 2024. (The assumption Q(t)=K(t)=V(t)=IdQ(t)=K(t)=V(t)=Id was used.)

[C] Geshkovski et al. A Mathematical Perspective on Transformers. Bulletin of the AMS 2024. (The assumption Q(t)=K(t)=V(t)=IdQ(t)=K(t)=V(t)=Id was used.)

[D] Karagodin et al. Clustering in Causal Attention Masking. NeurIPS 2024. (The assumption Q(t)=K(t)=constantQ(t)=K(t)=constant, and V(t)=IdV(t)=Id are used.)

[E] Abella et al. Consensus is all you get: The role of attention in transformers. ICLR 2025. (The assumptions V(t)V(t) symmetric and hemisphere initializations were used.)

[F] Pham et al., Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding. NeurIPS 2025. (The assumption Q(t)=K(t)=V(t)=constantQ(t)=K(t)=V(t)=constant, and V(t)V(t) symmetric are used.)

Importantly, our setting is already substantially broader than the most basic static case and we involve time-dependent parameters. By contrast, a full theory for completely general Transformer update and no structural restriction, appears highly challenging. To the best of our knowledge, no existing work has established such a result so far, and we view this as an important direction for future work.

AuthorsOpenReview ↗

Reply [2/5]

W2. Small-β\beta expansion is really controlled by β∣T∣\beta|T|, which is not small in trained models; this is not quantified anywhere.

Q1. What is the empirical size of β∣T∣\beta|T| in Llama's layer 20? If it is not small, how should we read the controlled nature of the expansion?

Answer to W2-Q1. We measure and report β∥T∥\beta\|T\| for every attention head of the pretrained Llama-3.1-8B model (Appendix D.5, Table 4) that β=1/dhead=1/128\beta = 1/\sqrt{d_{\text{head}}} = 1/\sqrt{128}; D=4096D = 4096; 32 query heads, 8 kv heads (grouped-query attention: query head hh uses key head ⌊h/4⌋\lfloor h/4\rfloor). In the following table, we report two quantities for each head:

  • Raw: β∥WQ,h⊤WK,g∥2\beta\|W_{Q,h}^\top W_{K,g}\|_2. This treats the vector entering the projections as if it had unit norm.
  • Unit sphere: β⋅D⋅∥ΓWQ,h⊤WK,gΓ∥2\beta\cdot D\cdot\|\Gamma W_{Q,h}^\top W_{K,g}\Gamma\|_2, where Γ=diag(γ)\Gamma=\mathrm{diag}(\gamma) holds the RMSNorm gains of the layer. Llama is pre-norm, so attention sees RMSNorm(x)=D Γx^\mathrm{RMSNorm}(x)=\sqrt{D}\,\Gamma\hat x with x^=x/∥x∥\hat x=x/\|x\|. Rewriting the score as β x^i⊤Tx^j\beta\,\hat x_i^\top T\hat x_j with unit tokens, as in our model, gives T=D ΓWQ⊤WKΓT=D\,\Gamma W_Q^\top W_K\Gamma.
layerβ∥T∥\beta \|T\| raw: median / maxβ∥T∥\beta \|T\| unit sphere: median / min / max
02.33 / 3.71739 / 321 / 2.08e+03
10.901 / 1.87126 / 49.1 / 440
20.455 / 0.633216 / 123 / 351
30.41 / 0.632211 / 122 / 315
40.45 / 0.709186 / 99 / 278
50.42 / 0.656235 / 105 / 409
60.378 / 0.841206 / 109 / 478
70.368 / 0.635207 / 90.7 / 369
8 **0.345 / 0.803238 / 118 / 579
90.373 / 0.682253 / 110 / 482
100.372 / 0.657276 / 129 / 476
110.328 / 0.547248 / 137 / 409
120.351 / 0.494222 / 132 / 317
130.392 / 0.596346 / 140 / 524
140.38 / 0.558323 / 132 / 474
150.411 / 0.755346 / 205 / 605
160.449 / 0.856342 / 155 / 686
170.48 / 0.69404 / 169 / 572
180.411 / 0.553364 / 235 / 488
190.371 / 0.708337 / 241 / 641
20 **0.407 / 0.58387 / 228 / 529
210.405 / 0.662377 / 185 / 591
220.408 / 0.584405 / 281 / 560
230.391 / 0.729412 / 286 / 729
240.329 / 0.634379 / 266 / 661
250.324 / 0.634386 / 194 / 706
26 **0.397 / 0.548411 / 199 / 554
270.313 / 0.515418 / 274 / 654
280.335 / 0.559398 / 253 / 608
29 **0.323 / 0.513423 / 250 / 624
300.329 / 0.584417 / 283 / 691
310.369 / 1.02321 / 196 / 760

We provide the bound in Proposition B.3 to make the dependence of the truncation error on β∥T∥\beta\|T\| explicit. As shown by the unit-sphere values in the table above, β∥T∥\beta\|T\| need not be small in practice. Consequently, this bound does not guarantee that the first-order truncation accurately approximates the nonlinear softmax dynamics in this regime.

Our approach should be understood as a perturbative theory treating β\beta small. This has appeared in Hinton et al. [1] and Inan et al. [2], where they use high-temperature softmax expansions to clarify knowledge distillation and word-embedding reuse. Low-order Taylor approximations also underpin attention architectures such as Based [3] and TaylorShift [4]. These works motivate our analytical approach. Strikingly, our Fourier analysis reveals, already at the linear level, the roles of the spectral gap and the positivity of the leading eigenvalue—two key spectral ingredients highlighted in the nonlinear clustering conjectures of Karagodin et al. [5].

References

[1] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network. arXiv:1503.02531, 2015.

[2] Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling. ICLR, 2017.

[3] Simran Arora et al. Simple Linear Attention Language Models Balance the Recall–Throughput Tradeoff. ICML, 2024.

[4] Tobias Christian Nauen, Sebastian Palacio, and Andreas Dengel. TaylorShift: Shifting the Complexity of Self-Attention from Squared to Linear (and Back) Using Taylor-Softmax. ICPR, 2024.

[5] Nikita Karagodin, Yury Polyanskiy, and Philippe Rigollet. Clustering in Causal Attention Masking. NeurIPS, 2024.

AuthorsOpenReview ↗

Reply [3/5]

W3. All Fourier-mode results are linear response near a constant eigenprofile; no nonlinear or global collapse results.

Answer. We agree that Theorems B.9–B.11 concern linear response near a constant eigenprofile and do not establish global nonlinear convergence. But linear response near a constant eigenprofile is sufficient for proving our main claims. A central strength of our analysis is that it reveals both the spectral mechanism of local alignment and the conditions for contraction already at the linear level. This provides a perspective that complements the nonlinear clustering studies of Karagodin et al. [KPR24] and Nguyen et al. [NPDNV26]. By treating token position as a continuous variable, our Fourier analysis makes these conditions explicit without requiring a token-by-token nonlinear argument. Interestingly, our linear continuous token positions reveals the exact condition set up for the nonlinear analysis developed in Karagodin et al. [KPR24] and Nguyen et al. [NPDNV26]. We find this correspondence particularly interesting: these spectral mechanisms become visible already through continuum Fourier analysis, without requiring a token-by-token nonlinear induction. We regard this as a strength compared to the nonlinear analysis works below.

[KPR24] Nikita Karagodin, Yury Polyanskiy, and Philippe Rigollet. Clustering in Causal Attention Masking. Advances in Neural Information Processing Systems (NeurIPS), 2024.

[NPDNV26] Trinh T. Nguyen, Tung D. Pham, Son H. Do, Tan M. Nguyen, Thieu N. Vo. Two-Clustering Regime of Token Dynamics in Causal Attention. Advances in Neural Information Processing Systems (NeurIPS), 2026.

W4. Stationary multi-cluster measures are existence only; no stability or attraction.

Answer. We agree that these results establish existence only. We include these constructions to illustrate the richness of the equilibrium structures admitted by the truncated causal-attention dynamics. Our purpose is to highlight interesting configurations and the further dynamical questions they raise; we do not attempt to establish their stability or attraction, which lie outside the scope of this paper. Whether these equilibria are stable, how they respond to perturbations, and whether they emerge dynamically from broader classes of initial configurations are interesting and nontrivial problems that we leave for future work.

W6. The Llama-3.1-8B experiment validates only Assumption 3.1, not any dynamical prediction of the theory. The replicated-layer-20 construction further distances the setup from real inference.

Answer. First, we have extended the Llama-3.1-8B experiments to include three additional layers (8, 26, and 29), spanning early and later depths. As shown in Figure 3 (Appendix D.2), the mean convergence exponents remain consistently below zero under both uniform and bump weightings. These results provide empirical support for the mean-field hypothesis and show that the qualitative conclusion is robust across layer choices.

Second, we have added an experiment using ordinary forward passes through the full pretrained Llama-3.1-8B model, retaining its distinct layers, residual connections, and MLPs. Figure 4 (Appendix D.3) shows that more repetitive prompts generally produce larger mean–covariance ratios ρ\rho, indicating a larger prefix mean relative to the covariance scale. These observations suggest a practical application to utilize ρ\rho together with cluster counts to monitor representation collapse and identify layers or input regimes that preserve or reduce token diversity. A systematic exploration of these potential applications is beyond the scope of the present paper, and we leave it for future work.

Together, these additions show that the mean-field evidence is robust across the tested layer choices and connect our framework to representation geometry during real model inference.

W7. Equating representation collapse with "attention sink" is loose; the paper is really about representation dynamics, and Appendix A hedges on this without resolving it in the main text.

Answer. We have replaced “attention sink” with “representation collapse” throughout the revised manuscript wherever we describe our theoretical and experimental results. This terminology reflects the phenomenon we study: token representations aligning toward a common direction.

Reviewer_aYcvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the need for a better understanding of the emergence of attention sinks vs. multi-cluster representations in causal attention and the transition regimes between them. For this, a mean-field transport equation is formulated that allows to measure how various modes decay through a model. Experimental results through controlled simulations on the sphere and layerwise probing of a pretrained model validate the formulated theory.

Critical strengths

The work seems to be well-positioned in the related work and the proposed models seem to predict the explored properties well, compared with the performed evaluations, whose findings could be potentially interesting for the community.

Critical weaknesses

The study presents multiple findings. However, it would be helpful to go back to the big picture and relate to the asked research question, related to actually employed transformer models. As the study limits the application to the iterative application of a single LLAMA layer, it remains open whether the findings generalize. This could be addressed in two complementary ways: First, studying how the selection of other layers impacts the conclusions and, second, discussing in what way the proposed approach brings the community a step closer to better answering the raised initial question and better understanding the mechanism in modern transformers. This can also connect to a discussion explaining why the study of collapsing and multi-cluster formation is crucial and relevant for this and related branches of research.

Questions for authors

What effects are expected when extending this to a full transformer? Many parameters were chosen for the analysis (eg layer 20 of Llama-3.1, repeated for 64 layers). What was the reasoning behind and how sensitive are the results with respect to the various hyperparameters?

4 comments

Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.


W1. [...] First, studying how the selection of other layers impacts the conclusions and, second, discussing in what way the proposed approach brings the community a step closer to better answering the raised initial question and better understanding the mechanism in modern transformers. This can also connect to a discussion explaining why the study of collapsing and multi-cluster formation is crucial and relevant for this and related branches of research.

Answer. Regarding the concern whether the selection of other layers impacts the conclusions, we first have extended the Llama-3.1-8B experiments to include three additional layers (8, 26, and 29), spanning early and later depths. As shown in Figure 3 (Appendix D.2), the mean convergence exponents remain consistently below zero under both uniform and bump weightings. These results provide empirical support for the mean-field hypothesis and show that the qualitative conclusion is robust across layer choices.

Second, we have added an experiment using ordinary forward passes through the full pretrained Llama-3.1-8B model, retaining its distinct layers, residual connections, and MLPs. Figure 4 (Appendix D.3) shows that more repetitive prompts generally produce larger mean–covariance ratios ρ\rho, indicating a larger prefix mean relative to the covariance scale. These observations suggest a practical application to utilize ρ\rho together with cluster counts to monitor representation collapse and identify layers or input regimes that preserve or reduce token diversity. A systematic exploration of these potential applications is beyond the scope of the present paper, and we leave it for future work.

Regarding the implications of our analysis for modern Transformers, our approach clarifies how causal self-attention can drive token representations toward a single cluster or allow multiple clusters to persist. A first-order expansion separates the attention update into an alignment term driven by the prefix mean and a query-dependent term shaped by the prefix covariance. Near a collapsed state aligned with the leading eigendirection of the value matrix, the covariance term is second order in the perturbation. Under our spectral assumptions, the value spectrum governs the decay of linearized perturbations. In balanced configurations, distinct clusters can cancel in the prefix mean while retaining nonzero covariance. We show that some of these configurations are stationary in the first-order model.

This decomposition gives us a way to test the proposed mechanism in pretrained Transformers. Measuring the two contributions at individual attention heads and relating them to changes in token geometry across layers would help assess when attention reinforces alignment and when distinct clusters persist.

We have added this discussion on the roles of prefix and covariance to page 2 of the revised manuscript.

Q1. What effects are expected when extending this to a full transformer?

Answer. We expect the two mechanisms identified in our analysis to remain relevant in full Transformers. The prefix mean can drive token alignment, while the prefix covariance can shape query-dependent updates and help preserve distinct clusters. For standard dot-product attention, the first-order decomposition applies to each head separately. The output projection combines the head contributions, which can reinforce or counteract one another. The MLP acts independently on each token, but it can amplify or damp differences between token representations and thereby change the prefix statistics seen by subsequent layers. We therefore expect the mechanisms to carry over qualitatively, with stability conditions and decay rates that depend on the full block. However, a rigorous analysis for full Transformers would require a new technique, and we leave this interesting direction for future study.

Q2. Many parameters were chosen for the analysis (eg layer 20 of Llama-3.1, repeated for 64 layers). What was the reasoning behind and how sensitive are the results with respect to the various hyperparameters?

Answer. We have extended the Llama-3.1-8B experiments to include three additional layers (8, 26, and 29), spanning early and later depths. As shown in Figure 3 (Appendix D.2), the mean convergence exponents remain consistently below zero under both uniform and bump weightings. These results provide empirical support for the mean-field hypothesis and show that the qualitative conclusion is robust across layer choices.


We hope we have cleared your concerns about our work. We have also revised our manuscript according to your comments, and we would appreciate it if we can get your further feedback at your earliest convenience.

AuthorsOpenReview ↗

Any Questions from Reviewer aYcv on Our Rebuttal?

Dear Reviewer aYcv,

We sincerely appreciate the time you have taken to provide feedback on our work, which has helped us greatly improve its clarity, among other attributes. This is a gentle reminder that the Paper Discussion Phase will end in less than 2 days from this comment, i.e., 11:59 pm AoE on November 18. We are happy to answer any remaining questions you may have or provide further clarification before then, but we will no longer be able to respond once the discussion phase has concluded.

If you agree that our responses to your reviews have addressed the concerns you listed, we kindly ask that you consider whether raising your score would more accurately reflect your updated evaluation of our paper. Thank you again for your time and thoughtful comments!

Sincerely,

Authors

Reviewer_aYcvOpenReview ↗

Feedback

Dear authors,

thank you for reaching out. I have read your rebuttal. It particularly clarified my concern about the stability of the results. Thank you for this. Given my comparably limited experience in the field, I will wait for the other reviewer's feedbacks before changing my opinion. Further, I will wait for the author's response to Reviewer 7n7s, as his concerns appear reasonable to me.

One last question from myside: About your answer about "Q1. What effects are expected when extending this to a full transformer?" "We therefore expect the mechanisms to carry over qualitatively" -> This is not very clear to me. Could re-explain your answer giving a slightly more specific explanation about the potentially observed effects and whether there is some evidence that this happens?

Thank you

Question. Could re-explain your answer giving a slightly more specific explanation about the potentially observed effects and whether there is some evidence that this happens?

Answer. Thank you for the follow-up. We would like to clarify the specific effects we expect in full Transformers and highlight the empirical evidence already provided in our paper.

First, regarding the expected effects, we expect mean–covariance interactions to influence how token clusters evolve across Transformer layers. When the prefix mean dominates, attention tends to promote token alignment, potentially reducing the number of clusters. Conversely, when the prefix mean is small but covariance remains significant, query-dependent interactions may help preserve distinct clusters or delay their merging. In a full Transformer, different attention heads may reinforce or counteract these effects, while the output projection, residual connections, and MLPs can further reshape token representations. Consequently, we expect input-dependent and layer-dependent clustering behaviour, rather than necessarily monotonic collapse.

Second, we already provide empirical evidence for such behaviour in Appendix D.3 (Figure 4). We examine the pretrained Llama-3.1-8B using its original Transformer layers, including residual connections and MLPs, on 500 prompts from five different families. We track the prefix mean-to-covariance ratio and the number of token clusters across layers. We observe that repetitive inputs generally have a larger mean-to-covariance ratio and fewer clusters, whereas random-token inputs maintain substantially more clusters. Interestingly, repetitive inputs also exhibit an increase in cluster count in intermediate layers, followed by a decrease in later layers. This non-monotonic behaviour illustrates the richer clustering dynamics that can arise in full Transformers.

Third, our controlled experiments provide more direct evidence of the underlying mechanism. In Appendix D.4.3 (Table 3 and Figure 7), we study perturbed two-cluster configurations using exact softmax attention. For a balanced configuration, increasing β\beta from 00 to 22 delays cluster merging from approximately 28 to 383 layers, while increasing the initial mean imbalance substantially accelerates merging. These results demonstrate that query–key interactions and mean imbalance can significantly influence cluster persistence.

Thus, our predictions for full Transformers are motivated not only by the theoretical decomposition but also by observations in a pretrained model and controlled dynamical experiments. Of course, the pretrained-model experiments do not isolate the causal contribution of each Transformer component, and a rigorous extension of our stability analysis to full blocks remains open. Nevertheless, we believe the existing evidence provides meaningful support for the relevance of the mechanisms identified in our paper beyond the simplified theoretical setting.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

Dear AC and reviewers,

Thanks for your thoughtful reviews and valuable comments, which have helped us improve the paper significantly. We are encouraged by the endorsements that our clean mean vs covariance decomposition is a useful organizing principle, and the log-depth transformation is a genuinely clever move (reviewer TTVS).

One common concern emerging from reviewers regarding strict assumptions (single-head, no MLP, spherical norm, shared eigenbases, algebraic conditions on VT⊤VT^{\top}) rather than a full Transformer. Another shared concern was on the replicated-layer-20 construction of the experiment with LLama. We address these concerns here.

Regarding the assumptions, we restrict our study to single-head attention without an MLP and impose assumptions on QQ, KK, and VV that make the analysis tractable and allow us to prove our main results. We agree that our theory does not yet cover the most general setting of Transformer and a simplified version of Transformer is used for the theoretical study. At the same time, we would stress that this kind of restriction is standard in theoretical studies of attention dynamics. It is common to work in a simplified but still expressive regime that makes rigorous analysis feasible while preserving the key qualitative mechanisms of interest. Several related works in this area follow the same philosophy, for example:

[A] Rigollet, The Mean-Field Dynamics of Transformers, Proceedings of the International Congress of Mathematicians 2026 - Volume 7. (The assumption Q(t)=K(t)=V(t)=IdQ(t)=K(t)=V(t)=Id was used.)

[B] Geshkovski et al. The Emergence of Clusters in Self-Attention Dynamics. NeurIPS 2024. (The assumption Q(t)=K(t)=V(t)=IdQ(t)=K(t)=V(t)=Id was used.)

[C] Geshkovski et al. A Mathematical Perspective on Transformers. Bulletin of the AMS 2024. (The assumption Q(t)=K(t)=V(t)=IdQ(t)=K(t)=V(t)=Id was used.)

[D] Karagodin et al. Clustering in Causal Attention Masking. NeurIPS 2024. (The assumption Q(t)=K(t)=constantQ(t)=K(t)=constant, and V(t)=IdV(t)=Id are used.)

[E] Abella et al. Consensus is all you get: The role of attention in transformers. ICLR 2025. (The assumptions V(t)V(t) symmetric and hemisphere initializations were used.)

[F] Pham et al., Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding. NeurIPS 2025. (The assumption Q(t)=K(t)=V(t)=constantQ(t)=K(t)=V(t)=constant, and V(t)V(t) symmetric are used.)

By contrast, a full theory for completely general Transformer update and no structural restriction, appears highly challenging. To the best of our knowledge, no existing work has established such a result so far, and we view this as an important direction for future work.

Regarding the replicated-layer-20 construction of the experiment with LLama, we first have extended the Llama-3.1-8B experiments to include three additional layers (8, 26, and 29), spanning early and later depths. As shown in Figure 3 (Appendix D.2), the mean convergence exponents remain consistently below zero under both uniform and bump weightings. These results provide empirical support for the mean-field hypothesis and show that the qualitative conclusion is robust across layer choices.

Second, we have added an experiment using ordinary forward passes through the full pretrained Llama-3.1-8B model, retaining its distinct layers, residual connections, and MLPs. Figure 4 (Appendix D.3) shows that more repetitive prompts generally produce larger mean–covariance ratios ρ\rho, indicating a larger prefix mean relative to the covariance scale. These observations suggest a practical application to utilize ρ\rho together with cluster counts to monitor representation collapse and identify layers or input regimes that preserve or reduce token diversity. A systematic exploration of these potential applications is beyond the scope of the present paper, and we leave it for future work.

We are glad to answer any further questions you have on our submission.

AuthorsOpenReview ↗

Summary of Revision

Incorporating the comments and suggestions from all reviewers, we have corrected typos and notation and made the following main changes to the revised paper:

  1. We have added a paragraph explaining the roles of the prefix mean and covariance in causal attention on page 2, at the end of the introduction.

  2. We have added Proposition B.4, together with its proof, to compare the exact and truncated analyses of token trajectories.

  3. We have added more experiments with the pretrained Llama-3.1-8B model in Appendix D.2 by repeating different layers, showing consistently negative mean convergence exponents.

  4. We have added another experiment with the pretrained Llama-3.1-8B model in Appendix D.3 to relate the mean-covariance ratio to the token cluster structure across prompt families.

  5. We have added additional simulation results on S1\mathbb{S}^1 in Appendix D.4 to show that our experiments on S1\mathbb{S}^1 are stringent tests.

  6. We have also added Appendix D.5 to measure the numerical values of β∣T∣\beta |T| in the pretrained Llama-3.1-8B model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.