- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work tries to figure out why the token collapse problem (whether tokens converge to a single cluster or multiple clusters) occurs under causal attention dynamics. To solve it, they decompose a special case of causal attention (in which every token is normalized (e.g., ), so that the updated representation lies on the unit hypersphere) into two components: mean-driven alignment and covariance-driven routing.
Then, for each case (when the tokens have already collapsed into either the single-cluster or multi-cluster regime), they analyze the local stability (using Fourier analysis or a first-order Taylor expansion) to show each component's effects on each regime (e.g., mean single and covariance multi).
Finally, by calculating the evolution of the trace and mean values, they try to show that such analysis can also be supported by real experiments.
Critical strengths
I think the overall problem setup and the intuition behind explaining the difference between single- and multi-cluster structures through the mean-covariance relationship are intuitively understandable (I like the characterization that the prefix mean handles alignment while the prefix covariance handles the routing term).
Critical weaknesses
I just wonder about the significance of this work.
-
From Eq. (5), under the Taylor expansion as . When we imagine a scenario like while , I think this is mathematically valid. However, I cannot be certain whether an analysis under such conditions could have further potential beyond this work. Plus, I think the normalization condition is also quite a strong assumption (although such a condition makes the update representable as a simple tangential term), so, in short, could the authors clarify my concerns about the significance of these conditions?
-
In Section 4.3, a stationary state exists under and . Thus, and . Regarding
lines 25-26, which state thatcovariance-driven routing remains active, allowing multi-cluster structures to persist, I just expect/hope that covariance dynamics makes multi-cluster structures persist (i.e., some non-zero dynamics keep each vector in such a system, for example, & but ). However, at stationarity, both terms are just zero, so it is hard to understand why this can be interpreted as stated inlines 25-26(including the conclusion or other strong statements). I personally think that to state it as such, one should at least imagine a perturbed scenario like and analyze the dynamics of . -
Continuing from (2), do we actually need the covariance-driven routing under the authors' assumptions? Because the authors consider , a stationary state exists even when the covariance term does not exist (e.g., ). i.e., my question is: do we really need an analysis of the covariance-driven routing term to validate the existence of a multi-cluster equilibrium?
-
For the Fourier analysis, the authors intended to show that, in a mean-dominant scenario (i.e., single-cluster cases), every harmonic of a small perturbation signal eventually converges under the constraint (w/ the additional spectral constraint ). However, these results are also established only when the vectors are already sufficiently aligned w/ the leading vector . Regarding
lines 56-57, I think this kind of analysis is not sufficient to establish causal effects (e.g., What mechanism determines whether causal attention collapses ...). It cannot explain 'how the tokens eventually collapse from their initial state', and it rather explains simple local stability, again w/ the strong assumption . Additionally, in the case of , shouldn't we consider its effects on the dynamics? Let's think about a conventional Hopfield-style network and the single vector's dynamics . Then, it is trivial that V's leading eigenvector becomes a minimum of a conventional energy function (like ), and thus, when the spectral gap is positive, the reduction of small perturbations is already known.
Overall, I feel the bunch of theoretical findings cannot fully support the paper's main claim and title.
Questions for authors
Please focus on clarifying the weaknesses above. I acknowledge weaknesses could be from my misleading so if author sufficiently clarify or correct my concerns, I will happy to change my score from reject to accept.
5 comments
Reply [2/2]
W3. Continuing from (2), do we actually need the covariance-driven routing under the authors' assumptions? Because the authors consider , a stationary state exists even when the covariance term does not exist (e.g., ). i.e., my question is: do we really need an analysis of the covariance-driven routing term to validate the existence of a multi-cluster equilibrium?
Answer. The reviewer is correct that, when , a vanishing prefix mean already gives a stationary state. Our contribution is to identify which multi-cluster configurations remain stationary when the covariance correction is included. For , zero prefix mean alone is generally insufficient, since the projected covariance contribution must also vanish at each support point.
When token representations are dispersed, covariance captures their second-order directional spread, which the prefix mean alone cannot describe. Through its interaction with the query–key matrix, it also captures how token deviations correlate with query-dependent similarity scores. Thus, configurations with the same prefix mean can exhibit different covariance-driven dynamics. Our stationary constructions identify geometric conditions under which this additional contribution is compatible with multi-cluster equilibria.
More broadly, our results concern linear response near alignment. Our strength is that, by treating token position as a continuum, Fourier analysis reveals already at the linear level why the spectral properties emphasized in [KPR24] and [NPDNV26] matter for collective alignment. The roles of the spectral gap and the leading eigenvalue become explicit in a single calculation. Our work therefore complements those nonlinear studies: we trade global nonlinear guarantees for a transparent explanation of the spectral mechanism and the conditions behind local contraction.
[KPR24] Nikita Karagodin, Yury Polyanskiy, and Philippe Rigollet. Clustering in Causal Attention Masking. Advances in Neural Information Processing Systems (NeurIPS), 2024.
[NPDNV26] Trinh Tien Nguyen, Duy-Tung Pham, Hoang-Son Do, Tan M. Nguyen, and Thieu Vo. Two-Clustering Regime of Token Dynamics in Causal Attention. Advances in Neural Information Processing Systems (NeurIPS), 2026.
W5. Overall, I feel the bunch of theoretical findings cannot fully support the paper's main claim and title.
Answer. We believe that our theoretical findings support the paper’s main claim and title. Equation (5) identifies two contributions to the same causal attention update, and our results show how their roles depend on the geometry of the token representations. Near an aligned state, the covariance contribution is second order in the perturbation. Under the stated spectral assumptions, the mean term controls local contraction through the spectrum of the value matrix. This linearization also holds for exact softmax at every fixed finite . In a balanced configuration, distinct clusters can cancel in the prefix mean while retaining nonzero covariance. Section 4.3 establishes that some such configurations are stationary.
We have revised the scope of the claim explicit in the conclusion. The theory identifies mechanisms that control local alignment and can change the stability of clustered states. A global classification of which initial configurations converge to each regime remains open.
We hope we have cleared your concerns about our work. We have also revised our manuscript according to your comments, and we would appreciate it if we can get your further feedback at your earliest convenience.
Sorry for my late response and Thank you for the author's detailed responses and revisions.
I acknowledge that I've misclassified such simplified settings (e.g., spherical normalization, structural assumptions) are common in theoretical studies of attention dynamics.
However, my central concern remains whether the theoretical findings sufficiently support the paper's main claim and title.
As I understand it, the main theoretical results establish:
- Local contraction near an already aligned state, characterized through Fourier analysis and spectral conditions.
- The existence of certain stationary multi-cluster configurations under specific algebraic conditions.
I acknowledge the technical contribution of the Fourier analysis.
However, the first result does not explain how tokens reach the aligned regime from their initial configurations.
For the second result, the authors explicitly agree that their stationary constructions do not establish stability or attraction.
They also acknowledge that a stationary state can already exist without the covariance term when and the prefix mean vanishes.
I appreciate the additional simulations and the perturbation toy example provided in response to other reviewers. Nevertheless, I remain unconvinced that these results establish how mean–covariance competition governs the formation or persistence of the two regimes.
The authors explicitly state:
"We believe that our theoretical findings support the paper’s main claim and title."
However, the authors also acknowledge that their stationary constructions do not establish stability or attraction, and that the suggested perturbation analysis would address these unresolved questions. Nevertheless, they maintain that the current theoretical results are sufficient to support the paper's main claim, how?
I would welcome the AC's (or other reviewer's) assessment of whether the current theoretical evidence sufficiently substantiates the paper's central claim, particularly the use of "governs" in the title.
For these reasons, if author cannot clarify these points, I lean toward to maintain my original rating.
Reply [1/2]
Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.
W1. From Eq. (5), x˙i=Pxi(Vxˉi)O(1)+βPxi(VΣiT⊤xi)O(β) under the Taylor expansion as β→0. When we imagine a scenario like xˉi≈0 while Σi=0, I think this is mathematically valid. However, I cannot be certain whether an analysis under such conditions could have further potential beyond this work. Plus, I think the normalization condition is also quite a strong assumption (although such a condition makes the update representable as a simple tangential term), so, in short, could the authors clarify my concerns about the significance of these conditions?
Answer. Regarding the potential beyond this work, the small-prefix-mean regime gives us a starting point for studying the stability of multi-cluster states. We establish that certain balanced configurations are stationary. When the projected mean term is of order β, even a small imbalance can make it comparable to the covariance correction. Perturbing the balanced equilibria would let us determine which imbalances decay and which grow, and investigate when that growth leads to clusters merging.
Regarding normalization, pretrained Transformers vary both in the rule they use, such as LayerNorm or RMSNorm, and in where they apply it, such as pre-norm or post-norm. Following the spherical post-norm model discussed by Rigollet [A], we use an idealized form of RMSNorm that rescales each token to unit norm after the attention residual update. This gives a closed equation for token directions on the sphere and makes the analysis tractable. Rigollet [A] shows that several idealized normalization schemes share the same projected attention field. This is why we expect the same competition to remain relevant under other normalization schemes of this form.
[A] Rigollet, The Mean-Field Dynamics of Transformers, Proceedings of the International Congress of Mathematicians 2026 - Volume 7.
W2. In Section 4.3, a stationary state exists under m(s,t)≈∑kαkvk=0 and Σ(s,t)≈∑kαkvkvk⊤=(1/d)⋅Id. Thus, Px(Vm)=0 and Px(VΣT⊤x)=Px(x/d)=0 (∵VT⊤=I). Regarding
lines 25-26, which state thatcovariance-driven routing remains active, allowing multi-cluster structures to persist, I just expect/hope that covariance dynamics makes multi-cluster structures persist (i.e., some non-zero dynamics keep each vector in such a system, for example, bmean=0 & bcov=0 but bmean+bcov=0). However, at stationarity, both terms are just zero, so it is hard to understand why this can be interpreted as stated inlines 25-26(including the conclusion or other strong statements). I personally think that to state it as such, one should at least imagine a perturbed scenario like xi=vk(i)+ϵyi+O(ϵ2) and analyze the dynamics of y˙=Jy.Answer. Our purpose in Section 4.3 is to use the mean–covariance decomposition to identify noncollapsed stationary configurations and illustrate the richness of the equilibrium structures admitted by the truncated dynamics. We did not intend to claim that covariance actively stabilizes these configurations. We agree that the phrase of our observation “routing remains active, allowing multi-cluster structures to persist” give the wrong impression what the stationary constructions can say, and we will delete this wording and the corresponding statements in the conclusion. What we meant is that, when tokens form multiple clusters in a nonsymmetric configuration, the covariance may contribute nontrivially to the routing dynamics, even at equilibrium. The reviewer's suggested perturbation analysis would address the further question of stability. This is an interesting and nontrivial problem that lies beyond the scope of our present equilibrium constructions.