acceptodds
Under review as a conference paper at ICLR 2027

Depth with Memory: Token Dynamics of Attention Residuals

Abstract

Attention Residuals aggregate stored layer outputs with normalized depth weights. This operation generally produces a history-dependent continuous-depth equation, rather than a closed ordinary differential equation. Starting from a normalized Volterra formulation, we identify a common-reweighting condition under which the dynamics close and derive the exact coupled equations for token directions and radii, with query, key, and value matrices fixed across depth. For spherical causal attention with a symmetric positive-definite value matrix having a simple largest eigenvalue, we impose a block structure on the query–key product and an explicit condition ensuring positive radial forcing. These conditions yield positive radial bounds and global well-posedness. When each history rate has an infinite integral, we prove by causal induction that, outside a Lebesgue-null set of admissible raw initial embeddings, every token direction converges to one of the two leading eigenvector poles. The prescribed history kernels are held fixed as the initial embeddings vary. The radial equation asymptotically averages the leading eigenvalue multiplied by the signed attention mass. We give conditions for radial convergence and an explicit softmax formula for the limiting lengths, which may differ within one directional cluster. The conclusions concern a specified normalized model and a sufficient clustering regime, not an arbitrarily trained Attention Residual networks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
attention residuals, token dynamics, transformers, interacting particle systems

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_uQQMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how replacing standard additive residual connections with Attention Residuals affects token dynamics. The authors introduce a continuous-depth model that accounts for the aggregation of previous layer outputs, and derive coupled equations for token directions and lengths. Under additional assumptions on the attention matrices and history weights, they show that almost every initial configuration converges directionally to at most two clusters, corresponding to the two opposite directions of the leading eigenvector of the value matrix.

Critical strengths

The motivation for the paper - studying how depth-wise aggregation of attention residuals affects token dynamics is a well-motivated direction. I think the derivation and setting the authors consider could be used to draw important insights.

Critical weaknesses
  1. I found the paper hard to read, both the main text but especially the proofs. I left some notable cases in the questions but this is true for most of the paper, it is very hard to follow the mathematical arguments that are presented densely.
  2. The contributions the author draw from their work is the analysis of the derivation of the dynamics, the authors do not draw any insight on what is the difference between Attention Residuals and the standard token dynamics analysis, this should be highlighted as it is the main motivation for the paper.
Questions for authors
  1. Could the authors either rephrase this paragraph or explain what they wanted to state here? (line 80) "The almost-everywhere conclusion concerns this raw initial state space, including the radii; it does not by itself give the same assertion for every fixed tuple of initial radii. The results do not claim a complete classification for arbitrary trained Transformers."
  2. Please explicitly refer to WW as the score matrix when introducing it, if this terminology is used later
  3. Please rewrite the second paragraph under assumption 1, it is very hard to follow (line 211).
  4. Most of the proofs are very hard to follow, for example the proof of Lemma 5.5 is described in large consecutive of text describing mathematical derivation but not showing it.
  5. The first two paragraphs of section 5 are hard to follow, it seems the result is introduced twice.
  6. Did you try to develop numerical simulations that show your results hold in practice?
6 comments
AuthorsOpenReview ↗

Response to Reviewer uQQM (1/2)

Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.


[W1] I found the paper hard to read, both the main text and especially the proofs. I left some notable cases in the questions, but this is true for most of the paper; it is very hard to follow the mathematical arguments that are presented densely.

Answer. Thanks for your comments. We have elaborated our results and our mathematical arguments throughout the revision; the specific cases you raised are answered in Q1–Q5 below.

[W2] The contributions the author draw from their work is the analysis of the derivation of the dynamics, the authors do not draw any insight on what is the difference between Attention Residuals and the standard token dynamics analysis, this should be highlighted as it is the main motivation for the paper.

Answer. Thanks for your suggestion. In our work, we analyze Attention Residuals as proposed by the Kimi Team [1]. Prior work models residual Transformers in continuous depth by x˙i=Pxi∑jpijVxj\dot x_i=P_{x_i}\sum_jp_{ij}Vx_j on the sphere [2, 3]. Our dynamics (Thm. 3.4) differ from this model in three ways.

  1. A history rate γi(t)\gamma_i(t). Residual connections add every layer output with unit weight. Attention Residuals instead re-weight the stored outputs, which produces the rate γi\gamma_i in z˙i=γi(t)(Fi(X)−zi)\dot z_i=\gamma_i(t)(F_i(X)-z_i). The rate decides the outcome:
    • if ∫γi=∞\int\gamma_i=\infty, the embedding is forgotten and tokens cluster (Thm. 5.1);
    • if ∫γi<∞\int\gamma_i<\infty, the embedding keeps a positive weight and the directions evolve only for a finite effective time; for example, the first token stops short of the pole (Lemma 4.3);
    • Γi=∫0tγi\Gamma_i=\int_0^t\gamma_i sets how fast tokens cluster.
  2. The state lives in Rd\mathbb R^d, not on Sd−1S^{d-1}. Each token has a radius coupled to its direction. The directional speed is γi/ri\gamma_i/r_i, so each token runs on its own clock. Radii stay bounded, whereas residual-stream norms grow at least linearly with depth: each layer adds a component of size at least η\eta along the token's direction (Lemma 4.1).
  3. Norms keep information after directional collapse. Tokens in the same cluster converge to different radii (20). These radii depend on each token's position and on how many of its predecessors sit at the opposite pole.

The limiting directions ±e\pm e, set by the value matrix, are shared with residual Transformers. Whether tokens reach them, how fast, and with which norms is specific to Attention Residuals. We added this comparison to the revised manuscript.

[Q1] Could the authors either rephrase this paragraph or explain what they wanted to state here? (line 80) "The almost-everywhere conclusion concerns this raw initial state space, including the radii; it does not by itself give the same assertion for every fixed tuple of initial radii. The results do not claim a complete classification for arbitrary trained Transformers."

Answer. Thanks for pointing this out. Here, "raw" means the original token vector ui∈Rdu_i\in\mathbb R^d, i.e., its direction and its length. Following your suggestion, we have rewritten that paragraph in our revision as follows:

These results give sufficient conditions for clustering in a normalized, single-head causal model whose query, key, and value matrices are shared across depth, with history kernels fixed in advance. Theorem 5.1 holds for Lebesgue-almost every initial configuration (u1,…,uN)∈(Rd)N(u_1,\dots,u_N)\in(\mathbb R^d)^N, where each uiu_i carries both a direction and a length. The theorem describes one scenario: a symmetric value matrix whose largest eigenvalue is simple and positive, together with Assumption 1. Other scenarios, such as a repeated or complex leading eigenvalue, and a complete picture of the token dynamics, require further analysis.

AuthorsOpenReview ↗

Response to Reviewer uQQM (2/2)

[Q2] Please explicitly refer to WW as the score matrix when introducing it, if this terminology is used later.

Answer. In the revised manuscript, we explicitly define W=Q⊤KW=Q^\top K as the score matrix in Section 3.2 and consistently use this terminology throughout the paper.

[Q3] Please rewrite the second paragraph under assumption 1, it is very hard to follow (line 211).

Answer. We have revised the corresponding paragraph in the manuscript as follows:

Requiring η>0\eta>0 reads m0>κ2(2dlog⁡N+4ρ)m_0>\kappa^2\bigl(2\sqrt d\log N+4\rho\bigr). Here κ=λ1/λd\kappa=\lambda_1/\lambda_d is the condition number of VV, and ρ=(m1−m0)2+∥B∥op2\rho=\sqrt{(m_1-m_0)^2+\|B\|_{\mathrm{op}}^2} measures how far the score matrix WW is from m0Im_0I. In words, the attention scores must be large compared with the spread of VV's eigenvalues, and more so for longer sequences (larger NN) or for score matrices further from isotropic (larger ρ\rho). In the simplest case W=m0IdW=m_0I_d (so ρ=0\rho=0), the condition is m0/d>2κ2log⁡Nm_0/\sqrt d>2\kappa^2\log N. The regime is not empty: with d=N=m=2d=N=m=2, V=diag(2,1)V=\mathrm{diag}(2,1) and Q=K=10 I2Q=K=\sqrt{10}\,I_2, we get W=10I2W=10I_2 and η≈0.114>0\eta\approx0.114>0.

[Q4] Most of the proofs are very hard to follow, for example the proof of Lemma 5.5 is described in large consecutive of text describing mathematical derivation but not showing it.

Answer. Thanks for your comments. We have revised the part of the proof that follows the linearization in the revision, so that every step is now argued explicitly:

  • the equation near the non-polar equilibrium set is written in the form of Lemma 5.4, with the explicit matrices A⪰κμIA\succeq\kappa_\mu I and C⪯0C\preceq0;
  • the derivatives of the remainders are bounded explicitly, and a path argument gives the Lipschitz constant that Lemma 5.4 requires;
  • for the measure-zero conclusion, the states that stay near the equilibrium set form a Borel null set, by Lemma 5.4 and Tonelli's theorem. The flow map is a local diffeomorphism by Liouville's formula, so the corresponding initial values also form a null set.

We have also rewritten the second half of the proof of Lemma 5.3 in the same way. It now proves explicitly that:

  • the potential converges along the trajectory, because after some time it can cross a non-critical level only upwards;
  • every limit point of the token direction is a critical point, and the limit set lies in a single connected component of the critical set;
  • in the non-polar case, the radius converges to α(qμ)μ\alpha(q_\mu)\mu, so the raw token approaches Kμ\mathcal K_\mu.

[Q5] The first two paragraphs of section 5 are hard to follow, it seems the result is introduced twice.

Answer. Thanks for pointing this out. In our revision, we have deleted the second paragraph, so the result is now introduced only once.

[Q6] Did you try to develop numerical simulations that show your results hold in practice?

Answer. Following your suggestion, we have added Appendix E to the revised manuscript, which presents the following numerical simulations.

  • An actual Attention Residual Transformer (Appendix E.1). We use a PyTorch implementation without modifying its architecture, with 1000 layers, d=128d=128 and N=256N=256 random tokens.
    • With parameter values satisfying our assumptions (Fig. 1), the tokens converge to the two poles ±e\pm e predicted by Thm. 5.1, and the split between the poles stays the same at every depth. With the standard residual, every token is within 1∘1^\circ of its pole at layer 1000. Full Attention Residuals are slower, as their history rate predicts.
    • With Assumption 1c violated (Fig. 3, Appendix E.2), the tokens still reach ±e\pm e, but in four of the six runs almost all of them gather at one pole.
    • From the random initialization (Fig. 2), which lies outside our assumptions, the tokens instead gather on one side of a single direction.
  • Pretrained models (Appendix E.3, Fig. 4). Three public Attention Residual language models, which also lie outside our assumptions, show no two-pole structure.

We hope we have cleared your concerns about our work. We have also revised our manuscript according to your comments, and we would appreciate it if we could get your further feedback at your earliest convenience.


References

  • [1] Kimi Team. Attention Residuals. arXiv:2603.15031, 2026.
  • [2] B. Geshkovski, C. Letrouit, Y. Polyanskiy, P. Rigollet. The emergence of clusters in self-attention dynamics. In NeurIPS, 2023.
  • [3] N. Karagodin, Y. Polyanskiy, P. Rigollet. Clustering in causal attention masking. In NeurIPS, 2024.
AuthorsOpenReview ↗

Any Questions from Reviewer uQQM on Our Rebuttal?

We would like to thank the reviewer again for your thoughtful review and valuable feedback.

We would appreciate it if you could let us know if our responses have addressed your concerns and whether you still have any other questions about our rebuttal.

We would be happy to do any follow-up discussion or address any additional comments.

Reviewer_wmGTOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This is a theory paper on attention residuals (AttnRes), which replace the fixed sum in standard residual connections with softmax attention over earlier layers’ outputs. It treats depth as continuous time and each token as a point moving under attention. To make the analysis tractable, it assumes a single attention head, the same Q/K/V in every layer, one depth query shared by all layers (the original AttnRes uses a different query per layer), and depth weights that follow a fixed schedule rather than depending on content. Under these assumptions, the authors prove that as depth goes to infinity, almost every initialization drives the tokens into at most two clusters, aligned with plus or minus the top eigenvector of the value matrix.

Critical strengths

This work focuses on the theory for AttnRes, a recently proposed algorithm that shows promising empirical success. Such theoretical analysis could lead to deeper understanding of the algorithm.

Critical weaknesses

The main weakness is that the analysis removes the features that distinguish AttnRes. With a depth query shared across layers, AttnRes reduces to a standard pre-norm residual with a scalar gate on each layer. The paper further fixes the depth weights as a schedule independent of content. Yet the motivation for AttnRes is that each layer selectively draws on earlier layers’ outputs, with weights that differ by layer and depend on content. As a result, its clustering results apply equally to ordinary pre-norm transformers and say little about AttnRes specifically.

Questions for authors

Please see the weakness above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer wmGT (1/2)

Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.

Critical Weaknesses

[W1] The main weakness is that the analysis removes the features that distinguish AttnRes. With a depth query shared across layers, AttnRes reduces to a standard pre-norm residual with a scalar gate on each layer. The paper further fixes the depth weights as a schedule independent of content. Yet the motivation for AttnRes is that each layer selectively draws on earlier layers’ outputs, with weights that differ by layer and depend on content. As a result, its clustering results apply equally to ordinary pre-norm transformers and say little about AttnRes specifically.

Answer. We agree that the analysis simplifies the original architecture. It uses a depth query shared across layers and a prescribed depth schedule. These simplifications are what make the memory equation close to an ODE: with a shared query, adding a layer rescales all stored weights by a common factor (Lemma 3.2), and the dynamics become Thm. 3.4. Extending the analysis to layer-specific and content-dependent depth weights is an important direction that we would like to pursue in future research.

However, even under these simplifications, the resulting dynamics are not those of pre-norm residual connections. They differ in two ways, and both enter our results.

AuthorsOpenReview ↗

Response to Reviewer wmGT (2/2)

[W1] (continued)

1. The history rate γi(t)\gamma_i(t). The Attention Residual dynamics are (Thm. 3.4) z˙i=γi(t)(Fi(X)−zi).\dot z_i=\gamma_i(t)\bigl(F_i(X)-z_i\bigr). The rate γi\gamma_i is produced by the depth weights. With a shared query, Γi(tn)=∫0tnγi=log⁡(Zi,n/Zi,0)\Gamma_i(t_n)=\int_0^{t_n}\gamma_i=\log(Z_{i,n}/Z_{i,0}), where Zi,nZ_{i,n} is the sum of the depth scores of token ii up to layer nn (Lemma 3.2). The rate is therefore specific to each token and set by the depth scores.

Pre-norm residual connections have no such rate. Every layer output is added with unit weight, zn+1=zn+F(Xn)z^{n+1}=z^n+F(X^n), and the continuous-depth model of residual Transformers [1, 2], Eq. (2), contains no rate either. (Dividing the pre-norm stream by n+1n+1 leaves directions unchanged and gives our recurrence with the fixed rate 1/(1+t)1/(1+t). Attention Residuals replace this fixed rate by one determined by the depth weights.)

The rate controls the outcome:

  • Whether tokens cluster. The weight on the embedding is ai(t)=e−Γi(t)a_i(t)=e^{-\Gamma_i(t)}. Thm. 5.1 requires ∫0∞γi=∞\int_0^\infty\gamma_i=\infty, i.e., the embedding is eventually forgotten. If instead ∫0∞γi<∞\int_0^\infty\gamma_i<\infty, the embedding keeps a positive weight e−Γi(∞)e^{-\Gamma_i(\infty)} at every depth, and the directions evolve only for a finite effective time. For example, the first token stops at eτ1(∞)Vu1/∥eτ1(∞)Vu1∥e^{\tau_1(\infty)V}u_1/\|e^{\tau_1(\infty)V}u_1\|, which is not a pole unless u1u_1 is already aligned with ee (Lemma 4.3).
  • How fast they cluster. Token directions follow the spherical flow in the effective time τi(t)=∫0tγi/ri ≥ Γi(t)/M\tau_i(t)=\int_0^t\gamma_i/r_i\ \ge\ \Gamma_i(t)/M (Lemma 4.2). For the first token, x1(t)=eτ1(t)Vu1/∥eτ1(t)Vu1∥x_1(t)=e^{\tau_1(t)V}u_1/\|e^{\tau_1(t)V}u_1\| (Lemma 4.3), so its distance to the pole decays like e−(λ−λ2)τ1(t)e^{-(\lambda-\lambda_2)\tau_1(t)}, with τ1\tau_1 comparable to Γ1\Gamma_1. Under the fixed rate of the pre-norm stream, Γ1=log⁡(1+t)\Gamma_1=\log(1+t), this decay is only polynomial in depth. Under a rate bounded below, it is exponential. The depth weights therefore set the speed of clustering.

2. The dynamics live in Rd\mathbb R^d, not on Sd−1S^{d-1}. An Attention Residual aggregate is a normalized average of the embedding and the stored outputs, so its norm is not fixed. Writing zi=rixiz_i=r_ix_i, Thm. 3.4 gives a coupled direction–radius system: x˙i=γiri PxiFi(X),r˙i=γi(⟨xi,Fi(X)⟩−ri).\dot x_i=\frac{\gamma_i}{r_i}\,P_{x_i}F_i(X),\qquad \dot r_i=\gamma_i\bigl(\langle x_i,F_i(X)\rangle-r_i\bigr). The spherical model of residual Transformers evolves directions only. Here, by contrast:

  • Norms stay bounded. By Lemma 4.2, min⁡{∥ui∥,η}≤ri≤max⁡{∥ui∥,λ}\min\{\|u_i\|,\eta\}\le r_i\le\max\{\|u_i\|,\lambda\}. In the pre-norm residual stream, norms grow at least linearly with depth: each layer adds a component of size at least η\eta along the token's direction (Lemma 4.1).
  • Norms keep information after directional collapse. The radii converge to explicit limits (20), which depend on each token's position and on how many of its predecessors sit at the opposite pole. Tokens in the same directional cluster therefore keep different norms. The first token's radius converges to λ\lambda (Lemma 4.3).

What this means for the reviewer's conclusion. The limiting directions, ±e\pm e from the value matrix, are shared with residual Transformers. The rest of our results are statements about the history rate and the radial dynamics, both of which come from the Attention Residual aggregation:

  • the clustering condition ∫γi=∞\int\gamma_i=\infty;
  • the speed of clustering, through Γi\Gamma_i and τi\tau_i;
  • the bounded, token-specific radii and their limits (20).

Pre-norm residual connections fix the history rate and let the norms grow. Attention Residuals let the learned depth weights set the rate, and they keep the norms bounded.

Revision. The revision adds a paragraph after Thm. 3.4, comparing our dynamics with the spherical model of residual Transformers. It covers the history rate, the radius, and the per-token clocks.


We hope we have cleared your concerns about our work. We have also revised our manuscript according to your comments, and we would appreciate it if we could get your further feedback at your earliest convenience.


References

  • [1] B. Geshkovski, C. Letrouit, Y. Polyanskiy, P. Rigollet. The emergence of clusters in self-attention dynamics. In NeurIPS, 2023.
  • [2] N. Karagodin, Y. Polyanskiy, P. Rigollet. Clustering in causal attention masking. In NeurIPS, 2024.
AuthorsOpenReview ↗

Any Questions from Reviewer wmGT on Our Rebuttal?

We would like to thank the reviewer again for your thoughtful review and valuable feedback.

We would appreciate it if you could let us know if our responses have addressed your concerns and whether you still have any other questions about our rebuttal.

We would be happy to do any follow-up discussion or address any additional comments.

AuthorsOpenReview ↗

Another Call for Your Response

Dear Reviewer wmGT,

We sincerely appreciate the time you have taken to provide feedback on our work, which has helped us greatly improve its clarity, among other attributes. This is a gentle reminder that the Paper Discussion Phase will end in less than 12 hours from this comment, i.e., 11:59 pm AoE on November 18. We are happy to answer any remaining questions you may have or provide further clarification before then, but we will no longer be able to respond once the discussion phase has concluded.

If you agree that our responses to your reviews have addressed the concerns you listed, we kindly ask that you consider whether raising your score would more accurately reflect your updated evaluation of our paper. Thank you again for your time and thoughtful comments!

Sincerely,

Authors

Reviewer_pmdCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies a continuous-depth model of Attention Residuals, where each layer combines outputs from earlier layers using normalized weights. Under a common-reweighting assumption, the authors derive coupled equations for token directions and magnitudes. With further assumptions on the attention matrices and history kernels, they establish global existence and uniqueness and show that almost every admissible initialization leads to at most two directional clusters, aligned with the leading eigenvector of the value matrix.

Critical strengths
  • The paper contains substantial technical work. The analysis of the coupled dynamics and the almost-everywhere clustering result seem mathematically valuable. I have not read the appendix, so I have not checked the detailed proofs.
  • Explicitly accounting for the reweighting of earlier layer outputs is a useful step toward understanding Attention Residuals. The paper explains when this history-dependent model can be reduced to an ODE.
  • The results describe both token directions and magnitudes. In particular, tokens in the same directional cluster can retain different magnitudes, which seems worth exploring further.
Critical weaknesses
  • The main conceptual takeaway is unclear: what do the results tell us about how Attention Residuals work. Establishing a continuous model, showing that its equations have solutions, and proving convergence are useful mathematical steps, but the paper should try to explain what we learn from them about the architecture.
  • The assumptions are not sufficiently motivated and discussed. The paper discusses some mathematical consequences of its assumptions, but their practical meaning and restrictiveness still need more attention. Some examples would help readers understand the scope of the results.
  • The paper includes no experiments. Even simple numerical examples would help illustrate the results, show whether clustering appears at reasonable depths, and test what happens when the assumptions are relaxed. A comparison with ordinary residual dynamics would also help explain what is specific to Attention Residuals.
  • The presentation is difficult to follow. Much of the main text is a sequence of lemmas and derivations, but very few words are spent on intuition and relevance of the result. Several intermediate results would fit better in the appendix. I would like the main text to spend more time explaining the central idea, the assumptions, and what the results mean for the model.
Questions for authors
  • How reasonable are the assumptions on the value matrix, the block structure of the query–key product, and the history kernels? Which assumptions are needed for the behavior itself, and which are needed for the proof?
  • What is the main insight about Attention Residuals that readers should take away from the analysis? Does this analysis explain why or in which regime does it perform better than additive residual connections?
  • What do the results tell us about the benefits or drawbacks of this architecture compared with ordinary residual connections?
  • Could you provide numerical examples showing how the predictions relate to finite-depth networks?
7 comments
AuthorsOpenReview ↗

Response to Reviewer pmdC (1/5)

Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.


[W1] The main conceptual takeaway is unclear: what do the results tell us about how Attention Residuals work. Establishing a continuous model, showing that its equations have solutions, and proving convergence are useful mathematical steps, but the paper should try to explain what we learn from them about the architecture.

Answer. Thanks for your suggestion. The revision states the following implications of the two-cluster regime explicitly, in the new paragraph "Why token clustering matters" in Sec. 2.

  1. The spectral picture survives the change of residual pathway. When the weight on the embedding vanishes, all tokens align in direction with the two poles ±e\pm e of the value matrix's top eigenvector (Thm. 5.1). The spectral insights known for standard residual connections therefore still apply to Attention Residuals. In the deep limit, a token's direction carries only one bit, its sign along ee. The first token's pole is set by sign⁡⟨u1,e⟩\operatorname{sign}\langle u_1,e\rangle (Lemma 4.3).
  2. Directional collapse is not complete collapse. Token norms converge to explicit limits (20). These depend on each token's position and on how many of its predecessors sit at the opposite pole, so tokens in the same cluster stay distinguishable by their norm. This is the effect the reviewer noted in the strengths, and a model of directions on the sphere cannot express it.
  3. The depth weights decide whether and how fast this happens. The depth weights set the history rate γi\gamma_i, and the embedding keeps weight e−Γi(t)e^{-\Gamma_i(t)} with Γi=∫0tγi\Gamma_i=\int_0^t\gamma_i.
    • If ∫0∞γi=∞\int_0^\infty\gamma_i=\infty, the embedding is forgotten and tokens cluster (Thm. 5.1).
    • If not, the directions evolve only for a finite effective time (Lemma 4.2). For example, the first token stops at eτ1(∞)Vu1/∥eτ1(∞)Vu1∥e^{\tau_1(\infty)V}u_1/\|e^{\tau_1(\infty)V}u_1\|, which is not a pole unless u1u_1 is already aligned with ee (Lemma 4.3).
    • The speed is governed by the same quantity, since directions move in effective time τi≥Γi/M\tau_i\ge\Gamma_i/M.
    • Residual connections have no such rate (Q3).
AuthorsOpenReview ↗

Response to Reviewer pmdC (2/5)

[W2] The assumptions are not sufficiently motivated and discussed. The paper discusses some mathematical consequences of its assumptions, but their practical meaning and restrictiveness still need more attention. Some examples would help readers understand the scope of the results.

Answer. The revision adds a paragraph on the assumptions after each assumption, with examples for these assumptions as well. Q1 explains which of these the behaviour itself needs and which only the proof needs.

Discussion on Assumption 1:

Assumption 1 is standard in the analysis of attention dynamics: Geshkovski et al. [13] take Q=K=V=IdQ=K=V=I_d on the sphere, Geshkovski et al. [12] take V=IdV=I_d or a VV with a simple positive top eigenvalue for unnormalized tokens, Karagodin et al. [5] take V=IdV=I_d, Abella et al. [4] take a symmetric VV and initial tokens in a common hemisphere, and Pham et al. [7] take constant Q,K,VQ,K,V for unnormalized tokens. The block form (10) makes ee an eigenvector of WW and VV, which reduces the limiting field of each token to the scalar q=⟨x,e⟩q=\langle x,e\rangle (Section 5). Beyond this setting, the dynamics remain open; Appendix E tests cases where Assumption 1 is partly violated.

References

  • [4] Á. Rodríguez Abella, J. P. Silvestre, P. Tabuada. Consensus is all you get: The role of attention in transformers. In ICML, 2025.
  • [5] N. Karagodin, Y. Polyanskiy, P. Rigollet. Clustering in causal attention masking. In NeurIPS, 2024.
  • [7] D.-T. Pham, A. T. Nguyen, V.-H. Tran, et al. Dynamical properties of tokens in self-attention and effects of positional encoding. In NeurIPS, 2025.
  • [12] B. Geshkovski, C. Letrouit, Y. Polyanskiy, P. Rigollet. The emergence of clusters in self-attention dynamics. In NeurIPS, 2023.
  • [13] B. Geshkovski, C. Letrouit, Y. Polyanskiy, P. Rigollet. A mathematical perspective on transformers. Bulletin of the American Mathematical Society, 62(3):427–479, 2025.

Discussion on Assumption 2:

Part (a) says that every new layer rescales all existing depth weights by the same factor, so they decay at the common rate γi(t)\gamma_i(t); e.g., equal depth weights ai(t)=Ki(t,s)=1/(1+t)a_i(t)=K_i(t,s)=1/(1+t) give γi(t)=1/(1+t)\gamma_i(t)=1/(1+t). A depth query shared across layers always yields such weights (Lemma 3.2). The divergence ∫0∞γi=∞\int_0^\infty\gamma_i=\infty holds if and only if the embedding weight ai(t)=e−∫0tγia_i(t)=e^{-\int_0^t\gamma_i} tends to zero, i.e., token ii eventually forgets its embedding. Since λ1\lambda_1 is simple, part (c) reduces to ⟨ui,e⟩≠0\langle u_i,e\rangle\ne0 for every ii, which excludes only a null set of embeddings.


[W3] The paper includes no experiments. Even simple numerical examples would help illustrate the results, show whether clustering appears at reasonable depths, and test what happens when the assumptions are relaxed. A comparison with ordinary residual dynamics would also help explain what is specific to Attention Residuals.

Answer. Thank you for the suggestion. The revision now includes numerical experiments in Appendix E.

  • An Attention Residual Implementation used at depth 1000 (Appendix E.1). We take a PyTorch implementation, keep its architecture unchanged, and run it with d=128d=128 and N=256N=256 random tokens.
    • When the parameters satisfy our assumptions (Fig. 1), the tokens settle at the two poles ±e\pm e given by Thm. 5.1, and the number of tokens on each side does not change with depth. Under the standard residual, all tokens lie within 1∘1^\circ of their poles at layer 1000, while Full Attention Residuals converge more slowly, as their history rate predicts.
    • When Assumption 1c fails (Fig. 3, Appendix E.2), the tokens still converge to ±e\pm e, but in most runs nearly all of them end up at the same pole.
    • Starting from a random initialization (Fig. 2), which lies outside our assumptions, the tokens do not split into two poles; instead, they collect on one side of a single direction.
  • Pretrained language models (Appendix E.3, Fig. 4). None of the three public Attention Residual models, which also fall outside our assumptions, shows a two-pole structure.
  • Figs. 2–4 test our assumptions: they show what happens when only Assumption 1c is relaxed (Fig. 3), when Assumption 1 is not covered entirely (Fig. 2), and in trained models outside all our assumptions (Fig. 4).
AuthorsOpenReview ↗

Response to Reviewer pmdC (3/5)

[W4] The presentation is difficult to follow. Much of the main text is a sequence of lemmas and derivations, but very few words are spent on intuition and relevance of the result. Several intermediate results would fit better in the appendix. I would like the main text to spend more time explaining the central idea, the assumptions, and what the results mean for the model.

Answer. Thanks for your comments. Following your suggestion, we have revised the main text to spend more words on intuition and on what the results mean:

  • Central idea (opening of Sec. 3). A new paragraph states the key observation before any technical result. When the depth query is shared across layers (Lemma 3.2), the memory term of the Volterra equation (5) can be differentiated away. Each token then simply relaxes toward the current attention output, z˙i=γi(t)(Fi(X)−zi)\dot z_i=\gamma_i(t)\bigl(F_i(X)-z_i\bigr) (Thm. 3.4). The whole effect of the depth weights is carried by the rate γi(t)\gamma_i(t), which decides whether, and how fast, the embedding is forgotten.
  • The assumptions (after Assumption 1). New text explains in words what the certificate (11) requires and gives an example where it holds. It also compares Assumption 1 with the assumptions of prior work. Finally, it explains why we need the block form (10): it makes ee an eigenvector of both WW and VV, which reduces the limiting field of each token to the scalar q=⟨x,e⟩q=\langle x,e\rangle.
  • Proof strategy (right after Thm. 5.1). The former bullet list is replaced by a plain-language explanation of the proof. The first token turns toward ±e\pm e. Each later token is then studied under a limiting field in which its predecessors are frozen at their poles. Under this field, the token ends either at a pole or at an unstable set, and only a zero-volume set of initial conditions reaches the unstable set. Repeating this argument token by token proves the theorem.
  • Relevance (Sec. 2). A new paragraph, "Why token clustering matters", discusses the costs and benefits of token clustering and states what our results mean for Attention Residuals.
  • Intermediate results. The derivation of the limiting field has moved to Appendix C; the main text keeps only its explicit formula (22).

[Q1] How reasonable are the assumptions on the value matrix, the block structure of the query–key product, and the history kernels? Which assumptions are needed for the behavior itself, and which are needed for the proof?

Answer. We agree that our theory does not yet cover the most general Transformer. However, theoretical studies of attention dynamics work in a simplified but still expressive regime that makes rigorous analysis feasible:

  • [1] uses tokens in Rd\mathbb R^d without normalization, with V=IdV=I_d or a VV whose top eigenvalue is simple and positive;
  • [2] uses Q=K=V=IdQ=K=V=I_d;
  • [3] uses constant Q,KQ,K and V=IdV=I_d;
  • [4] uses a symmetric VV and hemisphere initializations;
  • [5] uses constant Q,K,VQ,K,V and tokens in Rd\mathbb R^d without normalization, under symmetry or sign conditions on QK⊤QK^\top and VV.

Our setting is broader than the most basic static case. VV is a general diagonal matrix rather than the identity, and WW ranges over the block class (10). The initial tokens may lie in both hemispheres. We prove the harder two-cluster regime, and the history rate γi(t)\gamma_i(t) makes the dynamics non-autonomous. A full theory for completely general Q,K,VQ,K,V appears highly challenging. To our knowledge, no existing work establishes it.

Reviewer_PgaBOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study whether the token clustering observed under self-attention with standard residual connections persists under attention residuals. Recent works have linked token clustering and its geometry to the spectrum of the Value matrix. The standard residual connections accumulate layer outputs, thus increasing hidden-state norms and reducing the relative contribution of individual layers. Attention residuals instead use learned attention over the initial embedding and preceding layer outputs, allowing each layer to adaptively weigh representations from earlier layers. To study how this change affects representation dynamics, the authors derive coupled dynamical equations for both the directions and norms of token representations.

Critical strengths
  1. The paper addresses a theoretical question about token evolution across depth under attention residuals. Recent works have linked token clustering and its geometry under standard residual connections to the spectrum of the value matrix. Whether these relationships persist with attention residuals can determine whether existing spectral insights transfer to architectures that adaptively combine representations across layers.

  2. The authors show coupled dynamical equations for the evolution of token direction and its norm. They also characterize the long-term behavior of the first token. Under the given assumptions, the first token converges to the unit eigenvector corresponding to the largest eigenvalue λ1\lambda_1 in direction, and its representation norm converges to the largest eigenvalue of the value matrix. This connects the first token’s limiting behavior to the spectral properties of the Value matrix.

  3. Under attention residuals and Assumptions 1 (a)-(c), the tokens form at most two directional clusters. This establishes that the token clustering behavior does not change under the residual attention mechanism.

Critical weaknesses
  1. The paper lacks illustrations of the predicted token dynamics and clustering behavior, making the theoretical analysis difficult to read and follow. The plots showing how token directions and norms evolution with depth would provide useful intuition.

  2. The authors should add a clear motivation section on why studying clustering behavior in transformers with attention residual is important, and the consequences of the results discussed in the paper.

  3. The analysis relies on restrictive assumptions: QQ, KK, and VV are fixed across depth, and W=Q⊤KW = Q^{\top} K is assumed to have a particular structure in equation (10). The authors do not clearly motivate this structural assumption and its role in establishing the clustering behavior. The discussion of how the predicted token clustering behavior might change as assumptions are relaxed would clarify the theory's scope and limitations.

  4. The paper lacks experiments on pretrained transformers, leaving it unclear whether the predicted token dynamics and clustering behavior occur in models with attention residuals. The authors should add these experiments for the practical relevance of the theoretical findings, particularly when the assumptions underlying the analysis are not satisfied.

  5. Some of the terms are not discussed clearly in the paper, for example, the Volterra memory/formulation and positive radial forcing. The problem setting in Section 3.1 assumes familiarity with some of the equations/variables; for example, equation (4) uses the history kernel Ki(t,s)K_i(t,s) without defining it or giving any discussion on how it relates to the residual attention formulation. As the theory given in the paper relies on the history kernel and Volterra memory, it is very hard to follow the paper’s results.

Questions for authors
  1. What is the meaning of Volterra memory and history kernel, and how do these connect to the attention residual in equation (4)?

  2. How do the results of the paper change when assumption 1(c) is violated? Do the token clustering results transfer to transformer models with attention residuals under standard initializations?

8 comments
AuthorsOpenReview ↗

Response to Reviewer PgaB (1/6)

Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.


[W1] The paper lacks illustrations of the predicted token dynamics and clustering behavior, making the theoretical analysis difficult to read and follow. The plots showing how token directions and norms evolve with depth would provide useful intuition.

Answer: We added Appendix E in our revision to illustrate our predicted dynamics. We run an Attention Residual Transformer, a PyTorch implementation used without any change to its architecture, at depth L=1000L=1000 with d=128d=128 and N=256N=256 random tokens. We only set its parameters to values satisfying Assumption 1: one head, W=m0IW=m_0I, VV with eigenvalue 11 on ee and all others in [0.10,0.20][0.10,0.20], and η=0.03>0\eta=0.03>0. The circle plots show the direction of every token at successive layers (Fig. 1, Appendix E.1).

  • Tokens that initially spread over the sphere move to the two poles ±e\pm e predicted by Thm. 5.1. With the standard residual stream, every token is within 1∘1^\circ of its pole at layer 10001000.
  • Every token ends at the pole of the hemisphere in which it starts, so the split between the two poles is fixed by the input. In this regime, the first-token rule of Lemma 4.3 holds for all tokens.
  • The same holds for Full and Block Attention Residuals and when Q,K,VQ, K, V change from layer to layer, but at very different speeds, which the history rate predicts.
AuthorsOpenReview ↗

Response to Reviewer PgaB (2/6)

[W2] The authors should add a clear motivation section on why studying clustering behavior in transformers with attention residual is important, and the consequences of the results discussed in the paper.

Answer: We added a remark Why token clustering matters to Section 2 Background in our revision to motivate our study on token dynamics. We also include that discussion below.

Why token clustering matters. In continuous depth, token clustering is the dynamical counterpart of rank collapse and over-smoothing, and these phenomena have concrete costs. Rank collapse at initialization makes the gradients of the query and key parameters vanish [1]. In decoder-only models, distinct inputs can be mapped to nearly identical last-token representations, which leads to failures on counting and copying [2]. Representations in trained language models are also strongly anisotropic [3]. Clustering further limits the usable depth: once tokens reach consensus, an overly deep model ignores its prompt [4]. Since depth plays the role of time, how fast tokens cluster therefore matters as much as where they end up [5, 16]. Clustering is not uniformly harmful, however. Grouping tokens around a few distinguished ones can aid classification [6], and dynamics that form a few clusters tend to help performance [7]. Existing clustering theorems [4, 5, 12, 13] assume the additive running sum. An Attention Residual aggregate is instead a normalized, re-weighted average of stored outputs, which in continuous depth yields the Volterra system (5) with memory (Section 3). Prior results therefore do not transfer automatically. We show that the spectral picture nevertheless survives: when the weight on the embedding vanishes, the tokens cluster at ±e\pm e, where ee is the leading eigenvector of VV (Theorem 5.1). Whether and how fast this happens is set by the depth weights through the history rate γi\gamma_i, which residual connections lack. Token norms stay bounded, and tokens in the same cluster converge to different norms (Eq. (20)), so directional collapse is not representational collapse.

References

[1] L. Noci, S. Anagnostidis, L. Biggio, et al. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. In NeurIPS, 2022.

[2] F. Barbero, A. Banino, S. Kapturowski, et al. Transformers need glasses! Information over-squashing in language tasks. In NeurIPS, 2024.

[3] K. Ethayarajh. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In EMNLP-IJCNLP, 2019.

[4] Á. Rodríguez Abella, J. P. Silvestre, P. Tabuada. Consensus is all you get: The role of attention in transformers. In ICML, 2025.

[5] N. Karagodin, Y. Polyanskiy, P. Rigollet. Clustering in causal attention masking. In NeurIPS, 2024.

[6] A. Alcalde, G. Fantuzzi, E. Zuazua. Clustering in pure-attention hardmax transformers and its role in sentiment analysis. SIAM Journal on Mathematics of Data Science, 7(3):1367–1393, 2025.

[7] D.-T. Pham, A. T. Nguyen, V.-H. Tran, et al. Dynamical properties of tokens in self-attention and effects of positional encoding. In NeurIPS, 2025.

[12] B. Geshkovski, C. Letrouit, Y. Polyanskiy, P. Rigollet. The emergence of clusters in self-attention dynamics. In NeurIPS, 2023.

[13] B. Geshkovski, C. Letrouit, Y. Polyanskiy, P. Rigollet. A mathematical perspective on transformers. Bulletin of the American Mathematical Society, 62(3):427–479, 2025.

[16] B. Geshkovski, H. Koubbi, Y. Polyanskiy, P. Rigollet. Dynamic metastability in the self-attention model. arXiv:2410.06833, 2024.

AuthorsOpenReview ↗

Response to Reviewer PgaB (3/6)

[W3] The analysis relies on restrictive assumptions: QQ, KK, and VV are fixed across depth, and W=Q⊤KW = Q^{\top}K is assumed to have a particular structure in equation (10). The authors do not clearly motivate this structural assumption and its role in establishing the clustering behavior. The discussion of how the predicted token clustering behavior might change as assumptions are relaxed would clarify the theory's scope and limitations.

Answer: We agree that our theory does not yet cover the most general Transformer. However, this kind of restriction is standard in theoretical studies of attention dynamics. One works in a simplified but still expressive regime that makes rigorous analysis feasible while preserving the qualitative mechanism of interest.

Keeping Q,K,VQ,K,V fixed across depth is the standard setting of continuous-depth token dynamics [4, 5, 7, 12, 13, 14]. Related works impose structural assumptions of the same kind:

  • [12] Geshkovski et al. (NeurIPS 2023) keep the tokens in Rd\mathbb R^d without normalization and use V=IdV=I_d with Q⊤KQ^\top K positive definite, or a VV whose top eigenvalue is simple and positive;
  • [13] Geshkovski et al. (Bull. AMS 2025) use Q=K=V=IdQ=K=V=I_d;
  • [5] Karagodin et al. (NeurIPS 2024) use constant Q,KQ,K and V=IdV=I_d;
  • [4] Abella et al. (ICML 2025) use a symmetric VV and hemisphere initializations;
  • [7] Pham et al. (NeurIPS 2025) use constant Q,K,VQ,K,V and tokens in Rd\mathbb R^d without normalization, under symmetry or sign conditions on QK⊤QK^\top and VV.

Our setting within this line of work. Our analysis is already broader than the most basic static case:

  • VV is a general diagonal matrix with a simple positive top eigenvalue, not the identity;
  • the score matrix WW ranges over the block class (10), which contains the multiples of the identity used in [13];
  • the initial tokens may lie in both hemispheres, with no hemisphere condition as in [4];
  • the dynamics are non-autonomous, because the history rate γi(t)\gamma_i(t) varies with depth and differs across tokens.

Role of the block form (10). It makes ee an eigenvector of WW as well as of VV. As a result, the limiting field of each token reduces to the scalar q=⟨x,e⟩q=\langle x,e\rangle (eqs. (22)–(25)). This reduction is what allows us to classify its equilibria and prove the almost-everywhere statement.

Beyond these assumptions. A full theory for completely general Q,K,VQ,K,V appears highly challenging. That would mean allowing a non-symmetric or layer-dependent VV and an unstructured WW. To the best of our knowledge, no existing work establishes clustering in that generality, and we view it as an important direction for future work. Our answer to your Q2 below discusses the difficulties and how we expect to handle them.

Revision. We added a paragraph right after Assumption 1 (Sec. 3.2) on its restrictiveness. It compares our assumptions with those of [4, 5, 7, 12, 13] and explains why we need the block form (10). Under these structural assumptions, our results establish, for the Attention Residual dynamics, the two-cluster behaviour conjectured by Karagodin et al. [5] for causal attention.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Summary of Revisions

Incorporating the comments and suggestions of the reviewers, we summarize the main changes in the revised paper. All changes are highlighted in blue.

  1. Motivation (PgaB W2, pmdC W1). A new paragraph in Sec. 2, "Why token clustering matters", explains the costs and benefits of clustering and what our results mean for Attention Residuals.
  2. Central idea (pmdC W4, uQQM W1). A new paragraph opens Sec. 3 and states the key observation before the technical results.
  3. Volterra memory and history kernel (PgaB W5, Q1). The beginning of Sec. 3.1 now defines the history kernel, the embedding weight, and the Volterra equation, and relates them to the discrete depth weights. A sentence before Lemma 4.1 explains positive radial forcing.
  4. Assumptions (PgaB W3; pmdC W2; uQQM Q2, Q3). New text after Assumptions 1 and 2 explains the certificate (11) in words with an example, compares our assumptions with prior work, and explains the role of the block form (10). WW is now called the score matrix where it is introduced.
  5. Comparison with residual Transformers (pmdC Q3, wmGT W1, uQQM W2). A new paragraph after Thm. 3.4 compares our dynamics with the spherical model: the history rate, the radius, and the per-token effective times.
  6. Proof strategy (pmdC W4, uQQM Q5). A plain-language explanation of the proof now follows Thm. 5.1. The result is introduced only once at the start of Sec. 5, and the derivation of the limiting field has moved to Appendix C.
  7. Proofs (uQQM W1, Q4). The proofs of Lemmas 5.3 and 5.5 are rewritten with every step argued explicitly.
  8. Scope (uQQM Q1, wmGT W1, PgaB Q2). The scope paragraph of the introduction is rewritten. The Limitations paragraph now states that the analysis uses a shared depth query.
  9. Experiments (PgaB W1, W4; pmdC W3, Q4; uQQM Q6). New Appendix E: the two-pole regime and random initialization (E.1), Assumption 1c violated (E.2), and pretrained models (E.3).

References

  • [1] Kimi Team. Attention Residuals. arXiv:2603.15031, 2026.
  • [2] B. Geshkovski, C. Letrouit, Y. Polyanskiy, P. Rigollet. The emergence of clusters in self-attention dynamics. In NeurIPS, 2023.
  • [3] B. Geshkovski, C. Letrouit, Y. Polyanskiy, P. Rigollet. A mathematical perspective on transformers. Bulletin of the American Mathematical Society, 62(3):427–479, 2025.
  • [4] N. Karagodin, Y. Polyanskiy, P. Rigollet. Clustering in causal attention masking. In NeurIPS, 2024.
  • [5] Á. Rodríguez Abella, J. P. Silvestre, P. Tabuada. Consensus is all you get: The role of attention in transformers. In ICML, 2025.
  • [6] D.-T. Pham, A. T. Nguyen, V.-H. Tran, et al. Dynamical properties of tokens in self-attention and effects of positional encoding. In NeurIPS, 2025.
AuthorsOpenReview ↗

Global Response (1/3)

Dear AC and reviewers,

Thank you for your thoughtful reviews and valuable comments, which have helped us improve the paper significantly. We are encouraged by the endorsements that 1) establishing the continuous model, its well-posedness, and the convergence result are useful mathematical steps, and 2) the token norms carry information beyond the directions (reviewer pmdC).

Three questions were shared by reviewers:

  • what the results tell us about Attention Residuals, and how they differ from residual connections (PgaB W2; pmdC W1, Q2, Q3; wmGT W1; uQQM W2);
  • how restrictive the assumptions are, and what happens when they are relaxed (PgaB W3, Q2; pmdC W2, Q1);
  • whether numerical experiments support the theory (PgaB W1, W4; pmdC W3, Q4; uQQM Q6).

We address the first two in parts 1 and 2, and the third in parts 2 and 3. The changes to the presentation are listed in the Summary of Revisions.

What the results tell us about Attention Residuals. We analyze Attention Residuals as proposed by Kimi Team [1]. We do not propose the architecture, and we do not claim an empirical advantage over residual connections. Our results give three insights.

  1. The spectral picture survives. When the weight on the embedding vanishes, all token directions converge to the two poles ±e\pm e, where ee is the leading eigenvector of the value matrix (Thm. 5.1).
  2. The depth weights decide whether and how fast this happens. With a depth query shared across layers, the memory equation reduces to z˙i=γi(t)(Fi(X)−zi)\dot z_i=\gamma_i(t)\bigl(F_i(X)-z_i\bigr) (Thm. 3.4). The history rate γi\gamma_i is set by the depth weights, and the embedding keeps the weight e−∫0tγie^{-\int_0^t\gamma_i}. If ∫0∞γi=∞\int_0^\infty\gamma_i=\infty, the tokens cluster. If not, the first token stops short of its pole (Lemma 4.3). A pre-norm residual stream corresponds to the fixed rate 1/(1+t)1/(1+t), which gives only polynomial convergence in depth, whereas a rate bounded below gives exponential convergence.
  3. Directional collapse is not representational collapse. The tokens live in Rd\mathbb R^d, not on the sphere. Their norms stay bounded (Lemma 4.2) and converge to explicit limits (20), which differ between tokens of the same cluster.

Hence, even with a shared depth query (reviewer wmGT), our dynamics are not those of the spherical model of residual Transformers [3, 4]. The poles ±e\pm e are shared with that model. The clustering condition, the speed, and the norms come from the Attention Residual aggregation. The Limitations paragraph now states that we use a shared depth query.

AuthorsOpenReview ↗

Global Response (2/3)

The assumptions. Restrictions of this kind are standard in the theory of attention dynamics. Most results fix Q,K,VQ,K,V across depth and restrict them further:

  • Q=K=V=IdQ=K=V=I_d on the sphere. Single-cluster convergence for a temperature β≲n−1\beta\lesssim n^{-1} or β≳n2\beta\gtrsim n^2, or for d≥3d\ge3 [3, 7]; metastable clusters from separated or i.i.d. uniform initial tokens [8, 9]; quantitative mean-field clustering for tokens in a cap or a small β\beta [10].
  • V=IdV=I_d with general Q,KQ,K. One cluster for causal attention from almost every initial condition [4], and consensus for tokens in one open hemisphere [5].
  • VV tied to Q⊤KQ^\top K. V=Q⊤K=K⊤QV=Q^\top K=K^\top Q, in a comparison of normalization schemes [11]; V=−K⊤QV=-K^\top Q with K⊤QK^\top Q symmetric, which makes Sinkhorn attention a Wasserstein gradient flow [12]; V=αIdV=\alpha I_d with Q⊤KQ^\top K symmetric positive definite, for hardmax attention [13].
  • Other VV. Without normalization, [2] treats a VV with a simple positive top eigenvalue and states clustering for the rescaled tokens e−tVxie^{-tV}x_i, [6] imposes symmetry or sign conditions on QK⊤QK^\top and VV, and [14] takes Gaussian initial tokens with VV commuting with their covariance. On the sphere, [5] treats causal attention with a symmetric VV whose top eigenvalue is simple and positive, for a single head and tokens in one open hemisphere.

For VV with distinct positive eigenvalues, [4] conjectures that causal attention sends almost every initial configuration to two clusters at ±e\pm e. Our setting allows a symmetric VV with a simple positive top eigenvalue, score matrices in the block class (10), which contains W=m0IdW=m_0I_d, initial tokens in both hemispheres, and a history rate that varies with depth and across tokens. Under these assumptions, Thm. 5.1 gives this two-pole picture for Attention Residuals. It does not resolve the conjecture of [4], which concerns residual attention with general Q,KQ,K.

  • What the behaviour needs. The divergence ∫0∞γi=∞\int_0^\infty\gamma_i=\infty and nonzero leading projections ⟨ui,e⟩≠0\langle u_i,e\rangle\ne0. The latter excludes only a null set of embeddings.
  • What only our proof needs. The block form (10) makes ee an eigenvector of both WW and VV, which reduces the limiting field of each token to one scalar. The certificate (11) keeps the norms away from zero.
  • Beyond them. The poles remain equilibria for every WW. Our response to reviewer PgaB (Q2) explains where the proof breaks without Assumption 1c and how we expect to extend it. Clustering for general Q,K,VQ,K,V remains open, and to our knowledge no existing work establishes it.

Numerical experiments (new Appendix E).

We present in Appendix E the following table of experiments. We run a public PyTorch implementation of Attention Residuals, with its architecture unchanged, at depth 1000 with d=128d=128 and N=256N=256 tokens, and we analyse three publicly released pretrained Attention Residual language models. Here, RR is the largest distance between a token direction and its nearer pole (R=0R=0 at a two-pole configuration), and bal\mathrm{bal} is the fraction of tokens on the minority side of ee (0.50.5 for an even split, 0.000.00 for a one-sided cloud).

Table 1: Token geometry in an Attention Residual implementation of depth 1000 (Figs. 1–3) and in pretrained models (Fig. 4)

SettingTwo poles ±e\pm e?bal\mathrm{bal}: input → outputRR
Fig. 1: Assumption 1 holds (η=0.03\eta=0.03)Yes0.47 → 0.470.01 (standard residual) and 0.69 (Full AttnRes) at layer 1000
Fig. 3: certificate (11) violated (η≈−2.26\eta\approx-2.26)Yes0.47 → 0.02–0.030.02 at layer 109 (standard residual); 0.22 at layer 1000 (Full AttnRes)
Fig. 2: default random initializationNo, one-sided cloudabout 0.5 → 0.000.22 (standard residual) and ≥ 0.68 (Attention Residual runs) at layer 1000
Fig. 4: three pretrained models, 28 to 48 layersNo, one-sided cloud0.38, 0.23, 0.31 → 0.00≥ 0.74 at every layer
AuthorsOpenReview ↗

Global Response (3/3)

  • Parameters satisfying Assumption 1 (Fig. 1, App. E.1). The tokens converge within the network to the two poles ±e\pm e predicted by Theorem 5.1. With the standard residual, every token is within 1∘1^\circ of its pole at layer 1000, and the balance never changes, so the input fixes how many tokens end at each pole.
  • Finite depth (Fig. 1). With the standard residual, 97% of the motion happens within 37 layers. Full Attention Residuals have a smaller history rate and need about nine times more layers; at layer 1000, their farthest token is still about 40∘40^\circ from its pole. Thm. 5.1 thus predicts where the tokens go, and the history rate predicts how many layers this takes.
  • Assumption 1c violated (Fig. 3, App. E.2). We lower the attention scale by about 1000 times, so that η<0\eta<0. The tokens still converge to ±e\pm e, but in four of the six runs almost all of them end at the same pole. The poles, which come from VV, survive without the certificate. The certificate decides how the tokens are distributed between them.
  • Random initialization (Fig. 2, App. E.1). With all weights drawn from N(0,0.022)\mathcal N(0,0.02^2), 4 heads and the MLP active, Assumption 1 fails on several counts. No two-pole structure appears: the tokens gather on one side of a single direction. The standard residual concentrates most, with all tokens within 13∘13^\circ of that direction at layer 1000, while every Attention Residual run keeps some token at least 40∘40^\circ away.
  • Pretrained models (Fig. 4, App. E.3). We examine a 0.6B model with Block AttnRes, a 0.6B model with Full AttnRes, and a 3B model, with 28 to 48 layers. The token cloud does not collapse, and no model forms two antipodal clusters at any depth. These models have several heads, MLPs, layer-specific non-symmetric weights, and content-dependent depth weights, so Thm. 5.1 makes no prediction for them.

In general, the results agree with our theory where its assumptions hold, and they show which part of the picture each assumption is responsible for. Outside them, the experiments show what changes: near consensus instead of two populated poles without the certificate, and a one-sided geometry at random initialization and in trained models.

We hope that our rebuttal has helped to clear concerns about our work. We are glad to answer any further questions you have on our submission, and we would appreciate it if we could get your further feedback at your earliest convenience.

References

  • [1] Kimi Team. Attention Residuals. arXiv:2603.15031, 2026.
  • [2] B. Geshkovski, C. Letrouit, Y. Polyanskiy, P. Rigollet. The emergence of clusters in self-attention dynamics. In NeurIPS, 2023.
  • [3] B. Geshkovski, C. Letrouit, Y. Polyanskiy, P. Rigollet. A mathematical perspective on transformers. Bulletin of the American Mathematical Society, 62(3):427–479, 2025.
  • [4] N. Karagodin, Y. Polyanskiy, P. Rigollet. Clustering in causal attention masking. In NeurIPS, 2024.
  • [5] Á. Rodríguez Abella, J. P. Silvestre, P. Tabuada. Consensus is all you get: The role of attention in transformers. In ICML, 2025.
  • [6] D.-T. Pham, A. T. Nguyen, V.-H. Tran, et al. Dynamical properties of tokens in self-attention and effects of positional encoding. In NeurIPS, 2025.
  • [7] C. Criscitiello, Q. Rebjock, A. D. McRae, N. Boumal. Synchronization on circles and spheres with nonlinear interactions. arXiv:2405.18273, 2024.
  • [8] B. Geshkovski, H. Koubbi, Y. Polyanskiy, P. Rigollet. Dynamic metastability in the self-attention model. arXiv:2410.06833, 2024.
  • [9] G. Bruno, F. Pasqualotto, A. Agazzi. Emergence of meta-stable clustering in mean-field transformer models. In ICLR, 2025.
  • [10] S. Chen, Z. Lin, Y. Polyanskiy, P. Rigollet. Quantitative clustering in mean-field transformer models. arXiv:2504.14697, 2025.
  • [11] N. Karagodin, S. Ge, Y. Polyanskiy, P. Rigollet. Normalization in attention dynamics. In NeurIPS, 2025.
  • [12] M. E. Sander, P. Ablin, M. Blondel, G. Peyré. Sinkformers: Transformers with doubly stochastic attention. In AISTATS, 2022.
  • [13] A. Alcalde, G. Fantuzzi, E. Zuazua. Clustering in pure-attention hardmax transformers and its role in sentiment analysis. SIAM Journal on Mathematics of Data Science, 7(3):1367–1393, 2025.
  • [14] V. Castin, P. Ablin, J. A. Carrillo, G. Peyré. A unified perspective on the dynamics of deep transformers. arXiv:2501.18322, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.