acceptodds
Under review as a conference paper at ICLR 2027

MLP Layers Refine and Expand Clusters in Causal Attention Dynamics

Abstract

Modern transformers exhibit rich dynamical phenomena such as clustering, alignment, and dimensional collapse. These effects are often explained through attention-only dynamics, where spectral properties of the value matrix drive tokens toward dominant eigendirections. However, in practical transformer blocks, attention is coupled with a nonlinear multilayer perceptron (MLP), whose dynamical effect remains poorly understood. In this work, we study a minimal model of causal attention interacting with a nonlinear MLP. We show that the MLP can fundamentally reshape attention dynamics: even when attention alone selects dominant eigendirections, the coupled system can develop new attracting states that are not aligned with any eigenvector of the value matrix. These non-eigendirection attractors have basins of attraction of positive Lebesgue measure and therefore produce genuinely new clustering behavior. We identify the underlying mechanism: the MLP nonlinearly deforms the attention vector field, creates new equilibria, stabilizes them locally, and, through causal propagation, synchronizes tokens around these new states. We construct explicit two-dimensional examples and extend the analysis to multi-token causal dynamics. Our results demonstrate that MLP blocks are not merely auxiliary nonlinearities but can qualitatively change the long-time behavior of transformer dynamics, generating clustering patterns beyond those predicted by attention alone.

Primary area:
learning theory
Keywords:
token dynamic, token clustering, multilayer perceptron, causal attention

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uFYbOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper investigates the roles of MLP layers in a Transformer model and shows that MLP layers may induce feature clustering behaviors that are different from attention-only models. This work focuses on an analytical toy model that interprets the forward pass through Transformer layers as continuous dynamics. Under these dynamics, the authors show that specific choices of MLP weights result in equilibrium points that cannot be represented in an attention-only model. The paper concludes with experiments demonstrating that the theoretical findings are consistent with the pre-trained models found in practice.

Critical strengths

The approach to understand model behavior through the lens of dynamical systems is very interesting and can be very useful. Furthermore, the main hypothesis posed by this paper is also worthy of further research. I hope that the authors can continue on this work and present more comprehensive results in a paper with better exposition.

Critical weaknesses

The paper is poorly written and has major technical limitations. First of all, there is very little discussion surrounding the main theorems, and this makes interpreting the consequences of the work extremely difficult. Also, the proof sketch is quite messy and hard to map to the various steps in the full proof.

Most importantly, I have serious doubts about the model formulation of the work.

  • The Transformer block definition (1-2) deviates from the standard implementation in popular models such as Llama or Qwen. The choice to route the residual connection around the whole block instead of individually for the attention and MLP layers are baffling to me. Using post-norm instead of pre-norm is also less common but more forgivable.

  • In (7), the projection onto the orthogonal direction makes little sense given that the author decided not include residual connection for the attention layer. And adding a residual connection would require a change to the LHS of (7), which is also nontrivial to fix.

  • The projection operator defined on line 160 is not norm-invariant, which invalidates the assumption on line 180.

Also, the depth of the technical results are quite limited and do not provide substantial insights.

  • I feel that Theorem 4.1 does not add much on top of Theorem 3.1. Specifically, Thm 4.1 only discusses equilibrium points where all tokens are the same. This basically renders the Q and K matrices vacuous.

  • The results only discuss specific weights construction for the MLP. In particular, Thm 3.1 and 4.1 assume that VV, W1W_1 and W2W_2 are in the neighborhood of diagonal matrices, and Thm 4.2 assumes that W2W_2 is in the neighborhood of a rank-one matrix. I feel that, at minimum, there needs to be additional analysis showing the relevance of these constructions.

  • Only two equilibrium points were studied by each of the theorems. The paper did not consider other possible equilibrium points and thus gave an incomplete picture.

  • The paper only studies the case where all layers share the same weights.

  • There is also no mention of training dynamics. While this may not be necessary to have a strong paper, proving such results could be a valid way to address my prior concerns.

Overall, I found there are significant limitations in the presentation and technical sophistication of this paper. There remains a lot of work to be done to have a strong paper. I recommend clear rejection.

Questions for authors

I have no questions that can help improve the current draft.

6 comments
AuthorsOpenReview ↗

Reply [1/3]

Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.


W1. The paper is poorly written and has major technical limitations. First of all, there is very little discussion surrounding the main theorems, and this makes interpreting the consequences of the work extremely difficult. Also, the proof sketch is quite messy and hard to map to the various steps in the full proof.

Answer. We have added a remark after Theorem 3.1 to discuss the changes of attractors when adding MLP. Theorem 3.1 says that adding a GELU MLP can change where the first token converges, even with the same value matrix and the same initial token. Indeed, in the attention alone, almost every initial direction converges to e1e_1 or −e1-e_1, the two orientations of the dominant eigenvector of V0V^0. While in attention with the specified MLP, the coupled flow has two additional attracting directions, ±u=±(e1+e2)/2\pm u=\pm(e_1+e_2)/\sqrt2. The attractors are non-spectral because the attention-only velocity is nonzero at both points, even after perturbation. The simulation in Section 6.1 verifies the predicted change in attracting direction numerically in the projected model.

Another discussion has been added after Theorem 4.1 to explain the effect of MLP in Transformer's dynamic. while Theorem 3.1 shows that the MLP can create a new attracting direction for the first token, Theorem 4.1 shows that the same direction can attract an entire causal prefix, even when the tokens start at distinct points. For example, at the reference weights, if all LL tokens start sufficiently close to u=(e1+e2)/2u=(e_1+e_2)/\sqrt2, they all converge exponentially to uu. If they start sufficiently close to −u-u, they all converge to −u-u. These are two separate basins, each leading to one shared limit. The shared limit is non-spectral, meaning attention alone would still move the tokens even once they were synchronized there.

We have also updated the proof sketch in Section 5 to improve readability and clarify how it corresponds to the steps in the full proof.

W2. The Transformer block definition (1-2) deviates from the standard implementation in popular models such as Llama or Qwen. The choice to route the residual connection around the whole block instead of individually for the attention and MLP layers are baffling to me. Using post-norm instead of pre-norm is also less common but more forgivable.

W3. In (7), the projection onto the orthogonal direction makes little sense given that the author decided not include residual connection for the attention layer. And adding a residual connection would require a change to the LHS of (7), which is also nontrivial to fix.

Answer to W2-W3. Regarding the normalization, we follow Rigollet [A] in adopting a post-normalized spherical model for our theoretical analysis. In the attention-only setting, Rigollet shows that pre-norm and post-norm share the same projected attention field, with normalization-dependent factors that change the speed of each token. This common structure allows both schemes to be studied within the same analytical framework, even though their convergence rates can differ.

Regarding the residual connections and the projection in Equation (7). We use a single residual connection around the composed block to obtain a tractable model that retains causal token mixing and a tokenwise MLP. This lets us isolate whether the MLP can create new attracting directions and whether those directions can attract subsequent tokens. In Equation (7), the projected attention vector is the instantaneous change in token direction supplied to the MLP. For a unit token xℓx_\ell and attention output aℓa_\ell, this interpretation follows from

=P_{x_\ell}a_\ell.$$ This identity motivates the tangent projection as a modeling choice in the reduced system. We agree that giving attention and the MLP separate residual updates would change the dynamics. If the attention substep were modeled as a separate continuous-depth flow, the intermediate token state would satisfy an evolution equation with $\partial_t$ on the left-hand side, in place of the algebraic relation in Equation (7). We conjecture that the MLP can also create non-spectral attractors in that setting and that these attractors can persist across subsequent tokens. We have added a paragraph after Equation (9) to justify the tangent projections in Equations (6)–(7). [A] Rigollet, The Mean-Field Dynamics of Transformers, Proceedings of the International Congress of Mathematicians 2026 - Volume 7.
AuthorsOpenReview ↗

Reply [2/3]

W4. The projection operator defined on line 160 is not norm-invariant, which invalidates the assumption on line 180.

Answer. We respectfully disagree with the reviewer. The tangent projection in line 160 is the projection onto the tangent plane at x(t)x(t) of the unit sphere, therefore it is not norm invariant. But we do not need it to be norm-invariant. The unit-norm conclusion in line 180 concerns the token trajectory and follows from the tangency of its velocity.

To see this directly, let g(t)=∥xℓ(t)∥2−1g(t)=\|x_\ell(t)\|^2-1. Along the flow dynamic of xℓx_\ell, which can be rewritten shortly in the form x˙ℓ=Pxℓf(xℓ)\dot x_\ell=\mathsf P_{x_\ell} f(x_\ell) for a suitable function ff, we have

=2\langle x_\ell(t),\mathsf P_{x_\ell(t)}f(x_\ell(t))\rangle =-2\langle x_\ell(t),f(x_\ell(t))\rangle\,g(t).$$ Since $g(0)=\|x_\ell(0)\|^2-1=0$, this scalar equation gives $g(t)=0$ throughout the solution’s interval of existence and $x_\ell(t) \in \mathbb{S}^{d-1}$. Thus the outer tangent projection preserves the unit-sphere constraint. To clarify wording that may have been misleading, we have added an explanation after line 180 of the revised manuscript showing why $x_\ell(t)$ remains on the unit sphere. **W5. Also, the depth of the technical results are quite limited and do not provide substantial insights. I feel that Theorem 4.1 does not add much on top of Theorem 3.1. Specifically, Thm 4.1 only discusses equilibrium points where all tokens are the same. This basically renders the Q and K matrices vacuous.** **Answer.** We respectfully disagree that Theorem 4.1 adds little to Theorem 3.1. The main contribution of Theorem 4.1 is to show that an attractor created by the MLP for the first token can draw initially distinct tokens to the same non-spectral direction. We agree that the existence of the synchronized equilibrium follows from Theorem 3.1. The stability of that equilibrium in the full product space is the new result. For $L>1$, configurations with identical tokens form a set of zero product surface measure. Theorem 4.1 instead gives an open basin of positive measure in which the tokens converge exponentially to the common direction. This full-prefix stability can fail even when Theorem 3.1 applies. To see why, at the reference MLP weights with $V^0=\operatorname{diag}(-1,-2,-3)$, Theorem 3.1 gives a first-token sink at $u$. At the corresponding synchronized three-token equilibrium, however, a perturbation of the third token in the $e_3$ direction has linearized growth rate $-\frac{\lambda_1+\lambda_2-2\lambda_3/3}{4}=\frac14>0$. Thus the synchronized equilibrium is unstable in this example. The additional hypothesis in Theorem 4.1 controls these later-token perturbations and ensures stability for every finite prefix length and every fixed choice of $Q,K$, with a common radius for parameter perturbations. We have added this counterexample to Remark C.1 in appendix. We have also added a remark after Theorem 3.1 explaining how the MLP changes the attracting directions of the first token, and a remark after Theorem 4.1 explaining how these attracting directions persist across subsequent tokens.
AuthorsOpenReview ↗

Reply [3/3]

W6. The results only discuss specific weights construction for the MLP. In particular, Thm 3.1 and 4.1 assume that VV, W1W_1 and W2W_2 are in the neighborhood of diagonal matrices, and Thm 4.2 assumes that W2W_2 is in the neighborhood of a rank-one matrix. I feel that, at minimum, there needs to be additional analysis showing the relevance of these constructions.

W8. The paper only studies the case where all layers share the same weights.

W10. Overall, I found there are significant limitations in the presentation and technical sophistication of this paper. There remains a lot of work to be done to have a strong paper.

Answer to W6-W8-W10. We agree that our theory does not yet cover the most general setting of Transformer and a simplified version of Transformer is used for the theoretical study. At the same time, we would stress that this kind of restriction is standard in theoretical studies of attention dynamics. It is common to work in a simplified but still expressive regime that makes rigorous analysis feasible while preserving the key qualitative mechanisms of interest. Several related works in this area follow the same philosophy, for example:

[A] Rigollet, The Mean-Field Dynamics of Transformers, Proceedings of the International Congress of Mathematicians 2026 - Volume 7. (The assumption Q(t)=K(t)=V(t)=IdQ(t)=K(t)=V(t)=Id was used.)

[B] Geshkovski et al. The Emergence of Clusters in Self-Attention Dynamics. NeurIPS 2024. (The assumption Q(t)=K(t)=V(t)=IdQ(t)=K(t)=V(t)=Id was used.)

[C] Geshkovski et al. A Mathematical Perspective on Transformers. Bulletin of the AMS 2024. (The assumption Q(t)=K(t)=V(t)=IdQ(t)=K(t)=V(t)=Id was used.)

[D] Karagodin et al. Clustering in Causal Attention Masking. NeurIPS 2024. (The assumption Q(t)=K(t)=constantQ(t)=K(t)=constant, and V(t)=IdV(t)=Id are used.)

[E] Abella et al. Consensus is all you get: The role of attention in transformers. ICLR 2025. (The assumptions V(t)V(t) symmetric and hemisphere initializations were used.)

[F] Pham et al., Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding. NeurIPS 2025. (The assumption Q(t)=K(t)=V(t)=constantQ(t)=K(t)=V(t)=constant, and V(t)V(t) symmetric are used.)

Importantly, our setting is already substantially broader than the most basic static case: we analyse the dynamical role of the MLP, and inparticular its interaction with attention, which is already a nontrivial generalization of prior analyses. This role and interaction remains much less understood from the dynamical system point of view in the literature. By contrast, a full theory for completely general Transformer update and no structural restriction, appears highly challenging. To the best of our knowledge, no existing work has established such a result so far, and we view this as an important direction for future work.

W7. Only two equilibrium points were studied by each of the theorems. The paper did not consider other possible equilibrium points and thus gave an incomplete picture.

Answer. Under the assumptions of Theorems 3.1 and 4.1, only two types of equilibria arise. These equilibria have the properties needed to prove our main claims, that the MLP can create new non-spectral attractors for the first token and preserve them across subsequent tokens. Beyond these assumptions, additional equilibria may exist. The study for more general assumption which may lead to more equilibria lies beyond the scope of the present paper and is a direction for future work.

W9. There is also no mention of training dynamics. While this may not be necessary to have a strong paper, proving such results could be a valid way to address my prior concerns.

Answer. Regarding the role of training dynamics in our analysis, we treat the weights as fixed parameters and allow them to vary over the model’s parameter space independently of any particular training procedure. Incorporating training would shift the focus from the asymptotic behavior of tokens at fixed weights to how changes in the weights during training shape token behavior at finite depth. This question lies beyond the scope of the present paper and is a direction for future work.


We hope we have cleared your concerns about our work. We have also revised our manuscript according to your comments, and we would appreciate it if we can get your further feedback at your earliest convenience.

Reviewer_H4b4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies a simplified continuous-time model of causal attention coupled with an MLP. The authors explicitly construct MLP weight matrices that introduce locally stable equilibria outside the eigendirections of the attention value matrix. They then establish local stability of synchronised configurations for finite causal token sequences. Synthetic simulations illustrate these constructions. Experiments on four pretrained language models show that removing MLPs substantially increases token alignment, although these experiments do not directly test the attractor mechanism established by the theory.

Critical strengths
  • The paper provides explicit mathematical constructions showing that feed-forward transformations can introduce stable non-eigendirection attractors in a simplified attention dynamical system.
  • The authors also establish local stability under parameter perturbations and extend the analysis to finite causal token sequences. These results appear technically sound and offer an interesting dynamical-systems perspective, although their relevance to actual transformers remains unclear.
Critical weaknesses
  • The proposed ODE is not derived from the transformer update. For example, the equilibrium analysis assumes fixed weights across depth, whereas standard transformers use different weights at successive layers. These are fundamental modelling differences, leaving it unclear whether the proved attractors have any correspondence to actual transformer dynamics.
  • Limited scientific insight into the claimed ML phenomenon. The main results do not explain why training would produce such weights or whether these attractors exist in trained models. The pretrained-model experiments demonstrate increased token alignment when MLPs are removed, but do not establish the proposed mechanism. The mathematical results offer limited explanatory insight into the empirical phenomenon that motivates the paper.
Questions for authors
  • Can the authors derive Equations (6)–(7) from the transformer update in Equations (1)–(2), particularly justifying the tangent projection before the MLP? How do the equilibrium results extend to the depth-dependent weights of standard transformers?
  • Can the authors identify stable non-eigendirection attractors in the pretrained models, rather than inferring their existence from token cosine similarities?
4 comments
AuthorsOpenReview ↗

Reply [1/2]

Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address each of your concerns in detail.


W1. The proposed ODE is not derived from the transformer update. For example, the equilibrium analysis assumes fixed weights across depth, whereas standard transformers use different weights at successive layers. These are fundamental modelling differences, leaving it unclear whether the proved attractors have any correspondence to actual transformer dynamics.

Answer. We agree that our theory does not yet cover the most general setting of Transformer and a simplified version of Transformer is used for the theoretical study. At the same time, we would stress that this kind of restriction is standard in theoretical studies of attention dynamics. It is common to work in a simplified but still expressive regime that makes rigorous analysis feasible while preserving the key qualitative mechanisms of interest. Several related works in this area follow the same philosophy, for example:

[A] Rigollet, The Mean-Field Dynamics of Transformers, Proceedings of the International Congress of Mathematicians 2026 - Volume 7. (The assumption Q(t)=K(t)=V(t)=IdQ(t)=K(t)=V(t)=Id was used.)

[B] Geshkovski et al. The Emergence of Clusters in Self-Attention Dynamics. NeurIPS 2024. (The assumption Q(t)=K(t)=V(t)=IdQ(t)=K(t)=V(t)=Id was used.)

[C] Geshkovski et al. A Mathematical Perspective on Transformers. Bulletin of the AMS 2024. (The assumption Q(t)=K(t)=V(t)=IdQ(t)=K(t)=V(t)=Id was used.)

[D] Karagodin et al. Clustering in Causal Attention Masking. NeurIPS 2024. (The assumption Q(t)=K(t)=constantQ(t)=K(t)=constant, and V(t)=IdV(t)=Id are used.)

[E] Abella et al. Consensus is all you get: The role of attention in transformers. ICLR 2025. (The assumptions V(t)V(t) symmetric and hemisphere initializations were used.)

[F] Pham et al., Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding. NeurIPS 2025. (The assumption Q(t)=K(t)=V(t)=constantQ(t)=K(t)=V(t)=constant, and V(t)V(t) symmetric are used.)

Importantly, our setting is already substantially broader than the most basic static case: we analyse the dynamical role of the MLP, and inparticular its interaction with attention, which is already a nontrivial generalization of prior analyses. This role and interaction remains much less understood from the dynamical system point of view in the literature. By contrast, a full theory for completely general Transformer update and no structural restriction, appears highly challenging. To the best of our knowledge, no existing work has established such a result so far, and we view this as an important direction for future work.

W2. Limited scientific insight into the claimed ML phenomenon. The main results do not explain why training would produce such weights or whether these attractors exist in trained models. The pretrained-model experiments demonstrate increased token alignment when MLPs are removed, but do not establish the proposed mechanism. The mathematical results offer limited explanatory insight into the empirical phenomenon that motivates the paper

Q2. Can the authors identify stable non-eigendirection attractors in the pretrained models, rather than inferring their existence from token cosine similarities?

Answer to W2-Q2. Regarding the role of training in our analysis, we treat the weights as fixed parameters and allow them to vary over the model’s parameter space independently of any particular training procedure. Incorporating training would shift the focus from the asymptotic behavior of tokens at fixed weights to how changes in the weights during training shape token behavior at finite depth. This question lies beyond the scope of the present paper and is a direction for future work.

Regarding the identification of stable non-eigendirection attractors, our results establish that such attractors can arise in the model studied here. Developing a criterion for identifying them in pretrained models remains an open problem.

Regarding the increase of token alignment when MLPs are removed, Theorems 3.1 and 4.1 show that MLPs can create new attracting directions and sustain them across subsequent tokens. Removing the MLPs can therefore eliminate these attractors, offering a possible explanation for the stronger token alignment observed after MLP ablation.

AuthorsOpenReview ↗

Reply [2/2]

Q1. Can the authors derive Equations (6)–(7) from the transformer update in Equations (1)–(2), particularly justifying the tangent projection before the MLP? How do the equilibrium results extend to the depth-dependent weights of standard transformers?

Answer. Regarding the derivation of Equations (6)-(7) from Equations (1)-(2), our spherical formulation in Equations (6)-(7) is motivated by Rigollet’s treatment of normalized attention [A]. In the attention-only setting, several normalization variants, including pre-norm and post-norm, lead to the same underlying projected attention field for token directions. We follow this approach by modeling the evolution of token directions on the unit sphere.

The tangent projections follow from the geometry of this spherical model. For a unit token xℓx_\ell and an ambient update vv,

=P_{x_\ell}v.$$ Applied to the MLP output, this identity gives the outer projection in Equation (6), which keeps the token velocity tangent to the sphere. Applied to the attention output, it identifies the instantaneous change in token direction induced by attention. Equation (7) makes the modeling choice to feed this directional change to the MLP. This choice is additional to the RMSNorm operation in Equation (2), which acts on the attention output itself. Regarding the equilibrium results in the depth-dependent weights scenario, if the value or MLP weights vary with depth, the corresponding frozen-weight equilibria may move, and tracking them requires additional control of the weight variation. We leave this study for future work. Section 6.2 studies the layer-dependent setting empirically. We have added a paragraph after Equation (9) to justify the tangent projections in Equations (6)–(7). The following paragraph explains why the outer tangent projection in Equation (6) ensures that each token remains on the unit sphere throughout its evolution. [A] Rigollet, The Mean-Field Dynamics of Transformers, Proceedings of the International Congress of Mathematicians 2026 - Volume 7. --- We hope we have cleared your concerns about our work. We have also revised our manuscript according to your comments, and we would appreciate it if we can get your further feedback at your earliest convenience.
AuthorsOpenReview ↗

Any Questions from Reviewer H4b4 on Our Rebuttal?

We would like to thank the reviewer again for your thoughtful review and valuable feedback.

We would appreciate it if you could let us know if our responses have addressed your concerns and whether you still have any other questions about our rebuttal.

We would be happy to do any follow-up discussion or address any additional comments.

AuthorsOpenReview ↗

Another Call for Your Response

Dear Reviewer H4b4,

We sincerely appreciate the time you have taken to provide feedback on our work, which has helped us greatly improve its clarity, among other attributes. This is a gentle reminder that the Paper Discussion Phase will end in less than 12 hours from this comment, i.e., 11:59 pm AoE on November 18. We are happy to answer any remaining questions you may have or provide further clarification before then, but we will no longer be able to respond once the discussion phase has concluded.

If you agree that our responses to your reviews have addressed the concerns you listed, we kindly ask that you consider whether raising your score would more accurately reflect your updated evaluation of our paper. Thank you again for your time and thoughtful comments!

Sincerely,

Authors

Reviewer_cb2rOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper analyzes how MLP layers can change how token representations form clusters. The authors are able to show that adding an MLP can create stable states that the attention mechanism by itself wouldn't be to produce. They provide a few proofs and numerical examples that demonstrate this point. In particular, they show that these effects appear in ~7B language models so that the effect is relatively robust.

Critical strengths

It is clear that their theoretical result shows that the new attracting states survive small perturbations to the weights and this is reinforced from the series of small experiments and also the larger pretrained language models.

Critical weaknesses

There are no substantial weaknesses.

Questions for authors

It would be interesting to see whether or not these effects also appear within different language model architectures.

4 comments

Thank you for your thoughtful review and constructive feedback. We have revised our manuscript in response to your suggestions, with all changes highlighted in blue. Below, we address your concern in detail.


Q1. It would be interesting to see whether or not these effects also appear within different language model architectures.

Answer. We conjecture that similar effects of MLPs also occur within different language model architectures. In the following, we give an example showing this effects in state space models (SSMs), where an MLP can create a new attracting direction for the first token and preserve it across subsequent tokens.

Consider a sequence of tokens xi(t)∈S1x_i(t)\in S^1, indexed by i=1,…,Li=1,\ldots,L. At each depth tt, apply the two-dimensional causal recurrence h0=0,hi=Ahi−1+xi,yi=Chi,h_0=0,\quad h_i=Ah_{i-1}+x_i,\quad y_i=Ch_i, where A=diag⁡(1/2,1/3)A=\operatorname{diag}(1/2,1/3) and C=diag⁡(2,1)C=\operatorname{diag}(2,1). The SSM-only flow is x˙i=Pxiyi.\dot x_i=\mathsf P_{x_i}y_i. Coupling it to GELU MLP with two weight matrices W1=[[1,1],[1,1]]W_1=[[1,1],[1,1]] and W2=[[1,0],[0,0]]W_2=[[1,0],[0,0]] yields the coupled flow dynamic: x˙i=PxiW2GELU⁡(W1Pxiyi).\dot x_i=\mathsf P_{x_i}W_2\operatorname{GELU}(W_1\mathsf P_{x_i}y_i). We can use the same arguments in the proofs of Theorems 3.1 and 4.1 to derive the attracting directions of these SSM-only and the coupled flows. Indeed, writing xi=(cos⁡θi,sin⁡θi)x_i= (\cos\theta_i,\sin\theta_i), let qi=(−sin⁡θi,cos⁡θi)⊤yiq_i=(-\sin\theta_i,\cos\theta_i)^\top y_i denote the SSM-only angular velocity. For the first token, y1=Cx1y_1=Cx_1, so q1=−sin⁡θ1cos⁡θ1q_1=-\sin\theta_1\cos\theta_1 and the only attracting directions are ±e1\pm e_1. With the MLP, the angular equation becomes

\operatorname{GELU}\bigl((\cos\theta_i-\sin\theta_i)q_i\bigr).$$ Let $s=(1,1)/\sqrt{2}$. For any finite prefix length $L$, both $(s,\ldots,s)$ and $(-s,\ldots,-s)$ are equilibria of the coupled flow because $\cos\theta_i-\sin\theta_i=0$ at every token. At either configuration, unrolling the recurrence gives $$q_i=-\frac12\sum_{k=0}^{i-1} \left(2(1/2)^k-(1/3)^k\right)\le -\frac12.$$ Thus the SSM-only flow has nonzero velocity at both configurations. In the coupled flow, every derivative through $q_i$ is multiplied by the vanishing factor $\cos\theta_i-\sin\theta_i$. The angular Jacobian is therefore diagonal, with entries $q_i/2<0$ by $\operatorname{GELU}'(0)=1/2$. Both synchronized configurations are hyperbolic sinks. In particular, the MLP creates the attracting directions $\pm s$ for the first token and preserves them across subsequent tokens. We have added this interesting discussion to Remark 5.1 in the revised version of the paper. --- We hope we have cleared your concern about our work. We have also revised our manuscript according to your comment, and we would appreciate it if we can get your further feedback at your earliest convenience.
AuthorsOpenReview ↗

Any Questions from Reviewer cb2r on Our Rebuttal?

We would like to thank the reviewer again for your thoughtful review and valuable feedback.

We would appreciate it if you could let us know if our responses have addressed your concerns and whether you still have any other questions about our rebuttal.

We would be happy to do any follow-up discussion or address any additional comments.

Reviewer_cb2rOpenReview ↗

Good example.

This is a very nice example. Thanks.

AuthorsOpenReview ↗

Thanks for your endorsement!

Thank you for your thoughtful comments and constructive feedback. We truly appreciate your endorsement.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

Dear AC and reviewers,

Thanks for your thoughtful reviews and valuable comments, which have helped us improve the paper significantly. We are encouraged by the endorsements that: 1) our results are very interesting, can be very useful, worthy of further research (reviewer uFYb), technically sound, and offer an interesting dynamical-systems perspective (reviewer H4b4); 2) our theory provides explicit mathematical constructions showing that MLP can introduce stable non-eigendirection attractors in a simplified attention dynamical system (reviewer H4b4); 3) clearly establish local stability under parameter perturbation (reviewers cb2r, H4b4).

One common concern emerging from reviewers regarding a simplified version of Transformer is used for the theoretical study rather than a full Transformer update without any assumptions on Q,K,V,WQ,K,V,W. Another shared concern was on the involvement of training dynamics. An additional concern is whether our simplified model is sufficient to answer the paper’s motivating question. We address these concerns here.

Regarding the use of a simplified version of Transformer for the theoretical study, we study a continuous-depth model with single-head causal attention, spherical normalization, and weights fixed across depth. These choices let us isolate how a tokenwise MLP changes the dynamics induced by causal attention. We would stress that this kind of restriction is standard in theoretical studies of attention dynamics. It is common to work in a simplified but still expressive regime that makes rigorous analysis feasible while preserving the key qualitative mechanisms of interest. Several related works in this area follow the same philosophy, for example:

[A] Rigollet, The Mean-Field Dynamics of Transformers, Proceedings of the International Congress of Mathematicians 2026 - Volume 7. (The assumption Q(t)=K(t)=V(t)=IdQ(t)=K(t)=V(t)=Id was used.)

[B] Geshkovski et al. The Emergence of Clusters in Self-Attention Dynamics. NeurIPS 2024. (The assumption Q(t)=K(t)=V(t)=IdQ(t)=K(t)=V(t)=Id was used.)

[C] Geshkovski et al. A Mathematical Perspective on Transformers. Bulletin of the AMS 2024. (The assumption Q(t)=K(t)=V(t)=IdQ(t)=K(t)=V(t)=Id was used.)

[D] Karagodin et al. Clustering in Causal Attention Masking. NeurIPS 2024. (The assumption Q(t)=K(t)=constantQ(t)=K(t)=constant, and V(t)=IdV(t)=Id are used.)

[E] Abella et al. Consensus is all you get: The role of attention in transformers. ICLR 2025. (The assumptions V(t)V(t) symmetric and hemisphere initializations were used.)

[F] Pham et al., Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding. NeurIPS 2025. (The assumption Q(t)=K(t)=V(t)=constantQ(t)=K(t)=V(t)=constant, and V(t)V(t) symmetric are used.)

Importantly, our setting is already substantially broader than the most basic static case: we analyse the dynamical role of the MLP, and inparticular its interaction with attention, which is already a nontrivial generalization of prior analyses. This role and interaction remains much less understood from the dynamical system point of view in the literature. By contrast, a full theory for completely general Transformer update and no structural restriction, appears highly challenging. To the best of our knowledge, no existing work has established such a result so far, and we view this as an important direction for future work.

Regarding the involvement of training dynamics, we treat the weights as fixed parameters and allow them to vary over the model’s parameter space independently of any particular training procedure. Incorporating training would shift the focus from the asymptotic behavior of tokens at fixed weights to how changes in the weights during training shape token behavior at finite depth. This lies beyond the scope of the present paper and is a direction for future work.

Regarding the concern whether our simplified model is sufficient to answer the paper’s motivating question, we would like to clarify the model is sufficient to answer our motivating question of how a tokenwise MLP can change where tokens cluster under causal attention. Indeed, Theorem 3.1 shows that, with the reference value matrix held fixed, adding an MLP can change the first token’s limit from a dominant eigendirection to a stable non-eigendirection. Theorem 4.1 gives sufficient conditions under which an entire finite causal prefix of initially distinct tokens converges to that same direction. Remark C.1 shows why this requires a separate analysis: a stable first-token equilibrium can correspond to an unstable three-token synchronized state. Theorem 4.3 and Corollary C.3 further show that, for any nonscalar reference value matrix, suitable MLP weights can make any prescribed non-eigendirection a locally attracting destination for a synchronized prefix.

We are glad to answer any further questions you have on our submission.

AuthorsOpenReview ↗

Summary of Revision

Incorporating the comments and suggestions from all reviewers, we have corrected typos and notation and made the following main changes in the revised paper.

  1. We have added an explanation on page 4, after Equation (9), of why the tangent projections appear in the dynamical system. We also explain why the outer tangent projection preserves the unit-sphere constraint for the tokens.

  2. We have added Remark 3.2 to discuss the consequences of Theorem 3.1.

  3. We have added Remark 4.2 to explain the consequences of Theorem 4.1 and how it extends Theorem 3.1.

  4. We have updated the proof sketch in Section 5 to improve readability and clarify how it corresponds to the steps in the full proof.

  5. We have added Remark 5.1 to show how the same mechanism can occur in a causal state-space model.

  6. We have added Remark C.1 to show that the first-token equilibrium can be stable while the corresponding synchronized prefix equilibrium is unstable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.