acceptodds
Under review as a conference paper at ICLR 2027

Clustering Thresholds for Homogenized Transformers

Abstract

We study the evolution of spherical token representations under randomly resampled attention and MLP layers, in a continuous-depth model where all tokens are driven by the same random fields. Our main result is that the long-time behavior is decided by the sign of a single explicit quantity. When this quantity is negative, tokens started close together collapse to a single cluster with probability tending to one as their initial diameter tends to zero. When it is positive, distinct tokens almost surely do not converge. We also give a sufficient condition under which the mean deficit decays exponentially and synchronization holds almost surely from every initial configuration, and we show that the same rate governs the Jacobian of the token flow. We connect the thresholds separating different clustering behavior with (quenched and annealed) Lyapunov exponents, and identify a regime where token representations can cluster even as small perturbations of their shared representation are amplified.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mean-field transformers, lyapunov exponents, clustering, homogenized dynamics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EDMdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors consider the clustering evaluation of token representations for transformers whose parameter are randomly initialized. They find that the long-time behavior is controlled by λMLP\lambda_{\rm MLP} and λATT\lambda_{\rm ATT} which are explicit quantities. They also connect token synchronization and Lyapunov exponents based on the classical theory of random dynamical system.

Critical strengths

The strength of this paper is to characterize clustering behavior in the long time regime. Some quantities appeared in several sufficient conditions are explicit, which directly provides the practical benefits. The experiments provide good empirical support for the theoretical results.

Critical weaknesses

The weakness of this paper is that, although the sufficient conditions are explicit, the analysis depends on Gaussian initialization, as the authors already acknowledged. Also, this paper studies the long-time clustering behavior of tokens in deep random transformers at initialization, while it does not study the evolution of model parameters during training.

Questions for authors
  1. Is there any intrinsic difficulty to extend the result to non-Gaussian settings ? It might be difficult to discuss general distributions, but what about some concrete distribution, or the setting close to Gaussian, for instance, sub-Gaussian distributions ?

  2. To be closer to practical settings, could authors discuss the evolution of model parameters during training ? I do not ask the authors to add details, but just a brief discussion on this possibility.

1 comment

We thank the reviewer for their positive feedback and their comments, which we address below.

  • The homogenization result of Koubbi et al. (2026), from which this part of the model is derived, holds for general sub-Gaussian distributions. Under this constraint, the sampling on VV can be generalized without many consequences if we keep the entries i.i.d. and centered, so that Lemma 11 and Lemma 12 still work. Similarly, for the homogenization results we may assume a general distribution with fine moments for the Query and Key matrices, as long as we keep the independence from the MLP sampling and the Value matrix. Our clustering and Lyapunov results, however, rely heavily on the isotropy of the covariance kernel of the limiting random field. In particular, considering the MLP, the identity E[G(x)G(y)⊤]=κ(⟨x,y⟩)Id\mathbb{E}[G(x)G(y)^\top]=\kappa(\langle x,y\rangle)I_d holds because the rows of UU have a rotation-invariant law. This isotropy makes the coefficients Λ∗\Lambda^* and Σ\Sigma of Lemma 15 the same at every point of the sphere, yielding an explicit, constant threshold λMLP+λATT\lambda_{\mathrm{MLP}}+\lambda_{\mathrm{ATT}}. This does not hold, e.g., for iid Bernoulli (or Rademacher) weights: in that case a simple calculation shows that the local contraction rate depends on where the cluster sits and on the direction of separation, significantly complicating the computation of the exponents. Our results, at least for the MLP, can therefore be extended to rotation-invariant row laws (e.g., uniform on dSd−1\sqrt d \mathbb{S}^{d-1} ) for UU provided the induced kernel has the required regularity.
  • The difficulty of extending our results to training is that gradient updates couple the initially independent weights across layers. Early in training, while these updates only weakly perturb the blocks’ input Jacobians, we expect the initialization exponents to remain informative about which gradient components are attenuated or amplified through depth. Our experiments also suggest that the thresholds identified at initialization are predictive of trainability over longer training horizons. How long the exponents remain predictive of signal propagation as the weights evolve, and whether that evolution admits a tractable stochastic description, remain interesting and important open questions. We added a sentence in this sense in the conclusions.
Reviewer_sDqcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors look at the long term behavior of a transformer model with infinite depth and width. The authors derive conditions under which the tokens cluster and diverge. A Lyapunov type argument is used in the proofs. Relation of these parameters is derived to Lypaunov exponents of the system.

Critical strengths
  • The paper establishes conditions under which the token evolution results in clustering or divergence.
  • The authors relate this to the Lyapunov exponent, which provides a way to compute these parameters.
Critical weaknesses

First critical weakness is philosophical. After my reading, I did not get a good sense of why the results of the paper are important for understanding transformers. The behavior of token evolution of an arbitrary untrained transformer for arbitrarily large depth seems to not have been justified well enough. Is there a analogue of such results of purely MLPs?

Many places, the authors have used terminologies that are not clearly defined.

The mathematical exposition of the paper seems to require more work. (See questions)

  • For instance, they seem to not have referred to how their arguments are connected to other long term behavior of stochastic processes.
  • Another weakness, the authors' sketch of proof seems very disconnected from their actual proof. They should explain more what the extra steps are doing and what gaps are being filled in the sketch.

In the numerical experiments, the authors start quite abruptly. Here again, explanation is missing as to what are the experiments supposed to be testing.

Questions for authors

Could the authors provide a motivation for studying the problem they have?

Some terms, phrases or sentences require clarification:

In the abstract: -"mean deficit" -"same rate governs the Jacobian of the token flow."

  • what is the motivation for this, what does it mean and how is it related to the previous clustering cases identified: "identify a regime where token representations can cluster even as small perturbations of their shared representation are amplified."

Rest of the paper:

  • In the contribution, what is "residual-stream dynamics of deep transformers."?
  • Could the authors cite which equation of Koubai they are using?
  • What is a "Gaussian Brownian field"? Is there a reference for this?
  • Could the authors clarify in what sense (5) is to be understood given we have a brownian field instead of usual brownian motion?
  • In definition 8, what is the perturbation being applied to?
  • In eq (12), what is A. There are a family of maps A_t. What is A?
  • What is the "annealed limit"? The quenched limit seems to have been defined. But I am not clear as to what the "annealed limit" is.
  • What does it mean in line 315: "λ describes a single network"?
  • I know what the generator of a process is. But what is "GENERATOR ON THE OVERLAPS"?

What is the motivation for section 4? Sure these exponents are related to the parameters that classify the clustering phenomena. But what additional information is this providing?

2 comments

Dear Reviewer, thank you for the comments and observations that helped us improve our work. Below we report the answers to your questions:

  • Connection with the long-time behaviour of stochastic processes. We added a section in the appendix explaining in detail the relationship between long-time behavior of stochastic processes and the results presented in the paper. This gives us the chance to introduce the inexperienced reader to the topic.
  • Motivation for studying transformers at initialization. One main reason to study initialization in deep architectures is to identify hyperparameter regimes leading to successful training at large depth (which in turn depends on how the signal propagates through the network). For instance, for fully connected MLPs, [1] shows that the characteristic depth over which signals propagate diverges at a critical initialization, while gradients vanish or explode exponentially away from criticality. For ResNets, taking the infinite-depth limit leads to a neural ODE description of the layer dynamics [2]. This makes it possible to ask whether that description remains valid as the weights evolve, and [3] shows that it can persist during gradient-flow training. In Transformers, self-attention couples tokens within the same sequence, so the analogous question is whether token representations remain distinguishable as depth increases. This analysis has important consequences for training because token collapse at initialization can cause query and key gradients to vanish [4], while the initial rates of token separation and gradient propagation have been shown to correlate with final test loss [5]. Another reason to study infinite depth is to understand how to scale a model so that hyperparameters tuned at smaller depths can be reused in deeper networks. Depth-μP\mu P for ResNets [6] and CompleteP for Transformers [7] provide scaling rules that enable this transfer while preserving feature learning. We have added some of these references to the Related Work in the updated paper, while the connection between our analysis and trainability is strengthened by new experiments (see answer to Reviewer 7QM1).

References

[1] Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep Information Propagation.2017.

[2] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K. Duvenaud. Neural Ordinary Differential Equations. 2018.

[3] Pierre Marion, Yu-Han Wu, Michael E. Sander, and Gérard Biau. Implicit Regularization of Deep Residual Networks towards Neural ODEs. 2024.

[4] Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurélien Lucchi. Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse. 2022.

[5] Aditya Cowsik, Tamra Nebabu, Xiaoliang Qi, and Surya Ganguli. Geometric Dynamics of Signal Propagation Predict Trainability of Transformers. 2025.

[6] Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks. 2024.

[7] Nolan Dey, Bin Zhang, Lorenzo Noci, Mufan Li, Blake Bordelon, Shane Bergsma, Cengiz Pehlevan, Boris Hanin, and Joel Hestness. Don’t Be Lazy: CompleteP Enables Compute-Efficient Deep Transformers. 2025.

  • In the abstract the "mean deficit" stood for the expected squared distance between tokens in the manifold metric as defined at line 174 (183 in the new version). Furthermore, by the "same rate that governs the Jacobian of the token flow" we were trying to summarize in one sentence that the threshold of the main theorem, namely λMLP+λATT\lambda_{MLP}+\lambda_{ATT}, is the Lyapunov exponent of the restriction of Jacobian to the tangent space. We amended this sentence in the updated version of the paper.

  • With the phrase "identify a regime where token representations can cluster even as small perturbations of their shared representation are amplified." we wanted to stress the fact that the distinction between J⊥J^\perp and J∥J^\parallel is important as in the regime λ(J⊥)<0<λ(J∥)\lambda(J^\perp) <0<\lambda (J^\parallel) one may have clustering, but still the perturbations common to all tokens are amplified. For sake of clarity, we have deleted this part from the abstract.

  • The residual-stream dynamics refers to the depth-wise dynamics of the tokens in Residual Networks. This has been made more clear in the new version.

  • In the new version we have reported the specific equation of Koubbi et al. that we have used, namely Equation 2.5 of their work. While their equation models only the Attention dynamics, it is straightforward to derive the Equation 5, as discussed in Remark 2.1 in Koubbi et al., as argued in the new version of the paper.

  • We have updated our nomenclature with a more standard one: the field Bt(x)B_t(x) is a centered Gaussian field with covariance structure determined by equation 4 in the submitted version, hence its definition is unambiguous.

  • The stochastic integral with respect to this field can be interpreted more intuitively with the Karhunen-Loève expansion of the field in the spatial variable: the integral with respect to dBt(x)dB_t(x) is the countable sum of integrals with respect to σ(x)dBtk,\sigma(x)dB_t^k, where σk(x)\sigma_k(x) are the coefficients of the expansion and BtkB_t^k are a countable collection of standard Brownian motions. In the new version, this intuition is recalled in a footnote, while more details about the expansion are given in Appendix A.

  • In Definition 8 of the submitted version the perturbation is an element of the tangent space at X0X_0 which is applied to the initial condition X0X_0. We clarified this in the updated version of the paper.

  • A=(At)t≥0A=(A_t)_{t\ge 0} is a general family of linear (random) maps with the cocycle property. We now clarify this in the paper.

  • Equation 12 describes two limits: the annealed limit, denoted λˉ\bar{\lambda} and the quenched limit, denoted λ\lambda. Hence, when in Theorem 9 we refer to the quenched and annealed limits we refer to these quantities. We tried to be more clear in the new version.

  • With the expression "λ\lambda describes a single network" we wanted to underline the fact that the quenched exponent λ\lambda describes the pathwise dynamics: a single network stands for a realization of the network's parameters, hence a single path. For sake of clarity we have reformulated the sentence.

  • With "Generator on the overlaps" we wanted to refer to the Action of the generator of the tokens' dynamics over the overlap functions ρij(X):=⟨Xti,Xtj⟩,\rho_{ij}(X):=\langle X_t^i, X_t^j\rangle, defined in line 175 of the updated paper. We tried to be more precise in the new version.

  • Section 4 identifies the quantity that determines our local clustering threshold with a standard object in the theory of stochastic dynamical systems, the transverse Lyapunov exponent. This naturally leads us to compute the corresponding longitudinal exponent and ask what it tells us about gradient propagation through the architecture. Through the chain rule, we obtain a second, closely related threshold below which parameter gradients in early blocks vanish for synchronized inputs. Our new experiments show that networks initialized near this longitudinal threshold tend to train better.

Reviewer_7QM1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a simple threshold that determines if nearby tokens collapse or stay apart in transformers. The threshold connects to Jacobian growth; Simulations that match the predicted rates.

Critical strengths

The paper contributes nice and useful framework and results to the topic of token dynamic.

  • The threshold is explicit and easy to interpret.
  • The proofs are for the nonlinear dynamics, with small perturbations.
  • The whole work provides useful picture of gradient behavior and token separation from movement of the total token cluster.
Critical weaknesses

I feel like the experimental part is very weak.

  • Experiments mostly check local behavior with short depth. Moreover, I can see if the threshold remains accurate at widths and step sizes used in real practice.
  • For work in this topic of token dynamic, do the results support the benefit for learning, as suggested? There is none experiment for this purpose in the paper.
  • There is no code submission.
Questions for authors

Please answer the first two bullets raised in the Weaknesses.

1 comment

Dear Reviewer, thank you for the constructive feedback. Motivated by the concerns about connections with practical transformers, we expanded our experiments in the revised manuscript:

  • Realistic dimensions and step sizes (Figures 1 and 2(a)-(b)): We updated our numerical experiments to use more realistic dimensions and step sizes, specifically depth L=64L = 64, embedding dimension d=64/128d = 64/128, and MLP width m=256/512m = 256/512. Even in these finite, practical regimes, the empirical separation and gradient rates continue to closely match our asymptotic predictions.

  • Empirical link to learning and trainability (Figures 2(c) and 3): We added two CIFAR-10 training experiments demonstrating how our initialization rates affect optimization:

    • Figure 2(c) evaluates 8080 initializations of a 1616-layer transformer (L=16L = 16, d=m=64d = m = 64), showing that strongly negative longitudinal quenched rates induce vanishing gradients and stall learning, while the validation loss reaches its minimum right around the critical threshold λ^∥≈0\hat{\lambda}_\parallel \approx 0.
    • Figure 3 examines a deep 6464-layer transformer (L=64L = 64, d=m=64d = m = 64, T=4T = 4) trained across initializations that share nearly identical annealed rates (≈0\approx 0) but differ in their quenched rates. This experiment points to the quenched exponent (as opposed to the annealed one), as the deterministic limit point of the (random) finite-depth expansion rates, as the "right" quantity predicting the propagation of gradients in the network and, as a consequence, its training behavior.
  • We updated the supplementary material with the requested code.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.