Clustering Thresholds for Homogenized Transformers

We study the evolution of spherical token representations under randomly resampled attention and MLP layers, in a continuous-depth model where all tokens are driven by the same random fields. Our main result is that the long-time behavior is decided by the sign of a single explicit quantity. When this quantity is negative, tokens started close together collapse to a single cluster with probability tending to one as their initial diameter tends to zero. When it is positive, distinct tokens almost surely do not converge. We also give a sufficient condition under which the mean deficit decays exponentially and synchronization holds almost surely from every initial configuration, and we show that the same rate governs the Jacobian of the token flow. We connect the thresholds separating different clustering behavior with (quenched and annealed) Lyapunov exponents, and identify a regime where token representations can cluster even as small perturbations of their shared representation are amplified.