Mean–Covariance Competition Governs Collapse and Multi-Cluster States in Causal Attention Dynamics

Causal self-attention can exhibit two distinct regimes: token representations may collapse toward a single dominant direction, forming attention sinks, or they may remain organized into multiple clusters. Despite extensive empirical observations, the mechanism governing the transition between these regimes remains poorly understood. In this work, we develop a dynamical and mean-field framework to explain this transition. We first use a first-order expansion of the causal softmax interaction to decompose the dynamics into two competing terms: a mean-driven alignment term, which depends on the prefix mean of previous tokens, and a covariance-driven routing term, which depends on the prefix covariance and the query–key interaction. We then show that, near an eigendirection of the value matrix, a spectral gap suppresses transverse perturbations and drives collapse toward a single dominant cluster, providing a mechanism for attention-sink formation. Conversely, when the cluster geometry keeps the prefix mean close to zero while maintaining nontrivial covariance, the alignment effect is weakened and covariance-driven routing remains active, allowing multi-cluster structures to persist. A key novelty of our approach is the use of Fourier analysis in the depth variable, which reveals the frequency-dependent structure of causal attention. These results identify causal attention dynamics as a competition between mean-driven spectral alignment and covariance-driven routing. These mechanisms are then examined by ${\mathbb{S}}^1$ simulations and layerwise analyses of pretrained Llama-3.1-8B model.