MLP Layers Refine and Expand Clusters in Causal Attention Dynamics

Modern transformers exhibit rich dynamical phenomena such as clustering, alignment, and dimensional collapse. These effects are often explained through attention-only dynamics, where spectral properties of the value matrix drive tokens toward dominant eigendirections. However, in practical transformer blocks, attention is coupled with a nonlinear multilayer perceptron (MLP), whose dynamical effect remains poorly understood. In this work, we study a minimal model of causal attention interacting with a nonlinear MLP. We show that the MLP can fundamentally reshape attention dynamics: even when attention alone selects dominant eigendirections, the coupled system can develop new attracting states that are not aligned with any eigenvector of the value matrix. These non-eigendirection attractors have basins of attraction of positive Lebesgue measure and therefore produce genuinely new clustering behavior. We identify the underlying mechanism: the MLP nonlinearly deforms the attention vector field, creates new equilibria, stabilizes them locally, and, through causal propagation, synchronizes tokens around these new states. We construct explicit two-dimensional examples and extend the analysis to multi-token causal dynamics. Our results demonstrate that MLP blocks are not merely auxiliary nonlinearities but can qualitatively change the long-time behavior of transformer dynamics, generating clustering patterns beyond those predicted by attention alone.