Two-Clustering Regime of Token Dynamics in Causal Attention

We partially resolve Conjecture 1 of Karagodin et al., Clustering in Causal Attention Masking (NeurIPS 2024) and extend it to time-dependent parameter regimes. The conjecture predicts that causal attention dynamics converge to two clusters aligned with the eigenvectors associated with the largest eigenvalue ${\lambda }_{\max }$ of the value transformation. We prove that this behavior holds when ${\lambda }_{\max }>0$ and is generally false when ${\lambda }_{\max }\le 0$. The resulting two-cluster regime is common in practice and substantially more challenging than the classical single-cluster setting studied previously. A central contribution of our work is the identification of a rigorous dimension deduction phenomenon: despite evolving in a high-dimensional embedding space, token trajectories collapse onto a two-dimensional structure governed by a small number of dominant directions. Building on this insight, we introduce a new analytical framework–instability within the flow–inspired by instability phenomena in physical systems. This framework enables the analysis of steady-regime behavior in non-monotone, non-autonomous attention dynamics. Our theoretical findings are illustrated through simulations and experimental studies using Group Query Attention (GQA) layers.