Pairwise Anti-Collapse Regularization Restores Representational Diversity in Pure-Attention Transformers

Pure self-attention without skip connections converges to a rank-one subspace at a doubly exponential rate in depth—a pathology known as representational collapse. For a decade the remedy has been architectural: route around attention, treating collapse as an inevitability to be engineered against. We show it is instead learned: the representation begins near full rank and contracts as training proceeds, because cross-entropy finds that aligning tokens cheaply reduces loss. From one initialisation, two gradients therefore steer the same network to opposite fixed points—the rank-one corner, or an orthonormal frame. Reaching the frame means penalising the right object: not the $D\times D$ channel covariance of feature-decorrelation methods, since decorrelated channels do not stop two tokens from sharing a direction, but the $T\times T$ token cosine matrix, whose mean absolute off-diagonal entry we prove is uniquely minimised by an orthonormal frame. The resulting regulariser, (), is four lines of PyTorch—one hyper-parameter, no parameters, no inference cost. On the canonical nine-cell benchmark it reduces cosine collapse by 99.2% and lifts the entropy-based effective rank from $2.74$ to $15.99$ of a $T=16$ ceiling. Against fifteen baselines spanning $2016$–$2026$ it alone eliminates cosine collapse and restores near-full rank on every cell, with $16\times$ less collapse than the best. The separation is categorical: the two nine-cell samples do not overlap ($81$ of $81$ pairwise comparisons, permutation $p=4.1\times 10^{-5}$), it holds across depths and widths at an untuned $\lambda =1$, and the spectrum is left essentially flat (${\sigma }_{16}/{\sigma }_1=0.92$)—the signature of the predicted frame. Added to a standard $+$MLP$+$Skip block, yields the single best point of the entire benchmark on every metric, at $4.67\%$ test error for a measured $+4.3\%$ training overhead. Representational collapse is thus an optimisation attractor, governable at the level of the loss.