On attention heads and bilinear forms

We study the symmetric and antisymmetric parts of bilinear forms encoded by the query and key maps of attention heads, separating properties imposed by the architecture from patterns associated with training. We prove that, whenever the stacked query–key map is surjective, the symmetric part has exactly as many positive as negative eigenvalues. This condition is generic when the embedding dimension is at least twice the head dimension, and the predicted signature occurs in all $9,512$ heads of fourteen pretrained language models. To describe variation beyond this structural constraint, we introduce an orthogonally invariant profile map into a three-dimensional simplex. In thirteen of the fourteen models, trained profiles accumulate near the one-parameter family of profiles of rank-one forms. A limit theorem partially explains this phenomenon: profiles of symmetric matrices with proportionally distributed positive and negative eigenvalues converge to the projection of the rank-one profile family onto the symmetric face.