Attention Invariants: Weight-Space Interpretability Without a Forward Pass

Attention heads in transformers can be characterized through basis-invariant quantities from weights alone, without any data or forward pass. We derive two invariants of the QK sector: $\rho$, which measures a head's positional capacity, and $\alpha$, the definiteness of the symmetric part of its QK form. We prove a necessity condition on positional computation. A head with $\rho =0$ cannot depend on relative position, so positional function requires $\rho >0$, a structural constraint we find is obeyed without exception across every model. On five models from four architecture families (Qwen2.5-1.5B/7B, Mistral-7B, Llama-3.1-8B, Gemma-2-2B), $\rho$ and $\alpha$ separate three canonical head types (induction, duplicate-token, previous-token) in distinct regions of a two-dimensional plane. We show that a logistic regression trained on $(\rho ,\alpha )$ separates every head type pair at ROC-AUC $\ge 0.97$, and unsupervised clustering recovers the types at $0.86$–$1.00$ purity. Transferring to a different architecture, we obtain a median of $0.94$ balanced accuracy ($16$ of $20$ pairs $\ge 0.87$). As an application, we apply YaRN only to the top $50\%$ heads by $\rho$, arriving at better results than a standard uniform application.