Cone Geometry is a Routing-Controlled State
Abstract
Transformer layers are prone to over-mixing: token representations can lose distinguishability with depth or context, appearing as rank collapse, spectral compression, or narrow cone-like directional concentration. This concentration is usually described through coupled correlates–attention sinks, norm outliers, massive activations, and spectral compression–yet it remains unclear which is a controllable cause of the cone rather than a co-occurring symptom. We test one candidate: recipient-side structural sink routing. We suppress incoming attention to structural sink tokens and measure the resulting change in cone geometry, applying the same intervention to a text-to-image diffusion transformer (Stable Diffusion 3) and to a 16-model language-model suite. In language models, norm controls (output-norm replay, output-norm stress, and residual-stream norm stress) do not reproduce the effect, even though residual norm stress moves spectral-compression metrics. Thus, in the controllable sense, the cone is routing-controlled rather than norm-locked. Across both domains, this routing handle is signed: special/EOT-like SD3 text carriers open the image-token cone while padding-like carriers close it, and LM first-token routes form a carrier- and depth-dependent landscape rather than a uniformly positive sink effect. A pre-registered two-dataset breadth audit shows that these LM signs are mostly dataset-stable, and a first-order value-replacement probe acts as a cheap pre-intervention sign diagnostic: without running the full routing suppression, it predicts whether candidate edits will open or narrow the cone above chance. The same routing interventions also move next-token logit distributions far more than matched controls. Recipient-side routing is therefore a separable, predictable, and signed causal axis for cone geometry–and for immediate output behavior–in the models we study.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.