The Low-Rank Canonical Geometry of LLMs

Large language models predict the next token from vocabularies exceeding 10^5 tokens, suggesting an extremely high-dimensional output space. Yet it remains unclear whether their predictive distributions exploit this enormous space, or instead vary within a much smaller structure as context varies. We develop a spectral framework that characterizes this geometry directly in probability space. By separating the model's predictive marginal from context-dependent probability variation, the framework decomposes LLM next-token prediction into orthogonal token-side canonical modes, revealing both the dimensionality and orientation of the predictive geometry. Surprisingly, we find that LLM prediction is organized around a much smaller canonical geometry despite their vast vocabulary. Across model families and context distributions, nearly all context-dependent predictive variation is captured by a small set of dominant token modes: only 9.7%–24.0% of the token-space dimensions account for \(99.9%\) of this variation. More surprisingly, this geometry remains largely intact under downstream adaptation. Fine-tuning changes the model's conditional predictions while largely preserving the concentration of the canonical spectrum and the dominant canonical token subspace, even across different downstream tasks. These observations motivate the Canonical Subspace Hypothesis: pretrained language models organize next-token prediction around a low-dimensional canonical token geometry that downstream adaptation largely reuses rather than reconstructs. More broadly, our framework provides a general tool for studying how the geometry of LLM prediction is formed, preserved, and transformed throughout training.