Predictive Rank Language Modeling: Measuring Representation Capacity in Language Models

Language models learn to predict future tokens by encoding information about their context. However, how the dimensionality and strength of predictive information interact with limited representation capacity remains unclear. To address this, we introduce Predictive Rank Language Modeling (PRLM), a controlled next-token learning environment with known predictive variables and a tunable distribution over future tokens. PRLM varies predictive rank, spectrum, and signal strength alongside residual width, while exposing only token sequences to the model. Experiments across 20 rank–width configurations reveal a gradual capacity frontier: increasing predictive rank reduces both predictive gain capture and global state decodability, whereas increasing residual width improves both. Matched controls further show that concentrating predictive energy raises gain capture from 27.2% to 82.6% at fixed rank and approximately matched Bayes KL. Under overload, weak global decoding nevertheless coexists with stable local effects and causally used predictive components. Sparse autoencoders likewise reconstruct activations accurately without reliably recovering the known concepts. Together, these findings show how predictive demand shapes learned representations and, using known ground truth, clarify what global decoding, local structure, sparse reconstruction, and causal use can establish about them.