acceptodds
Withdrawn from ICLR 2027

Predictive Rank Language Modeling: Measuring Representation Capacity in Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language models learn to predict future tokens by encoding information about their context. However, how the dimensionality and strength of predictive information interact with limited representation capacity remains unclear. To address this, we introduce Predictive Rank Language Modeling (PRLM), a controlled next-token learning environment with known predictive variables and a tunable distribution over future tokens. PRLM varies predictive rank, spectrum, and signal strength alongside residual width, while exposing only token sequences to the model. Experiments across 20 rank–width configurations reveal a gradual capacity frontier: increasing predictive rank reduces both predictive gain capture and global state decodability, whereas increasing residual width improves both. Matched controls further show that concentrating predictive energy raises gain capture from 27.2% to 82.6% at fixed rank and approximately matched Bayes KL. Under overload, weak global decoding nevertheless coexists with stable local effects and causally used predictive components. Sparse autoencoders likewise reconstruct activations accurately without reliably recovering the known concepts. Together, these findings show how predictive demand shapes learned representations and, using known ground truth, clarify what global decoding, local structure, sparse reconstruction, and causal use can establish about them.

Primary area:
interpretability and explainable AI
Keywords:
language modeling, representation learning, representation capacity, mechanistic interpretability, controlled experiments

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_rD2QOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper introduces a synthetic language modeling task where tokens are sampled from softmax⁡(Azt+b)\operatorname{softmax}(A z_t + b). The authors train many four-layer transformers on sequences that describe and update the latent state ztz_t and study how performance changes as a function of the rank and singular value spectrum of the data generated log-odds matrix and residual width. There are two main findings. First, increasing the rank of the data-generating matrix reduces performance at fixed model width. Likewise, increasing the width or concentrating the singular value spectrum improves performance. Second, the authors find that models still achieve nontrivial performance despite poor recovery of ΣV⊤zt\Sigma V^\top z_t using global probes over hidden states. They further show that SAEs can reasonably reconstruct activations without recovering the underlying ztz_t.

Critical strengths

I appreciate the comprehensive and rigorous nature of the experiments. Additionally, the implementation is not left ambiguous: the appendix contains all training configs, controls, and full sweep results.

PRLM very clearly allows one to isolate and mechanistically study the various levers that dictate model performance – this is genuinely useful for the community.

Critical weaknesses

This paper was unnecessarily difficult to follow, even after careful reading. The submission would benefit from a substantial rewrite.

  • The data-generation process and training setup are not clearly explained in the main text. This paper would benefit from a concrete example of a complete training sequence outside the appendix, and perhaps some pseudocode describing the training loop/what the transformer is trained to predict.
  • This paper introduces a substantial amount of jargon/terminology. Furthermore, the text is littered with technical/overly complex words that describe relatively simple things. This greatly reduces the reading experience and limits accessibility. For example: “Capture” is just normalized improvement in KL, “Predictive overload” means rank⁡(B)\operatorname{rank}(B) > width, “Payload KL” means evaluation restricted to generated token vocab. There are many more such examples
  • Section 3 is borderline unreadable. For example, the discussion after equation 9 is extremely convoluted, and the authors introduce several layers of terminology. It only needs to say, "We vary the singular values of B while keeping its Frobenius norm fixed. We use ΣV⊤z\Sigma V^\top z as the target for the probes because these coordinates determine the log odds."
  • Four research questions feel excessive. More generally, there are several distinct experimental threads, but there’s no clear central argument connecting them. It feels like the authors conducted many experiments, threw them together, explained them, but don’t actually synthesize them – what should the community learn from this paper? More on this below

I’m not sure what new understanding this paper provides beyond already well-established results/intuitions. Said differently, I can’t identify any sufficiently novel conceptual or theoretical contribution.

  • The authors find that increasing predictive rank at fixed width reduces performance, while increasing width improves it. Why should these results surprise me? This relationship seems largely anticipated by existing theory. Classical low rank approximation results (like Eckart-Young Theorem) characterize how approximation error depends on the discarded singular values, while Saxe et al. (2013) analyze how undercomplete deep linear networks learn the leading singular modes of a target matrix. Extensions to exponential-family losses (Collins et al., 2002), together with the well-established softmax bottleneck (Yang et al., 2017; Ganea et al., 2019), provide further context for precisely this type of experiment. The authors demonstrate similar empirical trends in small Transformers, but it remains unclear what new understanding these experiments provide beyond existing approximation and expressivity results.
  • The authors find that poor global linear decodability and high model performance can coexist, but it’s well known that a model’s ability to make accurate predictions does not mean its entire underlying state can be inferred through linear regression. Further, the authors show that accurate SAE reconstruction need not imply recovery of the underlying latent, but this is again a known limitation of SAEs.

How much of the model performance is actually attributable to the representation bottleneck?

  • Given the known generator (p^*(y\mid z)=\operatorname{softmax}(Bz)), the minimum expected KL achievable by a predictor with a (d)-dimensional representation and linear output head is [ \min_{W,h}\mathbb E_z\left[D_{\mathrm{KL}}\left(\operatorname{softmax}(Bz),|,\operatorname{softmax}(Wh(z))\right)\right], \qquad h(z)\in\mathbb R^d. ] The SVD truncation is an okay proxy, but doesn’t actually solve the objective, and the models seem to improve with longer training (as in Figure 6 and Table 6). Further, models with enough width to represent full rank are still quite suboptimal. Without establishing what performance is actually achievable under the rank constraint, it’s hard to disambiguate whether representation capacity, optimization, or the task itself are the main driver of experimental results.

References:

Collins, Michael, Sanjoy Dasgupta, and Robert E. Schapire. "A generalization of principal components analysis to the exponential family." Advances in neural information processing systems 14 (2001).

Ganea, Octavian, et al. "Breaking the softmax bottleneck via learnable monotonic pointwise non-linearities." International Conference on Machine Learning. PMLR, 2019.

Godey, Nathan, Éric de la Clergerie, and Benoît Sagot. "Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck." arXiv preprint arXiv:2404.07647 (2024).

Saxe, Andrew M., James L. McClelland, and Surya Ganguli. "Exact solutions to the nonlinear dynamics of learning in deep linear neural networks." arXiv preprint arXiv:1312.6120 (2013).

Wies, Noam, et al. "Which transformer architecture fits my data? A vocabulary bottleneck in self-attention." International Conference on Machine Learning. PMLR, 2021.

Yang, Zhilin, et al. "Breaking the softmax bottleneck: A high-rank RNN language model." arXiv preprint arXiv:1711.03953 (2017).

Questions for authors

See the section above for what to address. I’m very open to changing my score!

Reviewer_CszbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PRLM, a synthetic language modeling setting with controllable predictive rank, spectrum, and signal strength. Experiments study how these factors and model width affect prediction performance. The paper also uses known ground truth to evaluate linear probes, causal interventions, and SAEs.

Critical strengths
  1. Controlled evaluation setting. PRLM provides known predictive ground truth under standard next-token training, allowing systematic studies of representation learning.

  2. Interpretability insights. The experiments show that poor linear probe performance does not necessarily mean predictive information is unused, and good SAE reconstruction does not guarantee recovery of true predictive variables.

Critical weaknesses

Limited Insights into Representation Capacity and Learning Efficiency

The main rank–width and spectral-concentration findings are largely expected from classical low-rank approximation: higher predictive rank makes fixed-width approximation harder, while a concentrated spectrum reduces effective dimensionality. More importantly, the experiments do not distinguish intrinsic capacity limitations from incomplete learning. For example, even when predictive rank equals model width (128), the model captures only 41.3% of the available predictive gain. Doubling training from 72 to 144 epochs still improves performance, with the best checkpoints near the end of training, leaving convergence unclear.

Since the data-generating distribution is known, the authors could compare against a capacity-constrained oracle: a predictor given the true latent state but restricted to a rank-(d) output log-odds mapping, where (d) matches model width. Together with compute/data scaling experiments, this would clarify how much of the observed degradation reflects unavoidable dimensionality constraints versus difficulty learning the predictive representation. Without such evidence, it is difficult to identify new insights into representation learning or actionable model-design guidelines beyond the expected approximation trends.

Questions for authors
  1. Capacity vs. Learning: How much of the observed performance gap can be closed with more training compute/data, and how much remains relative to the capacity-constrained oracle described in weakness?

  2. Practical Significance: Can predictive rank or spectrum provide actionable guidance for model width or compute allocation beyond expected low-rank approximation behavior?

Reviewer_fwoUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I will let the paper (try to) speak for itself and provide a portion of the paper verbatim as the paper summary.

Abstract: "To address this, we introduce Predictive Rank Language Modeling (PRLM), a controlled next-token learning environment with known predictive variables and a tunable distribution over future tokens. PRLM varies predictive rank, spectrum, and signal strength alongside residual width, while exposing only token sequences to the model. Experiments across 20 rank–width configurations reveal a gradual capacity frontier: increasing predictive rank reduces both predictive gain capture and global state decodability, whereas increasing residual width improves both. Matched controls further show that concentrating predictive energy raises gain capture from 27.2% to 82.6% at fixed rank and approximately matched Bayes KL. Under overload, weak global decoding nevertheless coexists with stable local effects and causally used predictive components."

Conclusion: "We introduced PRLM to control predictive rank, spectrum, and signal strength under ordinary next-token training, with known ground truth for representation analysis. Across 20 rank–width configurations and targeted controls, we find a gradual capacity frontier; concentrating predictive energy raises gain capture from 27.2% to 82.6% at fixed rank and approximately matched Bayes KL. Weak global decoding can coexist with predictive use, while accurate sparse reconstruction can coexist with weak recovery of the ground-truth predictive variables."

Critical strengths

None.

Critical weaknesses

The paper is a pain to read and a standard example of mostly end-to-end AI writing. How do I know? When AI writes a paper end to end, it often neglects the reader's state of mind, i.e., it assumes the reader is already fully aware of its content, and hence, does not define terminology chrnololgically. As we can above in the abstract and similarly in the introduction, we encounter new terms undefined: predictive rank, spectrum, siganl strength, residual width (of what??), capacity frontier, gain capture, global state decodability, predictive energy, weak global decoding and more. These are English words smashed into an alphabet soup that carries no substance and simply exists to obscure what is being done. The presentation obscures the concrete research question and overstates the explanatory scope of the experiments. Thus, I refuse to even discuss the remaining weaknesses as this paper is simply not at all close to the caliber of writing necessary to publish. Significant revision is required but an amount so significant that when it is properly revised, we would need new reviewers as it would essentially be a brand new paper.

Questions for authors

None.

Reviewer_MWzHOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work proposes a controlled environment for studying the way predictive information is represented in Transformers. It names the Predictive Rank Language Modeling (PRLM) as a technique for control model training and studies its development and learned features, looking into how the residual stream converges to the desired representations.

Critical strengths

It is useful to find that narrower models achieve worse predictive performance without pointing the cause to be representation dimensionality.

Critical weaknesses

The paper doesn't properly compare the work with prior literature nor explains the decisions taken from a theoretical perspective. Authors should look into literature in subspaces and low-rank structures in LLM interpretability and explain how the techniques presented distinguish themselves properly. In particular, [5-6] need attention because they provide formal treatments to counterfactual directions using residual differences and evaluate their consistency with a proper geometric inner product, and [7] shows subspace manipulation can alter outputs in ways that are not necessarily causal, being due to confounding effects. The current paper combines several existing ideas without clearly identifying them nor which empirical observation is truly unexpected, which limits the claims on novelty.

  • [1] Yang et al., Breaking the Softmax Bottleneck, ICLR 2018
  • [2] Liu et al., Superposition Yields Robust Neural Scaling, NeurIPS 2025
  • [3] Shai et al., Transformers Represent Belief State Geometry, NeurIPS 2024
  • [4] Sharma et al., Temporal Superposition and Feature Geometry, ICLR 2026
  • [5] Park et al., The Linear Representation Hypothesis and the Geometry of LLMs, ICML 2024
  • [6] Valois et al., Frame Representation Hypothesis, TACL 2025
  • [7] Makelov et al., Is This the Subspace You Are Looking For?, ICLR 2024
  • [8] Nelson et al., Toward Identifiable Sparse Autoencoders, ICML 2026

In that sense, there are several limitations in the current experiments and discussion that need to be addressed. The problem is that the experiment currently do not isolate the representation capacity because, for example, the rank-128/width-128 model achieves only 41.3% Capture, so there is clearly learning difficulty behavior and part of the results could be random. Also, changing residual widths impacts the model behavior way beyond representation capacity, altering how attention and MLP layers work, providing them with out-of-distribution data.

In the compositional evaluation, the default generator has 128 possible variables and 4 active variables per episode, giving 10M possible active sets, but the dataset contains only 12k training sequences and Appendix B.1 states replacement is disabled, so training data covers only approximately 0.11% of possible combinations. This greatly limits the extension of the results. The paper also needs to compare performance with an oracle model and fit causal intervention directions on independent training data, so then to intervene on held-out examples.

Finally, the writing is unorthodox, with unclear motivation, excessive terminology, and inaccessible explanations at parts, so the paper needs substantial improvement in clarity.

Questions for authors
  • What is fundamentally new compared with prior work?
  • How can you be sure the observed results are caused by representation capacity and not something else?
  • What makes compositional evaluation different from ordinary IID testing?
  • in Appendix F.3, Table 18, a substantial part of the observed performance deficit remains unexplained by rank compression, so what other effects could be involved and why does this method fail in that aspect?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.