- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper introduces a synthetic language modeling task where tokens are sampled from . The authors train many four-layer transformers on sequences that describe and update the latent state and study how performance changes as a function of the rank and singular value spectrum of the data generated log-odds matrix and residual width. There are two main findings. First, increasing the rank of the data-generating matrix reduces performance at fixed model width. Likewise, increasing the width or concentrating the singular value spectrum improves performance. Second, the authors find that models still achieve nontrivial performance despite poor recovery of using global probes over hidden states. They further show that SAEs can reasonably reconstruct activations without recovering the underlying .
Critical strengths
I appreciate the comprehensive and rigorous nature of the experiments. Additionally, the implementation is not left ambiguous: the appendix contains all training configs, controls, and full sweep results.
PRLM very clearly allows one to isolate and mechanistically study the various levers that dictate model performance – this is genuinely useful for the community.
Critical weaknesses
This paper was unnecessarily difficult to follow, even after careful reading. The submission would benefit from a substantial rewrite.
- The data-generation process and training setup are not clearly explained in the main text. This paper would benefit from a concrete example of a complete training sequence outside the appendix, and perhaps some pseudocode describing the training loop/what the transformer is trained to predict.
- This paper introduces a substantial amount of jargon/terminology. Furthermore, the text is littered with technical/overly complex words that describe relatively simple things. This greatly reduces the reading experience and limits accessibility. For example: “Capture” is just normalized improvement in KL, “Predictive overload” means > width, “Payload KL” means evaluation restricted to generated token vocab. There are many more such examples
- Section 3 is borderline unreadable. For example, the discussion after equation 9 is extremely convoluted, and the authors introduce several layers of terminology. It only needs to say, "We vary the singular values of B while keeping its Frobenius norm fixed. We use as the target for the probes because these coordinates determine the log odds."
- Four research questions feel excessive. More generally, there are several distinct experimental threads, but there’s no clear central argument connecting them. It feels like the authors conducted many experiments, threw them together, explained them, but don’t actually synthesize them – what should the community learn from this paper? More on this below
I’m not sure what new understanding this paper provides beyond already well-established results/intuitions. Said differently, I can’t identify any sufficiently novel conceptual or theoretical contribution.
- The authors find that increasing predictive rank at fixed width reduces performance, while increasing width improves it. Why should these results surprise me? This relationship seems largely anticipated by existing theory. Classical low rank approximation results (like Eckart-Young Theorem) characterize how approximation error depends on the discarded singular values, while Saxe et al. (2013) analyze how undercomplete deep linear networks learn the leading singular modes of a target matrix. Extensions to exponential-family losses (Collins et al., 2002), together with the well-established softmax bottleneck (Yang et al., 2017; Ganea et al., 2019), provide further context for precisely this type of experiment. The authors demonstrate similar empirical trends in small Transformers, but it remains unclear what new understanding these experiments provide beyond existing approximation and expressivity results.
- The authors find that poor global linear decodability and high model performance can coexist, but it’s well known that a model’s ability to make accurate predictions does not mean its entire underlying state can be inferred through linear regression. Further, the authors show that accurate SAE reconstruction need not imply recovery of the underlying latent, but this is again a known limitation of SAEs.
How much of the model performance is actually attributable to the representation bottleneck?
- Given the known generator (p^*(y\mid z)=\operatorname{softmax}(Bz)), the minimum expected KL achievable by a predictor with a (d)-dimensional representation and linear output head is [ \min_{W,h}\mathbb E_z\left[D_{\mathrm{KL}}\left(\operatorname{softmax}(Bz),|,\operatorname{softmax}(Wh(z))\right)\right], \qquad h(z)\in\mathbb R^d. ] The SVD truncation is an okay proxy, but doesn’t actually solve the objective, and the models seem to improve with longer training (as in Figure 6 and Table 6). Further, models with enough width to represent full rank are still quite suboptimal. Without establishing what performance is actually achievable under the rank constraint, it’s hard to disambiguate whether representation capacity, optimization, or the task itself are the main driver of experimental results.
References:
Collins, Michael, Sanjoy Dasgupta, and Robert E. Schapire. "A generalization of principal components analysis to the exponential family." Advances in neural information processing systems 14 (2001).
Ganea, Octavian, et al. "Breaking the softmax bottleneck via learnable monotonic pointwise non-linearities." International Conference on Machine Learning. PMLR, 2019.
Godey, Nathan, Éric de la Clergerie, and Benoît Sagot. "Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck." arXiv preprint arXiv:2404.07647 (2024).
Saxe, Andrew M., James L. McClelland, and Surya Ganguli. "Exact solutions to the nonlinear dynamics of learning in deep linear neural networks." arXiv preprint arXiv:1312.6120 (2013).
Wies, Noam, et al. "Which transformer architecture fits my data? A vocabulary bottleneck in self-attention." International Conference on Machine Learning. PMLR, 2021.
Yang, Zhilin, et al. "Breaking the softmax bottleneck: A high-rank RNN language model." arXiv preprint arXiv:1711.03953 (2017).
Questions for authors
See the section above for what to address. I’m very open to changing my score!