- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies the toy model of superposition introduced by Elhage et al. It considers at what width scale do all individual features get represented. The paper shows empirically that there exists a phase transition after which all individual features are represented. The paper shows that this behavior arises in a random model, in which a subset of features are represented and the rest are not. (Then the question becomes the optimal number of features to represent). The paper shows this value to be about E log (n/E) where n is the total number of features and E is the average number of active features. This matches classical compressed sensing results.
Critical strengths
While many papers have demonstrated that superposition is possible when the width is on the scale of E log (n/E) (using the paper's notation), the paper illustrates interesting behavior prior to this cutoff point, where subsets of features are well-represented and the rest are not (as shown by Figure 2(a)). I think the paper could perhaps further emphasize this aspect of the result, which is comparatively novel. Naively, one might expect that even below the E log (n/E) threshold, the model is best off trying to represent every feature (even if poorly). This fact is more surprising than on first read of the paper.
The partial random projection model is elegant, and gives a simple way to reach this conclusion.
Critical weaknesses
The implications for further developing a theory of language models or the linear representation hypothesis are not obvious, and could be further developed.
The paper would also benefit from exploring what happens under alternative "architectures" such as PCA. What aspect of the model yields the phase transition behavior?
Questions for authors
How do you see the model's core implications beyond establishing the E log n critical threshold, particularly in relation to further developing theory of language models?
What are the analogs of the results for PCA? What are the crucial aspects of the model that yield the phase transition behavior?
2 comments
Dear Reviewer mrik,
Thank you again for your time and effort in reviewing our paper.
With the discussion period closing in less than 24 hours (Nov. 18, AoE), we would like to follow up on our response to your questions on the implications of our results, the comparison with PCA, and the ingredients of the transition. Please let us know whether our replies answer them, or whether anything is still unclear; we would be happy to clarify.
If our responses address your concerns, we would be grateful if you could take them into account in your final assessment.
Best regards, The authors
We thank the reviewer for the thoughtful review, and in particular for pointing out that the behavior below the threshold, where a subset of features is well represented and the rest are not, is the more surprising part of our results.
How do you see the model's core implications beyond establishing the Eˉlogn critical threshold: We thank the reviewer for this question, and for the suggestion to further emphasize the behavior below the threshold, which we will follow in the revised version. Beyond the value of m∗, we see two implications for a theory of language models.
We will add further discussions about these points to the revised version. Testing these implications in language models requires feature-level measurements across model widths (e.g., with sparse autoencoders), which we leave to future work.
What are the analogs of the results for PCA? We read PCA as the linear version of the model (no ReLU), whose optimum is a principal-component projection of the covariance matrix Cov(x) of the input x (Baldi & Hornik, 1989). When features differ in importance or frequency, it keeps the m leading features and discards the rest (r=m), as in Elhage et al. (2022). This is the linear analog of our non-uniform result that frequent features are represented first, but the transition from partial to full representation cannot happen at m<n. For uniform features, Cov(x)∝I, so every rank-m projection is optimal and the linear model has no preferred number of represented features. Therefore, the phase transition cannot happen for uniform features either. We will add the discussion on PCA in the revised version.
What are the crucial aspects of the model that yield the phase transition behavior? The transition in our model requires three ingredients: 1) sparsity, so that interference grows with the number of active features Eˉ rather than n; 2) a nonlinear threshold that filters interference on inactive features. In our toy model it is ReLU with bias, which is where logn/Eˉ comes from; and 3) a width-dependent trade-off: the cost of representing a feature and the cost of discarding it depend differently on the width m, so their balance shifts with m and sets $m^* $.