- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes an atomic theory of language model representations. The authors show that the linear representation hypothesis implies the existence of atomic features. They then explore the implications of this for language model representations and SAEs trained on them. They make and validate 3 predictions based on their theory about the results of training SAEs on different data and with different scales.
Critical strengths
The paper is clear, well written and provides a particularly clean account of the consequences of the linear representation hypothesis that could be useful to the field.
This work also has implications for the promise of scaling SAEs or what would not be true of language model representations if scaling up SAEs does not work.
Critical weaknesses
I think the paper could do a better job acknowledging that the idea of discrete/atomic features encoded in superposition in transformer residual streams has been a common idea in the field and the main motivation for using SAEs and other dictionary leaning approaches. As such atomic features and their connection to the LRH are not new even if they have not been combined into a concrete theory with clean testable predictions before this work.
While this work provides a clear and useful theory, it provides more of a formalization and combination of existing ideas than a set of novel ideas. I nevertheless think this is a useful contribution to the field.
Questions for authors
Beyond my point under weaknesses, I would also like to see a bit more discussion of the point made in paragraph 3 of the introduction. After introducing Theorem 2, what is your view of the plausibility of atomic theory? What observations does your theory predict that, if found false in residual stream representations would undermine the LRH? Are there ways to soften the LRH that would make it more likely to fit with observations in real models?
2 comments
Thank you for your response.
While I believe this improves the paper, my previous concern about novelty still prevents me from raising to 4/4. I still think this is a useful contribution to the field and recommend acceptance.
New experiments with null model (enabling more falsifiability)
Thank you for the thoughtful review.
We agree that the idea of the idea of discrete features is not new, and that our main contribution is a way to make this precise and testable (which we see as a core step in resolving disagreements and building scientific theory).
Regarding your questions, we implemented an additional experiment that shows a comparison to a null model (Gaussian-distributed embeddings with matching mean and covariance). If the stability measures were the same or worse than in this null model, we would see that as evidence against the hypothesis. In experiments on this null model, we found 0% matches on the wiki vs no_wiki training data setting, suggesting that the amount of stability we found (26%) is substantial evidence in favor of the LRH.
We will add this discussion of falsifiability using a null model prominently after Theorem 2.
If you think this new addition helps address your concerns, we would appreciate it if you considered raising your score. Thanks again.