- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors conduct some intuitive case studies on sparse autoencoders, showing that there is a relationship between higher-order ground-truth feature magnitudes in the true data-generating process and the learned sparse codes. They generalize these case studies to some theoretical statements.
Critical strengths
- The paper's claims are correct, as far as I can tell. There is indeed a connection between the co-occurence statistics of data-generating concepts and the recovered concepts in an SAE.
- The conversational tone of the paper is nice when reading specific sections.
Critical weaknesses
- The overall structure of the paper is fairly atypical. I would find it easier to parse a more conventionally structured paper (think: Intro/elated work/Theory/Experiments/Conclusions). The related work in the introduction would be best separated to its own section, in my opinion, to better understand the contrast. In my opinion, this is absolutely necessary in a research area as active as SAEs.
- The main claim can be boiled down to: feature co-occurrence patterns (yes-no) are not the only aspect of importance when considering what SAEs learn relative to original representations, the higher-order co-activation statistics mattern, too. Unfortunately, this is not a new claim. For example, [1] says almost exactly this: "If co-occurring feature magnitudes vary relative to each other, we observe "partial absorption", where a latent tracking a main feature sometimes fires weakly instead of not firing at all, but sometimes does fully not fire." Additionally, [2] conducts experiments similar to the ones in this submission that manipulate these higher-order statistics to show that similar phenomena emerge. This submission cites a followup to [1] that unfortunately does not include the relevant commentary, likely simply an oversight. In addition there is substantial similarity to the empirical results of [2].
- At first glance, oth Theorems 2 and 3 can be read as a fairly strong identifiability claim on ReLU SAEs, which is potentially of interest. However, I think this reading is perhaps too gracious. Neither theorem applies to the overcomplete setting, which unfortunately limit their applicability to SAEs. This is of particular note because I think Theorem 2 in particular says very little beyond typical statements about -penalized models. Theorem 2 as written applies to the 2-dimensional case in the main text; one has to turn to the appendix to see the generalization to multiple dimensions. Theorem 3 has no generalization beyond 2 dimensions, and again seems to be a standard statement about -penalized models.
[1] Chanin et al. "Toy Models of Feature Absorption in SAEs", LessWrong (2024).
[2] Anders et al. "Sparse Autoencoders Find Composed Features in Small Toy Models", LessWrong (2024).
Questions for authors
- Can you clarify whether Theorem 2 or 3, as written hold, in the overcomplete setting?
- Do your experiments extend to cases beyond [1]/[2] referenced above? Can you add these references?
Dear Authors,
Thank you for your paper, but I don't seem to find any rebuttal to my comments? At any rate, thank you for your effort in writing this paper. Much appreciated!