- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper explores the question of identifiability in SAEs, presenting the idea that identifiability should be viewed as a 5-level spectrum rather than a binary question. The paper then explores how 3 different SAE training algorithms compare against the existing gemma-scope SAE on gemma-2-2b, at different layers and sparsity levels, and compares and contrasts them along these identifiability axes.
Critical strengths
-
The abstract and intro of the paper are mostly written well and present the motivation and research question in a way that made me interested and convinced this is an interesting research question to study.
-
Although I am not super familiar with the literature on identifiability, the claim that identifiability should be viewed as a spectrum with different levels of granularity, rather than a binary question, is interesting and brings nuance, which I think has potential to advance our understanding of model internals and SAEs specifically.
Critical weaknesses
-
No related work section in the main paper. The intro refers to the appendix for a related work section which is very limited - it mentions one study plus some concurrent studies. The authors provide no positioning of this work within existing literature on identifiability of model components / neurons, or other interpretability methods other than SAEs, which is critical for understanding the paper and its contributions. This is a major weakness which significantly lowered my score.
-
Overclaiming on applications to many interpretability methods, when the paper only discusses SAEs. The title and intro of the paper make it seem like the identifiability hierarchy is applicable to any (mechanistic) interpretability method, while the paper discusses SAEs alone. I would avoid this overclaiming, and a discussion on how this can be further applied/extended can be written in the discussion section.
-
The structure of the paper has major weaknesses that take away from its readability:
- The intro does not provide any actual details on the levels proposed: coefficient, support, subspace, behavioral and causal. That makes the list of contributions difficult to read, as it lists insights before I understand (at least at an intuitive level) what each level means.
- Moreover, one of the contributions refers to the appendix - contributions should all fit in the main text.
- Section 2.1 is very dense with definitions that are critical for the paper, I would take more space to explain them in detail. Specifically, the definition of “identifiable up to a class T”.
- Figure 1 is unclear (and AI generated?) - What are OMP/IHT/FISTA? Why is the encoder an arrow while the decoder is a box? The texts that accompany each level are unclear, and the bottom one is cut off.
- 3.1 lists four activation datasets but lists MMLU twice.
- In section 2, OMP and FISTA are again used but not defined/referenced, it's unclear what they are and very difficult to read. I later found the definitions and references in section 3.2, the definitions should come before the first mention of these methods.
- Section 2 does not give technical details on how each level is evaluated and measured, while section 3 refers to these non-existent definitions.
-
As far as I understand, Table 1 presents results for a single dataset (the pile), but 4 (or 3) datasets where mentioned in the experimental setup. There is no mention of the rest of the results in the text.
The writing of this paper is very difficult to follow, and major revisions are needed in order for it to be accepted (some are listed above). For these reasons I give this paper a score of clear reject.
Questions for authors
No Questions
We thank you for your time and help, especially when it comes to the detailed list of structural and legibility issues. We are glad to see that the introduction convinced you that this is a question worth studying, and that viewing identifiability in this way provides useful nuance to understanding model internals.
Before addressing the main comments, we do wish to start with an important clarification. OMP, IHT, and FISTA which are explained in section 3 (with added details in the appendix) are not SAE training methods. These are classic algorithms on sparse inference that we execute over an already trained dictionary (GemmaScope) that we freeze. To reiterate we do not ever train any new SAE. Our question is on whether with the same dictionary and the same activation, there exist other sparse coders as good as the encoder.
Introduction: We do not provide the full definitions and operationalizing details of each level in the hierarchy because we dedicate the whole subsequent section to this.
Related work: While our paper already integrated a lot of related work in the prose of sections 2 and 3 meant to function as background needed for the reader in order to understand the experimental sections, we have added a new related work subsection in section 2. We also have expanded the related work section in the appendix.
Scope: While the main experimental evidence in the main body of text concerns SAEs, we do still maintain that the framework is generalizable beyond these.The definitions in section 2 and proposition 1 in the appendix apply to any sparse code from a dictionary, such an SAE, a sparse probe, or selection of characteristics. While in the main body of text we provide experiments and results on SAEs only, Appendix G shows that the same dissociation we observed for SAEs appears in sparse probes with an external label, and even on decision trees.
Algorithm definition: We have revised the text so that OMP, IHT, and FISTA are presented on Section 2 with a reference to Section 3 for a more complete explanation.
How each level is measured: We have added a table on section 2 in addition to our original figure 2 which should make it easier to quickly grasp what is the test we use for each level and the verdict on our own experimental results.
Figure: We have fixed the cut-off text. And yes this was made with the assistance of AI tools, this was already explicitly disclosed in the required AI usage section. As for the visual representation of the encoder and decoder: What you see is two branching arrows, one for the encoder and one for the alternative sparse coding method, the decoder is shown in just one form because in our experimental protocol we freeze it.
Table 1: Table 1 shows the PILE results. The results on FACTUAL, MMLU, and CODE, and on a second model are provided in appendix D and show the same pattern (Jaccard 0.44–0.58, top-10 0.74–0.83). We have added a footnote on this.
Thank you again for taking the time to review the paper and providing helpful feedback!