- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This work aims at explaining features of trained sparse autoencoders for varying types of language models (both autoregressive and bidirectional). This is done by training a linear classifier with positive weights to predict a feature activation of an SAE feature solely from the one hot encoded input token matrix. In this way we gain insight into whether a specific token at a specific location elicits a specific SAE feature and we can also determine whether it alone suffices or whether a combination of multiple tokens is needed.
The central focus thereby is on both training and evaluating at "natural prevalence". Specifically, the authors identified a limitation of existing approaches, which often use an equal share of texts that elicit a specific SAE feature and texts that don't elicit it, and then use some method (like processing them with an LLM) to automatically determine the rule from these samples. The evaluation then usually works in the same way by using held out samples to check how often the identified rule can predict a feature activation, which is also usually done by using the same number of positive and negative samples (or a slightly skewed ratio). The problem with this set up is that a low false positive rate (e.g., 10%) will only have a minor influence on the evaluation score in this specific setting. But if the classifier is instead applied to a vast corpus of text without balancing positives and neagatives first, then the number of negatives will dominate (since most features appear only very rarely in the corpus), which drives up the absolute number of false positives in contrast to the true positives and leads to extremely low classifier performance.
The linear classifier proposed in this work can circumvent this issue by simply adapting the threshold required for a positive prediction (i.e., raising it so that a 90% confidence is needed instead of, for example, 50%). In this way the false positive rate can be lowered substantially (at the cost of also missing a few positives), which leads to much stronger performance if evaluated via F1 score under natural prevalence.
The performance of the proposed method is evaluated for existing SAEs from GemmaScope (for Gemma 2 2B), demonstrating competitive performance at a much lower cost for the existing evaluation methods, and superior performance if evaluated under natural prevalence. Further experiments aim at providing general insights into the patterns that elicit specific SAE features by evaluating the distribution of how many tokens are requred to elicit an SAE feature, showing that often a combination of multiple tokens is needed. Further, a case study presenting a selection of explicit insights gained from trained classifiers is also provided.
The last section then evaluates the classifier on existing SAEs trained on a protein language model instead of text. This evaluation therefore demonstrates that the framework is not limited to natural language, but now also offers the comparison of classifier annotations against real biological knowledge (and not just against the same activations they were fitted to). The authors show that the classifier is able to find known structural motifs.
Critical strengths
- In contrast to other sometimes very complex schemes, the proposed pipeline is extremely simple. This makes it broadly applicable without substantial effort and thus offers the possibility for wider adoption.
- The authors demonstrate the benefit of the pipeline very well, showing that an adaptable threshold is a benefit if a specific performance is desired for evaluations at natural prevalence.
- The proposed method is very cost effective, offering cheap and scalable training and inference.
- The different evaluations and case studies demonstrate the benefits and potential applications well.
- The figures were generally very insightful (especially Figure 3b which shows the distribution of requiring multiple tokens to activate).
Critical weaknesses
-
While the simplicity of the method is generally a large benefit, it slightly lowers the significance of the methodological contribution. Ultimately, the concept of using linear classifiers to explain neural network activations is very well known, so that this method simply applies a well-known paradigm to SAE features. The main contribution is therefore not the development of the general method, but rather 1) identifying the possibility of easily adapting the decision threshold to adapt classifier characteristics to feature distribution at evaluation and 2) the empirical evaluation and insights gained from using this method.
-
The evaluations in Section 3 compare the trained classifier against the existing scheme, which uses an LLM to determine the rule of the SAE activation from a selection of activating and non-activating texts. A problem is here that the classifier is explicitly adapted to the distribution of activating and non-activating samples in the evaluation: the decision threshold is explicitly chosen to maximize performance on the target distribution. The LLM-based methods, however, are not adapted in the same way, and thus face a structural disadvantage. This could have been mitigated by also introducing an adaptable threshold for the LLMs (for example, by letting them predict a score from 0 to 100 for each given sample and then determining the decision threshold in the same way to adapt it optimally to the target distribution). This is an issue both in the first evaluation that uses the original ratios of the benchmarks (since the classifier gets additional information that LLMs do not have) and in the second evaluation under natural prevalence (since the LLMs were not adapted at all to this setting which they were not designed for).
-
For SAEBench the evaluation reports only the originally published scores, for which the main text does not say with which LLM they were obtained, and scores produced by Haiku 4.5. Considering that the original study is from 2025, the model used there is likely far from state-of-the-art and so is Haiku 4.5. Further, we see from the other benchmark that a better model (Opus 5) leads to a clear improvement over the worse one. An evaluation with a more powerful model is therefore needed to evaluate whether the proposed method really produces state-of-the-art results on SAEBench.
-
Some parts of the paper are not particularly well written. Here are a few examples of where I stumbled on a first read, or where more info from the appendix was needed to understand things:
- Line 132 mentions the regularization path which was never introduced.
- At several points in the text it is claimed that the classifier is fitted at natural prevalence. But as far as I can see it from the appendix it is actually fitted on a more balanced sample and just the decision threshold is later adapted to perform optimally at natural prevalence which makes this statement rather misleading.
- Equation 3 shows a simplified F1 score approxmation that is intended to show how a reduced prevalence of a feature leads to a substantially reduced F1 score. The paper claims that this holds "In the limit that the positive class is rare", but this is only true if the prevalence of a class is much lower than the false positive rate. Only by stating the complete F1 formula and explaining the approximation does that become clear.
- The previous point is even more severe since it makes it impossible to understand why the proposed linear classifier can still achieve a good performance. The reason is that the false positive rate can be decreased substantially (at the cost of a lower true positive rate), which is one of the main benefits of the proposed method. Showing just the approximate formula makes gaining this insight impossible, and it should in general have been discussed extensively in the main text.
- The evaluation under natural prevalence introduces a different evaluation setup with slightly different data distribution and relevant activations being at specific locations. This is unexpected since the original F1 scores could have simply be re-weighted to yield estimates of the performance under natural prevalence, and since the current experiments still rely on this exact reweighting instead of actually evaluating under natural prevalence.
Questions for authors
Please feel free to treat any of the identified weaknesses as questions in case a clarification could resolve some of them.
2 comments
W3 SAEBench with Opus 5
We have added comparisons to Opus in Appendix C.2. The results are summarized below:
| SAE | Published | Haiku 4.5 | Opus 5 | Classifier |
|---|---|---|---|---|
| L19, 65k | 0.870 | 0.886 | 0.914 | 0.922 |
| L19, 16k | 0.846 | 0.854 | 0.889 | 0.892 |
| L12, 65k | 0.842 | 0.862 | 0.892 | 0.900 |
These scores are higher than Haiku 4.5 but slightly below the classifier.
W4 Writing.
Regularization path: we have removed the term from the main text, and the line now reads: "We minimize Equation 2 across a range of values, and score each solution by on a held-out tune split at natural prevalence."
Fitted at natural prevalence: we have reworded two sentences, "While the classifier's threshold is chosen on tokens at natural prevalence, the evaluations score excerpts drawn 1:1 in detection and 4:10 in SAEBench." and "The classifier, with its threshold chosen at natural prevalence, is also evaluated on the same held-out text."
Eq. 3: the condition is stated explicitly. The exact form is also available in Eq. 8 in Appendix B.
Why the classifier still scores well: this is the point of W2 above and of the new paragraph at the start of the discussion; for the LTF, tuning changes the annotation itself.
F1 reweighting: the at natural prevalence is hard to compute from benchmark scores because estimating an FPR at the scale of requires negatives (the relative error with negatives is ), which cannot be estimated from the number of negatives in the benchmarks (100 for detection and 10 for SAEBench). We use 2,000 negatives per feature, and 20,000 where needed (Appendix C.3). The reweighting is the Horvitz–Thompson estimator (Horvitz & Thompson, 1952), which is unbiased for a fixed FPR and TPR, and is the standard method for computing at a different prevalence rate.
We hope these additions address the points raised, and would appreciate your reconsidering the score. We are happy to clarify anything further during the discussion period.
References
- Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016.
- Daniel G. Horvitz and Donovan J. Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47(260):663–685, 1952.
- Gonçalo Paulo, Alex Mallen, Caden Juang, and Nora Belrose. Automatically interpreting millions of features in large language models. In Proceedings of the 42nd International Conference on Machine Learning, 2025.
- Adly Templeton et al. Scaling monosemanticity: Extracting interpretable features from Claude 3 Sonnet. Transformer Circuits Thread, 2024.
- Berk Ustun and Cynthia Rudin. Learning optimized risk scores. Journal of Machine Learning Research, 20(150):1–75, 2019.
Thank you for such a comprehensive and detailed read of our work, and we are glad to hear that you found the method simple and the evaluations insightful. We have added multiple experiments and changes to the text to address all the points raised, and we summarize them below:
W1: Contribution of the paper
We do agree that a major contribution of the paper is the evaluation framework and that the method has a long history in ML (Alain & Bengio, 2016; Ustun & Rudin, 2019).
We do believe that there's a notion of tractability of auto-interpretability that we did not frame clearly in our original work; specifically that descriptions themselves are judged as classifiers (in all the evaluations in autointerp pipelines), and for them to be accurate (given how rarely features activate under the sparsity constraint), they need to be both described and scored in a regime that obeys the class imbalance. The LTF has certain properties (convexity, tunability) that make it a natural fit for this regime.
W2: LLM descriptions were not given an adaptable threshold
We have added the suggested experiment in Appendix C.4 (Table 4) where we tune the descriptions first then evaluate them. The results are summarized below:
We find that adding a threshold does raise the F1 by ~1.5 times but is still lower than the classifier by 3–16 times.
The reason why the threshold does not close the gap is because descriptions cannot become more specific by tuning (for example, the Gate vs the Golden Gate example in Figure 1). This is a specific advantage of the method where the threshold directly corresponds to a more specific annotation (since more literals are required to cross the threshold).
Given the prevalence problem, annotation is essentially the act of classifying cases where a feature fires from cases where it doesn't, based on a limited number of samples. The LTF is one of the few examples where the prevalence can be explicitly accounted for after the fit, and where changing the threshold directly changes what the annotation is.
We have added a paragraph at the start of the discussion (Section 6) that addresses both W1 and W2, and we would like to thank you for helping us crystallize this point.