acceptodds
Under review as a conference paper at ICLR 2027

Executable annotations of SAE features

32 views

Abstract

Sparse autoencoder (SAE) features are used in practice through their descriptions. While features fire at a prevalence of , LLM descriptions are evaluated at 1:1. Rescored at natural prevalence, descriptions collapse to 0.01-0.05 and describe concepts that are 40-200 more prevalent than the feature. In this work, we show that annotating a feature is equivalent to learning a Boolean function over the one-hot encoding of a token and its context. We annotate every feature of three GemmaScope dictionaries by learning this function using a monotone linear threshold function (LTF) over (offset, token) literals, and evaluate it directly as a per-token detector on held-out text. These annotations outscore LLM descriptions on the field's own benchmarks, and score 5-24 higher at natural prevalence. Scored at this scale, the dictionary resolves into strata that balanced evaluation cannot distinguish, and the LTF weights expose multilingual concepts, boilerplate templates, and superposition directly. Applied without modification to a protein language model, the same method recovers established structural motifs such as zinc-finger spacing, Kelch repeats, and P-loop motifs from sequence-only representations. All 147k features across the three Gemma dictionaries are annotated and evaluated within 7 hours on a single quad-GH200 node, with no LLM in the pipeline. All annotations and code will be released publicly.

Primary area:
interpretability and explainable AI
Keywords:
sparse autoencoders, mechanistic interpretability, automated interpretability, feature annotations, superposition, evaluation, class imbalance, linear threshold functions, protein language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_YP2AOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This work aims at explaining features of trained sparse autoencoders for varying types of language models (both autoregressive and bidirectional). This is done by training a linear classifier with positive weights to predict a feature activation of an SAE feature solely from the one hot encoded input token matrix. In this way we gain insight into whether a specific token at a specific location elicits a specific SAE feature and we can also determine whether it alone suffices or whether a combination of multiple tokens is needed.

The central focus thereby is on both training and evaluating at "natural prevalence". Specifically, the authors identified a limitation of existing approaches, which often use an equal share of texts that elicit a specific SAE feature and texts that don't elicit it, and then use some method (like processing them with an LLM) to automatically determine the rule from these samples. The evaluation then usually works in the same way by using held out samples to check how often the identified rule can predict a feature activation, which is also usually done by using the same number of positive and negative samples (or a slightly skewed ratio). The problem with this set up is that a low false positive rate (e.g., 10%) will only have a minor influence on the evaluation score in this specific setting. But if the classifier is instead applied to a vast corpus of text without balancing positives and neagatives first, then the number of negatives will dominate (since most features appear only very rarely in the corpus), which drives up the absolute number of false positives in contrast to the true positives and leads to extremely low classifier performance.

The linear classifier proposed in this work can circumvent this issue by simply adapting the threshold required for a positive prediction (i.e., raising it so that a 90% confidence is needed instead of, for example, 50%). In this way the false positive rate can be lowered substantially (at the cost of also missing a few positives), which leads to much stronger performance if evaluated via F1 score under natural prevalence.

The performance of the proposed method is evaluated for existing SAEs from GemmaScope (for Gemma 2 2B), demonstrating competitive performance at a much lower cost for the existing evaluation methods, and superior performance if evaluated under natural prevalence. Further experiments aim at providing general insights into the patterns that elicit specific SAE features by evaluating the distribution of how many tokens are requred to elicit an SAE feature, showing that often a combination of multiple tokens is needed. Further, a case study presenting a selection of explicit insights gained from trained classifiers is also provided.

The last section then evaluates the classifier on existing SAEs trained on a protein language model instead of text. This evaluation therefore demonstrates that the framework is not limited to natural language, but now also offers the comparison of classifier annotations against real biological knowledge (and not just against the same activations they were fitted to). The authors show that the classifier is able to find known structural motifs.

Critical strengths
  1. In contrast to other sometimes very complex schemes, the proposed pipeline is extremely simple. This makes it broadly applicable without substantial effort and thus offers the possibility for wider adoption.
  2. The authors demonstrate the benefit of the pipeline very well, showing that an adaptable threshold is a benefit if a specific performance is desired for evaluations at natural prevalence.
  3. The proposed method is very cost effective, offering cheap and scalable training and inference.
  4. The different evaluations and case studies demonstrate the benefits and potential applications well.
  5. The figures were generally very insightful (especially Figure 3b which shows the distribution of requiring multiple tokens to activate).
Critical weaknesses
  1. While the simplicity of the method is generally a large benefit, it slightly lowers the significance of the methodological contribution. Ultimately, the concept of using linear classifiers to explain neural network activations is very well known, so that this method simply applies a well-known paradigm to SAE features. The main contribution is therefore not the development of the general method, but rather 1) identifying the possibility of easily adapting the decision threshold to adapt classifier characteristics to feature distribution at evaluation and 2) the empirical evaluation and insights gained from using this method.

  2. The evaluations in Section 3 compare the trained classifier against the existing scheme, which uses an LLM to determine the rule of the SAE activation from a selection of activating and non-activating texts. A problem is here that the classifier is explicitly adapted to the distribution of activating and non-activating samples in the evaluation: the decision threshold is explicitly chosen to maximize performance on the target distribution. The LLM-based methods, however, are not adapted in the same way, and thus face a structural disadvantage. This could have been mitigated by also introducing an adaptable threshold for the LLMs (for example, by letting them predict a score from 0 to 100 for each given sample and then determining the decision threshold in the same way to adapt it optimally to the target distribution). This is an issue both in the first evaluation that uses the original ratios of the benchmarks (since the classifier gets additional information that LLMs do not have) and in the second evaluation under natural prevalence (since the LLMs were not adapted at all to this setting which they were not designed for).

  3. For SAEBench the evaluation reports only the originally published scores, for which the main text does not say with which LLM they were obtained, and scores produced by Haiku 4.5. Considering that the original study is from 2025, the model used there is likely far from state-of-the-art and so is Haiku 4.5. Further, we see from the other benchmark that a better model (Opus 5) leads to a clear improvement over the worse one. An evaluation with a more powerful model is therefore needed to evaluate whether the proposed method really produces state-of-the-art results on SAEBench.

  4. Some parts of the paper are not particularly well written. Here are a few examples of where I stumbled on a first read, or where more info from the appendix was needed to understand things:

  • Line 132 mentions the regularization path which was never introduced.
  • At several points in the text it is claimed that the classifier is fitted at natural prevalence. But as far as I can see it from the appendix it is actually fitted on a more balanced sample and just the decision threshold is later adapted to perform optimally at natural prevalence which makes this statement rather misleading.
  • Equation 3 shows a simplified F1 score approxmation that is intended to show how a reduced prevalence of a feature leads to a substantially reduced F1 score. The paper claims that this holds "In the limit that the positive class is rare", but this is only true if the prevalence of a class is much lower than the false positive rate. Only by stating the complete F1 formula and explaining the approximation does that become clear.
  • The previous point is even more severe since it makes it impossible to understand why the proposed linear classifier can still achieve a good performance. The reason is that the false positive rate can be decreased substantially (at the cost of a lower true positive rate), which is one of the main benefits of the proposed method. Showing just the approximate formula makes gaining this insight impossible, and it should in general have been discussed extensively in the main text.
  • The evaluation under natural prevalence introduces a different evaluation setup with slightly different data distribution and relevant activations being at specific locations. This is unexpected since the original F1 scores could have simply be re-weighted to yield estimates of the performance under natural prevalence, and since the current experiments still rely on this exact reweighting instead of actually evaluating under natural prevalence.
Questions for authors

Please feel free to treat any of the identified weaknesses as questions in case a clarification could resolve some of them.

2 comments

Thank you for such a comprehensive and detailed read of our work, and we are glad to hear that you found the method simple and the evaluations insightful. We have added multiple experiments and changes to the text to address all the points raised, and we summarize them below:

W1: Contribution of the paper

We do agree that a major contribution of the paper is the evaluation framework and that the method has a long history in ML (Alain & Bengio, 2016; Ustun & Rudin, 2019).

We do believe that there's a notion of tractability of auto-interpretability that we did not frame clearly in our original work; specifically that descriptions themselves are judged as classifiers (in all the evaluations in autointerp pipelines), and for them to be accurate (given how rarely features activate under the sparsity constraint), they need to be both described and scored in a regime that obeys the class imbalance. The LTF has certain properties (convexity, tunability) that make it a natural fit for this regime.

W2: LLM descriptions were not given an adaptable threshold

We have added the suggested experiment in Appendix C.4 (Table 4) where we tune the descriptions first then evaluate them. The results are summarized below:

SAEDescription (Table 2)Description, tuned thresholdClassifier
L19, 65k0.00980.0150.24
L19, 16k0.0460.0670.21
L12, 65k0.0120.0190.19

We find that adding a threshold does raise the F1F_1 by ~1.5 times but is still lower than the classifier by 3–16 times.

The reason why the threshold does not close the gap is because descriptions cannot become more specific by tuning (for example, the Gate vs the Golden Gate example in Figure 1). This is a specific advantage of the method where the threshold directly corresponds to a more specific annotation (since more literals are required to cross the threshold).

Given the prevalence problem, annotation is essentially the act of classifying cases where a feature fires from cases where it doesn't, based on a limited number of samples. The LTF is one of the few examples where the prevalence can be explicitly accounted for after the fit, and where changing the threshold directly changes what the annotation is.

We have added a paragraph at the start of the discussion (Section 6) that addresses both W1 and W2, and we would like to thank you for helping us crystallize this point.

Annotating a feature means learning a Boolean function from examples. Doing so with high fidelity requires the examples to have the correct balance between positives and negatives, which for SAE features (that are already rare under sparsity penalty) means either having access to examples with 10410^4 negatives per positive, or correcting for the imbalance after the fit by tuning. LLM descriptions are hard to tune (descriptions cannot become more specific by tuning), and for tractability are evaluated on balanced accuracy. Symptoms of this problem have been noted by Templeton et al. (2024) (features may represent something more specific than their descriptions capture) and Paulo et al. (2025) (a random draw of negatives might not contain the hard cases that would reveal a feature's specificity).

A monotone LTF is a simple method that goes beyond token enrichment, can be run at scale, and can be tuned to give a more specific annotation that accounts for the class imbalance.

W3 SAEBench with Opus 5

We have added comparisons to Opus in Appendix C.2. The results are summarized below:

SAEPublishedHaiku 4.5Opus 5Classifier
L19, 65k0.8700.8860.9140.922
L19, 16k0.8460.8540.8890.892
L12, 65k0.8420.8620.8920.900

These scores are higher than Haiku 4.5 but slightly below the classifier.

W4 Writing.

Regularization path: we have removed the term from the main text, and the line now reads: "We minimize Equation 2 across a range of λ\lambda values, and score each solution by F1F_1 on a held-out tune split at natural prevalence."

Fitted at natural prevalence: we have reworded two sentences, "While the classifier's threshold is chosen on tokens at natural prevalence, the evaluations score excerpts drawn 1:1 in detection and 4:10 in SAEBench." and "The classifier, with its threshold chosen at natural prevalence, is also evaluated on the same held-out text."

Eq. 3: the condition FPR≫πi\mathrm{FPR} \gg \pi_i is stated explicitly. The exact form is also available in Eq. 8 in Appendix B.

Why the classifier still scores well: this is the point of W2 above and of the new paragraph at the start of the discussion; for the LTF, tuning changes the annotation itself.

F1 reweighting: the F1F_1 at natural prevalence is hard to compute from benchmark scores because estimating an FPR at the scale of πi\pi_i requires ∼1/πi\sim 1/\pi_i negatives (the relative error with nn negatives is ∼1/nFPR\sim 1/\sqrt{n \mathrm{FPR}}), which cannot be estimated from the number of negatives in the benchmarks (100 for detection and 10 for SAEBench). We use 2,000 negatives per feature, and 20,000 where needed (Appendix C.3). The reweighting is the Horvitz–Thompson estimator (Horvitz & Thompson, 1952), which is unbiased for a fixed FPR and TPR, and is the standard method for computing F1F_1 at a different prevalence rate.

We hope these additions address the points raised, and would appreciate your reconsidering the score. We are happy to clarify anything further during the discussion period.

References

  • Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016.
  • Daniel G. Horvitz and Donovan J. Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47(260):663–685, 1952.
  • Gonçalo Paulo, Alex Mallen, Caden Juang, and Nora Belrose. Automatically interpreting millions of features in large language models. In Proceedings of the 42nd International Conference on Machine Learning, 2025.
  • Adly Templeton et al. Scaling monosemanticity: Extracting interpretable features from Claude 3 Sonnet. Transformer Circuits Thread, 2024.
  • Berk Ustun and Cynthia Rudin. Learning optimized risk scores. Journal of Machine Learning Research, 20(150):1–75, 2019.
Reviewer_euw3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for generating SAE features using linear threshold functions. The authors identify limitations in existing LLM-generated feature descriptions, particularly their poor accuracy under feature prevalence. By modeling feature activations as Boolean functions over tokens and their contexts, the proposed method provides interpretability. Experiments demonstrate improved accuracy over LLM-based, while applications to protein language models show the ability to recover known biological motifs.

Critical strengths
  • Well-motivated. The paper identifies an important limitation of existing SAE feature, particularly their poor performance at feature.

  • Well-written. The paper is clearly written and well organized, with intuitive examples that effectively explain the motivation, methodology, and key findings.

Critical weaknesses
  • Limited expressivity of the model. The proposed method relies solely on monotone linear threshold functions to approximate SAE feature activations. However, such a simple model may struggle to capture more complex activation patterns. Comparisons with more expressive model structures are needed to justify this design choice and investigate the trade-off between accuracy and interpretability.

  • Limited validation on protein language task. Although the proposed method successfully recovers several known protein motifs, the evaluation mainly relies on qualitative case studies. More systematic experiments are needed to demonstrate the generality and biological validity of the resulting annotations.

Questions for authors
  • Have the authors considered comparing the proposed linear threshold functions with more expressive model structures? How would these alternatives affect the trade-off between annotation accuracy and interpretability?
  • Could the authors provide more systematic quantitative evaluations against known protein motifs or biological task to demonstrate the generality and biological validity of the proposed method?
3 comments

Thank you for reviewing our work. We are glad that you found the work well-motivated and well-written. Our comments are given below.

W2/Q2: Protein validation

We have added two new sections for validation.

In Appendix H.3 (Table 13), we compare the classifier to descriptions written by Claude Haiku 4.5 for 100 random InterPLM features. The results are summarized below:

TPRFPRF1F_1
LLM description0.522.7×10−12.7\times10^{-1}0.0069
Classifier0.161.9×10−31.9\times10^{-3}0.11

Similar to text, we find the classifier to be ~16 times better compared to LLM descriptions.

In Appendix H.2 (Table 12), we have also shown the features that match to Swiss-Prot concepts (the criteria chosen is features with F1F_1 > 0.5 where at least 50% of the activation falls within the region of Swiss-Prot annotation). The table is shown below:

Swiss-Prot conceptAccessionFeatures
Zinc finger: C2H2-typeInterPro IPR01308729
Repeat: ANKInterPro IPR00211024
Repeat: LRRInterPro IPR00161127
Repeat: PPRInterPro IPR00288520
Repeat: WDInterPro IPR0016809
Binding site: ATPPROSITE PS0001724
Transmembrane–15
Active site: Proton acceptorInterPro IPR00827112

W1/Q1: More expressive model classes

We actively chose to restrict the paper to a single method so that we could (1) explore and highlight the shortcomings of standard evaluations, and because (2) the method is cheap and has nice properties (convexity, tunability, monotonicity) and can be fit to scale fairly quickly. In the revised draft we also added a new section (4.3) showing how the method can be used to grep for features, perform attribution, and detect spurious correlations (Bias in Bios; Marks et al., 2025a).

In terms of richer classes, we do believe that picking a better choice of literals while keeping the same method is particularly promising, given the protein results where adding Swiss-Prot concepts increased the median F1F_1 from 0.11 to 0.21. Other promising methods also include short decision lists and k-DNF rule sets (Valiant, 1984; Rivest, 1987) and thresholds over low-degree conjunctions of literals (Klivans & Servedio, 2004); all methods should be scored the same way, at natural prevalence. These are now explicitly mentioned in the manuscript (Section 6), and the line reads: "The feature is a Boolean function of its window, so the gap is closable in principle, and richer executable classes, such as decision lists and kk-DNF (Valiant, 1984; Rivest, 1987) or thresholds over conjunctions of literals (Klivans & Servedio, 2004), inherit the same evaluation."

We are happy to clarify any of these points, and are open to adding more changes if required.

References

  • Adam R. Klivans and Rocco A. Servedio. Learning DNF in time 2O~(n1/3)2^{\tilde{O}(n^{1/3})}. Journal of Computer and System Sciences, 68(2):303–318, 2004.
  • Samuel Marks et al. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. In International Conference on Learning Representations, 2025a.
  • Ronald L. Rivest. Learning decision lists. Machine Learning, 2(3):229–246, 1987.
  • Leslie G. Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134–1142, 1984.

Thank you for the rebuttal and clarifications. I have two questions:

1. Comparability of the methods in Table 1. The proposed LTF is trained with explicitly labeled SAE activations, whereas the Dual-LLM approach relies on in-context examples without supervised parameter training. Why are these two fundamentally different learning paradigms directly compared, and how should we interpret the performance gap?

2. Attribution of performance gains. Without additional model comparisons under comparable supervision settings, the current experiments cannot separate the contribution of the LTF architecture itself from that of its access to more extensive and direct supervision.

Thank you for the follow-up questions. Our responses are given below.

1. Comparability of the methods in Table 1

We do agree that the two methods are learned in different ways. The reason why we compare them together is that the field evaluates descriptions (Paulo et al., 2025; Karvonen et al., 2025) by how well the description of a feature can separate instances where the feature is present from where it is not (inherently a classification problem), which we discuss in the Introduction and Sections 2 and 3.1. Following another reviewer's suggestions, we also added a paragraph at the start of the discussion (Section 6) to clarify this point:

Annotating a feature means learning a Boolean function from examples. Doing so with high fidelity requires the examples to have the correct balance between positives and negatives, which for SAE features (that are already rare under sparsity penalty) means either having access to examples with 10410^4 negatives per positive, or correcting for the imbalance after the fit by tuning. LLM descriptions are hard to tune (descriptions cannot become more specific by tuning), and for tractability are evaluated on balanced accuracy.

We included Table 1 to first compare the two methods on the evaluations as they are used in the field, before evaluating at natural prevalence. We do not put much weight on the gap there (0.02–0.07), since these evaluations are balanced; the main comparison is at natural prevalence (Table 2), which we discuss below.

2. Attribution of performance gains

We chose to focus on descriptions since they are the predominant method for understanding what SAE features detect (Lin, 2023; Karvonen et al., 2025), with limited alternatives.

We do agree that the two methods differ in the amount of supervision, and based on other reviewers' comments we did add multiple experiments in the revised draft, where we tried to put LLM descriptions on an equal footing with the classifier. The results (F1F_1 at natural prevalence) are summarized below:

ExperimentWhat it equalizesDescriptionClassifier
Main result (Table 2)–0.0098 / 0.046 / 0.0120.24 / 0.21 / 0.19
Description from 500 + 500 examples (Appendix C.5)supervision0.019 / 0.072 / 0.0160.24 / 0.21 / 0.19
Description with a tuned threshold (Appendix C.4)tuning to prevalence0.015 / 0.067 / 0.0190.24 / 0.21 / 0.19
Classifier on the judge's window (Appendix C.3)context0.0098 / 0.046 / 0.0120.26 / 0.23 / 0.21
Classifier at description length (Appendix C.6)annotation length0.0098 / 0.046 / 0.0120.208 / 0.195 / 0.178

(values are for the L19 65k / L19 16k / L12 65k SAEs)

Specifically, the descriptions written from 500 positives and 500 negatives (compared to 40 for detection and 15 for SAEBench) are a stronger version of the description baseline; going much further is limited by what an explainer can read in context. We find that this stronger baseline raises the F1F_1 by 1.3–1.9 times, while the classifier still scores 3–13 times higher.

In regards to other model classes, we chose the monotone LTF since it is the simplest class that is readable, can be fit at scale, and can be tuned to account for the class imbalance (Section 6). As per your earlier comment, Section 6 explicitly mentions richer classes (decision lists, kk-DNF, thresholds over conjunctions of literals), all of which inherit the same evaluation.

We are happy to further clarify any of these points during the remaining discussion period.

References

  • Adam Karvonen et al. SAEBench: A comprehensive benchmark for sparse autoencoders in language model interpretability. In Proceedings of the 42nd International Conference on Machine Learning, 2025.
  • Johnny Lin. Neuronpedia: Interactive reference and tooling for analyzing neural networks. 2023. https://www.neuronpedia.org
  • Gonçalo Paulo, Alex Mallen, Caden Juang, and Nora Belrose. Automatically interpreting millions of features in large language models. In Proceedings of the 42nd International Conference on Machine Learning, 2025.
Reviewer_cLg7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors show that LLM descriptions of SAE features are usually evaluated at a 1 to 1 ratio, although in reality features fire much more rarely. If descriptions are evaluated at the real firing rate their F1 falls dramatically. Instead of a LLM description, the authors propose to train for each feature a simple sparse linear classifier over tokens from the context. This classifier works better than descriptions on standard benchmarks and gets a 5–24 times higher F1 at the real firing rate. With this method, the authors annotate 147 thousand GemmaScope features and also apply it to a protein model, where they find known motifs.

Critical strengths

The authors raise a very important problem: current metrics almost do not punish descriptions of features that are too broad and the paper shows this well with concrete numbers. The method itself is simple, fast and works without an LLM, and the experiments are described very carefully. The results on proteins also look interesting.

Critical weaknesses

The main problem is that it is not clear why such a method is needed and what to do with it next. A feature already has an exact executable form (namely the SAE itself together with the model) so simply predicting the firings of a feature is not very useful. At the same time, the classifier catches only about a quarter of the firings, and the median annotation has several hundred literals, which are hard for a human to read. The paper has no single practical task where these annotations help. It seems to me that the method could be much more useful as a tool for analyzing what features SAEs learn at all. For example, it could be used to check whether many features are just trivial detectors of tokens and short phrases, and to look at how the features that such a classifier cannot describe are different. But the paper almost does not explore this question, and in its current form the work raises the problem well, but does not show what to do with the proposed solution.

I am open to discussion. If the authors can show that I have misunderstood something or that my concerns are wrong, I will be happy to respond and reconsider my evaluation.

Questions for authors

Is there a concrete interpretability task, such as finding the feature for a given concept or predicting the effect of steering a feature, where using your annotations gives better results than using LLM descriptions or the activations of the feature directly? And what does your method say about what features SAEs learn, and how trivial are these features?

4 comments

Thank you so much for the review, and for the suggestion to add concrete tasks to show utilities of the method. We have added a new section, 4.3 (and Appendix G) where we highlight three utilities, inspired by your comments:

Bias in Bios (Section 4.3 and Appendix G.3)

We ran the Bias in Bios setup, in which we trained a linear probe on the three SAEs considered in the work to classify biographies of male professors from female nurses, and then evaluated the probe on balanced profession accuracy and gender accuracy. The trained probe has a profession accuracy of 0.77–0.82 and gender accuracy of 0.68–0.73.

Next, we grepped for features where the literals in the annotation corresponded to the 11 gender indicators of De-Arteaga et al. (2019) (he, she, her, his, him, hers, himself, herself, mr, mrs, ms) and had weights > 1 (i.e. the literals cleared the threshold by themselves). This gave us $\sim$90 features, and we zeroed out those features in the linear probe and recomputed the metrics. Those are given below:

SAEFeatures zeroedProfession acc.Gender acc.
L19, 65k940.812 → 0.9390.686 → 0.532
L19, 16k860.821 → 0.9420.677 → 0.524
L12, 65k950.770 → 0.8720.726 → 0.547

Zeroing out the same number of random features did not change the metrics. The same 11 token query over the Neuronpedia descriptions (layer-19, 16k) returned 10 features, removal of which left the probe unchanged. All results are in Appendix G.3 (Table 10).

Beyond this experiment, we also showed other utilities, one on searching for features that correspond to tokens (single or multiple), or phrases. For example, searching for the word "charge" gives us features corresponding to three different meanings (payment, responsibility and allegation) (Appendix G.2). Finally, we added examples of how the annotations can be used to understand how features capture the context of tokens in sentences (Appendix G.1).

Using the method to study what SAEs learn

We had already included an analysis of the dictionary as enabled by the annotations in Section 4.1, and case studies of feature annotations in 4.2. We have added a line at the start of Section 4.1 to clarify this point: "We turn to characterizing what SAE features capture."

Point on annotation and SAEs are already executable

The annotations through the LTF provide a simple way of understanding why features activate in a natural basis (the input tokens), without rerouting the text through the LLM and the SAE, where this is hard (and is a goal of interpretability). Descriptions are used for the same reason.

Quarter of firings caught

While this is true, the threshold of the classifier can always be tuned to provide a higher recall (and doing so provides a different annotation); this is hard to do with descriptions. It should also be possible to fit the false negatives with a second LTF, as is standard in boosting (Freund & Schapire, 1997) and separate-and-conquer rule learning (Fürnkranz, 1999).

Number of literals

We have added results in Appendix C.6 where we sweep over the number of literals (by tune-set usage) and compute the test-set F1F_1 (Table 6, Figure 7). The results are summarized below:

SAEDescription1 literal51020At description lengthFull
L19, 65k0.00980.0380.1290.1710.1820.2080.238
L19, 16k0.0460.0520.1110.1450.1820.1950.213
L12, 65k0.0120.0480.1070.1300.1520.1780.192

We find that at a comparable number of literals to the description (each word taken as a literal), the F1F_1 is 4–21 times higher.

We believe the new Section 4.3 along with all other additions directly address the concerns raised, and given your openness to reconsidering, we would greatly appreciate an updated assessment. We are open to any other clarifications during the discussion period if anything remains unclear.

References

  • Maria De-Arteaga et al. Bias in Bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 120–128, 2019.
  • Yoav Freund and Robert E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1):119–139, 1997.
  • Johannes Fürnkranz. Separate-and-conquer rule learning. Artificial Intelligence Review, 13(1):3–54, 1999.
Area_Chair_aFDiOpenReview ↗

Reminder to engage in reviewer-author discussion

Hi cLg7,

As there are only two days left in the author-reviewer discussion period, I would like to remind you to kindly respond to the authors' rebuttal, posted at the end of last week. After November 18, reviewer responses are not be allowed and you will no longer be able to ask any follow-up questions. If you have no follow-ups, a short and polite response acknowledging you have read and understood the rebuttal and have no further questions is sufficient.

Thanks, AC

Thank you for the detailed response and the additional experiments! The Bias in Bios experiment addresses my main concern about what these annotations can be used for. The results with fewer literals also make the concern about annotation length less significant.

I still think the findings are interesting, but the contribution is somewhat limited. The new experiment shows a useful application, although the comparison with LLM descriptions only uses keyword matching, so it is hard to conclude how much better the annotations are for feature selection more generally. Still, the authors have addressed a substantial part of my concerns, and I am raising my score to 3.

Thank you for the response, and for raising the score. We are glad that the Bias in Bios experiment and the results with fewer literals addressed your concerns.

We noticed that the rating on the original review still shows 2, and we would appreciate it if you could update it there as well.

Reviewer_R3cpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles a valid concern with descriptions of features, which focus on the top activating examples, but fail to capture the lowest activating examples, what the paper describes as being evaluated '1:1'. To address the issue with natural language explanations, the authors propose using a linear threshold function to anotate features as boolean functions. The authors claim that this technique has leads to better explanations, or explanations that score higher in benchmarks, and that they are better even in the low activating contexts. They apply the same technique for protein language models.

Critical strengths

It presents an interesting alternative to natural language explanations of features.

Critical weaknesses

The authors show 2 scores, detection and SAEbench autointerp, but they are very similar so they are not really representative showing of how good the proposed explanations are. These two scores are also very simple and one would hope that good explanations would be good at simulation scoring (from "Language models can explain neurons in language models").

It is unclear how interpretable are the LFTs? The point of these interpretations is that humas should be able to read them. Are we comparing descriptions of the same 'lenght'? It seems possible that one could generate much longer textual descriptions that would score higher, but most people would prefer short sentence explanations.

The thresholds of the classifier are trained on the same neurons they are evaluated, even if on different contexts. The LLM that is generating explanations was not finetuned and the classifier has a overall knowledge avantage over the explainer. The fact that the treshold is tuned depending on the evaluation set also makes it easier for the classifier to have better scores.

The authors present as a result the fact that natural language explanations have high recal but low precision and that in the real world we would need high precision because features fire rarely. It is unclear to me if this is not something that is expected by everyone or is a real contribution.

Questions for authors

Why do you center windows around tokens, instead of showing only previous tokens? As you mention in 2.1, the activation of the feature is only dependent on the previous tokens.

Pi and Ni are mentioned after equation 2 but they don't show up. What are they?

Are the features you find in PLMs something that LLM explanations would not find?

6 comments

Thank you for the questions. We have added new results to the manuscript to address the issues raised, and clarified any points that were unclear in the manuscript.

Summary and 1:1 evaluation

We wanted to clarify that by 1:1, we meant that descriptions are evaluated on balanced accuracy, i.e. an equal number of activating and non-activating examples. It does not refer to low activating examples.

We have changed the abstract to clarify this point: "While features fire at a prevalence of 10−410^{-4}, LLM descriptions are evaluated on balanced accuracy."

Threshold tuned per feature and knowledge advantage

We have added multiple experiments to address this point. This includes Appendix C.4 (Table 4) where we threshold descriptions with a score between 0-100 (as per reviewer YP2A's comments), as well as descriptions written with 500 positive and 500 negative samples (compared to 40 for detection, 15 for SAEBench and 5 for simulation; Appendix C.5, Table 5). The results are shown below:

SAEDescription (Table 2)Description, 500 + 500 examplesDescription, tuned thresholdClassifier
L19, 65k0.00980.0190.0150.24
L19, 16k0.0460.0720.0670.21
L12, 65k0.0120.0160.0190.19

We also recomputed the F1F_1 for LTF where we restrict the window to −16≤r≤16-16 \le r \le 16 (Appendix C.3, Table 3) and the results are shown below.

SAEFull window−16≤r≤16-16 \le r \le 16
L19, 65k0.240.26
L19, 16k0.210.23
L12, 65k0.190.21

Finally, for all experiments the threshold of the classifier is chosen on a held-out tune split, and not on the evaluation set.

How interpretable are the LTFs

Monotone linear threshold functions are among the oldest models that are used for interpretability and read directly, with examples including clinical point scores such as CHADS2 (Gage et al., 2001) and their learned counterparts (Ustun & Rudin, 2016; 2019), position weight matrices for sequence motifs (Stormo et al., 1982; Stormo, 2000), and sparse linear and rule-based explanations of neural networks (Ribeiro et al., 2016; 2018). We have shown examples of annotations in Section 4.2.

Annotation description length

For the description length, we added new results in Appendix C.6 (Table 6, Figure 7), where we consider the F1F_1 vs the number of literals. For the same number of words as the descriptions (each word taken as a literal), the F1F_1 of the two methods is given below:

SAEDescriptionClassifier at description length
L19, 65k0.00980.208
L19, 16k0.0460.195
L12, 65k0.0120.178

We find that at a comparable number of literals to the description, the F1F_1 is 4–21 times higher.

Is low precision at natural prevalence already expected

The concern has been noted qualitatively (Templeton et al., 2024; Paulo et al., 2025), but the standard benchmarks still score descriptions at fixed ratios of 1:1 (Paulo et al., 2025) and 4:10 (Karvonen et al., 2025) positive to negative ratio. Even for simulation (Bills et al., 2023), the correlation is computed on 5 top-activating and 5 random excerpts. To our knowledge, Table 2 is the first measurement of SAE annotations at natural prevalence, where we find that F1F_1 falls to 0.01–0.05 and descriptions cover concepts 40–200× more prevalent than the feature.

Detection, SAEBench and Simulation scoring

Since the paper is concerned with class imbalance rather than relative activations, we evaluate whether a description can predict whether features activate or not, which is what detection and SAEBench measure (Table 1). The main evaluation beyond these two methods is detection at natural prevalence (Table 2). Since simulation is evaluated on 5 top-activating and 5 random excerpts, it has the same imbalance problem as the other methods.

As for the fidelity of descriptions, we included the score of descriptions written by an explainer that looks at 500 positives and 500 negatives (compared to 5 for simulation; Appendix C.5, Table 5), where we found that classifier still scores 3–13 times higher.

Why center windows instead of using previous tokens only

We have added results in Appendix C.3 (Table 3) where we restrict the classifier to only previous tokens:

SAEFull windowPast tokens only (r≤0r \le 0)
L19, 65k0.240.21
L19, 16k0.210.18
L12, 65k0.190.19

P and N after Equation 2

We have clarified this line in the manuscript, which now reads "Here, Pi\mathcal{P}_i and Ni\mathcal{N}_i are the sets of positive and negative samples in the training set, respectively. ci=∣Ni∣/∣Pi∣c_i = |\mathcal{N}_i| / |\mathcal{P}_i| is the class reweighting factor, where ∣⋅∣|\cdot| denotes the size of a set."

Comparison to descriptions for protein

We have added a new comparison in Appendix H.3 (Table 13), summarized below:

TPRFPRF1F_1
LLM description0.522.7×10−12.7\times10^{-1}0.0069
Classifier0.161.9×10−31.9\times10^{-3}0.11

We find that the performance by description is worse compared to text, and the classifier scores ~16 times higher.

We hope these experiments and clarifications address the weaknesses raised in your review, and that you will reconsider the score. We are happy to clarify anything further during the discussion period.

References

  • Steven Bills et al. Language models can explain neurons in language models. OpenAI, 2023.
  • Brian F. Gage et al. Validation of clinical classification schemes for predicting stroke: results from the National Registry of Atrial Fibrillation. JAMA, 285(22):2864–2870, 2001.
  • Adam Karvonen et al. SAEBench: A comprehensive benchmark for sparse autoencoders in language model interpretability. In Proceedings of the 42nd International Conference on Machine Learning, 2025.
  • Gonçalo Paulo, Alex Mallen, Caden Juang, and Nora Belrose. Automatically interpreting millions of features in large language models. In Proceedings of the 42nd International Conference on Machine Learning, 2025.
  • Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should I trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144, 2016.
  • Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: High-precision model-agnostic explanations. In Proceedings of the AAAI Conference on Artificial Intelligence, 32(1):1527–1535, 2018.
  • Gary D. Stormo, Thomas D. Schneider, Larry Gold, and Andrzej Ehrenfeucht. Use of the 'Perceptron' algorithm to distinguish translational initiation sites in E. coli. Nucleic Acids Research, 10(9):2997–3011, 1982.
  • Gary D. Stormo. DNA binding sites: representation and discovery. Bioinformatics, 16(1):16–23, 2000.
  • Adly Templeton et al. Scaling monosemanticity: Extracting interpretable features from Claude 3 Sonnet. Transformer Circuits Thread, 2024.
  • Berk Ustun and Cynthia Rudin. Supersparse linear integer models for optimized medical scoring systems. Machine Learning, 102(3):349–391, 2016.
  • Berk Ustun and Cynthia Rudin. Learning optimized risk scores. Journal of Machine Learning Research, 20(150):1–75, 2019.
Area_Chair_aFDiOpenReview ↗

Reminder to engage in reviewer-author discussion

Hi R3cp,

As there are only two days left in the author-reviewer discussion period, I would like to remind you to kindly respond to the authors' rebuttal, posted at the end of last week. After November 18, reviewer responses are not be allowed and you will no longer be able to ask any follow-up questions. If you have no follow-ups, a short and polite response acknowledging you have read and understood the rebuttal and have no further questions is sufficient.

Thanks, AC

Comments on the paper

7 comments
AuthorsOpenReview ↗

Summary of changes

We thank all the reviewers for the detailed and thoughtful comments. We have made substantial changes to address the points made by the reviewers, and believe that their comments have made the paper stronger. The changes to the manuscript (marked in blue) are summarized below, and we have responded to all the reviewers individually.

AddedWhereReviewer
Utilities, including Bias in BiosSection 4.3, Appendix G (Table 10)cLg7
Opening paragraph of the discussion; richer classesSection 6YP2A, euw3
Classifier with a restricted windowAppendix C.3 (Table 3)Lj39, R3cp
Descriptions with a tuned thresholdAppendix C.4 (Table 4)YP2A, R3cp
Descriptions from 500 + 500 examplesAppendix C.5 (Table 5)Lj39, R3cp
Classifiers with fewer literalsAppendix C.6 (Table 6, Figure 7)cLg7, R3cp
SAEBench with Opus 5Appendix C.2YP2A
Label threshold 0 vs 0.15Appendix A.0Lj39
Conjunctive share against F1F_1, with a direct test on GemmaAppendix F (Table 9)Lj39
Protein motifs and comparison to descriptionsAppendix H.2, H.3 (Tables 12, 13)euw3, R3cp
Abstract and wording fixesAbstract, Sections 2.2, 3.1, 3.2, 4.1, 5R3cp, YP2A, cLg7

We thank you for the questions. We have added new results to the manuscript to address the issues raised.

Q1/W1. Advantage under the same context as the LLM judge

We have added results for the classifier for two cases: one where tokens are restricted to the window −16≤r≤16-16 \le r \le 16 (same as descriptions), and one where only past tokens are used. These results are added in Appendix C.3 Table 3, and the F1F_1 is summarized below:

SAEFull windowPast tokens only (r≤0r \le 0)−16≤r≤16-16 \le r \le 16
L19, 65k0.240.210.26
L19, 16k0.210.180.23
L12, 65k0.190.190.21

In both cases the F1F_1 score stays within ~0.03 points of the classifier with the full window, and is still considerably higher than descriptions.

We have also compared to another run where we added 500 positives and 500 negatives for descriptions (compared to 40 for detection and 15 for SAEBench), and the results are given below (Appendix C.5, Table 5):

SAEDescription (Table 2)Description, 500 + 500 examplesDescription, tuned thresholdClassifier
L19, 65k0.00980.0190.0150.24
L19, 16k0.0460.0720.0670.21
L12, 65k0.0120.0160.0190.19

While the FPR decreases with added examples, the classifier still scores 3–13 times above the description.

As per another reviewer's comments, we have also added experiments where LLM descriptions are tuned with a threshold, and then scored (Appendix C.4, Table 4). We found that tuning only increased the score by ~1.5 times, still considerably below the score of the classifier.

Q2/W2. Threshold at >0 vs >0.15

We added the classifier scores for the two thresholds in Appendix A.0 (scored for the 300 features in Table 2). The results are summarized below:

SAELabel threshold 0Label threshold 0.15
L19, 65k0.240.32
L19, 16k0.210.29
L12, 65k0.190.22

In regards to the point that receiving an annotation is not enough to tell whether a feature's behavior has been adequately described, we would argue that for a fixed threshold, the F1F_1 score on held-out text at natural prevalence is one of the strongest metrics one can write down. All the annotations in this work are certified with that metric.

Q3/W3. Conjunctive features

This is an important point, and we thank the reviewer for pointing this out.

We have added the k∗k^* vs F1F_1 in Appendix F (Table 9), and we indeed do find that features with high F1F_1 can activate on a single token. At the same time, the classifier we choose is naturally equipped to capture single literals or tokens, vs capturing phrases / combination of different phrases. As such, the cases where the LTF scores lower correspond to cases that are phrasal, and we check this directly by checking whether a feature fires at all, if all possible tokens in vocabulary are added after <bos> in Gemma. The results are added in the same Table, and are shown below for the layer-19 65k SAE:

Annotation F1F_1Featuresk∗≥2k^* \ge 2Need more than one token (direct test)
all65,13977%62%
> 0.152,49074%58%
> 0.2532,14764%52%
> 0.516,54746%37%
> 0.758,38126%20%

We find that 62% of all features in this SAE require more than 1 token when directly checked with Gemma, and this percentage tracks the trend of the classifier when considering features resolved at different fidelity.

We hope these additions have made the paper stronger, and would appreciate your reconsidering the score. We are happy to clarify anything further during the discussion period.

Area_Chair_aFDiOpenReview ↗

Reminder to engage in reviewer-author discussion

Hi Lj39,

As there are only two days left in the author-reviewer discussion period, I would like to remind you to kindly respond to the authors' rebuttal, posted at the end of last week. After November 18, reviewer responses are not be allowed and you will no longer be able to ask any follow-up questions. If you have no follow-ups, a short and polite response acknowledging you have read and understood the rebuttal and have no further questions is sufficient.

Thanks, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.