- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the autointerpretability paradigm used to evaluate Sparse Autoencoder (SAE) interpretability, a framework where one language model generates a natural language description for an activated SAE feature, and a second model evaluates how accurately that description predicts feature activations on held-out data. The authors conduct systematic experiments across two language models (Pythia-160M and Apertus-8B-Instruct), testing four metrics across six axes of methodological variation. Rather than assessing construct validity (whether these metrics truly measure interpretability), the study focuses on statistical reliability across pipeline configurations.
Critical strengths
- Addresses an important and timely challenge in Sparse Autoencoder (SAE) interpretability by systematically analyzing the statistical reliability of standard evaluation pipelines. Since mechanistic interpretability relies heavily on SAE autointerpretation pipelines to benchmark new architectures and dictionary learning techniques, revealing that pipeline hyperparameters introduce noise that swamps architectural variance provides an intresting point of view.
- Evaluates metrics across multiple benchmark corpora, including The Pile, C4, and Wikipedia.
- The experimental framework is structured around formal statistical metrics: Intraclass Correlation Coefficients (ICC) for measurement reliability, Top-k Jaccard Lift for ranking reproducibility, and variance decomposition. The study evaluates two distinct language models across four interpretability metrics, six pipeline parameters, and five SAE architectures.
Critical weaknesses
-
The activation corpus accounts for a substantial share of score variance. However, feature behavior on distinct corpora reflects true semantic variation rather than pure pipeline measurement noise. The paper treats the choice of activation corpus as a source of evaluation pipeline instability. While corpus variation accounts for a substantial share of total score variance, a latent feature's behavior on distinct datasets reflects true, meaningful semantic variation across domains rather than measurement noise in the autointerpretability pipeline. By grouping legitimate domain-shift variance (CORPUS) under "pipeline unreliability", the authors artificially inflate the apparent instability of autointerpretability metrics.
-
Furthermore, by focusing strictly on the statistical reliability of a metric rather than its actual validity within specific pipeline configurations, the authors limit the broader impact of their work.
-
Crucial context regarding prior SAE evaluation practices is relegated to Appendix A. Without a clear grounding in prior work within the main text, it is difficult to evaluate the paper's novel contributions, positioning, or alignment with standard evaluation pipelines. Furthermore, the four evaluated metrics (e.g., SIMULATION) lack definitions and theoretical justifications in the main text. The paper does not include comparisons from prior literature, making it difficult to contextualize the magnitude and significance of the observed pipeline variations.
-
The broad conclusion that cross-paper autointerpretability score comparisons reflect pipeline variations rather than model architecture differences is derived from only two models (Pythia-160M and Apertus-8B-Instruct). To substantiate these claims, the study requires broader experimentation on widely adopted SAE setups (e.g., Gemma Scope) and direct comparison with existing literature.
Questions for authors
-
What types of latent features are being evaluated, and do they align with feature types commonly analyzed in the SAE literature? Is there baseline probing evidence for how these features manifest across the target corpora?
-
What is the rationale behind selecting Pythia-160M and Apertus-8B-Instruct? The authors claim their work "is the first work to train and evaluate SAEs on APERTUS-8B-INSTRUCT", but what is the added value of this choice? Given the focus on general metric reliability, would not it be more insightful to evaluate standard, widely benchmarked models across the SAE ecosystem (e.g., Gemma-based or Llama-based SAEs)?
-
How did the two-order-of-magnitude difference in model scale (160M vs. 8B parameters) impact your evaluation methodology, hyperparameter configurations, and the observed metric reliability?
6 comments
Thanks for the response, but this does not address my main concern on the practical contribution of this work in its current version. While appreciating the authors' efforts, the current version resembles more a position paper rather than providing a concrete methodological or experimental contribution in the field. Additionally, the authors use past tenses to report changes, yet the current version of their manuscript does not include the reported changes.
-
W1: The fact that "studies that use different corpora measure different things" is a trivial consideration, and everyone would agree ideally that "the corpus must therefore be fixed and reported". While the authors did show that the evaluation corpus has an impact on the evaluation pipeline of SAEs, "presenting it as an evaluation condition" represents a marginal and expected finding.
-
W2. Your scope was already cleared to my mind. My concern is that this is quite limited. A comprehensive evaluation should take into consideration both the reliability and validity of a metrics.
-
W3: I appreciate you are exploiting the additional page to include crucial related work. However, the authors should have taken into account its inclusion beforehand, using the original 9-page submission.
-
W4: I appreciate the explanation and the planned addition. I recommend the authors to include the underlying rational for model choice in the revised version.
-
Q1: Thanks for the further analysis. I would consider include it and its experimental findings within the main text.
-
Q2/Q3: Thanks for the clarifications
Thank you for the fast reply! We just uploaded the new version which includes all the changes.
We thank the reviewer for the feedback and address the weaknesses and questions below.
W1. We agree that part of the corpus effect is real, as features behave differently across domains. App. Figure 6 shows that the number of eligible features differs strongly per corpus, and on Pythia-160M mean simulation drops from 0.52 on the Pile to 0.35 on code (App. Table 14). Our point is about comparability rather than the source of the variance, as a score describes a feature on a particular corpus. Studies that use different corpora measure different things, whether this is noise or domain shift. Prior work found that the training corpus has little effect on scores (Paulo et al., 2025) and we show that the evaluation corpus does. The corpus must therefore be fixed and reported, and we now present it as an evaluation condition. Our conclusions do not depend on it, because without the corpus, a single methodological factor still explains more variance than architecture in five of eight model–metric pairs (App. Tables 22–23).
W2. We agree our scope is reliability rather than validity. Reliability is a precondition for validity. If scores change with the scorer or explainer, they cannot validly rank SAEs. We make this more explicit in Section 5 (Towards Reliable Evaluation) and in our response to reviewer 2.
W3. Background and metrics in the main text. We agree the main text should be self-contained. Using the extra page, we added a background section (prior work, known sources of variation, the gap) to the main text, and will extend the metric definitions in Section 3 (Preliminaries).
W4. Two models and no widely used SAEs. We combined public SAEBench SAEs (Pythia-160M) with our own (Apertus-8B) because our goal of comparing pipeline choices against architectures requires several architectures trained on identical data. To our knowledge, no public suite offers this for a larger model. Training our own SAEs also reflects common practice, for example for domains without public SAEs. We agree widely used SAEs matter, so we added the public Gemma Scope 2 SAE (JumpReLU, Matryoshka loss) on Gemma 3 4B-IT layer 17 and a second Apertus-8B layer. In both cases, paraphrase, draw and example count stay close to the paper’s reported ICC. Scorer, explainer and corpus remain the least reliable factors and purity the least reliable metric (full table in our response to Reviewer 1, W1).
Q1. Features are selected only on activation statistics, never on explanations or scores. Per feature, we record the maximum activation over each of 100k sequences. A feature is eligible if it exceeds a threshold T on at least 190 sequences, enough for 40 explanation and 50 scoring examples across 10 deciles plus repeated draws, and it fires on at most 2,000 (Pythia) or 1,000 (Apertus) sequences, to exclude very broad features. T is set once per model to maximise the number of eligible features. For both models, we end up with roughly 500 features. To classify each feature, we used two LLM classifiers that assigned each explanation to one of seven types (Cohen's κ between classifiers gives 0.70 for Pythia-160M and 0.58 for Apertus-8B):
Pythia features appear to be mostly lexical, while Apertus features more often code, domain or abstract. As a response to the probing remark, we would like to mention that SAE activations are observed directly, so no probe is needed to see how a feature behaves on a corpus. App. Figure 6 reports how many features are eligible per corpus.
Q2. The Pythia-160M SAEBench SAEs are public and cover multiple architectures trained under matched settings. Apertus-8B-Instruct is 50x larger, instruction-tuned, and fully open including its training data. Training the SAEs ourselves gave five architectures on identical data. The new setups add a second layer and public a Gemma Scope SAE.
Q3. The pipeline and selection rule are identical for both models. Scale changes the dictionary size (16k vs 65k), activation magnitude (T = 0.5 vs 63; Table 8) and feature types (Q1 answer). Despite this, the pattern we observe regarding reliability is the same in both models. Purity is the least reliable metric, the corpus has the lowest ICC in six of eight metric–model pairs, and the largest methodological factor explains more variance than architecture.
Paulo, G., Mallen, A., Juang, C., & Belrose, N. (2025). Automatically interpreting millions of features in large language models. ICML 2025.