acceptodds
Under review as a conference paper at ICLR 2027

Autointerpreting Sparse Autoencoders Is a Measurement Problem

Abstract

Autointerpretability scores, where large language models (LLMs) explain and score features, are the predominant way to evaluate sparse autoencoder (SAE) interpretability. Comparing methods with these scores depends on the assumption that they reflect stable feature properties rather than confounding aspects of the evaluation pipeline. Through systematic experiments across four metrics, two models (Pythia-160M, Apertus-8B-Instruct), and six axes of methodological variation, we show that this assumption rarely holds. We find that a single methodological choice explains more score variance than the SAE architecture. Excluding the activation corpus, which is arguably a property of the feature rather than the pipeline, weakens the effect, but it still holds for most model-metric pairs. Moreover, we find that metric reliability is a property of the metric–factor pair, not of the metric, and score agreement does not imply agreement about the top features; a failure that cannot be detected by similar mean scores or highly correlated per-feature scores. These findings suggest that cross-paper comparisons based on autointerpretability scores may reflect pipeline differences rather than architectural differences, providing evidence to an open debate: whether SAE latents are stable, semantically meaningful units for human interpretation. Beyond diagnosing these problems, we propose concrete recommendations for more reliable SAE evaluation.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, sparse autoencoders, evaluation reliability, autointerpretability, LLM-as-a-judge

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hqtqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the autointerpretability paradigm used to evaluate Sparse Autoencoder (SAE) interpretability, a framework where one language model generates a natural language description for an activated SAE feature, and a second model evaluates how accurately that description predicts feature activations on held-out data. The authors conduct systematic experiments across two language models (Pythia-160M and Apertus-8B-Instruct), testing four metrics across six axes of methodological variation. Rather than assessing construct validity (whether these metrics truly measure interpretability), the study focuses on statistical reliability across pipeline configurations.

Critical strengths
  • Addresses an important and timely challenge in Sparse Autoencoder (SAE) interpretability by systematically analyzing the statistical reliability of standard evaluation pipelines. Since mechanistic interpretability relies heavily on SAE autointerpretation pipelines to benchmark new architectures and dictionary learning techniques, revealing that pipeline hyperparameters introduce noise that swamps architectural variance provides an intresting point of view.
  • Evaluates metrics across multiple benchmark corpora, including The Pile, C4, and Wikipedia.
  • The experimental framework is structured around formal statistical metrics: Intraclass Correlation Coefficients (ICC) for measurement reliability, Top-k Jaccard Lift for ranking reproducibility, and η2η^2 variance decomposition. The study evaluates two distinct language models across four interpretability metrics, six pipeline parameters, and five SAE architectures.
Critical weaknesses
  • The activation corpus accounts for a substantial share of score variance. However, feature behavior on distinct corpora reflects true semantic variation rather than pure pipeline measurement noise. The paper treats the choice of activation corpus as a source of evaluation pipeline instability. While corpus variation accounts for a substantial share of total score variance, a latent feature's behavior on distinct datasets reflects true, meaningful semantic variation across domains rather than measurement noise in the autointerpretability pipeline. By grouping legitimate domain-shift variance (CORPUS) under "pipeline unreliability", the authors artificially inflate the apparent instability of autointerpretability metrics.

  • Furthermore, by focusing strictly on the statistical reliability of a metric rather than its actual validity within specific pipeline configurations, the authors limit the broader impact of their work.

  • Crucial context regarding prior SAE evaluation practices is relegated to Appendix A. Without a clear grounding in prior work within the main text, it is difficult to evaluate the paper's novel contributions, positioning, or alignment with standard evaluation pipelines. Furthermore, the four evaluated metrics (e.g., SIMULATION) lack definitions and theoretical justifications in the main text. The paper does not include comparisons from prior literature, making it difficult to contextualize the magnitude and significance of the observed pipeline variations.

  • The broad conclusion that cross-paper autointerpretability score comparisons reflect pipeline variations rather than model architecture differences is derived from only two models (Pythia-160M and Apertus-8B-Instruct). To substantiate these claims, the study requires broader experimentation on widely adopted SAE setups (e.g., Gemma Scope) and direct comparison with existing literature.

Questions for authors
  1. What types of latent features are being evaluated, and do they align with feature types commonly analyzed in the SAE literature? Is there baseline probing evidence for how these features manifest across the target corpora?

  2. What is the rationale behind selecting Pythia-160M and Apertus-8B-Instruct? The authors claim their work "is the first work to train and evaluate SAEs on APERTUS-8B-INSTRUCT", but what is the added value of this choice? Given the focus on general metric reliability, would not it be more insightful to evaluate standard, widely benchmarked models across the SAE ecosystem (e.g., Gemma-based or Llama-based SAEs)?

  3. How did the two-order-of-magnitude difference in model scale (160M vs. 8B parameters) impact your evaluation methodology, hyperparameter configurations, and the observed metric reliability?

6 comments

We thank the reviewer for the feedback and address the weaknesses and questions below.

W1. We agree that part of the corpus effect is real, as features behave differently across domains. App. Figure 6 shows that the number of eligible features differs strongly per corpus, and on Pythia-160M mean simulation drops from 0.52 on the Pile to 0.35 on code (App. Table 14). Our point is about comparability rather than the source of the variance, as a score describes a feature on a particular corpus. Studies that use different corpora measure different things, whether this is noise or domain shift. Prior work found that the training corpus has little effect on scores (Paulo et al., 2025) and we show that the evaluation corpus does. The corpus must therefore be fixed and reported, and we now present it as an evaluation condition. Our conclusions do not depend on it, because without the corpus, a single methodological factor still explains more variance than architecture in five of eight model–metric pairs (App. Tables 22–23).

W2. We agree our scope is reliability rather than validity. Reliability is a precondition for validity. If scores change with the scorer or explainer, they cannot validly rank SAEs. We make this more explicit in Section 5 (Towards Reliable Evaluation) and in our response to reviewer 2.

W3. Background and metrics in the main text. We agree the main text should be self-contained. Using the extra page, we added a background section (prior work, known sources of variation, the gap) to the main text, and will extend the metric definitions in Section 3 (Preliminaries).

W4. Two models and no widely used SAEs. We combined public SAEBench SAEs (Pythia-160M) with our own (Apertus-8B) because our goal of comparing pipeline choices against architectures requires several architectures trained on identical data. To our knowledge, no public suite offers this for a larger model. Training our own SAEs also reflects common practice, for example for domains without public SAEs. We agree widely used SAEs matter, so we added the public Gemma Scope 2 SAE (JumpReLU, Matryoshka loss) on Gemma 3 4B-IT layer 17 and a second Apertus-8B layer. In both cases, paraphrase, draw and example count stay close to the paper’s reported ICC. Scorer, explainer and corpus remain the least reliable factors and purity the least reliable metric (full table in our response to Reviewer 1, W1).

Q1. Features are selected only on activation statistics, never on explanations or scores. Per feature, we record the maximum activation over each of 100k sequences. A feature is eligible if it exceeds a threshold T on at least 190 sequences, enough for 40 explanation and 50 scoring examples across 10 deciles plus repeated draws, and it fires on at most 2,000 (Pythia) or 1,000 (Apertus) sequences, to exclude very broad features. T is set once per model to maximise the number of eligible features. For both models, we end up with roughly 500 features. To classify each feature, we used two LLM classifiers that assigned each explanation to one of seven types (Cohen's κ between classifiers gives 0.70 for Pythia-160M and 0.58 for Apertus-8B):

TypePythia-160MApertus-8B
Lexical44–45%5–6%
Syntax, formatting9–18%10–23%
Code, maths, technical4–6%24%
Named entities8–10%4–9%
Topic or domain6–7%21–25%
Abstract or semantic15–27%14–32%
Unclear0–1%0–3%

Pythia features appear to be mostly lexical, while Apertus features more often code, domain or abstract. As a response to the probing remark, we would like to mention that SAE activations are observed directly, so no probe is needed to see how a feature behaves on a corpus. App. Figure 6 reports how many features are eligible per corpus.

Q2. The Pythia-160M SAEBench SAEs are public and cover multiple architectures trained under matched settings. Apertus-8B-Instruct is 50x larger, instruction-tuned, and fully open including its training data. Training the SAEs ourselves gave five architectures on identical data. The new setups add a second layer and public a Gemma Scope SAE.

Q3. The pipeline and selection rule are identical for both models. Scale changes the dictionary size (16k vs 65k), activation magnitude (T = 0.5 vs 63; Table 8) and feature types (Q1 answer). Despite this, the pattern we observe regarding reliability is the same in both models. Purity is the least reliable metric, the corpus has the lowest ICC in six of eight metric–model pairs, and the largest methodological factor explains more variance than architecture.

Paulo, G., Mallen, A., Juang, C., & Belrose, N. (2025). Automatically interpreting millions of features in large language models. ICML 2025.

Thanks for the response, but this does not address my main concern on the practical contribution of this work in its current version. While appreciating the authors' efforts, the current version resembles more a position paper rather than providing a concrete methodological or experimental contribution in the field. Additionally, the authors use past tenses to report changes, yet the current version of their manuscript does not include the reported changes.

  • W1: The fact that "studies that use different corpora measure different things" is a trivial consideration, and everyone would agree ideally that "the corpus must therefore be fixed and reported". While the authors did show that the evaluation corpus has an impact on the evaluation pipeline of SAEs, "presenting it as an evaluation condition" represents a marginal and expected finding.

  • W2. Your scope was already cleared to my mind. My concern is that this is quite limited. A comprehensive evaluation should take into consideration both the reliability and validity of a metrics.

  • W3: I appreciate you are exploiting the additional page to include crucial related work. However, the authors should have taken into account its inclusion beforehand, using the original 9-page submission.

  • W4: I appreciate the explanation and the planned addition. I recommend the authors to include the underlying rational for model choice in the revised version.

  • Q1: Thanks for the further analysis. I would consider include it and its experimental findings within the main text.

  • Q2/Q3: Thanks for the clarifications

Thank you for the fast reply! We just uploaded the new version which includes all the changes.

Reviewer_FdANOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses that current dominant instruments for feature interpretability assessment in SAEs are unreliable, and argue that evaluating SAEs with autointerpretability is a measurement problem. The paper (1) proposes that a single methodological choice explains more variance than the SAE architecture; (2) shows that metric reliability is a property of the metric–factor pair, not of the metric; (3) shows that score agreement does not imply agreement about the top feature ranking.

Critical strengths
  1. It shows a lower bound of variance in the autointerpretability pipeline, as the experiment with a large subset of potential variance contributors, but not all of them.
Critical weaknesses
  1. The findings of the paper seems to be a numerical results rather than a theoretical results?
  2. For a reliable evaluation, the paper focuses only on two interventions: standardisation and averaging. This seems too simplistic and obvious.
Questions for authors
  1. I am wondering whether the findings of the paper are theoretical results. If they are, please show me the details of how the bound is obtained.
  2. More detailed discussion about the reliable evaluation is necessary. At least show me, why you focus only on these two interventions.
3 comments

We would like to thank the reviewer for the feedback. We noticed that the weaknesses are related to the questions, so we address them together.

W1 and Q1. The first weakness and question concern whether our findings are theoretical, and ask for a derivation of the lower bound. Our findings are empirical and the paper does not derive a theoretical lower bound. We will make it more explicit, and state that we investigate an empirical lower bound. The paper concerns empirical results of the autointerpretability pipeline choices, and not the theoretical framing. We realise the framing suggests a formal bound, and we rephrase it in the paper.

We believe empirical results are what is needed here, because the question has a practical implication whether autointerpretability scores can support comparisons between SAEs. Our results show they cannot.

W2 and Q2. The second weakness and question raised concerns the interventions proposed, and a more detailed discussion on why reliable evaluation is needed.

First, why reliable evaluation is needed. Autointerpretability scores are often used for different kinds of decisions, and these decisions break down when scores are only reflecting pipeline choices. We highlight three of those decisions, which are comparing SAEs, comparing across papers and selecting features. First, new architectures are justified by higher scores. However, if the order of two architectures can reverse when only the scorer changes, which we show happens in 13 out of 80 comparisons, a claim depends more on the scorer than the SAE architecture. Secondly, comparing SAEs across papers based on autointerpretability scores depend on different explainers and scorers. Since these choices can explain more variance than architectures, scores cannot be compared well. Lastly, metrics are used for selecting features. We show in Section 4 finding R3 that different top features can be selected when the pipeline differs. Reliable evaluation is therefore what makes these decisions reproducible. We added this motivation at the start of new Section 5.

Secondly, we would like to address why we focus on standardising and averaging. For every source of variation in the pipeline, a practitioner has two options: standardising or averaging. These are two options to deal with a source of variation, without changing the pipeline itself. Other solutions, such as a stronger scorer model or new metrics change the pipeline itself, rather than how it is used. We tested a third option of normalisation of the scorer’s scores, suggested by reviewer 1. This removes the calibration differences between scorers, but not their disagreement about individual features and leaves the order of architectures mostly scorer-dependent (from 13 out of 80 reversals to 10 out of 80 reversals). We would like to highlight that Section 5 does more than just naming the two options, as it mentions what each of the two options can offer conceptually. Standardising removes a factor’s contribution to the variance with no extra cost. For averaging, Table 4 shows how many conditions have to be averaged to reach good reliability for each metric and factor.

Thank you for the reply. I will keep my score.

Thank you for the reply. Could you elaborate on any concerns we have not addressed that result in no change in the score?

Reviewer_PDvaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether autointerpretability scores are reliable enough to compare different SAEs. The authors test the pipeline on 2 models with 5 SAE architectures and 4 metrics, and change 6 parts of the pipeline one at a time, such as the explainer, the scorer, and the corpus. They find that a single pipeline choice can affect the scores more than the SAE architecture, and that settings with similar mean scores often select different top features. They suggest fixing and reporting the scorer, the explainer, and the corpus in future evaluations.

Critical strengths

The experiments are systematic, changing one pipeline factor at a time across 2 models, 4 metrics, and 5 SAE architectures, with clear statistical measures. The paper is solid in its experimental design. The authors also use several statistical measures, such as ICC with confidence intervals, top-k Jaccard, and η². The appendix gives full results, prompts, and statistical details, which makes the work easy to follow.

Critical weaknesses
  1. The scale of the experiments is quite limited. The paper uses only two models, Pythia-160M and Apertus-8B-Instruct, and only one layer for each (layer 8 and layer 16). The authors note that the layer can affect the scores, so it is unclear whether the findings hold for other layers or for larger models. The explainer and scorer are also limited to three relatively small and cheap LLMs. Since these LLMs are not the strongest available, part of the instability may come from the limited ability of the LLMs. Maybe stronger LLMs might give more stable scores. And all experiments on pipeline factors are done on BatchTopK with k = 160, it is unclear whether the conclusions apply to other SAEs.
  2. The paper only studies the reliability of autointerpretability scores, not their validity. The authors clearly state that they do not test whether the scores truly capture interpretability. This limits the value of their paper. If the chosen scorer has a bias, for example toward simple token-level features, every study will share the same bias. The paper also does not say which setting is closer to the truth. For example, on Pythia-160M simulation, gemini-2.5-flash gives an average score of 0.528 while gpt-4o-mini gives 0.299, and it is unclear which one should be trusted. Also, high stability does not mean the scores are meaningful. Because a scorer that ignores the explanation could also be very stable.
  3. The novelty of the paper is limited. As the authors said in the introduction, prior work has already shown that sample selection, the explainer and scorer models, and prompt construction can affect autointerpretability scores. The main new part is quantifying these factors together and comparing them with SAE architecture. The recommendations, such as fixing and reporting the scorer, the explainer, and the corpus, are also fairly standard practice. The paper does not propose a new method to make the scores more stable or more valid.
Questions for authors
  1. Could the authors provide some causal tests? For example, use the explanation to predict the effect of steering a feature, and check whether the prediction is accurate or not.
  2. What temperature and decoding settings were used for the LLMs?
  3. Have the authors compared the scores with human judgments? Which scorer agrees more with humans?
  4. Beyond reporting the pipeline settings, can the authors suggest a concrete method to make scores comparable across scorers, such as score normalization?
  5. In Table 10, the architecture η² for some metrics is close to or below the random-split baseline. Does this mean the five architectures barely differ on these metrics? If so, how should Finding 1 be interpreted?
3 comments

We would like to thank the reviewer for the questions and the concerns, which led to new analyses.

W1. We added two setups, new SAEs on Apertus-8B layer 25 (A25), and one public Gemma Scope 2 SAE (JumpReLU) on Gemma 3 4B-IT layer 17 (G). Given budget constraints, we report detection, fuzzing, and purity. Gemini-2.5-flash is no longer available, so gemini-3.8-flash replaces it as main explainer and scorer, with gpt-4o-mini as second.

ICCDet. new (G / A25)Det. paper (P / A)Fuzz. newFuzz. paperPur. newPur. paper
Paraphrase.93 / .89.87 / .92.85 / .95.95 / .93.81 / .78.64 / .81
Draw.81 / .78.79 / .78.75 / .84.79 / .78.44 / .59.46 / .51
Example count.81 / .74.72 / .73.73 / .73.70 / .72.41 / .48.45 / .51
Scorer.75 / .66.88 / .81– / .46.57 / .44.06 / .16.33 / .38
Explainer.62 / .33.46 / .66.51 / .47.55 / .72.11 / .18.26 / .34
Corpus.41 / .61.55 / .46.29 / .57.49 / .34.02 / .28.22 / .23

Gemma fuzzing is scored by gpt, because gemini description fails for fuzzing here. The paper results hold. In both setups, paraphrase, draw and example count reproduce the paper closely, while the scorer, explainer and corpus remain least reliable. Again, the largest methodological factor matches or exceeds architectures’ variance explained.

Other architectures. Using the data we have from the scorer-architecture experiment (App. E.9), we computed the scorer ICC per architecture. The ordering of metrics (detection > fuzzing > purity) holds for every architecture, on both models and at A25. The scores of the five SAEs lie between the following ranges: detection .63–.67, fuzzing .39–.45, purity .17–.24). Full tables are in App. E.9.1.

Stronger scorer. We chose affordable models deliberately, as autointerpretability needs many calls per feature. Scoring all experiments with GPT-5 would cost around 3300 without simulation, and 12,000 with it. We therefore added GPT-5 (reasoning "low") as a fourth scorer in the scorer experiment (around $130):

Scorer ICC (P / A)DetectionFuzzingPurity
Best paper pair, per metric.88 / .81.77 / .65.47 / .62
GPT-5 + gemini-2.5-flash.90 / .85.91 / .82.48 / .61
All three paper scorers.86 / .80.67 / .58.33 / .38
All four, with GPT-5.85 / .77.70 / .57.30 / .39

The pooled ICC does not change. It seems that strong scorers agree more on detection and fuzzing, but not on purity. Also, the means of GPT-5 differ slightly (e.g. Pythia purity 0.85 vs 0.77). This indicates that the scorer must still be fixed and reported.

W2. We agree we do not test validity. Reliability is a precondition, because if a score changes with the scorer, it cannot validly rank SAEs, whichever scorer is right. We give a more detailed answer to reviewer 2. The second question asks which scorer to trust. Our answer on Q4 shows the gap is mostly calibration. Also, we want to highlight that a scorer that ignores the explanation would score around 0.5 on detection and fuzzing. Our means range from .73–.82.

W3. Prior work has studied factors in isolation, and we would like to explain what we do differently. First, we measure all factors on the same features, adding corpus, draw and paraphrase. Secondly, we compare them with architecture variance, the comparison these scores are used for (13/80 architecture pairs reverse with the scorer). Third, we show similar means can select different top features. Lastly, we quantify how many conditions to average (Spearman–Brown). Q4 shows that normalisation does not solve this.

Q1. Steering tests (AxBench, Wu et al., 2025) address validity and are complementary. The results would need the same reliability analysis, which we leave up to future work.

Q2. The temperature is 0 for all explainers and scorers (except for GPT-5: 1). gemini-2.5-flash scores without thinking, but explains with default thinking. We added the settings to Appendix B.

Q3. We have not conducted a human study. Human ratings need their own reliability study and we leave this up to future work.

Q4. We used z-score and rank transform on each scorer over the pooled features of all five architectures. Scorer ICC rises for simulation on both models, but barely for detection and fuzzing. Reversals stay 13/80 under z, and 10/80 under ranks. It also needs both scorers on the same features, which is often not possible in cross-paper comparisons. Full results are in App. E.10.

Q5. Yes, for Pythia detection and purity and Apertus purity, the architecture η² is within the null. This indicates that the architectures barely differ under a fixed pipeline. Where they do differ, a methodological factor still explains more. The null is conservative, as we used 100 features against 500 in the actual experiments. We clarified this in the paper.

Reviewer_PDvaOpenReview ↗

Official Comment by Reviewer PDva

Thank you for the reply. I will keep my score.

Thank you for the reply. Could you elaborate on any concerns we have not addressed that result in no change in the score?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We would like to thank all the reviewers for their feedback. Below we summarise the new experiments and changes we added to the revision.

New experiments:

  • We added five newly trained SAEs on Apertus-8b Layer 25 and a public Gemma Scope 2 SAE on Gemma 3 4B-IT layer 17. We repeated all factors on detection, fuzzing and purity. This showed that the findings replicate.
  • For the scoring experiment, we added GPT5 as a fourth scorer, scoring detection, fuzzing and purity. Stronger scorers appear to agree more on detection and fuzzing, but not purity. Their means still differ.
  • With earlier results, we showed scorer reliability per SAE architecture.
  • We added an experiment with per-scorer normalisation, showing that this can remove calibration differences, but not the features disagreements or architecture ranking reversals.
  • Feature types of both Pythia-160M and Apertus-8B are explored, showing some differences in the feature distributions over categories.

Revisions done:

  • New background section in the main text (section 2) and extended metric definition (section 3)
  • Rationale for model choice (section 3)
  • Motivation for reliable evaluation (section 5)
  • Lower bound rephrased as “empirical lower bound” (whole paper)
  • Decoding settings (appendix B3)
  • Scorer ICC per architecture (appendix E9.1)
  • Normalisation experiment (appendix E10)
  • Feature types (appendix F1)

For the final version:

  • A small section and table in the main paper with the generalisation results for the new setups and GPT5 (all details in appendix)
  • Reframe the corpus factor clearly as a domain-shift, instead of measurement noise

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.