acceptodds
Under review as a conference paper at ICLR 2027

Prediction Is Not Description: An External-Label Audit of Sparse Autoencoders

Abstract

Sparse autoencoders (SAEs) are evaluated almost entirely from inside the model. On those metrics they are hard to tell apart from random directions, and simple supervised baselines beat them. Yet that verdict conflates two properties: whether a direction separates a concept, and whether it can be described. Telling them apart needs a criterion the model never saw. MIMIC-IV admissions carry ICD-9 codes that human coders assign from the chart for billing, with no automated process involved. We train vanilla and JumpReLU SAEs on Gemma-2-2B activations from 50,000 discharge summaries, then audit them against a fixed 46-code panel on held-out notes, beside six alternatives: a general-purpose SAE, covariance- matched random directions, PCA, lexical n-grams, and two directions fitted on the codes themselves. Grounding against the codes fixes a common scale of correlation strength, on which the SAEs reach |rpb| = 0.86 against 0.43 for the random directions. Three tests then measure separate properties along that scale. Description and ablation both put the two domain-trained SAEs first. At matched correlation a blind judge recovers the code from an SAE feature’s unsupervised explanation 62–75% of the time, and from a direction fitted on that code only 17–38%, at or below the random floor. Ablation raises loss more on notes carrying the code for every domain-specific SAE target, and for no random direction. The general-purpose SAE, applied zero-shot, falls below both on every test it enters. Prediction reverses the order: the lexical n-grams classify the codes better than either SAE. Prediction and description are separate properties of a direction, and it is description that the sparse decomposition supplies.

Primary area:
interpretability and explainable AI
Keywords:
sparse autoencoders, interpretability evaluation, mechanistic interpretability, automated interpretability, concept directions, feature validation, causal ablation, external ground truth

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FUhpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Prior work on SAE has largely relied on model-internal metrics to evaluate features, often underperforming simple supervised baselines. To address this gap, the authors train vanilla and JumpReLU SAEs on Gemma-2-2B activations from MIMIC-IV discharge summaries and evaluate their associations with 46 selected ICD-9 codes. They conduct a comprehensive comparison against baselines including GemmaScope, covariance-matched random directions, PCA, and TF-IDF n-grams. Their findings reveal a disconnect between prediction and description: while TF-IDF achieves the strongest predictive performance, blind judges rate SAE features as more interpretable at matched correlation levels.

Critical strengths
  1. This paper effectively fills the a gap in previous intern metrics scoring by scoring features agains labels the model never saw
  2. The baselines and statistics are carefully chosen to rule out alternative explanations.
  3. The authors honestly reported the evaluated results on TF-IDF beating them on prediction.
Critical weaknesses
  1. The strong TF-IDF baseline is never tested on the judge task.
  2. There are some inconsistencies with the documented approach: section 4.3 says the judge chooses from the true code, seven same-chapter distractors and 'none'. But Appendix D says the slate is the feature's top-5 codes by correlation, three low-correlation negatives and 'none'.
  3. Some of the baselines may not be accurate to reflect the contribution of the paper: for example, the difference-in-means are fitted on note-level max-pooled activations. And then they are projected to token-level and then max-pooled. This causes inconsistency in evaluation and might introduce the low correlation. A more calibrated baseline can solve this concern.
  4. In appendix C, GemmaScope has an explained variance (held-out) of -4.21. If the reconstruction error is about 5 times the data's variance, simply outputting activation average would even do better. Although Appendix G investigates recentering, I think a further verification of this checkpoint might make the baseline result more convincing.
Questions for authors
  1. How does TF-IDF do on the judge task?
  2. What approach produces the result in table 3 (corresponding to weakness 2)
  3. Can you calibrate the baselines of GemmaScope and other methods such as difference-in-means?
Reviewer_3e7gOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the problem of evaluating Sparse Autoencoders (SAEs). The authors train both vanilla and JumpReLU SAEs on Gemma-2-2B activations derived from MIMIC-IV discharge summaries and then audit the learned features using ICD codes that are not observed during training.

By comparing the trained SAEs against six alternative approaches (e.g., a general-purpose SAE and TF-IDF), the authors conclude that prediction (whether the learned features can be used to predict labels) and description (whether a judge can name the ICD code from a feature explanation) are distinct properties. They further argue that domain-specific SAEs are particularly beneficial for the latter.

I found the paper difficult to understand, so my comments are based on an educated guess.

Critical strengths
  • Strengths
    • I think the distinction between prediction and description makes sense. The authors aim to understand what strengths domain-specific SAEs provide and where their limitations lie. I find this type of analysis interesting. However, it is not entirely clear to me how these findings can be leveraged to advance the field. For example, can they inform the design of better SAEs or lead to more effective training objectives?
    • The authors compare domain SAEs against multiple alternatives.
Critical weaknesses
  • Weaknesses
    • The authors are well aware of the limitations of their work and have already mentioned the two main weaknesses I was able to identify.
    • The first limitation is about the generalizability of the findings, as the experimental setup is somewhat arbitrary, focusing only on activations from a single layer of a single model. It is therefore unclear to what extent the conclusions would hold across different layers, architectures, or model scales. Also, I do not see any obvious obstacles to addressing this limitation through a broader set of experiments.
    • The second limitation relates to the reliability of the ICD codes. It would be helpful if the authors could validate their findings using additional datasets or sources to assess whether the same conclusions continue to hold.
    • I would encourage the authors to improve the readability of the paper. In particular, the paper should not assume that readers are already familiar with every analysis technique or understand the implications of all reported results.
Questions for authors

N/A

Reviewer_AyCaOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an external-label audit framework for evaluating sparse autoencoder (SAE) features using human-assigned ICD-9 codes in clinical notes. It compares domain-trained SAEs with several alternatives, including a general-purpose SAE, random directions, PCA components, label-fitted directions, and TF-IDF features. The evaluation considers prediction, description-based code retrieval, and activation ablation. The main finding is that predictive strength and describability do not necessarily align: domain- trained SAE features are more readily associated with external labels by LLM judges at matched correlation levels, whereas TF-IDF achieves stronger whole-representation prediction performance. The study highlights the importance of evaluating interpretability independently of predictive performance.

Critical strengths
  1. A useful evaluation framework: The paper evaluates SAE features against human-assigned clinical codes and compares them with multiple alternative feature sources under a shared evaluation protocol. This provides a concrete way to examine whether feature explanations correspond to externally specified concepts, rather than relying solely on model-internal interpretability metrics.
Critical weaknesses
  1. The figures in the paper have clarity and formatting issues. The visualizations should be improved in terms of clarity, presentation, and overall layout.
  2. The claimed dissociation is not established by the current matching procedure, matching correlation magnitudes does not ensure that features have comparable predictive information or comparable activation distributions.
  3. There is significant room for improvement in the paper's writing style and wording.
Questions for authors

Please provide confidence intervals or feature-level resampling estimates for the judge results and ablation effects, and clarify the inference scope given that ICD-9 codes may be correlated and may reflect billing practices rather than complete clinical ground truth.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.