acceptodds
Under review as a conference paper at ICLR 2027

Verifying the Linear Representation Hypothesis: How Interpretable Are Vision SAEs?

Abstract

Vision Sparse Autoencoders (SAEs) have become a popular tool in Mechanistic Interpretability due to their presumed ability to disentangle complex features learned by a model into monosemantic concepts. Despite their growing popularity, evaluating their interpretability remains an active topic of research. The bedrock motivating the adoption of SAEs is the Linear Representation Hypothesis (LRH), which claims that polysemantic features can be projected onto a (near) orthogonal basis of sparse, human-understandable representations. Yet, most current frameworks evaluate proxies such as the sparsity of SAE features or the coherence of the inferred dictionary, implicitly assuming that these reflect alignment with human perception. In this paper, we provide empirical evidence that measuring the interpretability of SAE concepts is more difficult than these proxies suggest. To this end, we adapt the Autointerpretability Score (AIS) - previously shown to align with human judgements in Natural Language Processing - to vision tasks and validate our approach in a dedicated user study. We evaluate SAE concept quality using both standard metrics and our adapted AIS. We find that established interpretability metrics for SAEs correlate neither with one another nor with AIS, indicating that no single reference-free metric, whether grounded in the LRH or not, is sufficient for verifying the interpretability of vision SAEs. We argue these findings support recent calls for more verifiable, ground-truth-anchored design and evaluation of explanation methods. All code and experiments can be found in our anonymized repository: https://anonymous.4open.science/r/XAI_SAE-0222.

Primary area:
interpretability and explainable AI
Keywords:
Sparse Autoencoders, Linear Representation Hypothesis, Autointerpretability Score, Human Perception

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_p5TaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This work introduces an auto-interpretability score (AIS) pipeline for evaluating features extracted from vision models, adapted from what was previously developed for language models. This was evaluated with a small-scale user study, comparing the performance of the proposed LLM-based AIS to an analogous human AIS protocol. The authors also benchmark 3 SAE models trained on 3 vision encoders over 3 datasets, evaluating multiple conventional SAE metrics in addition to their AIS. Results suggest these metrics are largely uncorrelated.

Critical strengths

I found the manuscript generally clear and well-written. In particular, Sections 1‒3 introduce the background and problem setting compellingly and motivate the research very well.

The experiments benchmarking common SAE metrics are interesting. The authors evaluated distinct types of SAE models on distinct vision encoders and distinct datasets. This comprehensive panel of results then allowed further analyses of relationships between the metrics. Even if discussion of the latter findings was fairly limited, the analysis invites further research on rigorous evaluation of interpretability methods.

Critical weaknesses

There are several elements of the visual AIS methodology that are not sufficiently justified or validated, even though this pipeline is one of the core paper contributions. For example, the explanation step takes only positive exemplars, although literature suggests that graded or contrastive panels can lead to more discriminative explanations (see e.g. Meissen et al. 2026, Automatic Interpretation of Visual Concepts). Only 5 instances are provided for explanation generation, increasing the risk of confounding with correlated concepts. It is also unclear why all patches of the top-average images are presented for explanation, instead of only top patches of images with highest max activation, which seems more intuitive. None of these design choices are evaluated against alternatives.

My other key concern is that the goal and implications of the user study are unclear, undermining confidence in the proposed method. The authors aimed to validate that their adapted vision AIS recipe "captures explanation quality as perceived by humans" (L474). However, the study has humans both generate explanations and simulate activations, instead of directly rating LLM explanations (as in Bills et al. 2023). The experimental design therefore seems to evaluate an abstract notion of interpretability as "feature is able to be encoded and decoded as language", regardless of the quality or human understanding of the LLM explanations. This is not necessarily an invalid question, but the framing is misleading. Further, the user study covers only two SAE features, with only 14 valid images in total, making the evidence seem anecdotal. Unless the focus of the study was specifically inter-observer variability, it would have been much preferable to increase coverage of SAE features as the primary unit of analysis, with fewer annotators. Lastly, in the LLM-vs-human AIS comparison, I expect Pearson correlation over ternary ordinal outcomes to be very noisy, and with such small sample size it's hard to draw any conclusions.

Questions for authors
  1. Please elaborate justifications for the numerous design choices in the proposed AIS pipeline, as exemplified above.
  2. Clarify the intended message of the user study and how exactly it corroborates AIS performance.
  3. Strongly consider running automated evaluations on intermediate steps of the pipeline to generate stronger empirical evidence. Ideally, supplement these with sensible ablations/baselines for individual components.
  4. (Not listed in critical weaknesses but would strengthen the paper:) Do you have hypotheses for the unexpected correlations observed in experiment #3?
6 comments

Dear Reviewer p5Ta,

Thank you for reviewing our paper. Please let us know whether our responses below addressed all of your questions or whether there are further questions we can answer so that you feel confident in increasing your score.

  1. “[...] only positive exemplars [...] only 5 instances provided for explanation”

    • For building the AIS pipeline, we took inspiration from Huben et al., Sparse Autoencoders Find Highly Interpretable Features in Language Models, 2024. There, the authors also select only 5 top-activating text samples from which the LLM-Explainer induces the common concept. Just as the text fragments there contain both high- and low-activating tokens, our 5 example images contain both high- and low-activating patches. Hence, the LLM-Explainer is also provided with negative exemplars via the low-activating patches. We added this clarification in Section 4.1 (Lines 263-266) and included the suggested reference (thank you for pointing us towards it!). Secondly, while we acknowledge that providing more examples and more contrast in the explanation phase would increase the context from which the Explainer is supposed to learn the concept, this would also noticeably increase the AIS computation time and considerably increase the prompt length (currently at 80 caption lines per neuron, given the 5 images x 16 patches per explanation batch).
  2. “unclear why all patches of the top-average images are presented for explanation, instead of only top patches of images with highest max activation, which seems more intuitive.”

    • Thank you for pointing this out. We admit that we did not find consistency in the literature regarding the best practice for sorting SAE neuron, e.g. Huben et al, 2024 are also not clear about it. But we understand that ranking images by the mean over cells favors features that cover many cells, such as grass, sky and water, while under-ranking more localized finer features. We therefore modified the ranking step in our AIS pipeline to sort according to max-pooled activations, not mean-pooled, and reran the AIS experiments. See the reformulated Max Filter in Appendix D for more details. We also ran a quick check with a Gemma4-2b, on Vanilla/CUB/DINO, and computed the AIS for the top 100 neurons sorted once on max-pooling and once on average-pooling. For max-pooling, we arrive at mean AIS 0.1347, median AIS 0.1508, 55% neurons with significant p-value (below 5%). For average-pooling, we arrive at mean AIS 0.1192, median AIS 0.1241, 51% significant neurons. So we notice slightly better statistics for the max-pooling. We also checked the IoU of the two sets of ranked neuron ids. It is 0.3605, so the two approaches do differ noticeably in their neuron selection. Nonetheless, our histogram showing the top 20 concepts/words in all the Gemma-generated explanations (previously Figure 6, now Figure 4), does not seem to have changed much.
  1. “[...] concerns regarding the goal and implications of the user study”

    • We agree that our human-in-the-loop experiment was conducted on a small scale, both in terms of number of subjects and neuron examples. Indeed, the user study was not at the center of the contribution we aimed at. It was intended as a sanity check to validate AIS and we felt that even though it was not conducted on a large pool of users and neurons, this is still an important aspect: many studies in the field often avoid a validation of XAI metrics and methods in user studies altogether. We would also like to note that scaling up the experiment in its current design would be rather challenging, given the non-triviality and length of the task (even though we have sharply simplified it down to scores from 0 to 2). For instance, annotating the 10 images of one neuron in the simulation phase requires exactly 160 clicks. For two neurons, including the explanation phase, we observed an average completion time of around 45 minutes per user. At this complexity, crowdsourcing the experiment would just not be feasible, hence our decision of conducting a small-scale study for now, leaving a broader human investigation as a goal for future work. While we were open about the uncertainty of the observed correlation between the human AIS and LLM AIS, given the width of the CI, we admit that we may have overstated the significance of our findings in terms of validating the AIS metric. The results of the user study should be seen more as having a descriptive role and providing a first piece of empirical evidence that LLM and human annotators perform similarly on the AIS task. Nonetheless, we did not aim at checking whether users understand the LLM explanations (as in Bills et al. 2023). We were indeed interested in whether an LLM can infer concepts and predict activations about as well as a human can. We have now reformulated Section 5.1 to reflect more clearly what we intended with this study and how the results should be interpreted.
  2. “[...] automated evaluations on intermediate steps of the pipeline [...]”

    • Thank you for raising this issue. We have now extended our experiments with the following supplementary comparisons: i) More caption grid sizes to motivate our choice for 4 x 4 (Table 4); ii) A caption-free test, where the LLM Explainer/SImulator infers the explanation and predicts the activation scores directly on images (Table 4); iii) A random baseline to check dictionary statistics of an untrained Vanilla SAE (Table 2); iv) We have changed the ranking strategy in our AIS pipeline (now max-pooling instead of average-pooling) - this was another reviewer’s hint, see Appendix D.
  3. “Do you have hypotheses for the unexpected correlations observed in experiment #3?”

    • We see some possible explanations for these unexpected correlations, which we have not fully tested. For instance, MS and AIS measure different things. MS measures embedding-space similarity weighted by activation products, so it tracks density, OOD and dataset homogeneity. AIS measures how describable a feature is in text. This may be a reason for the very poor correlation between the two across SAE families and suggests that similar samples (in whatever embedding space) are not necessarily perceived as containing a common dominant concept (at least not by an LLM explainer in our setting). Furthermore, some dictionary metrics partly reflect artifacts: Coherence is a maximum statistic, that can be saturated by a single duplicated atom and the negative OOD-Coherence relation may reflect a trade-off between grounding atoms in clustered patch embeddings and making them near-orthogonal, which is consistent with work questioning orthogonality assumptions (Costa et al., 2025, Hindupur et al., 2025). We set a more comprehensive evaluation of these hypotheses as a goal for future work.

I thank the authors for the detailed responses and revisions.

1. Thank you for the clarification. I still believe explaining only 5 unique contexts risks confounding, as the concepts in each image's 16 patches are likely strongly correlated. (Related to my clarified suggestion in the next point.)

2. The results with the max-pooling alternative are interesting, though my actual query was about showing only the top patches to the LLM explainer, without needing to present all patches of every selected image—allowing you to include a larger number of diverse contexts. I realise my phrasing could have been clearer, so I will not specifically penalise the authors for this misunderstanding.

3. I appreciate the authors' candour regarding the limitations of the user study and the issues with the previous framing. With the revision, the narrower scope is much clearer and my Question 2 is addressed.

4. While the extended comparisons are highly valuable (especially the caption-free baseline), my concern about lacking intermediate evaluations largely remains. As the authors acknowledge on L509–513, errors can compound over captioning, explanation, and simulation, but no way to measure the reliability of each component is presented yet.

5. Thank you for sharing these conjectures. They sound plausible and I believe would make an interesting addition to the discussion in the paper.

Reviewer_Yf37OpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adapts the Autointerpretability Score (AIS) from language-model SAEs to vision SAEs. A VLM captions patches of each image, and an LLM explainer and simulator use these captions to generate and score explanations of SAE features. The paper then compares AIS with several other SAE interpretability metrics and finds that they correlate neither with AIS nor with each other. The authors argue that this challenges the linear representation hypothesis (LRH) and that SAE interpretability evaluation needs rethinking.

Critical strengths
  1. Adapting AIS from language to vision makes vision and language SAEs directly comparable. The vision AIS is validated against human judgments, which supports using it as a reference metric.
  2. Sparsity, coherence and OOD score correlate in the opposite direction from what the LRH predicts. This is an interesting result. It suggests that current metrics miss what SAEs learn, and it raises the question of whether the LRH holds for vision models.
Critical weaknesses

AIS depends on captions lining up with patch activations, and the paper does not show that they do. The 4x4 captioning grid was chosen "after several iterations" (Sec. 4.2), but no results for other grid sizes are reported. Since AIS depends heavily on the captions, the paper should report results across grid resolutions and captioners. A 4x4 grid is also a coarse description of an image. In addition, self-attention in the image encoder mixes information across all patches, so a feature that fires strongly on a patch token is not necessarily responding to the content of that patch.

Questions for authors
  1. Why caption patches and pass the text to an LLM, rather than use a VLM directly as explainer and simulator? The paper mentions this option. Why was the caption-plus-LLM route chosen?
  2. How do the encoder's patches relate to the 4x4 caption grid? Step 3 of AIS (Sec. 4.1) does not say whether "patch activations" means encoder patches or caption cells. Appendices D and F suggest average-pooled encoder patch activations. Please clarify how the 4x4 captions are matched to encoder patches. (clarification)
2 comments

Dear Reviewer Yf37,

Thank you for reviewing our paper. Please let us know whether our responses below addressed all of your questions or whether there are further questions we can answer.

  1. “[...] the paper should report results across grid resolutions and captioners.”

    • Agreed, thank you for raising this point. We have run a comparison for Vanilla/ViT/CUB (Vanilla trained with λ=4.5×10−3\lambda = 4.5 \times 10^{-3}) between our initial choice for the captioner’s grid size - 4 x 4 - against two other ones - a coarser 2 x 2 and a finer 8 x 8. Table 4 shows that the 4 x 4 grid yields the highest AIS on this fixed configuration.
  2. “[...] self-attention in the image encoder mixes information across all patches, so a feature that fires strongly on a patch token is not necessarily responding to the content of that patch.”

    • Thank you for bringing up this important point. Local patch captions only have to carry enough information to predict activations. The shuffle control in Table 4 (-0.0043 vs. about 0.2784) is one piece of evidence that they do. Nonetheless, indeed, as AIS measures alignment with local content, contextual or global features will be more difficult to describe. We have appended this clarification to Section 5.4 (Lines 492-496).
  3. “Why caption patches and pass the text to an LLM, rather than use a VLM directly as explainer and simulator?”

    • Indeed, while having previously acknowledged this as a limitation and future work, we did not run any sanity check on it. But now we did conduct an experiment to compare our caption-based AIS approach against a caption-free alternative, where an LLM explainer (here Gemma4-12b and 4-4b) receives the images directly and is asked to induce a common concept on the explanation batch without captions. The LLM Simulator is also asked to predict the activation scores directly from images. For this, we annotated the images with a 4 x 4 grid and imprinted the activation scores within the image cells (for the explanation step); we added these details in Section 5.4. The results in Table 4 show that at least for the configuration tested (Vanilla/ViT/CUB), the caption-based approach achieves a higher AIS than the caption-free solution. Of course, one can also discuss alternative ways of showing the images and activation scores to the direct Explainer, e.g. image covered with an empty grid and activation scores in the prompt referencing to the cells in the image.
  4. “How do the encoder's patches relate to the 4x4 caption grid?”

    • Initially, the transformer backbones work with a finer patch grid: 16 x 16 in DINO and 14 x 14 in ViT and SigLIP. This is also the grid size used to train the SAEs, meaning that we feed 256 or 196, respectively, patch embeddings per image into the SAE. It also implies that a forward pass of the images through backbone and SAE results in 256 or 196 patch activations per neuron k; this is what we denote as a(k)a^{(k)} in Section 3.1. The captioner’s grid is coarser (4 x 4). In order to align it with the encoder’s grid, we average-pool the latter down to 4 x 4. Thus, we are able to build 16 pairs of captions and average-pooled activations per image. Appendix D gives more details about this process, as well as the main AIS script in our anonymized repository (ais_script.py).

The two new experiments do address our concerns. But we don't understand why the caption grid sizes and the VLM explainers are evaluated by their AIS values: a higher or lower AIS does not prove the validity of the approach. The authors should report how these scores correlate with human judgment, as in Figure 2. We are also concerned by the new random baselines: an untrained SAE matches or beats 12 of the 27 trained Vanilla configurations on AIS, and beats all of them on MS. If AIS barely separates trained from random SAEs, a low correlation with AIS says little about interpretability. We lower our score to 2.

Reviewer_UjPKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper adapts the Autointerpretability Score (AIS) from NLP to vision SAEs. Each image is split into a 4×4 grid, and LLaVA 1.5 captions every cell. Gemma explains each patch-level SAE neuron from the captions and activations, then simulates activations on 5 top and 5 random images; AIS is the Pearson correlation between simulated and true activations. The paper evaluates Vanilla, TopK, and MP SAEs on DINOv2, ViT, and SigLIP features of CUB, ImageNet100, and Caltech101. It reports that AIS and six standard SAE metrics do not correlate, and it supports AIS with a 16-user study.

Critical strengths
  1. The question is timely. Vision SAEs are mostly judged by dictionary statistics, and the paper adds a behavioral measure backed by a human study and a caption-shuffle null.
  2. The comparison is broad: 3 SAE families, 3 backbones, 3 datasets, and all metrics in one table (Table 2). This makes the question concrete and testable.
Critical weaknesses
  1. The most important opportunity, I think, is to evaluate SAEs that are actually sparse. Vanilla density is 0.50 (about 6,100 of 12,288 latents active per patch, more than pp) with λ=10−5\lambda = 10^{-5} (L1042), and TopK keeps 20% of latents active (L1032). Both violate the paper's own condition ∥z∥0≤K≪m\|z\|_0 \le K \ll m (L201). Fig. 3 therefore mostly reflects design choices:
  • Sparsity takes only three values, each set by hyperparameters.
  • Connectivity is 1.0000 in 23 of 27 rows.
  • Vanilla Coherence is identical across datasets and across ViT and SigLIP (0.2111). That is close to a random unit-norm dictionary of the same size (about 0.29 for p=384p=384 and 0.20 for p=768p=768 in my simulation).

With 27 clustered points, no ∣r∣<0.38|r| < 0.38 is significant. Retraining at standard L0 [3], sweeping sparsity within each family, and reporting within-family correlations with CIs plus a random-SAE baseline would make the central claim testable.

  1. The AIS validation is thin. The user study covers 2 easy MP/DINO/CUB neurons and 14 images (r = 0.62, CI [0.133, 0.866], with 5 of 21 users excluded post hoc). This shows agreement per image, not agreement on how AIS ranks neurons or SAEs. The explanations lean on generic words ("textures" 5.2%, Fig. 6), and L457 to 459 concede that a low AIS may not reflect the features. Two comparisons would separate captioner error from feature quality: MIS [1], which was validated against human psychophysics, and an image-input explainer as in MAIA [2] (Gemma 3 4B and 12B accept images).

[1] Zimmermann, R. S., Klindt, D., and Brendel, W. "Measuring Per-Unit Interpretability at Scale Without Humans." NeurIPS, 2024.
[2] Rott Shaham, T., et al. "A Multimodal Automated Interpretability Agent." ICML, 2024.
[3] Gao, L., et al. "Scaling and Evaluating Sparse Autoencoders." ICLR, 2025.

Questions for authors
  1. Does the Vanilla decoder stay near its initialization? Coherence before and after training, plus per-patch L0, would settle this.
  2. L470 says the paper challenges the LRH, yet the experiments only measure agreement among SAE metrics on fixed backbones. What result would have supported the LRH?
  3. How does mean AIS change if NaN neurons are scored rather than dropped? Table 4 drops 36 of 200 neurons, and Table 5 drops up to 181 of 200 for No SAE.

My rating could go up if the conclusions hold for sparse SAEs with confidence intervals.

4 comments

Dear Reviewer UjPK,

Thank you for reviewing our paper. Please let us know whether our responses below addressed all of your questions or whether there are further questions we can answer so that you feel confident in increasing your score.

  1. Sparsity levels in the SAEs

    • Thank you for bringing up this critical aspect. We have rerun the AIS experiments and recomputed all the metrics in Table 2, now split across Tables 2 (main text) and 5 and 6 (Appendix). We adjusted Section 5.2 to reflect the findings from the new sparsity sweeps. We also added a random untrained baseline on Vanilla/DINO/CUB (Table 2), thank you for this suggestion. Indeed, we can now see that the dictionary of our previous Vanilla configurations mainly stayed within its random initialization. We suppose this was mainly caused by a design fault: our previous SAE models had a BatchNorm following the encoder’s linear projection, which practically made the concept space dense again. This was more visible for Vanilla than the other two SAE types, since TopK and MP apply anyway a top-k-filter. We removed the BatchNorm from all our SAE models. Figure 3 now shows the Spearman rank correlations per SAE model (upper triangles), along with the widths of their corresponding 95-CIs (lower triangles). In short, we learn from the fresh results that our previous main messages remain valid: i) SAEs with high reconstruction potential are not necessarily interpretable, even in controlled sparse regimes; ii) SAE metrics (both from within the dictionary structure, as well as aside from it [MS and AIS]) do not correlate consistently across SAE families.
  2. AIS human-in-the-loop validation concerns

    • We agree that our human-in-the-loop experiment was conducted on a small scale, both in terms of number of subjects and neuron examples. Indeed, the user study was not at the center of the contribution we aimed at. It was intended as a sanity check to validate AIS and we felt that even though it was not conducted on a large pool of users and neurons, this is still an important aspect: many studies in the field often avoid a validation of XAI metrics and methods in user studies altogether. We would also like to note that scaling up the experiment in its current design would be rather challenging, given the non-triviality and length of the task (even though we have sharply simplified it down to scores from 0 to 2). For instance, annotating the 10 images of one neuron in the simulation phase requires exactly 160 clicks. For two neurons, including the explanation phase, we observed an average completion time of around 45 minutes per user. At this complexity, crowdsourcing the experiment would just not be feasible, hence our decision of conducting a small-scale study for now, leaving a broader human investigation as a goal for future work. While we were open about the uncertainty of the observed correlation between the human AIS and LLM AIS, given the width of the CI, we admit that we may have overstated the significance of our findings in terms of validating the AIS metric. The results of the user study should be seen more as having a descriptive role and providing a first piece of empirical evidence that LLM and human annotators perform similarly on the AIS task. We have now reformulated Section 5.1 to reflect more clearly what we intended with this study and how the results should be interpreted.
  1. “[...] separate captioner error from feature quality”

    • Thank you for the hints and the pointers to the two papers, they are both very relevant for embedding our contribution into the current literature. We have studied them in detail and referenced both in our manuscript, on Line 061 and Line 1030, respectively. As for the MAIA approach: We have now conducted an experiment ourselves with Gemma4-12b and 4b, by asking them to explain the images directly, without captions (see Section 5.4 and the new Table 4). For this test, we annotated the images with red 4 x 4 grids and activation scores (for the explanation step), similarly to the images shown to the users in our study. From the caption-based vs caption-free AIS comparison we learn that, at least on this fixed configuration, our caption-based approach achieves a higher AIS. Of course, as noted in Section 5.4, one can consider other ways of presenting the images and activation scores to the LLM, too (e.g. images covered only with an empty grid and the activation scores in the prompt).
  2. “Does the Vanilla decoder stay near its initialization?”

    • After removing the BatchNorm from the SAE, we notice clear differences in the dictionary statistics for Vanilla before and after training (see Table 2 and our answer to point 1 above).
  3. “What result would have supported the LRH?”

    • As stated in Section 3.2, the LRH assumes the existence of a sparse (near) orthogonal basis of human-interpretable concepts. Oftentimes, SAEs are only evaluated based on their dictionary statistics, without checking whether the concepts learned are indeed understandable to humans. Other interpretability metrics for SAEs, such as the Monosemanticity Score, have been shown to align with human perception. Our HIL experiment offers a first piece of empirical evidence in this direction for the adapted AIS for vision. Assuming the other dictionary metrics had correlated consistently with MS and/or AIS, this would have been an argument in favor of solely reporting dictionary statistics to support the LRH, since the ‘human understandability’ condition would have been implied from them. However, this was not the case in our experiments, which leads to the conclusion that properly evaluating the interpretability of SAEs requires a broader spectrum of measurements (also outside the dictionary structure). See our extended Sections 5.2 and 5.3.
  4. “How does mean AIS change if NaN neurons are scored rather than dropped?”

    • We are not sure whether we have correctly understood the implications of your question. Hopefully, the following answer clarifies it somewhat. NaNs arise only when Pearson correlation is undefined, i.e. when ground-truth or predicted activations are constant. In our runs, all such cases are all-zero pairs after the 0-10 normalization, so we deemed it safe to drop them from the AIS evaluation. But as an example, let us look at Table 8, at the SAE-AIS for layer 5: it is in our current recipe 0.2321 and has 58 NaNs in the top 100 neurons. If we were to score those neurons as 0, the AIS would drop to (42 * 0.2321 + 58 * 0) / 100 = 0.0974. If we were to score them as 1, the AIS would rise to (42 * 0.2321 + 58 * 1) / 100 = 0.6774. Side note: The former Table 4 was removed; p-value percents and NaN counts are now included directly in Tables 2, 5 and 6.

I thank the authors for the extensive revision. Retraining without BatchNorm, the sparsity sweeps, the cluster-bootstrap CIs, and the caption-free baseline (Table 4) cover most of what I asked for, and I appreciate the candid admission that the earlier Vanilla SAEs stayed near initialization. I will raise my Soundness score.

The most important points still keep me from raising the rating is: The random baseline is now the most important result. On DINO/CUB, the untrained SAE reaches AIS 0.338 with 82% significant neurons, above all three trained Vanilla SAEs on the same configuration (0.19 to 0.25, 61% to 72%), and its MS exceeds every trained SAE. The "7 of 27" comparison (L361) compares ViT and SigLIP SAEs against a DINO baseline. Could you report random baselines for all 9 embedder-dataset pairs? If AIS cannot separate trained from random SAEs, a low correlation with AIS says little about interpretability.

Thank you for your reply and raising the Soundness score, we are happy to see that we could improve on most of the issues that you raised.

As for the random Vanilla baseline: we understand and agree that it is now an important reference point and should be extended with the other Vanilla combinations to offer a more complete image. Therefore, we have now calculated all the metrics on all 9 embedder-dataset pairs for Vanilla. The updated Table 2 can be found in our new revised manuscript (uploaded here on OpenReview). Here we present results on only one random seed, but are happy to include more seeds in the manuscript after the rebuttal. We have also updated Lines 378-382 to reflect the fresh results.

Overall, 15 of the 27 trained Vanilla configurations achieve a higher AIS than their respective random baselines. This suggests that AIS is able to separate trained SAEs from random SAEs on most tested configurations. The random MS is still better than any actually trained configuration.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overview Rebuttal Changes

Dear reviewers,

Thank you for your valuable feedback. We are happy to see that you found our contribution “timely” (Reviewer UjPK), our results “interesting” (Reviewer Yf37) and our manuscript “generally clear and well-written” (Reviewer p5Ta).

We uploaded a revised version of our manuscript. In brief, we added and revised the following points and hope we could address all the concerns raised in the reviews:

  • We changed the ranking protocol in our AIS pipeline from mean-pooling to max-pooling (Reviewer p5Ta) and reformulated Appendix D to reflect these changes in our Max Filter.
  • We reran the AIS experiments with multiple sparsity levels per SAE family (Reviewer UjPK). The former Table 2 was completely recomputed and is now split between 3 Tables (2, 5 and 6) containing the results per SAE family. The new Table 2 also contains a random untrained baseline. Figure 3 now depicts correlations within SAE families, as well as the widths of their corresponding 95-CIs.
  • We tested our choice for the captioner’s grid size (4 x 4) against two others (Reviewer Yf37). We report the results in a new Table 4 (where also the random shuffling null has migrated).
  • In the same Table 4 we compare our caption-based approach against a caption-free alternative, namely asking Gemma4-12b and 4b (which are multimodal models) to directly explain images (Reviewers UjPK and Yf37).
  • We reformulated and clarified various parts in the manuscript according to reviewers’ feedback and questions. A main point of concern addressed in the reviews was related to the goal, claims and validity of our user study. We have, therefore, reframed our findings from this study (Section 5.1) and hopefully clarified our design choices.

Dear reviewers,

Thank you again for your insightful feedback and engaging discussions throughout the review process! Based on your initial reviews and subsequent questions, we have made the following revisions to our manuscript (new version already uploaded):

  1. We extended Table 2 with random baselines for all 9 embedder-dataset pairs in the Vanilla SAE (Reviewer UjPK). 15 of the 27 combinations have a higher AIS than random.
  2. We appended our conjectures regarding the unexpected correlations between dictionary statistics and MS and AIS in Section 5.3 (Reviewer p5Ta).
  3. We added a study over alternative explanation batch sizes and contrasts in Appendix J (Reviewer p5Ta): 5 positives vs top-random - no noticeable differences.
  4. We extended the ‘Vision AIS Limitations’ subsection with a discussion and results for automatic evaluations of intermediate steps in our pipeline, e.g. CLIP Scores for the captions (Reviewer p5Ta). On CUB, a 4x4 grid has a lower CLIP Score than 2x2, but a higher AIS.
  5. Minor: Algorithm 1 has migrated from Section 4.2 to Appendix E and the Paragraph ‘Image-level Representations’ from Section 3.1 to Appendix C.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.