- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for measuring the human interpretability of vision models. TopK SAE features are extracted from the final representation of six vision transformers (ViT-S/16, ViT-B/16, DINOv2, DINOv3, CLIP, SigLIP) and evaluated with two psychophysics protocols: localizability (clicking where a feature fires on a novel image, scored against a RISE heatmap with a chance-anchored normalisation) and semanticity (free-text description, scored by CLIP image–text similarity and inter-participant agreement). A control experiment shows that the existing forced-choice protocol yields model-dependent above-chance baselines. From 13,400 responses by 377 participants, the authors find that all four foundation models score below the size-matched supervised ViT-B, that interpretability is not significantly correlated with downstream performance, and that feature locality and coarse-grained alignment with human similarity judgements track interpretability, whereas fine-grained and visual-strategy alignment do not.
Critical strengths
[S1] A well-motivated correction to the dominant human-evaluation protocol. The control experiment in Section 3.1 is a clean demonstration that the forced-choice protocol admits a process-of-elimination shortcut, and the finding that the inflation is model-dependent (60.0% vs 53.6%) and can reverse a published ranking is a useful cautionary result for the field independently of the rest of the paper. The two replacement protocols remove the distractor by design, and the chance-anchored localizability score is a sensible way to make scores comparable across features with different activation profiles.
[S2] Scale and care of the behavioural data. With more than 15,000 responses, pre-specified exclusion criteria, IRB approval and above-minimum-wage compensation, this is one of the largest human studies of vision-model interpretability to date, and the breadth-over-depth allocation is justified by a pilot (Appendix D). The supplementary material is unusually thorough for a study of this kind: six alternative plausibility scorers, crop-size sweeps, HSIC rescoring, a two-seed stability check, and distributional plots that justify reporting medians. The limitations section is candid about the small number of models and the single scale.
[S3] Findings that go beyond the supervised–foundation gap. The dissociation between rater confidence and measured interpretability (DINOv2 elicits the highest confidence while ranking last on localizability and agreement) is a new observation with practical consequences for work that treats DINOv2 features as an interpretability reference. The locality result and the coarse-versus-fine alignment contrast are, to my knowledge, the first attempt to connect representational-alignment measures to human interpretability rather than to downstream performance, and the concordance of the two protocols on both the ranking and the predictors is a reasonable argument that a stable property is being measured.
Critical weaknesses
Weaknesses
[W1] No symmetric control experiment for the proposed protocol
In Section 3.1 the authors establish the flaw of the forced-choice protocol by showing that participants score above chance even when the maximally activating images carry no information about the feature. The same test is not applied to the proposed protocol. In the localizability task, a participant could plausibly score above without any explanation by clicking the centre of the most salient object, and this saliency prior may itself differ across models. In particular, models with more local features have heatmaps that concentrate on objects, so the model-dependent baseline identified in Section 3.1 may reappear in the new protocol.
[W2] Gap between the control experiment (neuron basis, two models) and the main study (SAE features, six models)
The control in Section 3.1 is run only on ResNet-50 and ViT-B/32 from Zimmermann et al. [6] and only on the neuron basis. The main study evaluates SAE features of six different models, so the motivation does not directly cover the setting in which the proposed protocol is applied. The size of the process-of-elimination inflation depends on how homogeneous the minimally activating images are, which can differ across feature bases. The subtraction-based renormalization ( vs ) also assumes an additive baseline without justification. Repeating the control on SAE features of at least two of the evaluated models, or narrowing the conclusion of Section 3.1 to neuron-basis forced-choice protocols, would close this gap.
[W3] Missing vision-SAE prior work and the novelty of contribution (i)
The related-work discussion of SAEs cites only language-model papers, although the evaluated objects are SAE features of vision transformers. Several prior works have already extracted and interpreted SAE features from supervised ViT, CLIP, DINOv2 and SigLIP [1, 2, 3, 4]. In particular, [1] reports with a human study that SAE features are more interpretable than neurons across all layers of three models, and [2] validates a monosemanticity score against a large-scale user study on CLIP and SigLIP and shows with it that SAE neurons are more monosemantic than the original ones. Contribution (i) is therefore not new in itself; what is new is placing SAE features on a cross-model scale. I would ask the authors to revise the related work and the wording of contribution (i) accordingly.
[W4] Single-layer evaluation with inconsistent layer choice across model families
All interpretability scores are measured at a single layer, and the position of that layer differs across model families (penultimate layer for supervised ViTs, last layer for DINO, encoder output for CLIP and SigLIP), with no argument that these are functionally corresponding positions. Zimmermann et al. [6] and Colin et al. [5], both cited by the authors, as well as ViT SAE work [1], report that feature interpretability varies substantially with depth, with intermediate layers typically harder to interpret.
[W5] Comparison at a single SAE operating point with no sensitivity analysis
All six SAEs use the same configuration (, expansion). Fixing is consistent with the matched- convention in the LLM literature [8], but the resulting SAEs end up at substantially different levels of reconstruction fidelity: ranges from (supervised ViTs) to (SigLIP) (Table A2). In other words, the models reported as most interpretable are exactly those whose SAEs reconstruct the least. The interpretability gap in Table 1 may therefore stem from differences in SAE fidelity rather than from properties of the representations themselves. is also known to affect interpretability directly. [8] reports that higher degrades the quality of pattern-based explanations, and recent work [7] argues that the appropriate is model-specific and that an inappropriate yields incorrect features. My concern is therefore that applying the same to models of different representational complexity can make the interpretability ranking across models depend on the chosen value of .
[W6] Appendix organisation and its connection to the main text (minor)
The appendix is split into fifteen sections, several of which contain a single paragraph or a single table (G, J, L, M, O) or only figures (C, E). More importantly, the Limitations (Appendix A) and the additional related work (Appendix B) are never referenced from the main text, and the main text itself contains no limitations paragraph. Consolidating the short sections and adding a pointer to the limitations from the conclusion would make the paper easier to follow and would place the paper's own caveats where readers will see them.
[W7] Section 3.1 and contribution (i) are overstated relative to Colin et al. [5]
The semantic confound in the forced-choice protocol, and a control for it, were already reported in Colin et al. [5], which also performed the first dictionary-basis model comparison. The present paper cites that work only for the superposition argument, while Section 3.1 presents the confound as a new observation and contribution (i) presents the move to a dictionary basis as a new design choice.
[W8] Insufficient scope of the SAE seed-reproducibility analysis
Only about 50% of CLIP and DINOv3 features have a counterpart in the second SAE seed (Table A3), and DINOv3 is the model on which the paper's most optimistic conclusion rests. The check in Appendix F filters the seed-1 responses to "stable" features, which does not test whether a different SAE draw would yield the same model ordering. I would ask the authors to re-run localizability on seed-2 features for at least two models, or to report the ordering under a stability-oriented SAE such as Archetypal SAE [9].
[W9] The predictor analyses rest on correlations over six models
The paper's most novel empirical claims, that locality and coarse-grained semantic alignment predict interpretability while fine-grained and strategy alignment do not, rest on Spearman correlations over six models, with more than fifteen correlations reported and no multiple-comparison correction.
After carefully considering the paper’s strengths and weaknesses, my initial recommendation is a weak reject.
References
[1] Jinyeong Kim, Junhyeok Kim, Yumin Shim, Joohyeok Kim, Sunyoung Jung, Seong Jae Hwang, Interpreting Vision Transformers via Residual Replacement Model, NeurIPS, 2025
[2] Mateusz Pach, Shyamgopal Karthik, Quentin Bouniot, Serge Belongie, Zeynep Akata, Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models, NeurIPS, 2025
[3] Hyesu Lim, Jinho Choi, Jaegul Choo, Steffen Schneider, Sparse Autoencoders Reveal Selective Remapping of Visual Concepts During Adaptation, ICLR, 2025
[4] Samuel Stevens, Wei-Lun Chao, Tanya Berger-Wolf, Yu Su, Interpretable and Testable Vision Features via Sparse Autoencoders, arXiv:2502.06755, 2025
[5] Julien Colin, Lore Goetschalckx, Thomas Fel, Victor Boutin, Thomas Serre, Nuria Oliver, Choosing the Right Basis for Interpretability: Psychophysical Comparison Between Neuron-Based and Dictionary-Based Representations, TMLR, 2026
[6] Roland S. Zimmermann, Thomas Klein, Wieland Brendel, Scale Alone Does Not Improve Mechanistic Interpretability in Vision Models, NeurIPS, 2023
[7] David Chanin, Adrià Garriga-Alonso, Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders, ICML, 2026
[8] Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, Jeffrey Wu, Scaling and Evaluating Sparse Autoencoders, ICLR, 2025
[9] Thomas Fel, Ekdeep Singh Lubana, Jacob S. Prince, Matthew Kowal, Victor Boutin, Isabel Papadimitriou, Binxu Wang, Martin Wattenberg, Demba Ba, Talia Konkle, Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models, ICML, 2025
Questions for authors
[Q1] The control experiment in Section 3.1 uses 62 and 46 participants (2,480 and 1,840 trials), considerably fewer than the main study (377 participants, 13,400 responses). Were the original and modified versions of a trial shown to the same participants or to separate groups? If within-subject, exposure to the original trial could inform the response to the modified one. If between-subject, group differences enter the estimated inflation. Please clarify the design, the ordering of conditions, and the number of features covered by the control trials.
4 comments
[W5] The choice of K in TopK might influence results
To test the sensitivity of our findings to SAE sparsity, we retrained TopK SAEs for three models (ViT-B, DINOv2, SigLIP) across five values of K = [10, 19, 38, 76, 152]. Since repeating the human experiments for each setting would be prohibitively expensive, we used as a proxy feature locality (Hoyer sparsity), which is our strongest predictor of human interpretability. Locality remained largely stable across K for a model as reflected in the table below, and the ranking between models remained unchanged, suggesting that our results are not primarily driven by the choice of sparsity.
| Model | K=10 | K=19 | K=38 | K=76 | K=152 |
|---|---|---|---|---|---|
| ViT-B | 0.095 | 0.098 | 0.102 | 0.103 | 0.101 |
| DINOv2 | 0.055 | 0.055 | 0.056 | 0.057 | 0.055 |
| SigLIP | 0.060 | 0.062 | 0.063 | 0.064 | 0.065 |
[W6] Appendix organisation
We thank the reviewer for these suggestions and will improve the appendix organization and its connection to the main text.
[W7] Section 3.1 contribution related to previous works
Colin et al. [5] identified a semantic confound in the protocol from Zimmermann et al. [6] and proposed to revise the protocol to address it. In Section 3.1 we test if the revised protocol from [5] could therefore be used to compare models, and find that the original shortcut remains, making the protocol unreliable for comparing models. Hence, our proposal of new human-centric, robust and fair protocols to measure and compare the interpretability of different models.
[W8] Stability of results with TopK instability
Running additional human experiments for every SAE ablation is prohibitively expensive. Instead, we use a proxy analysis in Appendix F to test whether SAE instability affects our conclusions. Restricting the analysis to features stable across SAE seeds changes median localizability by at most 0.033 and preserves our main findings.
Thank you again for the questions and feedback, which we hope to have successfully addressed in our responses.
Thank you for your detailed responses and substantial additional experiments.
After reviewing your responses, I believe that many of my original concerns have been adequately addressed. In particular, I greatly appreciate the additional human control experiments conducted during the rebuttal period. I believe these experiments provide important additional evidence supporting the reliability of the proposed evaluation protocol. However, despite these improvements, I feel that some of my concerns remain unresolved (particularly W4 and, to a lesser extent, W8).
[W4] Single-layer evaluation
The authors explain that they evaluate the last layer of the vision encoder across all models, thereby comparing the same architectural position. However, the same architectural position does not necessarily imply the same functional role. In particular, models trained with different objectives may exhibit different patterns in how feature semantics and interpretability develop across layers. I therefore believe that the validity of comparing the same architectural position across different models should be supported by relevant references or further discussion. If I have overlooked any relevant analyses or explanations provided during the rebuttal period, I would appreciate it if the authors could point me to them.
[W8] SAE seed reproducibility
The authors explain that restricting the analysis to features that are stable across SAE seeds results in only small changes in median localizability and preserves the model rankings. While this is a useful additional analysis, I do not believe it directly addresses my original concern. If I understand correctly, this analysis filters the features evaluated using the first SAE based on their stability across seeds, rather than testing whether the same interpretability results can be reproduced when independently evaluating features extracted from the second SAE. In other words, restricting the analysis to stable features does not sufficiently establish how differences in feature extraction across SAE seeds might affect cross-model interpretability comparisons.
I understand that the remaining discussion time is limited, but I would greatly appreciate any brief additional clarifications on these two points, as they would be very helpful in informing my final assessment.
Thank you for taking the time to carefully read our responses. We are glad to hear that they have adequately addressed many of your original concerns. Please find below our brief additional clarifications on W4 and W8
[W4] Single-layer evaluation
We agree that the same architectural position does not necessarily imply the same functional role, and we will clarify this distinction in the paper. However, extending our human evaluation of thousands of features (1 layer; >8,000 features; 8 models; >500 participants) to every layer would be prohibitively expensive, requiring us to select one. We chose the final vision-encoder layer for three reasons:
- Vision SAE literature: Previous studies involving human evaluations or cross-model comparisons have focused on final-layer representations ([1] Kim et al., 2025; [2] Pach et al., 2025; Thasarathan et al., 2025).
- Vision evaluation more broadly: Comparing final encoder representations is common practice when evaluating model capabilities and human alignment (Oquab et al., 2024; Muttenthaler et al., 2025).
- Cross-model comparison: The final layer provides a well-defined endpoint of visual processing for every model, regardless of architecture or training objective. Selecting corresponding intermediate layers would require an additional, potentially arbitrary matching criterion.
We will explicitly limit our conclusions to final-layer representations, without assuming they generalize across depth. We recognize that intermediate layers are also important to study, as demonstrated by recent work (e.g., Bolya et al., 2025).
[W8] SAE seed reproducibility
We agree that restricting our analysis to features stable across seeds is not equivalent to independently evaluating a second SAE. However, repeating our large-scale human evaluation across multiple SAE seeds would be prohibitively expensive. We therefore conducted two complementary robustness analyses:
- Seed stability: Our analysis in Appendix F tests whether features not reproduced across seeds drive our conclusions. Excluding these features preserves our main findings.
- Proxy analysis: We used feature locality (Hoyer sparsity), our strongest automated predictor of human interpretability, to evaluate sensitivity to different TopK sparsity settings (K=10–152). Locality varies little within models, and their ranking remains unchanged across settings.
We recognize that neither analysis establishes full seed reproducibility. To address this concern more directly, we can additionally report full-set locality scores from independently trained second-seed SAEs in the revised paper.
We will explicitly acknowledge single-seed human evaluation as a limitation.
Thank you for the extensive and detailed review.
[W2] Gap between control experiment (neuron) and main study (SAE features)
We agree that the control in Section 3.1 focuses on neuron features. However, showing that a protocol fails in one setting raises questions about its reliability more generally. Moreover, as explained below, our proposed protocol is robust to the controls.
[W1] Control experiment for proposed protocols
We have carried out additional control experiments to estimate the empirical chance-level performance for our protocols:
Localizability: We have conducted a control experiment similar to Section 3.1, testing whether participants can perform above chance without information about the correct feature. For this experiment, we have evaluated two models with different interpretability levels (ViT-B and DINOv2, sitting at both ends), and have evaluated a subset of their features in a new study with 30 participants (450 responses per model). In the control condition with mismatched explanations, the participants’ performance is at chance for both models (ViT-B: 48.3%; DINOv2: 48.1%; chance: 50%). Hence, participants need information about the feature to localize it, further rejecting the hypothesis that “clicking on the main object may beat chance”.
Semanticity: To perform a control experiment regarding semanticity we did not need to carry out any new user study, as in this case there is no notion of a feature explained vs tested from the point of view of participants. In this case chance levels can be directly estimated by rescoring their descriptions against other features' images (plausibility) or descriptions (agreement). Chance levels vary slightly across models (plausibility: 0.193–0.205; agreement: 0.149–0.165), but chance-corrected scores preserve our main findings and largely preserve model rankings. Moving forward, we will report chance-corrected scores in the paper.
Plausibility
Agreement
[W3] Related work and novelty
Note that our contribution is not to compare neuron-based and SAE-based features. Instead, we build on existing findings from the literature that SAE features are more interpretable than neurons and propose new human-centric protocols to measure and fairly compare their interpretability across models. Nevertheless, we thank the reviewer for highlighting these relevant works, which we will include in the related work section.
Moreover, in the context of the literature in human-centric interpretability, including suggested references [1] and [2], our paper establishes a substantially more rigorous and extensive human-centric framework for model interpretability. By evaluating eight computer vision model architectures—encompassing three standard Vision Transformers, two DINOv variants, two SIGLip models, and one CLIP backbone—our study captures a broader spectrum of visual representations than related works. This multi-architecture breadth is paired with a large-scale human evaluation protocol involving over 400 participants and 2 carefully designed protocols with their 3 associated metrics capturing something about human understanding. When combined with the added control condition, the research presented in our paper delivers a level of empirical validation beyond previous works in human-centric interpretability.
[W4] We study the same layer across models
We consistently study the last layer of the vision encoder across models, corresponding to the same architectural position in ViT-Base models trained with different paradigms. While interpretability varies with depth, evaluating all layers is beyond the scope of this work. Note that this choice of layer has been reported in the interpretability literature, including [2] recommended by the reviewer.