acceptodds
Under review as a conference paper at ICLR 2027

Capability ≠ Interpretability: Human Interpretability of Vision Foundation Models

Abstract

How interpretable are the features of leading vision models? The question is increasingly pressing as these models move from research benchmarks into high-stakes deployments, yet existing methods cannot answer it reliably. We close this gap with a framework for measuring and comparing the human interpretability of vision models, built around two complementary psychophysics protocols: (1) *localizability*—can an observer predict *where* a feature fires on a novel image?—and (2) *semanticity*—can an observer accurately describe *what* the feature represents? Features are recovered via sparse autoencoders, and a chance-anchored scoring function places every model on a common scale. Applying the framework to six vision transformers—two supervised ViTs and four foundation models (DINOv2, DINOv3, CLIP, SigLIP)—we collected more than behavioral responses, of which we analyzed responses from the participants who passed our pre-specified quality checks. We find that foundation models are consistently *less* interpretable than their size-matched supervised counterparts, and the gap is not a capability tradeoff: interpretability does not correlate with downstream task performance on any benchmark we examine. What does correlate is the *locality* of a feature's activations and *coarse-grained* semantic alignment with humans—models with focal activations and representations that reflect the world's broad categorical structure produce more interpretable features, whereas fine-grained perceptual alignment does not. The two protocols yield strongly correlated rankings and share the same predictors, establishing interpretability as an independent, measurable dimension of representation quality—and, surprisingly, one on which every foundation model we tested falls below the supervised baselines that came before. Capability alone cannot close that gap, but locality and coarse-grained alignment might.

Primary area:
interpretability and explainable AI
Keywords:
human-interpretability, foundations models, evaluation, computer vision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6rh9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for measuring the human interpretability of vision models. TopK SAE features are extracted from the final representation of six vision transformers (ViT-S/16, ViT-B/16, DINOv2, DINOv3, CLIP, SigLIP) and evaluated with two psychophysics protocols: localizability (clicking where a feature fires on a novel image, scored against a RISE heatmap with a chance-anchored normalisation) and semanticity (free-text description, scored by CLIP image–text similarity and inter-participant agreement). A control experiment shows that the existing forced-choice protocol yields model-dependent above-chance baselines. From 13,400 responses by 377 participants, the authors find that all four foundation models score below the size-matched supervised ViT-B, that interpretability is not significantly correlated with downstream performance, and that feature locality and coarse-grained alignment with human similarity judgements track interpretability, whereas fine-grained and visual-strategy alignment do not.

Critical strengths

[S1] A well-motivated correction to the dominant human-evaluation protocol. The control experiment in Section 3.1 is a clean demonstration that the forced-choice protocol admits a process-of-elimination shortcut, and the finding that the inflation is model-dependent (60.0% vs 53.6%) and can reverse a published ranking is a useful cautionary result for the field independently of the rest of the paper. The two replacement protocols remove the distractor by design, and the chance-anchored localizability score is a sensible way to make scores comparable across features with different activation profiles.

[S2] Scale and care of the behavioural data. With more than 15,000 responses, pre-specified exclusion criteria, IRB approval and above-minimum-wage compensation, this is one of the largest human studies of vision-model interpretability to date, and the breadth-over-depth allocation is justified by a pilot (Appendix D). The supplementary material is unusually thorough for a study of this kind: six alternative plausibility scorers, crop-size sweeps, HSIC rescoring, a two-seed stability check, and distributional plots that justify reporting medians. The limitations section is candid about the small number of models and the single scale.

[S3] Findings that go beyond the supervised–foundation gap. The dissociation between rater confidence and measured interpretability (DINOv2 elicits the highest confidence while ranking last on localizability and agreement) is a new observation with practical consequences for work that treats DINOv2 features as an interpretability reference. The locality result and the coarse-versus-fine alignment contrast are, to my knowledge, the first attempt to connect representational-alignment measures to human interpretability rather than to downstream performance, and the concordance of the two protocols on both the ranking and the predictors is a reasonable argument that a stable property is being measured.

Critical weaknesses

Weaknesses

[W1] No symmetric control experiment for the proposed protocol

In Section 3.1 the authors establish the flaw of the forced-choice protocol by showing that participants score above chance even when the maximally activating images carry no information about the feature. The same test is not applied to the proposed protocol. In the localizability task, a participant could plausibly score above 0.50.5 without any explanation by clicking the centre of the most salient object, and this saliency prior may itself differ across models. In particular, models with more local features have heatmaps that concentrate on objects, so the model-dependent baseline identified in Section 3.1 may reappear in the new protocol.

[W2] Gap between the control experiment (neuron basis, two models) and the main study (SAE features, six models)

The control in Section 3.1 is run only on ResNet-50 and ViT-B/32 from Zimmermann et al. [6] and only on the neuron basis. The main study evaluates SAE features of six different models, so the motivation does not directly cover the setting in which the proposed protocol is applied. The size of the process-of-elimination inflation depends on how homogeneous the minimally activating images are, which can differ across feature bases. The subtraction-based renormalization (83−6083 - 60 vs 80−53.680 - 53.6) also assumes an additive baseline without justification. Repeating the control on SAE features of at least two of the evaluated models, or narrowing the conclusion of Section 3.1 to neuron-basis forced-choice protocols, would close this gap.

[W3] Missing vision-SAE prior work and the novelty of contribution (i)

The related-work discussion of SAEs cites only language-model papers, although the evaluated objects are SAE features of vision transformers. Several prior works have already extracted and interpreted SAE features from supervised ViT, CLIP, DINOv2 and SigLIP [1, 2, 3, 4]. In particular, [1] reports with a human study that SAE features are more interpretable than neurons across all layers of three models, and [2] validates a monosemanticity score against a large-scale user study on CLIP and SigLIP and shows with it that SAE neurons are more monosemantic than the original ones. Contribution (i) is therefore not new in itself; what is new is placing SAE features on a cross-model scale. I would ask the authors to revise the related work and the wording of contribution (i) accordingly.

[W4] Single-layer evaluation with inconsistent layer choice across model families

All interpretability scores are measured at a single layer, and the position of that layer differs across model families (penultimate layer for supervised ViTs, last layer for DINO, encoder output for CLIP and SigLIP), with no argument that these are functionally corresponding positions. Zimmermann et al. [6] and Colin et al. [5], both cited by the authors, as well as ViT SAE work [1], report that feature interpretability varies substantially with depth, with intermediate layers typically harder to interpret.

[W5] Comparison at a single SAE operating point with no sensitivity analysis

All six SAEs use the same configuration (K=38K = 38, ×10\times 10 expansion). Fixing KK is consistent with the matched-L0L_0 convention in the LLM literature [8], but the resulting SAEs end up at substantially different levels of reconstruction fidelity: R2R^2 ranges from 0.7150.715 (supervised ViTs) to 0.8960.896 (SigLIP) (Table A2). In other words, the models reported as most interpretable are exactly those whose SAEs reconstruct the least. The interpretability gap in Table 1 may therefore stem from differences in SAE fidelity rather than from properties of the representations themselves. L0L_0 is also known to affect interpretability directly. [8] reports that higher L0L_0 degrades the quality of pattern-based explanations, and recent work [7] argues that the appropriate L0L_0 is model-specific and that an inappropriate L0L_0 yields incorrect features. My concern is therefore that applying the same KK to models of different representational complexity can make the interpretability ranking across models depend on the chosen value of KK.

[W6] Appendix organisation and its connection to the main text (minor)

The appendix is split into fifteen sections, several of which contain a single paragraph or a single table (G, J, L, M, O) or only figures (C, E). More importantly, the Limitations (Appendix A) and the additional related work (Appendix B) are never referenced from the main text, and the main text itself contains no limitations paragraph. Consolidating the short sections and adding a pointer to the limitations from the conclusion would make the paper easier to follow and would place the paper's own caveats where readers will see them.

[W7] Section 3.1 and contribution (i) are overstated relative to Colin et al. [5]

The semantic confound in the forced-choice protocol, and a control for it, were already reported in Colin et al. [5], which also performed the first dictionary-basis model comparison. The present paper cites that work only for the superposition argument, while Section 3.1 presents the confound as a new observation and contribution (i) presents the move to a dictionary basis as a new design choice.

[W8] Insufficient scope of the SAE seed-reproducibility analysis

Only about 50% of CLIP and DINOv3 features have a counterpart in the second SAE seed (Table A3), and DINOv3 is the model on which the paper's most optimistic conclusion rests. The check in Appendix F filters the seed-1 responses to "stable" features, which does not test whether a different SAE draw would yield the same model ordering. I would ask the authors to re-run localizability on seed-2 features for at least two models, or to report the ordering under a stability-oriented SAE such as Archetypal SAE [9].

[W9] The predictor analyses rest on correlations over six models

The paper's most novel empirical claims, that locality and coarse-grained semantic alignment predict interpretability while fine-grained and strategy alignment do not, rest on Spearman correlations over six models, with more than fifteen correlations reported and no multiple-comparison correction.

After carefully considering the paper’s strengths and weaknesses, my initial recommendation is a weak reject.

References

[1] Jinyeong Kim, Junhyeok Kim, Yumin Shim, Joohyeok Kim, Sunyoung Jung, Seong Jae Hwang, Interpreting Vision Transformers via Residual Replacement Model, NeurIPS, 2025
[2] Mateusz Pach, Shyamgopal Karthik, Quentin Bouniot, Serge Belongie, Zeynep Akata, Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models, NeurIPS, 2025
[3] Hyesu Lim, Jinho Choi, Jaegul Choo, Steffen Schneider, Sparse Autoencoders Reveal Selective Remapping of Visual Concepts During Adaptation, ICLR, 2025
[4] Samuel Stevens, Wei-Lun Chao, Tanya Berger-Wolf, Yu Su, Interpretable and Testable Vision Features via Sparse Autoencoders, arXiv:2502.06755, 2025
[5] Julien Colin, Lore Goetschalckx, Thomas Fel, Victor Boutin, Thomas Serre, Nuria Oliver, Choosing the Right Basis for Interpretability: Psychophysical Comparison Between Neuron-Based and Dictionary-Based Representations, TMLR, 2026
[6] Roland S. Zimmermann, Thomas Klein, Wieland Brendel, Scale Alone Does Not Improve Mechanistic Interpretability in Vision Models, NeurIPS, 2023
[7] David Chanin, Adrià Garriga-Alonso, Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders, ICML, 2026
[8] Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, Jeffrey Wu, Scaling and Evaluating Sparse Autoencoders, ICLR, 2025
[9] Thomas Fel, Ekdeep Singh Lubana, Jacob S. Prince, Matthew Kowal, Victor Boutin, Isabel Papadimitriou, Binxu Wang, Martin Wattenberg, Demba Ba, Talia Konkle, Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models, ICML, 2025

Questions for authors

[Q1] The control experiment in Section 3.1 uses 62 and 46 participants (2,480 and 1,840 trials), considerably fewer than the main study (377 participants, 13,400 responses). Were the original and modified versions of a trial shown to the same participants or to separate groups? If within-subject, exposure to the original trial could inform the response to the modified one. If between-subject, group differences enter the estimated inflation. Please clarify the design, the ordering of conditions, and the number of features covered by the control trials.

4 comments

Thank you for the extensive and detailed review.

[W2] Gap between control experiment (neuron) and main study (SAE features)

We agree that the control in Section 3.1 focuses on neuron features. However, showing that a protocol fails in one setting raises questions about its reliability more generally. Moreover, as explained below, our proposed protocol is robust to the controls.

[W1] Control experiment for proposed protocols

We have carried out additional control experiments to estimate the empirical chance-level performance for our protocols:

  • Localizability: We have conducted a control experiment similar to Section 3.1, testing whether participants can perform above chance without information about the correct feature. For this experiment, we have evaluated two models with different interpretability levels (ViT-B and DINOv2, sitting at both ends), and have evaluated a subset of their features in a new study with 30 participants (450 responses per model). In the control condition with mismatched explanations, the participants’ performance is at chance for both models (ViT-B: 48.3%; DINOv2: 48.1%; chance: 50%). Hence, participants need information about the feature to localize it, further rejecting the hypothesis that “clicking on the main object may beat chance”.

  • Semanticity: To perform a control experiment regarding semanticity we did not need to carry out any new user study, as in this case there is no notion of a feature explained vs tested from the point of view of participants. In this case chance levels can be directly estimated by rescoring their descriptions against other features' images (plausibility) or descriptions (agreement). Chance levels vary slightly across models (plausibility: 0.193–0.205; agreement: 0.149–0.165), but chance-corrected scores preserve our main findings and largely preserve model rankings. Moving forward, we will report chance-corrected scores in the paper.  

Plausibility

ModelChanceRealChance-corrected
ViT-S0.2040.2740.070
ViT-B0.2020.2730.071
ViT-L0.1930.2600.067
DINOv20.1950.2590.064
DINOv30.1940.2600.066
CLIP0.2050.2660.062
SigLIP0.1960.2530.057
SigLIP20.1990.2530.054

Agreement

ModelChanceRealChance-corrected
ViT-S0.1560.4930.337
ViT-B0.1540.5190.366
ViT-L0.1650.4740.309
DINOv20.1490.4330.284
DINOv30.1580.4580.300
CLIP0.1580.4400.282
SigLIP0.1620.4430.281
SigLIP20.1570.4380.281

 

[W3] Related work and novelty

Note that our contribution is not to compare neuron-based and SAE-based features. Instead, we build on existing findings from the literature that SAE features are more interpretable than neurons and propose new human-centric protocols to measure and fairly compare their interpretability across models. Nevertheless, we thank the reviewer for highlighting these relevant works, which we will include in the related work section.

Moreover, in the context of the literature in human-centric interpretability, including suggested references [1] and [2], our paper establishes a substantially more rigorous and extensive human-centric framework for model interpretability. By evaluating eight computer vision model architectures—encompassing three standard Vision Transformers, two DINOv variants, two SIGLip models, and one CLIP backbone—our study captures a broader spectrum of visual representations than related works. This multi-architecture breadth is paired with a large-scale human evaluation protocol involving over 400 participants and 2 carefully designed protocols with their 3 associated metrics capturing something about human understanding. When combined with the added control condition, the research presented in our paper delivers a level of empirical validation beyond previous works in human-centric interpretability.

[W4] We study the same layer across models

We consistently study the last layer of the vision encoder across models, corresponding to the same architectural position in ViT-Base models trained with different paradigms. While interpretability varies with depth, evaluating all layers is beyond the scope of this work. Note that this choice of layer has been reported in the interpretability literature, including [2] recommended by the reviewer.

[W5] The choice of K in TopK might influence results

To test the sensitivity of our findings to SAE sparsity, we retrained TopK SAEs for three models (ViT-B, DINOv2, SigLIP) across five values of K = [10, 19, 38, 76, 152]. Since repeating the human experiments for each setting would be prohibitively expensive, we used as a proxy feature locality (Hoyer sparsity), which is our strongest predictor of human interpretability. Locality remained largely stable across K for a model as reflected in the table below, and the ranking between models remained unchanged, suggesting that our results are not primarily driven by the choice of sparsity.

 

ModelK=10K=19K=38K=76K=152
ViT-B0.0950.0980.1020.1030.101
DINOv20.0550.0550.0560.0570.055
SigLIP0.0600.0620.0630.0640.065

 

[W6] Appendix organisation

We thank the reviewer for these suggestions and will improve the appendix organization and its connection to the main text.

[W7] Section 3.1 contribution related to previous works

Colin et al. [5] identified a semantic confound in the protocol from Zimmermann et al. [6] and proposed to revise the protocol to address it. In Section 3.1 we test if the revised protocol from [5] could therefore be used to compare models, and find that the original shortcut remains, making the protocol unreliable for comparing models. Hence, our proposal of new human-centric, robust and fair protocols to measure and compare the interpretability of different models.

[W8] Stability of results with TopK instability

Running additional human experiments for every SAE ablation is prohibitively expensive. Instead, we use a proxy analysis in Appendix F to test whether SAE instability affects our conclusions. Restricting the analysis to features stable across SAE seeds changes median localizability by at most 0.033 and preserves our main findings.

 

Thank you again for the questions and feedback, which we hope to have successfully addressed in our responses.

Thank you for your detailed responses and substantial additional experiments.

After reviewing your responses, I believe that many of my original concerns have been adequately addressed. In particular, I greatly appreciate the additional human control experiments conducted during the rebuttal period. I believe these experiments provide important additional evidence supporting the reliability of the proposed evaluation protocol. However, despite these improvements, I feel that some of my concerns remain unresolved (particularly W4 and, to a lesser extent, W8).

[W4] Single-layer evaluation

The authors explain that they evaluate the last layer of the vision encoder across all models, thereby comparing the same architectural position. However, the same architectural position does not necessarily imply the same functional role. In particular, models trained with different objectives may exhibit different patterns in how feature semantics and interpretability develop across layers. I therefore believe that the validity of comparing the same architectural position across different models should be supported by relevant references or further discussion. If I have overlooked any relevant analyses or explanations provided during the rebuttal period, I would appreciate it if the authors could point me to them.

[W8] SAE seed reproducibility

The authors explain that restricting the analysis to features that are stable across SAE seeds results in only small changes in median localizability and preserves the model rankings. While this is a useful additional analysis, I do not believe it directly addresses my original concern. If I understand correctly, this analysis filters the features evaluated using the first SAE based on their stability across seeds, rather than testing whether the same interpretability results can be reproduced when independently evaluating features extracted from the second SAE. In other words, restricting the analysis to stable features does not sufficiently establish how differences in feature extraction across SAE seeds might affect cross-model interpretability comparisons.

I understand that the remaining discussion time is limited, but I would greatly appreciate any brief additional clarifications on these two points, as they would be very helpful in informing my final assessment.

Thank you for taking the time to carefully read our responses. We are glad to hear that they have adequately addressed many of your original concerns. Please find below our brief additional clarifications on W4 and W8

 

[W4] Single-layer evaluation

We agree that the same architectural position does not necessarily imply the same functional role, and we will clarify this distinction in the paper. However, extending our human evaluation of thousands of features (1 layer; >8,000 features; 8 models; >500 participants) to every layer would be prohibitively expensive, requiring us to select one. We chose the final vision-encoder layer for three reasons:

  • Vision SAE literature: Previous studies involving human evaluations or cross-model comparisons have focused on final-layer representations ([1] Kim et al., 2025; [2] Pach et al., 2025; Thasarathan et al., 2025).
  • Vision evaluation more broadly: Comparing final encoder representations is common practice when evaluating model capabilities and human alignment (Oquab et al., 2024; Muttenthaler et al., 2025).
  • Cross-model comparison: The final layer provides a well-defined endpoint of visual processing for every model, regardless of architecture or training objective. Selecting corresponding intermediate layers would require an additional, potentially arbitrary matching criterion.

We will explicitly limit our conclusions to final-layer representations, without assuming they generalize across depth. We recognize that intermediate layers are also important to study, as demonstrated by recent work (e.g., Bolya et al., 2025).

 

[W8] SAE seed reproducibility

We agree that restricting our analysis to features stable across seeds is not equivalent to independently evaluating a second SAE. However, repeating our large-scale human evaluation across multiple SAE seeds would be prohibitively expensive. We therefore conducted two complementary robustness analyses:

  • Seed stability: Our analysis in Appendix F tests whether features not reproduced across seeds drive our conclusions. Excluding these features preserves our main findings.
  • Proxy analysis: We used feature locality (Hoyer sparsity), our strongest automated predictor of human interpretability, to evaluate sensitivity to different TopK sparsity settings (K=10–152). Locality varies little within models, and their ranking remains unchanged across settings.

We recognize that neither analysis establishes full seed reproducibility. To address this concern more directly, we can additionally report full-set locality scores from independently trained second-seed SAEs in the revised paper.

We will explicitly acknowledge single-seed human evaluation as a limitation.

Reviewer_YNAJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a psychophysics protocol to evaluate the interpretability of visual representations. It focuses on two aspects of interpretability:

  • localizability: can a user understand and predict where in an image a feature will fire?
  • semanticity: can the user describe, in natural language, what the feature represents? This is measured as plausibility (similarity of the human patch-based description with its CLIP representation) and agreement (similarity between two human descriptions of the same feature). The study concludes that there’s no correlation between performance and interpretability, but that there is a correlation between localizability and coarse-grained similarity. With respect to previous work (like Zimmermann et al. 2023), the authors point out that they don’t use distractors (examples where the feature is absent) that can lead to shifting baselines because the users can use information about “what is not” in the distractors rather than what they can actually see in the image.
Critical strengths
  1. Interpretability studies with humans in the loop are badly needed to understand the how well aligned are the representations of vision models with human interpretation.
  2. Localizability and semanticity seem like two meaningful properties that a feature should have to be called interpretable.
  3. The user study, with 13’400 responses, seems to have the right scale level.
Critical weaknesses
  1. The results are compared to those of, for instance Herbart et al. (2020), where a user study is used to understand the coarse-grained semantic structure of human concepts, to then compare these to a model representation. The results of the human study of Herbart et al. (2020) can be cheaply applied to a new model, which would allow to compare many models easily, while the approach in the submitted paper need a new user study for a new model, leading to high costs and potential comparability issues is a new cohort is recruited. I could not uderstand the advantage of the proposed approach wrt the one in Herbart et al. (2020).
  2. If I get it well, the definition of locality uses the RISE heatmaps, and then it is evaluated using a localizability definition that uses RISE heatmaps (or HSIC, which is based on a similar principle) together with the user clicks. This raises serious questions about potential circularity in the result that find these two quantities being correlated.
  3. Since the authors evaluate the interpretability of 6 models, all correlations are measured using n=6. With many p-values between 0.03 and 0.05, the statistical significance of the conclusions could depend on a single model. It is not clear that the current experimental setup allows to reach a significant conclusion.
  4. Cropping around RISE peaks to compute plausibility risks not providing enough context for users to interpret concepts that are not focalized in a small region, potentially making the correlation between localizability and plausibility larger than it truly is.
  5. Although the authors use an SAE to extract features, a feel like the related works lacks a bit the positioning to recent SAE-based vision interpretability metrics, such as Monosemanticity [1] and TAPAScore [2] (although the latter may have been published after the deadline, so probably no need to include it).
  6. Although six models are evaluated, only one SAE setting is used. It could be that the results obtained may depend to the specific SAE used and the choice of K, more than on the underling model.

[1] Pach et al., Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models, NeurIPS 2025.
[2] Klotz et al., Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations, ECCV 2026.

Questions for authors
  1. What is the main advantage of setting up a model-specific user study rather than a model-agnostic one, such as in Herbart et al. (2020) that can easily be scaled up to evaluate many models?
  2. Why do we need RISE/HSIC if the SAE is trained per-patch and thus we know where it fired?
  3. Did the authors investigate different SAE settings? Does the specific SAE have an impact on the results?
3 comments

Thank you for your thoughtful feedback and constructive suggestions. Below, we address your concerns and questions.

Hebart et al. protocol vs. ours

The two experimental protocols measure fundamentally different properties. Our protocols are designed as practical proxies for what humans want to understand about the features learned by a model. In contrast, Hebart et al.'s protocol measures how well the semantic organization of a model's representations aligns with that of humans. Since our protocols evaluate individual features, which are model-specific, they require a new human evaluation for each model.

A connection between the two has been hypothesized, but our work provides the first empirical evidence that they are correlated. However, we still do not know whether this relationship is merely correlational or causal. If future research establishes a causal relationship, we agree that measuring alignment could serve as a cheaper proxy for interpretability, allowing evaluations to scale to many more models.

Circularity of localizability

The importance maps used to score participants' clicks (query image) are different from those used as hints to explain the features and to compute locality scores (top-activating images). Moreover, we find that locality is correlated with all three interpretability measures, including description agreement, i.e., the degree to which people provide similar descriptions of a feature, which relies on a different protocol, scoring method and modality. This suggests that locality is broadly associated with interpretability, rather than being specific to one protocol, reducing concerns about circularity.

RISE maps vs. patch-activation maps

The two maps answer different questions. A token-activation heatmap shows where in the token grid a feature is expressed, whereas RISE shows which input regions the feature's activation depends on. Our protocols test whether participants can predict the latter, making RISE the more appropriate reference. Moreover, interpreting token activations as image locations assumes that each token encodes its corresponding image patch, which has been shown to not necessarily hold [a, b].

Results based on n=6 models

We agree. We are working on extending our evaluation to additional models and will update the rebuttal and paper with the results when available. Nevertheless, for the current set of models, we ran leave-one-out correlations and found that our positive correlations retain large magnitudes (|r| > 0.63), although some become non-significant, partly due to the even smaller sample size. The negative correlations with capability also remain negative across every leave-one-out subset.

Plausibility crops

The image hints are shown as full images to participants during the semanticity experiment. The cropping used for plausibility only occurs when evaluating their descriptions, and using full images instead maintains all trends.

Extended related work

Thank you for pointing out these related works. We will include them in the revised manuscript, especially Pach et al., and clarify the distinction:

"While Borowski et al. study the visual coherence of a feature's preferred images by asking humans to predict which images the feature prefers, Pach et al. instead rely on self-reported understanding, which has been shown to be unreliable [c, d]."

One SAE setup

This is a very good question. While we know that using different SAEs for the same model can yield different results, it remains an open question in the broader interpretability literature whether the choice of SAE changes comparisons across models. In this work, we focus on the narrower question: given a standard TopK SAE setting [e], held fixed across models (which we consider the fairest choice given our current understanding), how interpretable are the features of each model?

[a] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. "Vision Transformers Need Registers." ICLR. (2024).

[b] Han, Sangyu, Yearim Kim, and Nojun Kwak. "Causal interpretation of sparse autoencoder features in vision." ICCV eXCV Workshop. (2025).

[c] Chromik Michael, Malin Eiband, Felicitas Buchner, Adrian Krüger, and Andreas Butz. "I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI". Proceedings of the 26th International Conference on Intelligent User Interfaces. (2021).

[d] Rozenblit, Leonid, and Frank Keil. "The misunderstood limits of folk science: An illusion of explanatory depth." Cognitive Science. (2002).

[e] Thasarathan Harrish, Forsyth Julian, Fel Thomas, Kowal Matthew, and Derpanis, Konstantinos G. "Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment". Proceedings of the 42nd International Conference on Machine Learning. (2025).

I thank the authors for their clarifications. I think a paper like this needs to be supported by enough evidence to be able to make the claims, and I encourage the authors to keep working on this and resubmit once they have gathered enough empirical evidence. I will keep my original score.

Need for more empirical evidence

Following the reviewer's suggestion, we have expanded our evaluation from 6 to 8 models by adding ViT-L and SigLIP2. These additional results not only corroborate but strengthen our main findings:

  • ViT-B remains the most interpretable model overall, ranking first across all three measures, with only SigLIP2 matching its localizability score (at a similar scale).
  • Model capability is not significantly correlated with interpretability on any of our three measures.
  • Models with more local representations have more interpretable features: participants converge to similar descriptions (r=0.723, p<0.05), which are more plausible (r=0.842, p<0.01), and the features are easier to localize (r=0.876, p<0.001).
  • Models that organize their representations in a way that is aligned with human semantic organization have features that are easier to localize (THINGS: r=0.884, p<0.01; Levels: r=0.797, p<0.01).

 

SupervisedFoundation
ViT-SViT-BViT-LDINOv2DINOv3CLIPSigLIPSigLIP2
Localizability (↑)80.386.084.271.079.679.771.484.4
Semanticity (plausibility) (↑)0.2740.2730.2630.2590.2600.2660.2530.255
Semanticity (agreement) (↑)0.4930.5190.4870.4330.4580.4400.4430.429
Semanticity confidence (↑)3.423.603.773.593.543.493.443.54

 

The choice of K in TopK might influence results more than model choice

To test the sensitivity of our findings to SAE sparsity, we retrained TopK SAEs for three models (ViT-B, DINOv2, SigLIP) across five values of K = [10, 19, 38, 76, 152]. Since repeating the human experiments for each setting would be prohibitively expensive, we used as a proxy feature locality (Hoyer sparsity), which is our strongest predictor of human interpretability. Locality remained largely stable across K for a model as reflected in the table below, and the ranking between models remained unchanged, suggesting that our results are not primarily driven by the choice of sparsity.

ModelK=10K=19K=38K=76K=152
ViT-B0.0950.0980.1020.1030.101
DINOv20.0550.0550.0560.0570.055
SigLIP0.0600.0620.0630.0640.065

 

Thank you again for your feedback. We hope to have successfully addressed your concerns and hence would respectfully ask for reconsideration of the review score.

Reviewer_TuZhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a framework for comparing how interpretable vision models are to humans, using two psychophysics protocols. Localizability asks participants to click where an SAE feature fires on a novel image; clicks are scored against RISE heatmaps with chance-anchored normalization. Semanticity asks for free-text descriptions, scored for CLIP plausibility and MPNet agreement. Across six ViTs (ViT-S/B, DINOv2, DINOv3, CLIP, SigLIP; 13,400 responses, 377 participants), the foundation models score below supervised ViT-B. Interpretability is not significantly correlated with ImageNet or ADE20K performance, but it tracks locality and coarse-grained THINGS alignment.

Critical strengths
  1. The Section 3.1 control is a contribution in itself. With median-ranked exemplars, the forced-choice protocol still gives 60.0% (ResNet-50) and 53.6% (ViT-B/32) against 50% chance, so raw scores from that protocol are not comparable across models.
  2. The human study is large and carefully run: pre-specified exclusions, SAE features instead of neurons, and HSIC and six-encoder checks. Both protocols also rank the models similarly, so I take the model ranking seriously.
Critical weaknesses
  1. The most important opportunity, I think, is to match the capability claim to what six models can show. All six performance correlations are negative (ρ = -0.48 to -0.77, lines 313 to 320). At n = 6, non-significance therefore cannot show that the gap "is not a capability tradeoff" (line 24). Meanwhile, r = 0.80 with a similar p = 0.059 is read as "highly similar" (lines 291 to 292). One model family at several scales would test the claim properly. So would validating MIS [1] against these six human-rated models and then scoring many more. Otherwise, the title and abstract should be softened.

  2. The protocol has not yet been given the control that Section 3.1 applies to prior work. Locality is the Hoyer sparsity of the same RISE maps that are shown as hints (line 216), used to score clicks (line 222), and used to crop images for plausibility (line 251). So ρ = 0.91 and 0.99 (lines 360 to 361) may partly reflect shared measurement. Clicking on the main object may also beat chance at different rates for different models. In addition, features are selected with an ImageNet-1k probe (lines 196 to 210), which is the label space the supervised ViTs were trained on. Three checks would help: a mismatched-explanation control, locality computed from the SAE's own patch activations, and DINOv2 with registers (the version without registers has high-norm background artifact tokens [2]). Together they would test whether "foundation-style pretraining itself" (lines 434 to 435) is the cause.

[1] Roland S. Zimmermann, David Klindt, and Wieland Brendel. "Measuring Per-Unit Interpretability at Scale Without Humans." NeurIPS, 2024.
[2] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. "Vision Transformers Need Registers." ICLR, 2024.

Questions for authors
  1. (Clarification) Line 393 reports agreement vs. THINGS as ρ = 0.85, p = 0.03, but Fig. 3 shows r = 0.58, p = 0.228. The panels in Figs. 2 and 3 also print r while the text says Spearman ρ. Spearman's ρ cannot equal 0.91 or 0.99 with six untied ranks. Which statistic and which values are correct?

  2. (Clarification) Which supervised ViT-B checkpoint (ImageNet-1k or ImageNet-21k pretraining) and which DINOv2 checkpoint were used?

I'm willing to raise my score if the rebuttal addresses both weaknesses and Question 1.

4 comments

Thank you for your thoughtful feedback. Below, we address your questions and concerns.

Capability claims from six models

We are working on extending our evaluation to additional models and will update the rebuttal and paper with the results when available. Regardless, we acknowledge your concern and will soften the claim from “is not a capability tradeoff” to “we do not find significant evidence of a capability tradeoff.”

Leveraging MIS to scale model evaluations

The MIS metric relies on a protocol that, as we show—and as the reviewer acknowledges above, produces scores that “are not comparable across models”. Thus, we do not consider it suitable for our purposes.

Control for our protocols

Thank you for the suggestion! We are working on a mismatched-explanation control to empirically validate our chance level and will update the rebuttal and paper with the results as soon as they are available.

Clicking on the main object may beat chance

Using ImageNet bounding boxes, we tested whether sampling a point within the bounding box results in above-chance scores. We find that, for all models, the average scores are below chance. We will include this control in the paper's appendix.

Studying DINOv2 with registers

Our selection of models already includes DINOv3, which also uses registers, and we find that its features are still generally less interpretable than those of the supervised ViT-B.

Locality

The importance maps used to score participants' clicks (query image) are different from those used as hints to explain the features and to compute locality scores (nine top-activating images). These images are shown in full to participants during the semanticity experiment. The cropping used for plausibility only occurs when evaluating their descriptions, and using full images instead does not change the results (Appendix K).

Clarifications

  1. We will correct the discrepancy in line 393, as well as the references to Spearman correlations. All reported correlations use Pearson's r.
  2. We use ViT-B pretrained on ImageNet-1k and the DINOv2-Base checkpoint released by Meta.

We hope these responses address your concerns and questions.

Thank you for the response. The softer wording and the bounding-box check help, but my main concerns are still open. The mismatched-explanation control and the additional models are still pending, and the ImageNet-1k feature-selection bias and DINOv2 with registers were not addressed. On Q1, please confirm which agreement value is correct. Line 393 seems to repeat the plausibility numbers, and if Fig. 3 (r = 0.58, p = 0.228) is right, the claim that both protocols share the same predictors does not hold for agreement. For now I will keep my score.

Thank you for responding to our initial rebuttal and for your constructive feedback. Below, we provide a response to the open concerns.

Adding more models

Following the reviewer's suggestion, we have expanded our evaluation from 6 to 8 models by adding ViT-L and SigLIP2. These additional results not only corroborate but strengthen our main findings:

  • ViT-B remains the most interpretable model overall, ranking first across all three measures, with only SigLIP2 matching its localizability score (at a similar scale).
  • Model capability is not significantly correlated with interpretability on any of our three measures.
  • Models with more local representations have more interpretable features: participants converge to similar descriptions (r=0.723, p<0.05), which are more plausible (r=0.842, p<0.01), and the features are easier to localize (r=0.876, p<0.001).
  • Models that organize their representations in a way that is aligned with human semantic organization have features that are easier to localize (THINGS: r=0.884, p<0.01; Levels: r=0.797, p<0.01).

 

SupervisedFoundation
ViT-SViT-BViT-LDINOv2DINOv3CLIPSigLIPSigLIP2
Localizability (↑)80.386.084.271.079.679.771.484.4
Semanticity (plausibility) (↑)0.2740.2730.2630.2590.2600.2660.2530.255
Semanticity (agreement) (↑)0.4930.5190.4870.4330.4580.4400.4430.429
Semanticity confidence (↑)3.423.603.773.593.543.493.443.54

 

Chance-level performance - control condition (mismatched-explanation)

We have carried out additional control experiments to estimate the empirical chance-level performance for our protocols:

  • Localizability: We have conducted a control experiment similar to Section 3.1, testing whether participants can perform above chance without information about the correct feature. For this experiment, we have evaluated two models with different interpretability levels (ViT-B and DINOv2, sitting at both ends), and have evaluated a subset of their features in a new study with 30 participants (450 responses per model). In the control condition with mismatched explanations, the participants’ performance is at chance for both models (ViT-B: 48.3%; DINOv2: 48.1%; chance: 50%). Hence, participants need information about the feature to localize it, further rejecting the hypothesis that “clicking on the main object may beat chance”.

  • Semanticity: To perform a control experiment regarding semanticity we did not need to carry out any new user study, as in this case there is no notion of a feature explained vs tested from the point of view of participants. In this case chance levels can be directly estimated by rescoring their descriptions against other features' images (plausibility) or descriptions (agreement). Chance levels vary slightly across models (plausibility: 0.193–0.205; agreement: 0.149–0.165), but chance-corrected scores preserve our main findings and largely preserve model rankings. Moving forward, we will report chance-corrected scores in the paper.

 

Plausibility

ModelChanceRealChance-corrected
ViT-S0.2040.2740.070
ViT-B0.2020.2730.071
ViT-L0.1930.2600.067
DINOv20.1950.2590.064
DINOv30.1940.2600.066
CLIP0.2050.2660.062
SigLIP0.1960.2530.057
SigLIP20.1990.2530.054

Agreement

ModelChanceRealChance-corrected
ViT-S0.1560.4930.337
ViT-B0.1540.5190.366
ViT-L0.1650.4740.309
DINOv20.1490.4330.284
DINOv30.1580.4580.300
CLIP0.1580.4400.282
SigLIP0.1620.4430.281
SigLIP20.1570.4380.281

 

One model family at several scales would test the claim properly

Following the reviewer's suggestion (thank you!), we added ViT-L to our evaluation, allowing us to compare three scales of the same supervised ViT family (S, B, L). Despite increasing capability (ViT-L > ViT-B > ViT-S), we find no consistent trend in the interpretability as per our three measures: localizability (B = L > S), plausibility (B = S > L), and agreement (B > S > L). These results provide further evidence that higher capability does not necessarily imply higher—or lower—interpretability.

ImageNet-1k feature-selection bias

Using ImageNet to study model interpretability beyond supervised models, including selecting features through an ImageNet probe, is common practice in the literature (e.g., DINOv2 in Fel et al., 2026). In our case, we use it to select similar features across models, which enables fair comparisons across models. We are unclear how this choice would systematically bias our results, especially given that we find that accuracy on ImageNet-1k (the ability of models to capture its "label space") is not predictive of their interpretability. It is also unclear what specific ablation the reviewer would expect to address this concern.

DINOv2 with registers

We are unclear what additional insight evaluating DINOv2 with registers would provide, especially given the substantial cost of evaluating another model through human experiments. If the concern is that registers could improve interpretability to the level of supervised models, we already evaluate DINOv3 with registers. While DINOv3 is more interpretable than DINOv2, it still falls short of the supervised ViT-B.

Shared predictor

We apologize for the confusion. The correct correlation for agreement is indeed r=0.58, p=0.228. Overall, our updated results (N=8 models) suggest that locality is a shared predictor of all three interpretability measures, while alignment with human semantic organization predicts only localizability. We will clarify this distinction in the paper.

 

Thank you for pointing this out. We hope to have addressed all the open concerns and we would be grateful if the reviewer would consider updating their review score accordingly. Thank you again for the valuable feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

General response

We apologize for our late response, as addressing the reviewers' concerns required five additional psychophysics experiments.

 

First, we would like to thank the reviewers for their detailed and constructive feedback and for highlighting several strengths of our work:

  • Novelty and relevance: The importance of human-centered interpretability evaluation (YNAJ, S1–S2), the model-dependent bias identified in existing protocols (TuZh, S1; 6rh9, S1), and the connection between interpretability and representational properties (6rh9, S3).

  • Rigor and soundness: The scale of our human study (13,400 responses, 377 participants), careful experimental design, and extensive robustness analyses (TuZh, S2; YNAJ, S3; 6rh9, S2).

  • Empirical findings: The consistent model rankings across protocols and the relationship between feature locality, human semantic alignment, and interpretability (TuZh, S2; 6rh9, S3).

Below, we summarize the main concerns raised by the reviewers and how we addressed them.

 

1. Generalizability of our findings (TuZh, W1; YNAJ, W3; 6rh9, W9)

We expanded our evaluation from 6 to 8 models, adding ViT-L and SigLIP2. These additional results not only corroborate but strengthen our main findings:

  • ViT-B remains the most interpretable model overall (localizability, plausibility, agreement).
  • We find no significant correlation between capability and interpretability, including across three scales of supervised ViTs (Small, Base, Large).
  • Locality correlates with the three interpretability measures (r=0.72–0.88, all p<0.05), and human semantic alignment correlates with localizability both on the THINGS and Levels datasets (THINGS: r=0.884; Levels: r=0.797, both p<0.01).

 

2. Empirical validation of our protocols (TuZh, W2; 6rh9, W1–W2)

We conducted additional control experiments to test whether participants can perform above chance without information about the feature of interest:

  • Localizability: We ran new experiments with ViT-B and DINOv2. With mismatched explanations (control condition), performance was near chance for both ViT-B (48.3%) and DINOv2 (48.1%), across 900 responses from 30 participants. We also tested whether simply clicking on the main object led to performance exceeding chance and found that it did not.
  • Semanticity: We estimated chance levels in all models by rescoring descriptions against mismatched features. Chance-corrected scores preserve our main findings and largely preserve model rankings.

 

3. Sensitivity to SAE choices (YNAJ, W6; 6rh9, W5)

We retrained SAEs for three models across five sparsity settings (K=10–152). Feature locality, our strongest predictor of interpretability, remained largely stable across K, suggesting that our results are robust to the choice of SAE sparsity.

 

4. Contextualization of our work against existing work in human-interpretability

Our study builds on prior human-interpretability work focused on model comparisons, including studies cited by the reviewers (6rh9, W3), while extending both the scale and experimental rigor of these evaluations. We evaluate 8 vision models with over 500 participants, using 2 complementary protocols and 3 measures of interpretability, compared with previous studies that have evaluated as few as 2 models [2] or included as few as 16 participants [1], often relying on a single measure. We also conduct control experiments and extensive robustness analyses to assess the reliability of our protocols, aiming to make cross-model comparisons more reliable.

 

5. Additional ablations and methodological clarifications

We also addressed the other concerns raised by the reviewers, including the choice of RISE and potential measurement circularity (TuZh, W2; YNAJ, W2/W4) and the scope of our claims, which we will refine in the revised paper (TuZh, W1/Q1; 6rh9, W4/W9). We will incorporate the additional results and clarifications into the revised manuscript.

 

We hope to have successfully addressed all the questions and concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.