- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This work introduces an auto-interpretability score (AIS) pipeline for evaluating features extracted from vision models, adapted from what was previously developed for language models. This was evaluated with a small-scale user study, comparing the performance of the proposed LLM-based AIS to an analogous human AIS protocol. The authors also benchmark 3 SAE models trained on 3 vision encoders over 3 datasets, evaluating multiple conventional SAE metrics in addition to their AIS. Results suggest these metrics are largely uncorrelated.
Critical strengths
I found the manuscript generally clear and well-written. In particular, Sections 1‒3 introduce the background and problem setting compellingly and motivate the research very well.
The experiments benchmarking common SAE metrics are interesting. The authors evaluated distinct types of SAE models on distinct vision encoders and distinct datasets. This comprehensive panel of results then allowed further analyses of relationships between the metrics. Even if discussion of the latter findings was fairly limited, the analysis invites further research on rigorous evaluation of interpretability methods.
Critical weaknesses
There are several elements of the visual AIS methodology that are not sufficiently justified or validated, even though this pipeline is one of the core paper contributions. For example, the explanation step takes only positive exemplars, although literature suggests that graded or contrastive panels can lead to more discriminative explanations (see e.g. Meissen et al. 2026, Automatic Interpretation of Visual Concepts). Only 5 instances are provided for explanation generation, increasing the risk of confounding with correlated concepts. It is also unclear why all patches of the top-average images are presented for explanation, instead of only top patches of images with highest max activation, which seems more intuitive. None of these design choices are evaluated against alternatives.
My other key concern is that the goal and implications of the user study are unclear, undermining confidence in the proposed method. The authors aimed to validate that their adapted vision AIS recipe "captures explanation quality as perceived by humans" (L474). However, the study has humans both generate explanations and simulate activations, instead of directly rating LLM explanations (as in Bills et al. 2023). The experimental design therefore seems to evaluate an abstract notion of interpretability as "feature is able to be encoded and decoded as language", regardless of the quality or human understanding of the LLM explanations. This is not necessarily an invalid question, but the framing is misleading. Further, the user study covers only two SAE features, with only 14 valid images in total, making the evidence seem anecdotal. Unless the focus of the study was specifically inter-observer variability, it would have been much preferable to increase coverage of SAE features as the primary unit of analysis, with fewer annotators. Lastly, in the LLM-vs-human AIS comparison, I expect Pearson correlation over ternary ordinal outcomes to be very noisy, and with such small sample size it's hard to draw any conclusions.
Questions for authors
- Please elaborate justifications for the numerous design choices in the proposed AIS pipeline, as exemplified above.
- Clarify the intended message of the user study and how exactly it corroborates AIS performance.
- Strongly consider running automated evaluations on intermediate steps of the pipeline to generate stronger empirical evidence. Ideally, supplement these with sensible ablations/baselines for individual components.
- (Not listed in critical weaknesses but would strengthen the paper:) Do you have hypotheses for the unexpected correlations observed in experiment #3?
6 comments
-
“[...] concerns regarding the goal and implications of the user study”
- We agree that our human-in-the-loop experiment was conducted on a small scale, both in terms of number of subjects and neuron examples. Indeed, the user study was not at the center of the contribution we aimed at. It was intended as a sanity check to validate AIS and we felt that even though it was not conducted on a large pool of users and neurons, this is still an important aspect: many studies in the field often avoid a validation of XAI metrics and methods in user studies altogether. We would also like to note that scaling up the experiment in its current design would be rather challenging, given the non-triviality and length of the task (even though we have sharply simplified it down to scores from 0 to 2). For instance, annotating the 10 images of one neuron in the simulation phase requires exactly 160 clicks. For two neurons, including the explanation phase, we observed an average completion time of around 45 minutes per user. At this complexity, crowdsourcing the experiment would just not be feasible, hence our decision of conducting a small-scale study for now, leaving a broader human investigation as a goal for future work. While we were open about the uncertainty of the observed correlation between the human AIS and LLM AIS, given the width of the CI, we admit that we may have overstated the significance of our findings in terms of validating the AIS metric. The results of the user study should be seen more as having a descriptive role and providing a first piece of empirical evidence that LLM and human annotators perform similarly on the AIS task. Nonetheless, we did not aim at checking whether users understand the LLM explanations (as in Bills et al. 2023). We were indeed interested in whether an LLM can infer concepts and predict activations about as well as a human can. We have now reformulated Section 5.1 to reflect more clearly what we intended with this study and how the results should be interpreted.
-
“[...] automated evaluations on intermediate steps of the pipeline [...]”
- Thank you for raising this issue. We have now extended our experiments with the following supplementary comparisons: i) More caption grid sizes to motivate our choice for 4 x 4 (Table 4); ii) A caption-free test, where the LLM Explainer/SImulator infers the explanation and predicts the activation scores directly on images (Table 4); iii) A random baseline to check dictionary statistics of an untrained Vanilla SAE (Table 2); iv) We have changed the ranking strategy in our AIS pipeline (now max-pooling instead of average-pooling) - this was another reviewer’s hint, see Appendix D.
-
“Do you have hypotheses for the unexpected correlations observed in experiment #3?”
- We see some possible explanations for these unexpected correlations, which we have not fully tested. For instance, MS and AIS measure different things. MS measures embedding-space similarity weighted by activation products, so it tracks density, OOD and dataset homogeneity. AIS measures how describable a feature is in text. This may be a reason for the very poor correlation between the two across SAE families and suggests that similar samples (in whatever embedding space) are not necessarily perceived as containing a common dominant concept (at least not by an LLM explainer in our setting). Furthermore, some dictionary metrics partly reflect artifacts: Coherence is a maximum statistic, that can be saturated by a single duplicated atom and the negative OOD-Coherence relation may reflect a trade-off between grounding atoms in clustered patch embeddings and making them near-orthogonal, which is consistent with work questioning orthogonality assumptions (Costa et al., 2025, Hindupur et al., 2025). We set a more comprehensive evaluation of these hypotheses as a goal for future work.
I thank the authors for the detailed responses and revisions.
1. Thank you for the clarification. I still believe explaining only 5 unique contexts risks confounding, as the concepts in each image's 16 patches are likely strongly correlated. (Related to my clarified suggestion in the next point.)
2. The results with the max-pooling alternative are interesting, though my actual query was about showing only the top patches to the LLM explainer, without needing to present all patches of every selected image—allowing you to include a larger number of diverse contexts. I realise my phrasing could have been clearer, so I will not specifically penalise the authors for this misunderstanding.
3. I appreciate the authors' candour regarding the limitations of the user study and the issues with the previous framing. With the revision, the narrower scope is much clearer and my Question 2 is addressed.
4. While the extended comparisons are highly valuable (especially the caption-free baseline), my concern about lacking intermediate evaluations largely remains. As the authors acknowledge on L509–513, errors can compound over captioning, explanation, and simulation, but no way to measure the reliability of each component is presented yet.
5. Thank you for sharing these conjectures. They sound plausible and I believe would make an interesting addition to the discussion in the paper.
Dear Reviewer p5Ta,
Thank you for reviewing our paper. Please let us know whether our responses below addressed all of your questions or whether there are further questions we can answer so that you feel confident in increasing your score.
“[...] only positive exemplars [...] only 5 instances provided for explanation”
“unclear why all patches of the top-average images are presented for explanation, instead of only top patches of images with highest max activation, which seems more intuitive.”