acceptodds
Under review as a conference paper at ICLR 2027

Are Dictionary Learning Metrics for Neural Concepts Interchangeable Proxies for Human-Interpretability?

Abstract

Concept-based methods aim to make neural networks more interpretable by representing their behavior in terms of human-meaningful concepts. As learned concept representations have become more common, recent work has proposed reconstruction fidelity, sparsity, Representation Diversity and Stability metrics to evaluate their quality. However, it remains unclear whether such metrics are actually informative of how comprehensible learned concept spaces are to humans, a prerequisite for increasing neural network interpretability. We study this question empirically by adopting an activation-space factorization view of learned concept representations, following recent work that frames concept extraction as dictionary learning. We then test whether dictionary learning metrics of the resulting concept spaces are associated with human evaluation of those spaces through a study with four complementary tasks and five measures of concept comprehension, involving 351 participants and yielding 5,616 individual responses. Our results show that different metrics exhibit substantially different relationships to human evaluation across tasks, and that these relationships are robust to the influence of individual classes. While reconstruction-based and stability metrics show no consistent connection to human judgments, Gini Sparsity and Representation Diversity display associations with linguistic comprehension measures for the studied setting. Overall, standard dictionary-learning metrics do not provide sufficient evidence for the evaluation of human-comprehensible concept spaces and should not be treated as interchangeable proxies for interpretability. However, some of them can be used as candidate indicators thereof in a systematic, human-validated analysis.

Primary area:
datasets and benchmarks
Keywords:
Concept-based Models, Interpretability, Human Evaluation, User Study, Dictionary Learning Metrics, Neural Concepts

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_i7poOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether commonly used dictionary-learning metrics for learned neural concept spaces can serve as reliable proxies for human comprehensibility. The authors extract class-specific concept spaces from ResNet representations using NMF and evaluate them using six automatic metrics, including reconstruction fidelity, sparsity, representation diversity, and stability. These metrics are then compared against five human-centered measures obtained from four complementary tasks covering informativeness, understanding, and distinguishability. The human study involves 351 participants and 5,616 responses. The main finding is that the relationship between automatic metrics and human evaluation is highly metric- and task-dependent. Reconstruction-based metrics and stability show little consistent alignment with human judgments, while Gini Sparsity and Representation Diversity exhibit stronger associations with linguistic measures such as label consensus. Based on these results, the paper argues that dictionary-learning metrics should be treated as diagnostic indicators whose relevance needs human validation, rather than as universal proxies for interpretability.

Critical strengths
  1. The paper addresses an important and under-validated assumption in concept-based interpretability. Automatic structural metrics are widely used to compare learned concept spaces, yet it is unclear whether improvements in these metrics correspond to concepts that humans can actually understand. Directly testing this relationship is valuable, and the paper identifies a meaningful gap between representation-level evaluation and human interpretability.
  2. The human evaluation is reasonably broad and carefully designed. The study evaluates concept spaces through multiple complementary tasks rather than relying on a single subjective interpretability score. The use of 351 participants, several behavioral measures, leave-one-class-out analyses, and multiple-comparison corrections makes the empirical investigation substantially more convincing than a small qualitative user study. I also appreciate the authors' cautious framing of the proposed measures as diagnostic tools rather than a new benchmark.
Critical weaknesses
  1. Limited statistical support for the strongest positive conclusions. Although the human study contains a large number of participant responses, the core correlation analysis is conducted over only 12 concept spaces. This results in limited statistical power. Moreover, none of the 36 metric-measure associations reaches q<0.05after Benjamini-Hochberg correction, including the strongest Gini Sparsity–Label Consensus association (q=0.055). I therefore think the positive findings should be framed more consistently as suggestive or exploratory rather than as validated indicators of human comprehensibility.
  2. Limited generalizability of the empirical setting. The evaluation is restricted to concept spaces extracted using a single backbone, NMF, a fixed K=5, and 12 image classes. Since the paper draws broader conclusions about dictionary-learning metrics for learned concept spaces, it remains unclear whether the observed relationships persist across different concept extraction methods, concept granularities, architectures, or datasets. A sensitivity analysis over K, or an additional extraction method, would substantially strengthen the claims.
Questions for authors
  1. Generalization across concept extraction settings. How sensitive are the reported metric-human correlations to the choice of K, backbone, and factorization method? Even a limited sensitivity analysis over several values of K, or an additional concept extraction method, would help clarify whether the observed relationships are specific to the current NMF setup.
  2. Multiple-comparison correction. Could the authors clarify which statistical tests they consider confirmatory? In Table 6, Label Consensus × Gini Sparsity has q=0.055when correcting over the 36 measure-metric comparisons, while the paper reports q=0.012when correcting globally over 102 tests. Please explain the rationale for these different correction families and why the larger global family produces a smaller adjusted value. I would also encourage the authors to explicitly distinguish corrected significant findings from suggestive trends in the main text.
  3. Reliability of the human measures. Could the authors provide uncertainty estimates or bootstrap confidence intervals for the human-evaluation measures and the resulting correlations? This would be particularly helpful for interpreting the mostly null results from the visual tasks and determining whether they reflect a genuine lack of association or limited measurement sensitivity.
Reviewer_LZ4KOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Concept-based methods aim to make neural network representations more interpretable. A lot of methods have been proposed to do so, but due to the bottleneck of human evaluation, alternative, automated measures have been proposed as a proxy to tell apart good from better methods. Nevertheless, there is little work validating how good a proxy those automated measures are for human interpretability. This paper fills this gap by conducting human evaluations and testing how automated proxies relate to the results. Overall, they find that only two of those metrics display associations with interpretability, yet they do not provide sufficient evidence that they can be used as interchangeable proxies for interpretability

Critical strengths
  • Important research question. As the number of concept-based methods increases, there is a need to find reliable proxies for their goodness, not only as methods in a vacuum, but as methods used by humans to understand models.
  • The data collected by the large-scale user study is a valuable resource for the community, especially given how it aims to cover multiple dimensions of interpretability with complementary measures
Critical weaknesses
  • My main concerns revolve around the human evaluations and its results.
    • First, there is a rich literature on human evaluation that is neither cited nor leveraged in this paper.
      • Doshi-Velez, F., & Kim, B. Towards a rigorous science of interpretable machine learning. arXiv (2017)
      • Fel, et al. "Craft: Concept recursive activation factorization for explainability." CVPR (2023). Cited for the method, not for the evaluations
      • Kim, et al. "HIVE: Evaluating the human interpretability of visual explanations." ECCV (2022).
      • Borowski, et al. "Exemplary natural images explain CNN activations better than state-of-the-art feature visualization." ICLR (2021).
      • Colin et al. "Capability ≠ Interpretability: Human Interpretability of Vision Foundation Models". arXiv (2026).
    • Soundness.
      • Informativeness is defined relatively vaguely, and it is not clear to me why click frequency is a good measure of it (in the sense that it could have a lot of confounds). The illustration of the task in Figure 1 has a click on both fish fin, yet it would seem like one concept to me. And as small as the crops from CRAFT are, it is still expected that a lot of irrelevant (to the concept) details will be present (which I would assume is strongly connected to the class).
      • Understanding is measured through ambiguity, which is intuitively correlated and not a bad measure per say, but I would argue it only functions as a lower bound. It might make more sense to call it ambiguity directly in the text. It is also unclear why in the visual task, only one image is shown to participant, it is hard to learn a rule (or concept) from one examplar.
      • Distinguishability is only measured subjectively. Self-report has been shown to be an unreliable and misleading measure in interpretability.
        • Hase, et al. "Evaluating explainable AI: Which algorithmic explanations help users predict model behavior?." ACM (2020).
        • Buçinca, et al. "Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems." Proceedings of the 25th international conference on intelligent user interfaces. 2020.
        • Colin et al. "Capability ≠ Interpretability: Human Interpretability of Vision Foundation Models". arXiv (2026).
Questions for authors
  • How did the authors validate that the difference in interpretability is not driven by class?
  • Can the authors confirm whether they corrected for the significance of their results? Given the number of tests they did, a threshold of p=0.05 does not seem appropriate.
  • Could the authors clarify how the proposed evaluation protocols are an improvement over the ones already existing in the literature for what they want to evaluate?
Reviewer_KusMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies if concepts learned by the interpretability method of Dictionary Learning corresponds with human-understandable or comprehensible concepts. Empirical activation-space factorization view of learned concept representations is used by them. The work uses human validation using images and text to measure alignment of learned concepts with human comprehension. The paper roughly concludes that the standard dictionary-learning metrics do not provide sufficient evidence for the evaluation of human-comprehensible concept spaces and should not be treated as interchangeable proxies for interpretability. In particular, the reconstruction-based stability metrics are insufficient, but two new metrics they used, namely, Gini Sparsity & Representation Diversity display associations with linguistic comprehension measures for the studied setting.

Critical strengths
  • A systematic study connecting automatically derived concepts with human comprehensible concepts using a study involving 351 individuals and over 5600 responses. The framework is nice though several variations can be imagined.
  • Study of concepts along the axes of Informativeness, Understandability, and Distinguishability seems an appropriate way to understand concepts.
Critical weaknesses

I have several comments and observations about the presented framework and work, however.

  • Paper hypothesizes that "reconstruction loss metrics are negatively correlated and the others are positively correlated" (L234). I don't see this being established explicitly. What are "reconstruction loss metrics"?
  • Using Imagenette subset is fine, but it seems arbitrary to add two randomly selected ImageNet classes to "improve statistical power". What is statistical power here?
  • It is stated that Tasks 3 & 4 use a "representative group" of patches {top$_6$}. What does this mean? How do you select them? Is the 6th from top being used for it? 6th out of how many?
  • How are the classes and images for the experiments selected? The DL is formed for the whole set of images and not separated by classes, right? What does top$_1$ for a class mean then?
  • Isn't the instruction to "Click on each separate thing and visual property you see” ambiguous, especially the visual property side?
  • More clicks could mean the patch contains many distinct things and hence is NOT very informative, isn’t it? If so, more clicks may mean less informative situation. Please explain.
  • It is said that "the most information rich concept space belongs to the church class". This is confusing. Patches of church class may contain multiple distinct patterns while patches of parachute contain fewer. Only patches are shown I assume and not the whole church/parachute. Church images could be busy with multiple physical structures whereas parachute images may only have simple regions and the sky. I don’t see how the number of clicks relate to informativeness of that concept.
  • For Eval Measure 2a: Isn't too restrictive to use only the top$_1$ patch for this? The other patches may also be relevant. Was some study done on this aspect?
  • For Eval Measure 2b: What images or patches are shown to them? How are they selected? Is that class wise? Top several patches? This information is not given in the paper.
  • Also, how is geometric similarity (L340) measured? In which space? How many patches or images are shown and how are they selected?
  • The analysis in Section 5.1 seems to go all over the place. What -- if any -- is the central message here? How good are Gini Sparsity and Representative Diversity?
  • The recommendations ion Sec 5.2 also don't seem sufficiently actionable. Given that the learned dictionaries do not correlate well with human comprehension, which concepts would/could?

Overall, I see the study creating a few interesting observations that perhaps have never been made before. However, I don't see it being "actionable". I would have expected the study to provide directions on how human comprehensible concepts can be elicited or extracted out of different parameters of the model. While the study seems comprehensive, it leaves a lot to be desired.

Questions for authors

See the part on weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.