acceptodds
Under review as a conference paper at ICLR 2027

SPACE: Shapley Projection Attribution for Concept Explanations

Abstract

Concept-based explanations connect a model's prediction to ideas people can name. Scored one at a time, they overcount: two concepts that point half the same way are each credited with the sensitivity they share, and the scores sum to no meaningful total. **SPACE** measures how much of the model's local gradient each group of concepts can express, and from that one game answers three questions. (1) How much of the model does the vocabulary reach? A coverage score, in closed form. (2) How does the captured sensitivity divide among the concepts? A Shapley allocation that is non-negative, sums exactly to the captured total, and credits shared sensitivity once. (3) Which few concepts should a reader keep? A subset selection on the same game, with a greedy guarantee, which the allocation does not solve. The game costs one gradient, one factorisation, and small per-coalition algebra, with no masked inputs, reference examples, or per-subset fits, at a per-coalition cost independent of the representation dimension. On ImageNet, a ResNet50 explained through activation coordinates with fifteen texture and colour concepts, the shares add up exactly and the coverage is stated first. Keeping the three concepts ranked highest, the allocation's three explain % of what the vocabulary can, against % for one-at-a-time scores. The gap holds across five refits of every concept and is confirmed by moving the network itself. Selection on the same game comes within one point of the best possible three. On a rendered benchmark with planted factors, the allocation recovers the planted factor as often as any rule. When a factor has several near-synonymous descriptions, selection finds both planted factors on every image, while every per-concept ranking, the allocation included, fills the shortlist with synonyms. SPACE turns any set of concept directions, however obtained, into one accounted explanation.

Primary area:
interpretability and explainable AI
Keywords:
concept-based explanations, Shapley value, attribution methods, explainable AI, model interpretability, local gradient decomposition

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kpnYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper fixes the "double-counting" problem in concept-based explanations, which happens when concept directions are correlated rather than orthogonal. The authors propose SPACE, a method that treats concepts as players in a game, where the value of a group of concepts is the length of the local gradient projected onto their combined subspace. This single setup does three things: measures how much of the gradient the vocabulary covers, assigns additive Shapley credit to each concept, and greedily selects a compact, non-redundant list of top concepts. By using a thin SVD step upfront, the method works in a small subspace of size KK (the number of concepts) instead of the huge feature space. Experiments on synthetic data and ImageNet show that SPACE avoids double-counting and handles duplicate concepts well.

Critical strengths
  1. The core game value, v(S)=∥PSg∥v(S)=\|P_Sg\|, is easy to understand: it measures the maximum local change we can get by moving within the space spanned by a subset of concepts SS. It naturally guarantees non-negative Shapley values without needing fake baseline inputs or model retraining.

  2. The paper clearly separates "splitting credit" (Shapley allocation) from "picking a diverse shortlist" (subset selection). The experiment with duplicate concepts nicely shows why just taking the top Shapley concepts still keeps near-duplicates, whereas greedy selection picks distinct concepts.

  3. The thin SVD preprocessing shrinks the problem down to an r≤Kr \le K dimensional space. This avoids building large projection matrices, prevents numerical issues with Gram matrices, and only needs a single backward pass.

  4. The authors are upfront about the method's limitations, openly discussing why the game is not submodular, how zero-sensitivity concepts can still get credit, and how noise in estimating concept vectors affects the results.

Critical weaknesses
  1. Exact calculation is too slow for large concept sets. Shrinking the feature dimension does not avoid doing a matrix factorization for every single subset: Algorithm 1 still needs an SVD of a ∣S∣×r|S|\times r matrix MSM_S for each coalition. Because of this, exact calculation scales as O(2KK3)O(2^K K^3). As shown in Appendix B, exact runtime jumps from 3 seconds at K=15K=15 to 100 seconds at K=20K=20. At K=52K=52 (Appendix F), the method has to rely on random sampling, which takes 12 seconds per image and still leaves an 8% difference between runs. So, exact accounting only works for small vocabularies.

  2. Greedy selection has clear theoretical and practical limits. Because the game is not submodular, greedy selection can miss concepts that look weak alone but work well together, as seen in the gap between greedy and optimal subsets in Table 2 (and Table 8). Also, when concepts are nearly identical, the theoretical lower bound (1−e−γm\sqrt{1-e^{-\gamma_m}}) drops toward zero and becomes useless (e.g., guaranteeing only 26% on ImageNet when it actually gets 99%). Finally, among near-duplicate concepts, tiny noise in training the probes can flip which concept gets picked first, which then wipes out the scores of its duplicates.

  3. "Helper" credit can be mistaken for direct influence. A concept with almost zero direct sensitivity (ak=⟨g,dk⟩≈0a_k=\langle g,d_k\rangle\approx0) can still get a large Shapley score simply by helping cancel out noise and completing another concept's subspace. On the worst ImageNet example, such concepts get up to 31% of the total credit. While this makes sense mathematically, users might wrongly think that changing this concept directly changes the model's prediction.

  4. Experiments are limited to one model and one type of concept. The real-world tests only look at a single layer of a ResNet50 on four ImageNet classes, using simple example-based linear probes (K=15K=15 in the main text and K=52K=52 in Appendix F). The paper does not test modern architectures like ViTs or popular text-based CLIP concept banks, which often have hundreds of concepts and different correlation patterns.

Questions for authors
  1. Building on your runtime numbers for K∈{8,15,20,52}K \in \{8, 15, 20, 52\}, could you show a clearer curve of how sampling error and runtime trade off at K∈{25,50,100}K \in \{25, 50, 100\}? How many samples do we typically need to get reliable concept rankings and low error?

  2. How useful is the greedy theoretical guarantee when there are near-duplicate concepts? Since the eigenvalue-based bound is very loose on ImageNet (predicting 26% vs. 99% actual performance), can you separate how much of that gap comes from the loose math bound versus the actual submodularity ratio on your data?

  3. In Appendix F, you check how often the #1 concept stays the same when retraining probes. How stable is the entire greedy shortlist (say, top 3 or top 5 concepts) across retrainings (e.g., using Jaccard similarity)? Would it help to also list the "suppressed near-duplicates" next to each selected concept so users know what was left out?

  4. Figure 1 nicely shows the SPACE scores next to the raw derivatives aka_k. To stop users from confusing "subspace-helping credit" with direct influence, could your method automatically flag concepts that get high Shapley credit (ϕk\phi_k) despite having near-zero direct sensitivity (∣ak∣|a_k|)?

Reviewer_wQ5cOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SPACE, a Shapley-based framework for concept attribution and selection. The method defines the value of a concept coalition as the norm of the model's output gradient projected onto the subspace spanned by the corresponding concept directions. Based on this value function, the authors use Shapley values to allocate sensitivity among concepts while accounting for overlapping directions. For concept selection, the paper considers both Shapley-based ranking and greedy forward selection to identify a small subset of concepts that captures a large portion of the joint sensitivity. The methods are evaluated on ImageNet and a controlled synthetic benchmark, with experiments comparing attribution and selection strategies, sensitivity coverage, and the effects of concept redundancy.

Critical strengths
  1. The use of Shapley values is intuitive and provides a reasonable way to account for overlapping concept directions. The formulation is straightforward and has clear mathematical properties.

  2. The paper considers both Shapley-based ranking and greedy selection, and compares them with independent sensitivity ranking and optimal selection. The experiments on ImageNet and the synthetic benchmark provide useful comparisons, particularly in showing how different methods handle redundant concepts.

Critical weaknesses
  1. The quality of the CAVs is not sufficiently evaluated. Each CAV is trained with only 120 positive and 120 negative examples, using DTD textures or synthetic color patches. These examples are considerably simpler than natural ImageNet images, but the paper does not report the classification accuracy of the learned CAVs on ImageNet or other natural images. Since the proposed method relies on these concept directions, it is unclear how reliably they represent the intended concepts in the target domain.

  2. The evaluation of explanation quality is limited. The selection methods are evaluated mainly using the same sensitivity coverage objective they are designed to optimize. While this shows that the methods can select concepts with high joint sensitivity, it does not necessarily demonstrate that the resulting explanations are more faithful. The synthetic experiments also show relatively weak agreement between Shapley attribution weights and the effects of actual semantic factor changes. This raises questions about whether the proposed attribution scores provide more meaningful explanations beyond the mathematical decomposition.

  3. The experiments are relatively small in terms of concept vocabulary size. The main experiments use only 15 concepts, with an additional experiment involving 52 concepts. With such a small number of concepts, it is not clear why selecting a small subset is necessary, as presenting all concept attribution scores may already be manageable. It would be useful to evaluate the method with a larger concept vocabulary, where redundancy and the need for selection become more significant. This would also help assess the scalability of the proposed approach.

Questions for authors

Please address my concerns in weaknesses.

Reviewer_cYcsOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the concept attribution problem and proposes SPACE, which quantifies the importance of defined concepts requiring a single backward pass. The authors argue that conventional concept attribution approaches mix contributions of overlapping concepts and address this limitation through a novel definition of the value function for the investigated cooperative game. The authors establish several properties of the resulting approach and later provide empirical evidence for its effectiveness through experiments on ImageNet.

Critical strengths
  • Coherent mathematical formulation. Different from conventional feature attribution, the paper introduces a new coalition value function for investigating concept attribution. Its definition is justified and allows SPACE to quantify the contributions of concepts without repeatedly querying the target model.
  • Properly established theoretical foundations. The analysis is insightful and settles several properties that support the interpretation and application of the method. This paper defines a novel and sufficiently justified concept attribution approach. While there is still space for improvements in terms of the presentation and experimental scope, I consider the methodological contribution and its theoretical support sufficient to justify a positive recommendation.
Critical weaknesses
  • Presentation and organization could be improved. The overall structure of the paper and narrative flow would benefit from further refinement. For example, the introduction section mixes too many different contents and fails to provide a full picture of the target research topic (e.g. the background information about concept-level explanations and how they are defined/derived). Some of the content in the introduction reads more like detailed related work or methodological discussions. Additionally, a smoother transition and logical connection between sentences and paragraphs will enhance the readability of this paper.
Questions for authors
  1. Lines 165-166 claim that "SPACE needs nothing else from ff". Given that ff is only the differentiable scalar head, how are the gradients acquired without knowing the model internals?
  2. What is DD, and how does it relate to the internal representation in a classifier?
  3. Section 4.6 refers to pairwise interactions; how does this part relate to other discussions, and how are its insights reflected by the experiments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.