- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper fixes the "double-counting" problem in concept-based explanations, which happens when concept directions are correlated rather than orthogonal. The authors propose SPACE, a method that treats concepts as players in a game, where the value of a group of concepts is the length of the local gradient projected onto their combined subspace. This single setup does three things: measures how much of the gradient the vocabulary covers, assigns additive Shapley credit to each concept, and greedily selects a compact, non-redundant list of top concepts. By using a thin SVD step upfront, the method works in a small subspace of size (the number of concepts) instead of the huge feature space. Experiments on synthetic data and ImageNet show that SPACE avoids double-counting and handles duplicate concepts well.
Critical strengths
-
The core game value, , is easy to understand: it measures the maximum local change we can get by moving within the space spanned by a subset of concepts . It naturally guarantees non-negative Shapley values without needing fake baseline inputs or model retraining.
-
The paper clearly separates "splitting credit" (Shapley allocation) from "picking a diverse shortlist" (subset selection). The experiment with duplicate concepts nicely shows why just taking the top Shapley concepts still keeps near-duplicates, whereas greedy selection picks distinct concepts.
-
The thin SVD preprocessing shrinks the problem down to an dimensional space. This avoids building large projection matrices, prevents numerical issues with Gram matrices, and only needs a single backward pass.
-
The authors are upfront about the method's limitations, openly discussing why the game is not submodular, how zero-sensitivity concepts can still get credit, and how noise in estimating concept vectors affects the results.
Critical weaknesses
-
Exact calculation is too slow for large concept sets. Shrinking the feature dimension does not avoid doing a matrix factorization for every single subset: Algorithm 1 still needs an SVD of a matrix for each coalition. Because of this, exact calculation scales as . As shown in Appendix B, exact runtime jumps from 3 seconds at to 100 seconds at . At (Appendix F), the method has to rely on random sampling, which takes 12 seconds per image and still leaves an 8% difference between runs. So, exact accounting only works for small vocabularies.
-
Greedy selection has clear theoretical and practical limits. Because the game is not submodular, greedy selection can miss concepts that look weak alone but work well together, as seen in the gap between greedy and optimal subsets in Table 2 (and Table 8). Also, when concepts are nearly identical, the theoretical lower bound () drops toward zero and becomes useless (e.g., guaranteeing only 26% on ImageNet when it actually gets 99%). Finally, among near-duplicate concepts, tiny noise in training the probes can flip which concept gets picked first, which then wipes out the scores of its duplicates.
-
"Helper" credit can be mistaken for direct influence. A concept with almost zero direct sensitivity () can still get a large Shapley score simply by helping cancel out noise and completing another concept's subspace. On the worst ImageNet example, such concepts get up to 31% of the total credit. While this makes sense mathematically, users might wrongly think that changing this concept directly changes the model's prediction.
-
Experiments are limited to one model and one type of concept. The real-world tests only look at a single layer of a ResNet50 on four ImageNet classes, using simple example-based linear probes ( in the main text and in Appendix F). The paper does not test modern architectures like ViTs or popular text-based CLIP concept banks, which often have hundreds of concepts and different correlation patterns.
Questions for authors
-
Building on your runtime numbers for , could you show a clearer curve of how sampling error and runtime trade off at ? How many samples do we typically need to get reliable concept rankings and low error?
-
How useful is the greedy theoretical guarantee when there are near-duplicate concepts? Since the eigenvalue-based bound is very loose on ImageNet (predicting 26% vs. 99% actual performance), can you separate how much of that gap comes from the loose math bound versus the actual submodularity ratio on your data?
-
In Appendix F, you check how often the #1 concept stays the same when retraining probes. How stable is the entire greedy shortlist (say, top 3 or top 5 concepts) across retrainings (e.g., using Jaccard similarity)? Would it help to also list the "suppressed near-duplicates" next to each selected concept so users know what was left out?
-
Figure 1 nicely shows the SPACE scores next to the raw derivatives . To stop users from confusing "subspace-helping credit" with direct influence, could your method automatically flag concepts that get high Shapley credit () despite having near-zero direct sensitivity ()?