- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies a limitation of Shapley-style feature attribution in the presence of contextual distractors. Because Shapley values and other semivalues average marginal contributions over contexts, they may assign substantial positive attribution to such features despite their harmful contribution in strong contexts. The paper proposes the least core as a complementary attribution principle because it emphasizes coalitional stability rather than average marginal contribution.
The main theoretical contribution is a bound on the semivalue-least-core attribution gap under an average marginal monotonicity assumption. The authors further show that placing more semivalue weight on larger contexts decreases the signed gap.
The empirical validation has two main parts. First, in a controlled dog/cat/bird experiment, an artificial red marker is correlated with dogs and introduced into a subset of cat images. The marker becomes beneficial in weak contexts but harmful for the cat logit when sufficient cat evidence is present. Shapley generally gives the marker greater attribution than the least core. Second, the authors construct naturally occurring contextual distractors from Salient ImageNet.
Critical strengths
The strongest part for me is that the paper does more than empirically show another case where Shapley explanations look questionable. It formally defines a specific phenomenon and connects the Shapley/semivalue behavior to its averaging over context sizes. Theorem 2 and the context-weighting result make this mechanism much more concrete. This is consequential because it provides an explanation for when and why the two attribution principles diverge, rather than only demonstrating disagreement between attribution maps.
The marker experiment is simple but effective. The marker contribution decreases with stronger surrounding context, and shifting semivalue mass toward larger contexts systematically decreases its attribution relative to the least core, exactly matching the theoretical prediction. This makes the paper's central mechanism convincing rather than relying only on qualitative heat maps.
Critical weaknesses
The natural-image evaluation is partly conditioned on the phenomenon the paper wants to demonstrate. This is my main concern. The natural-image pipeline first searches for regions satisfying a contextual sign reversal, and the subsequent analysis shows that these selected contextual distractors tend to receive more attribution from Shapley than Core. The authors are transparent about the selection procedure and include random-SAM controls, which helps. However, it is still difficult to interpret this experiment as evidence that the proposed approach is broadly better at identifying problematic/spurious features in natural images. It establishes behavior on a deliberately selected population of contextual distractors.
Practical least-core results depend on a sampled approximation whose robustness is not sufficiently established. The theoretical object considers all coalitions, whereas experiments approximate the least core with randomly sampled coalition constraints. The natural-image experiments additionally choose an egalitarian solution from the resulting sampled least core. I think this creates an important practical question: how sensitive are the resulting attribution differences to the coalition budget, random seed, and selector?
Questions for authors
-
How robust are the least-core attribution results to the approximation used in practice? In particular, could the authors report sensitivity to the number of sampled coalition constraints and different random seeds? Since the theoretical least core considers all coalitions, I would like to understand whether the observed Shapley-Core separation remains stable when the sampling budget changes substantially.
-
How should the natural-image experiment be interpreted given that contextual distractors are selected using the same context-dependent sign reversal that motivates the expected Shapley–Core separation? Could the authors provide an evaluation that does not condition the test population directly on this property, for example using independently annotated spurious regions and then measuring how often each attribution method identifies/downweights them?
Response to Reviewer NvDu
We thank the reviewer for the positive assessment of the paper and for these constructive questions. We would like to clarify that the original sign-reversal experiment was intended to study attribution behavior within a deliberately identified contextual-distractor regime, rather than to establish that the least core is broadly superior on arbitrary natural images. Nevertheless, we agree that evaluating the broader, unfiltered population is important for understanding how common the phenomenon is and whether the observed separation extends beyond the selected cases. We also agree that the sampled least-core approximation should be tested for robustness.
1. Robustness of the sampled least core.
We varied the number of sampled coalition constraints over 1,600, 3,200, 6,000, and 12,000, using five random seeds for each budget:
Thus, the qualitative Shapley-Core separation remains stable across substantial changes in the sampling budget and across random seeds. As a complementary check, on smaller instances where all coalitions can be enumerated, the mean ℓ2 error to the exact least-core solution decreases from 1.390 at 256 samples to 0.568 at 2048 samples.
Regarding the selector, the least core is set-valued and the theory is not tied to the egalitarian selector: the bound
x(D)≤ΔD(C)+ϵ∗
holds for every least-core allocation x. In the experiments, we use the egalitarian solution only to report a single attribution vector consistently. We do not claim empirical invariance across selectors, and we clarify this distinction in the revision.
2. Evaluation without sign-reversal conditioning.
To address the selection concern directly, we repeated the analysis on all 40,950 human-annotated Salient ImageNet image-feature candidates, without filtering for contextual sign reversal. These regions are independently annotated as spurious features; we then measure their model-dependent contextual effect.
We define CD+ by ΔD(C)<0 and CD- by ΔD(C)≥0. Overall, 19,902 (48.60%) candidates are CD+. Among the 38,001 correctly classified candidates, the prevalence remains 48.65%:
Importantly, even CD+ candidates that do not satisfy sign reversal exhibit a mean gap of +0.367, with Shapley > Core in 70% of cases. Hence, the observed separation is not restricted to the population selected by the stronger sign-reversal criterion.
We have added the dataset-wide evaluation to the main paper, with the detailed sampling and robustness analyses provided in the appendix.