acceptodds
Under review as a conference paper at ICLR 2027

Seeing Through Distractions: Context-Stable Attribution via the Least Core

Abstract

Shapley-value-based explanations are widely used for feature attribution in machine learning. However, in computer vision, a feature may reduce the target-class score when added to the complete genuine image, even when the predicted label remains unchanged. We formalize such features as _contextual distractors_ and investigate their attributions under semivalues and the least core. Under a weak average marginal monotonicity assumption, we derive quantitative semivalue–least-core gap bounds and sufficient conditions based on the probability of sampling smaller contexts. We further show that shifting semivalue weights toward larger contexts decreases the signed gap. Finally, controlled image-classification experiments and evaluations on Salient ImageNet provide strong empirical support for our theoretical findings.

Primary area:
interpretability and explainable AI
Keywords:
Feature Attribution, Explainable AI, Shapley Value, Least Core, Cooperative Game Theory, Spurious Correlations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NvDuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a limitation of Shapley-style feature attribution in the presence of contextual distractors. Because Shapley values and other semivalues average marginal contributions over contexts, they may assign substantial positive attribution to such features despite their harmful contribution in strong contexts. The paper proposes the least core as a complementary attribution principle because it emphasizes coalitional stability rather than average marginal contribution.

The main theoretical contribution is a bound on the semivalue-least-core attribution gap under an average marginal monotonicity assumption. The authors further show that placing more semivalue weight on larger contexts decreases the signed gap.

The empirical validation has two main parts. First, in a controlled dog/cat/bird experiment, an artificial red marker is correlated with dogs and introduced into a subset of cat images. The marker becomes beneficial in weak contexts but harmful for the cat logit when sufficient cat evidence is present. Shapley generally gives the marker greater attribution than the least core. Second, the authors construct naturally occurring contextual distractors from Salient ImageNet.

Critical strengths

The strongest part for me is that the paper does more than empirically show another case where Shapley explanations look questionable. It formally defines a specific phenomenon and connects the Shapley/semivalue behavior to its averaging over context sizes. Theorem 2 and the context-weighting result make this mechanism much more concrete. This is consequential because it provides an explanation for when and why the two attribution principles diverge, rather than only demonstrating disagreement between attribution maps.

The marker experiment is simple but effective. The marker contribution decreases with stronger surrounding context, and shifting semivalue mass toward larger contexts systematically decreases its attribution relative to the least core, exactly matching the theoretical prediction. This makes the paper's central mechanism convincing rather than relying only on qualitative heat maps.

Critical weaknesses

The natural-image evaluation is partly conditioned on the phenomenon the paper wants to demonstrate. This is my main concern. The natural-image pipeline first searches for regions satisfying a contextual sign reversal, and the subsequent analysis shows that these selected contextual distractors tend to receive more attribution from Shapley than Core. The authors are transparent about the selection procedure and include random-SAM controls, which helps. However, it is still difficult to interpret this experiment as evidence that the proposed approach is broadly better at identifying problematic/spurious features in natural images. It establishes behavior on a deliberately selected population of contextual distractors.

Practical least-core results depend on a sampled approximation whose robustness is not sufficiently established. The theoretical object considers all coalitions, whereas experiments approximate the least core with randomly sampled coalition constraints. The natural-image experiments additionally choose an egalitarian solution from the resulting sampled least core. I think this creates an important practical question: how sensitive are the resulting attribution differences to the coalition budget, random seed, and selector?

Questions for authors
  1. How robust are the least-core attribution results to the approximation used in practice? In particular, could the authors report sensitivity to the number of sampled coalition constraints and different random seeds? Since the theoretical least core considers all coalitions, I would like to understand whether the observed Shapley-Core separation remains stable when the sampling budget changes substantially.

  2. How should the natural-image experiment be interpreted given that contextual distractors are selected using the same context-dependent sign reversal that motivates the expected Shapley–Core separation? Could the authors provide an evaluation that does not condition the test population directly on this property, for example using independently annotated spurious regions and then measuring how often each attribution method identifies/downweights them?

1 comment
AuthorsOpenReview ↗

Response to Reviewer NvDu

We thank the reviewer for the positive assessment of the paper and for these constructive questions. We would like to clarify that the original sign-reversal experiment was intended to study attribution behavior within a deliberately identified contextual-distractor regime, rather than to establish that the least core is broadly superior on arbitrary natural images. Nevertheless, we agree that evaluating the broader, unfiltered population is important for understanding how common the phenomenon is and whether the observed separation extends beyond the selected cases. We also agree that the sampled least-core approximation should be tested for robustness.

1. Robustness of the sampled least core.
We varied the number of sampled coalition constraints over 1,600, 3,200, 6,000, and 12,000, using five random seeds for each budget:

Coalition budgetMean Shapley-Core gapGap > 0
1,600+0.33870%
3,200+0.36968%
6,000+0.33968%
12,000+0.36872%

Thus, the qualitative Shapley-Core separation remains stable across substantial changes in the sampling budget and across random seeds. As a complementary check, on smaller instances where all coalitions can be enumerated, the mean ℓ2\ell_2 error to the exact least-core solution decreases from 1.390 at 256 samples to 0.568 at 2048 samples.

Regarding the selector, the least core is set-valued and the theory is not tied to the egalitarian selector: the bound

x(D)≤ΔD(C)+ϵ∗x(D)\leq \Delta_D(C)+\epsilon^*

holds for every least-core allocation xx. In the experiments, we use the egalitarian solution only to report a single attribution vector consistently. We do not claim empirical invariance across selectors, and we clarify this distinction in the revision.

2. Evaluation without sign-reversal conditioning.
To address the selection concern directly, we repeated the analysis on all 40,950 human-annotated Salient ImageNet image-feature candidates, without filtering for contextual sign reversal. These regions are independently annotated as spurious features; we then measure their model-dependent contextual effect.

We define CD+ by ΔD(C)<0\Delta_D(C)<0 and CD- by ΔD(C)≥0\Delta_D(C)\geq 0. Overall, 19,902 (48.60%) candidates are CD+. Among the 38,001 correctly classified candidates, the prevalence remains 48.65%:

RegionPrevalenceMean Shapley-Core gapShapley > Core
CD+18,489 (48.65%)+0.41973.8%
CD-19,512 (51.35%)+0.19058.3%

Importantly, even CD+ candidates that do not satisfy sign reversal exhibit a mean gap of +0.367, with Shapley > Core in 70% of cases. Hence, the observed separation is not restricted to the population selected by the stronger sign-reversal criterion.

We have added the dataset-wide evaluation to the main paper, with the detailed sampling and robustness analyses provided in the appendix.

Reviewer_zjvrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper looks at whether Shapley-style feature attributions can overemphasize features that help predict a class in isolation but actually hurt the prediction when the rest of the image is already informative. The authors call these contextual distractors and compare semivalue attributions with least-core attributions. They provide a theoretical gap bound and evaluate idea on a synthetic marker-bias task and Salient ImageNet.

Critical strengths

The main question is relevant. Averaging-based explanations may not reflect a feature’s actual effect in the full-image context, and this is genuinely interesting problem. The controlled marker experiment clearly illustrates the intended context-dependent behavior. The Salient ImageNet experiments suggest the idea may extend beyond synthetic settings, which is promising.

Critical weaknesses

The paper is hard to follow. The central idea is intuitive, but the writing is overly formal and the flow is not smooth. The mathematical presentation feels heavier than necessary. There is a lot of notation introduced before the reader gets a clear intuitive understanding of the main point. The main theorem is difficult to parse and would benefit from a simple worked example or clearer plain-language explanation.

The experiments are limited. The controlled experiment uses one synthetic marker and a small number of images. The Salient ImageNet evaluation is more relevant, but the distractor regions are selected using a sign-reversal criterion that already matches the paper’s hypothesis, making the comparison somewhat circular. The paper also does not discuss when the least core itself may produce misleading explanations.

Practical concerns are not addressed enough, including runtime, scalability, sensitivity to the number of sampled coalitions, and the effect of the masking baseline. The comparison is mostly limited to Shapley and Banzhaf. Standard vision attribution methods such as integrated gradients, or occlusion-based methods are not sufficiently included.

Questions for authors

Questions:

  • Can the authors explain the main theorem more intuitively, ideally with a simple example?
  • How sensitive are the results to the masking baseline?
  • How were the Salient ImageNet distractors selected, and could this selection bias the comparison?
  • Why are common vision attribution baselines not included in main evaluation?
1 comment
AuthorsOpenReview ↗

Response to Reviewer zjvr

We thank the reviewer for these comments, especially regarding presentation. At a high level, our paper first identifies the contextual-distractor phenomenon, then characterizes when semivalue averaging and coalition-stability principles can assign systematically different credit, and finally tests this mechanism in controlled and natural-image settings. We have revised the discussion around the contextual-distractor definition to make this intuition more explicit before the formal theorem.

1. Intuition behind the main theorem.
Let CC denote the genuine image context and DD the distractor. For any least-core allocation xx, efficiency together with the least-core constraint on CC gives

x(D)≤ΔD(C)+ϵ∗.x(D)\leq \Delta_D(C)+\epsilon^*.

Thus, the least core directly constrains how much credit DD can receive based on its effect in the complete genuine context. In contrast, a semivalue such as Shapley averages ΔD(T)\Delta_D(T) over many partial contexts T⊆CT\subseteq C. Hence, if DD appears useful in partial contexts but becomes less useful or harmful as genuine context grows, the two attribution principles can systematically assign different credit. The formal theorem gives sufficient conditions under which this gap is guaranteed.

2. Salient ImageNet selection and possible selection bias.
We agree that restricting evaluation to examples satisfying the stronger sign-reversal criterion could make the original comparison appear biased toward our hypothesis. We therefore repeated the analysis on all 40,950 human-annotated Salient ImageNet image-feature candidates, without sign-reversal filtering.

We denote candidates satisfying ΔD(C)<0\Delta_D(C)<0 as CD+ (contextual distractors), and the remaining candidates satisfying ΔD(C)≥0\Delta_D(C)\geq 0 as CD-. Overall, 19,902 (48.60%) candidates are CD+. Restricting to the 38,001 correctly classified candidates gives essentially the same prevalence, 48.65%.

For correctly classified candidates:

RegionPrevalenceMean Shapley-Core gapShapley > Core
CD+18,489 (48.65%)+0.41973.8%
CD-19,512 (51.35%)+0.19058.3%

Importantly, CD+ candidates without sign reversal still exhibit a mean gap of +0.367, with Shapley > Core in 70% of cases. Thus, the observed separation is not restricted to examples selected by the stronger sign-reversal criterion.

3. Sensitivity to the masking choice.
We repeated the comparison on the same 100 contextual-distractor images using mean-color masking, black masking, and Gaussian blur:

Masking choiceMean Shapley-Core gapGap > 0
Mean color+0.44878%
Black+0.27162%
Gaussian blur+0.28770%

The mean gap remains positive under all three choices, although its magnitude changes. We therefore do not claim invariance to masking: different masking strategies induce different coalition games. The experiment instead shows that the qualitative separation is not specific to the mean-color choice used in the main experiments.

4. Standard vision attribution baselines.
We have added Integrated Gradients and Occlusion to the direct attribution comparison:

ModelShapleyIGOcclusionLeast Core
ResNet-502.3382.5070.0101.262
ViT-B/162.1773.1391.5621.645

These values are the mean attribution assigned to the annotated distractor region. We additionally evaluate Shapley, least core, Integrated Gradients, Occlusion, Grad-CAM, and Grad-CAM++ using insertion/deletion metrics that are independent of the least-core objective. The least core achieves the highest insertion AUC on both ImageNet and Salient ImageNet, while Shapley achieves the lowest deletion AUC on both datasets. We therefore view the methods as exhibiting complementary strengths rather than claiming that the least core is universally superior.

5. Sampling, scalability, and limitations of the least core.
Exact least-core computation is exponential in the number of coalitions, so our image experiments use sampled constraints. Across sampling budgets and random seeds, the qualitative Shapley-Core separation remains stable; on smaller fully enumerable instances, the mean ℓ2\ell_2 error to the exact least-core solution decreases from 1.390 at 256 samples to 0.568 at 2048 samples.

We also emphasize that the least core is not a universally correct explanation: it optimizes coalition stability and remains dependent on the coalition game induced by masking. The insertion/deletion results further show complementary strengths across attribution methods.

The main paper now makes the contextual-distractor intuition more explicit and includes the dataset-wide Salient ImageNet analysis; detailed masking, sampling, baseline, and insertion/deletion results are in the appendix.

Reviewer_w3wCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies contextual distractors, i.e., features whose marginal contributions vary across contexts and may become harmful in the complete image context. The paper formalizes this phenomenon in a cooperative-game framework and analyzes the attribution gap between semivalues and the least core. The authors derive quantitative bounds on the semivalue–least-core gap and characterize how the gap changes with different context-size weighting schemes. The theoretical findings are further examined through controlled marker-bias experiments and experiments on selected natural-image regions from Salient ImageNet.

Critical strengths
  1. Clear problem formulation. The paper provides a precise mathematical definition of contextual distractors, enabling systematic theoretical analysis.

  2. Theoretical analysis. The paper derives quantitative bounds on the attribution gap between semivalues and the least core, and analyzes the effects of context weighting.

  3. Empirical validation. The synthetic marker-bias experiments provide an intuitive illustration of the theoretical phenomenon, and the experiments with both ResNet and ViT architectures show that the predicted attribution gap can indeed arise in practice.

Critical weaknesses
  1. The practical significance is not sufficiently established. Semivalues and the least core follow different attribution principles, so their disagreement does not necessarily indicate that semivalue attribution is incorrect or that least-core attribution is better. The paper lacks an independent criterion for explanation quality to justify interpreting the observed gap as Shapley “over-crediting.”

  2. The prevalence of this phenomenon is unclear. In the natural-image experiments, the paper only keeps regions that already show the expected context-dependent behavior. Could the authors report how many candidate regions were examined before this filtering, and what fraction actually exhibit this behavior? This would help clarify whether contextual distractors are a common phenomenon or only occur in a relatively small subset of selected examples.

  3. The evaluation is not independent of the least-core objective. The paper uses the worst coalition deficit ϵ\epsilon to compare explanation methods, but minimizing this quantity is exactly the objective of the least-core method itself. Therefore, it is expected by construction that the least core achieves a lower ϵ\epsilon than Shapley or other methods. This result does not provide independent evidence that the least-core explanation is actually better.

  4. The applicability of the main theoretical assumption is unclear. The theoretical results require the distractor’s average contribution to decrease monotonically as more context is added. It would be useful to report how often this assumption actually holds in real data and whether the conclusions remain valid when it is violated.

Questions for authors
  1. How can the authors demonstrate that the least-core attribution is more meaningful, rather than simply different from semivalue attribution?

  2. How common are contextual distractors in real-world data?

  3. Can the authors evaluate explanation quality using a metric that is independent of the least-core objective?

  4. How broadly does the average marginal monotonicity assumption hold in practice, and what happens when it is violated?

1 comment
AuthorsOpenReview ↗

Response to Reviewer w3wC

We thank the reviewer for these important comments. We agree that disagreement between Shapley and the least core, by itself, does not establish that one explanation method is better than the other. We would also like to clarify that our paper does not make such a general claim. Rather, we study a specific contextual-distractor regime and characterize why the two attribution principles can systematically diverge there.

1. Meaningfulness of the Shapley--least-core gap.
We would like to clarify that our goal is not to argue that the least core is universally superior. We focus on a specific regime in which its stability-based viewpoint may be particularly useful: the model predicts the correct class, yet an annotated spurious feature lowers the logit of that same true class in the complete genuine context,

ΔD(C)=v(C∪D)−v(C)<0.\Delta_D(C)=v(C\cup D)-v(C)<0.

This harmfulness criterion is determined directly from the model response and is independent of either attribution method. Our theory characterizes why semivalue averaging and least-core stability can behave differently in precisely this regime.

2. How common is the phenomenon?
We now evaluate all 40,950 human-annotated Salient ImageNet image-feature candidates without sign-reversal filtering. We denote candidates satisfying ΔD(C)<0\Delta_D(C)<0 as CD+ (contextual distractors), and the remaining candidates satisfying ΔD(C)≥0\Delta_D(C)\geq 0 as CD-. Overall, 19,902 (48.60%) candidates are CD+. Restricting to the 38,001 correctly classified candidates gives essentially the same prevalence, 48.65%.

For the correctly classified population:

RegionPrevalenceMean Shapley-Core gapShapley > Core
CD+18,489 (48.65%)+0.41973.8%
CD-19,512 (51.35%)+0.19058.3%

Thus, the separation is substantially stronger in the contextual-distractor regime. Importantly, CD+ candidates without sign reversal still exhibit a mean gap of +0.367, with Shapley > Core in 70% of cases. Hence, the result is not restricted to examples selected by the stronger sign-reversal criterion.

3. Independent explanation-quality evaluation.
We agree that the worst coalition deficit is not an independent quality metric, since minimizing it is precisely the least-core objective. We therefore added insertion/deletion evaluation using Shapley, least core, Integrated Gradients, Occlusion, Grad-CAM, and Grad-CAM++:

MethodImageNet Ins. ↑ImageNet Del. ↓Salient Ins. ↑Salient Del. ↓
Shapley0.5080.1210.5440.178
IG0.3850.2660.4500.336
Occlusion0.4730.2180.5030.297
Grad-CAM0.3830.1830.4210.247
Grad-CAM++0.3710.2010.4210.262
Least Core0.5200.1900.5610.250

The least core achieves the highest insertion AUC on both datasets, while Shapley achieves the lowest deletion AUC. We therefore interpret the methods as having complementary strengths, rather than claiming universal superiority of the least core.

4. Average marginal monotonicity.
We evaluated all 40,950 candidates using 3,000 sampled contexts per candidate. Population-average marginal contributions decrease as genuine context grows; for CD+ candidates they move from positive in smaller contexts to negative near the full context, while CD- remains positive. This empirical pattern motivated average marginal monotonicity as a modeling abstraction.

We do not claim strict monotonicity for every individual example. More importantly, we clarify that the basic semivalue--least-core gap bound does not require this assumption. Average marginal monotonicity is only used for the simpler sufficient conditions and results concerning context weighting.

The dataset-wide results and context-size analysis are now included in the main paper; the complete insertion/deletion comparison and additional experimental details are provided in the appendix.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

Common response to all reviewers.

We thank the reviewers for the constructive feedback. The main questions concern the interpretation of the Shapley--least-core gap, the prevalence of contextual distractors, and the robustness of the empirical results.

Our goal is not to argue that the least core is universally superior. We study a specific regime where its stability-based viewpoint may be particularly useful, i.e., in the situation where the model predicts the correct class, yet a contextual feature lowers the logit of that same true class in the complete genuine context,

ΔD(C)=v(C∪D)−v(C)<0.\Delta_D(C)=v(C\cup D)-v(C)<0.

This criterion depends only on the model response, not on either attribution method. Our theory characterizes why semivalue averaging and least-core stability can assign systematically different credit in this regime.

To address the reviewers' concerns, we performed several new experiments:

  • Dataset-wide prevalence and attribution gap. We evaluated all 40,950 human-annotated Salient ImageNet image-feature candidates without sign-reversal filtering. We denote candidates satisfying ΔD(C)<0\Delta_D(C)<0 as CD+ (contextual distractors), and the remaining candidates satisfying ΔD(C)≥0\Delta_D(C)\geq 0 as CD-. Overall, 48.60% are CD+, and the prevalence remains 48.65% among correctly classified images. For correctly classified CD+ candidates, Shapley assigns greater attribution than the least core in 73.8% of cases, with a mean gap of +0.419. CD+ candidates without sign reversal show the same qualitative separation.

  • How the contextual effect arises. Across all 40,950 candidates, average marginal contributions decrease as genuine context grows. CD+ regions can appear helpful in partial contexts but become harmful near the full context, while CD- regions remain positive. This directly illustrates the mechanism motivating our analysis.

  • Independent evaluation and vision baselines. We added comparisons with Integrated Gradients, Occlusion, Grad-CAM, and Grad-CAM++, together with insertion/deletion evaluation. On both ImageNet and Salient ImageNet, the least core achieves the highest insertion AUC, while Shapley achieves the lowest deletion AUC. Thus, the two methods exhibit complementary strengths under evaluation criteria independent of the least-core objective.

  • Robustness and approximation. The qualitative Shapley--least-core separation persists across mean-color, black, and blurred masking choices, and across coalition-sampling budgets and random seeds. Separately, on smaller instances where all coalitions can be enumerated, increasing the sampling budget reduces the approximation error of the sampled least-core attribution relative to the full-coalition solution.

We also clarify the role of average marginal monotonicity. This is an empirically motivated modeling assumption: the decreasing trend with context size observed in our Salient ImageNet experiments motivated this structural abstraction. Importantly, the basic gap bound does not require this assumption; it is only used for simpler sufficient conditions and results concerning context weighting.

Changes to the paper. In the main paper, we added the dataset-wide Salient ImageNet evaluation, the CD+/CD- comparison on correctly classified images, and the marginal-contribution analysis illustrating how the contextual effect changes with context size.

In the appendix, we provide the complete Salient ImageNet selection statistics and unfiltered evaluation, comparisons with standard vision attribution methods, the full independent insertion/deletion evaluation, masking sensitivity, coalition-sampling and random-seed robustness, exact-versus-sampled least-core comparisons, and additional experimental details.

For ease of review, all newly added or revised text in the main paper and appendix is highlighted in blue. Figures, tables, captions, and numerical entries are left in the standard formatting for readability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.