acceptodds
Under review as a conference paper at ICLR 2027

On the Hidden Interaction Structure of Shapley Value Explanations

Abstract

The Shapley value summarizes a feature's marginal contribution by averaging it over different contexts, but this scalar attribution does not reveal how the underlying effect varies with the surrounding features. We study the higher-order interaction structure compressed by this averaging. We derive an order-wise representation in which the ordinary Shapley value, at any order , is expressed through lower-order empty-context effects and expected context-dependent order- effects. This makes explicit how higher-order interaction structure is aggregated into scalar Shapley summaries. For cardinal interaction indices, we further show that efficiency is equivalent to Shapley recovery under uniform member-wise allocation. We then ask when these scalar expectations adequately represent their underlying context-wise effects. We quantify the remaining variability by Shapley context variance. We show that conditional context variance decomposes into nonnegative squared contributions from higher-order interactions, and that full context variance implies a lower bound on the aggregate second moment of immediate next-order extensions. Thus, large context variance requires substantial higher-order effects, whereas small variance indicates that the scalar expectation closely represents the underlying context-wise effects. Analytical and empirical studies further show that similar scalar summaries can mask markedly different context dependence and that context variance identifies cases where higher-order inspection is informative.

Primary area:
interpretability and explainable AI
Keywords:
Shapley value, Feature attribution, Feature interactions, Explainable AI, Higher-order interactions

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RS62OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies a phenomenon, I am now naming here "interaction collapse", in attribution scores. Meaning, that an attribution score (i.e. Shapley value) is just an average of higher-order interactions and hence collapses these interaction onto a scalar value. The paper then expresses the Shapley value (and Shapley Taylor interactions) as an average of context-dependent interaction effects and express the variance of the effects over multiple contexts as a measure of how representative the scalar summary is.

Critical strengths
  1. This paper studies an important area of interpretability and particularly Shapley-based explanations. Having a measure of telling practitioners when our explanations are good or good enough is valuable work. This also brings explanations more together with research fields such as uncertainty quantification (and representation), which I think is quite nice (though not at all connected, see weakness section).
  2. Therein, the paper has also, while underdeveloped, a nice solution that more or less comes for free if you do a permutation-based estimation of the Shapley values. I like having uncertainty values of this sort, but I do not think this paper does anything meaningful with it to make it worthwhile.
  3. Overall the writing of the paper is clear.
Critical weaknesses

While I think this work honestly touches on interesting, albeit well understood, concepts I see many weaknesses with it:

  1. Purely descriptive Results: This work reads very much like an essay on interpretability and not like a technical paper with a well-defined methodological contribution. The descriptive findings are very much well understood. A scalar average, is averaging away effects. Same for higher-order interactions. This is the main motiviation for studying higher-order interactions and/or regional explanations [1]. While this work here does mention these streams of research as related work. It does nothing to actually connect the results here to anything concretely. For me what would make this more interesting is not just a descriptive study of the "phenomenon", but a technical contribution of an adaptive algorithm that uses knowledge about uncertainties of some players and a) either samples more carefully around these coalitions or b) adequately expands the representation space of the explanation from attribution to higher-order interactions. This however, would require lots of theoretical work on describing what the new explanation is. I could see this turning in a pruned version of FaithSHAP or Shapley Taylor, but have not spent too much time thinking on the implications. This is not done in the work.
  2. Weak Empirical Evaluation: The empirical evaluation is very rudimentary and not at all adequate for this level of venue. The language example is a BERT sentiment classification model which seems to only be applied on four hand-picked movie reviews. The Tabular example is only california housing (8 features) and the synthetic experiment in section 5.1 instantiates the intro illustration with four hand-crafted coalition values.
  3. No Code: The submission does not contain any source code or anything for validating the results. I wanted to checkout if there is more than the mere 4 imdb sentence, but couldn't. While it might be "easy" nowadays to code-up things with agents, I think it's unacceptable to submit a research paper and not submit the means to reproduce its content.

References

Questions for authors

Q1: Can the uncertainty measure be used to prune what kind of coalitions one should need to evaluate to build better explanations?

2 comments

We thank the reviewer for recognizing the value of assessing when Shapley explanations are representative and for suggesting a concrete connection to adaptive interaction analysis. We agree that this practical connection deserves greater emphasis. Accordingly, we have revised the manuscript to clarify our theoretical contributions and substantially expanded Section 5 with a variance-guided inspection algorithm and quantitative studies on vision, language, and tabular models.

Scientific contribution. Prior work has documented that Shapley averaging can conceal opposing effects. Our contribution is to characterize the higher-order interaction structure behind this loss of information and establish when and why it warrants further analysis. Section 3 derives an order-wise representation showing how context-dependent higher-order effects are aggregated into ordinary Shapley values. Section 4 then establishes that large Shapley context variance implies substantial aggregate interaction energy among immediate higher-order extensions (Theorems 4.2–4.3). These results give the diagnostic a structural justification for prioritizing coalitions whose scalar summaries warrant closer inspection.

Variance-guided inspection (Q1). To address the reviewer's question of whether variance can guide coalition selection, we added Algorithm 1 in Section 5.4. The procedure starts with individual features, selects the top-KK candidate coalitions by context variance, evaluates their one-feature extensions, and repeats this greedy selection at successive interaction orders. In this way, the variance–energy relationship guides inspection toward promising branches of a search space that grows combinatorially with order.

Section 5.4 evaluates this procedure (K=3K=3, up to order four) on 30 ImageNet images and 30 IMDB reviews. On ImageNet, it inspects only 0.51% of possible fourth-order coalitions per input on average. The energy-ranked Top-10 interactions it discovers have 6.34× (ImageNet) and 2.83× (IMDB) higher interaction energy than a seeded random reference inspecting twice as many final-order coalitions. Their absolute Shapley–Taylor interaction scores are also 3.40× and 2.57× higher, respectively (see Section 5.4 and Appendix E.1 for experimental details).

Additional empirical evidence. We also extended the exact tabular evaluation (Section 5.2, Appendix E.2) from California Housing to Bike Sharing and Adult Census. Variance-based selection captures substantially more next-order interaction energy than ranking by scalar mean effects on California Housing (56.7% versus 34.5%) and Bike Sharing (68.7% versus 55.7%). The improvement is smaller on Adult Census (72.4% versus 69.4%), where large scalar effects already track context variance more closely. This pattern shows that variance-guided inspection is especially informative when scalar effect magnitudes do not reflect the underlying context dependence.

Connection to GRANITE. We appreciate the reference to GRANITE and share its motivation that feature interactions can complicate the interpretation of feature-level explanations. We have added it in the revised Introduction. GRANITE seeks regions where different influence operators agree, whereas our analysis examines the context dependence hidden within a fixed Shapley attribution. These questions are complementary. For example, consider v(S)=a1[1,2⊆S]−a1[1,3⊆S]v(S)=a\mathbf{1}[\\{1,2\\}\subseteq S]-a\mathbf{1}[\\{1,3\\}\subseteq S]. The pure, partial, and full influence scores for feature 1 all equal zero, yet its Shapley context variance is a2/3a^2/3. Thus, even agreement among influence operators can conceal substantial context-dependent effects.

Reproducibility. We provide an anonymized supplementary code package implementing context-dependent interaction effects, permutation-based estimators of their mean, variance, and energy, and variance-guided higher-order inspection. Two executable notebooks demonstrate these diagnostics and inspection procedures using analytical and synthetic examples, as well as cached effects from five IMDB sentences and five ImageNet images. Dataset and model sources, preprocessing details, and game specifications are also provided.

I thank the authors for their response. I already like the direction where the evaluation is going. I still think that this paper needs a stronger emphasize on building on your tool, as you already say "[...] These results give the diagnostic a structural justification for prioritizing coalitions whose scalar summaries warrant closer inspection." Your added Algorithm 1 is one way of doing something with this information, but I still think more needs to be done for making this an impactful explanation method.

I will increase my score to weak reject to reflect that this an interesting line of research, which just needs more to make it a full-fledge paper.

Notes:

  • It is very hard to take a look at the updates/changes in the paper as it is not highlighted what changed.
  • Code should really be submitted at submission time, sucht that one can actually review the work carefully.
Reviewer_4LtVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies what information scalar Shapley explanations lose about higher-order feature interactions. It first derives an order-wise representation of the Shapley value that connects to existing coalition interaction indices such as Shapley–Taylor, Faith-Shap, and n-Shapley. It then proposes the context variance as a diagnostic for when the scalar summary is representative, with theory guaranteeing that large variance implies substantial higher-order interaction energy. Experiments show that similar scalar attributions can hide very different interaction structure, and that ranking by this variance efficiently identifies where higher-order inspection is worthwhile.

Critical strengths
  1. The diagnostic is theoretically grounded rather than heuristic. The relationship between variance and interaction is supported by an exact nonnegative decomposition and a lower bound, and the order-wise representation also establishes connections with various existing metrics.
  2. The variance estimates come from the same permutation samples as standard Shapley estimation at essentially no extra cost.
Critical weaknesses
  1. The empirical scope is too narrow to support the paper's claims. The motivating example is image classification, yet no vision experiments are run. The evidence consists of one small tabular dataset with 8 features and only four IMDB sentences, with no statistics or confidence intervals in the language setting.
  2. Section 5.3 motivates a "scalable higher-order exploration" strategy, but demonstrates it only on a single illustrative sentence. There is no algorithm specification, no cost analysis, and no comparison against existing any-order interaction estimators under the same budget.
Questions for authors
  1. Do the authors have any results on image models or settings with substantially more features where enumeration is infeasible? How reliable are the variance estimates and resulting rankings under permutation sampling noise?
  2. Can the implied "scalable higher-order exploration" be stated as an actual algorithm and compared against an any-order estimator such as SHAP-IQ under the same evaluation budget?
2 comments

We thank the reviewer for recognizing the theoretical foundation of our variance diagnostic and for raising valuable questions about its practical use and empirical support. To address these concerns, we have substantially revised the manuscript, expanding the evaluation across vision, language, and tabular models and introducing a concrete higher-order inspection procedure to demonstrate the practical value of our theoretical results.

Variance-guided higher-order inspection (Q2). Theorem 4.3 connects high parent context variance to substantial aggregate interaction energy among immediate extensions, motivating a concrete higher-order inspection strategy. To address the reviewer's request, we added Algorithm 1 in Section 5.4. The procedure starts with individual features, selects the top-KK candidate coalitions by context variance, evaluates their one-feature extensions, and repeats this greedy selection at successive interaction orders. In this way, the variance–energy relationship guides inspection toward promising branches of a search space that grows combinatorially with order.

Evidence from image and language models (CW1/Q2). Section 5.4 evaluates this procedure (K=3K=3, up to order four) on 30 ImageNet images and 30 IMDB reviews. On ImageNet, using 25 image patches, it inspects only 0.51% of the 12,650 possible fourth-order coalitions per input on average. The energy-ranked Top-10 interactions it discovers have 6.34× (ImageNet) and 2.83× (IMDB) higher interaction energy than a seeded random reference inspecting twice as many final-order coalitions. Their absolute Shapley–Taylor interaction scores are also 3.40× and 2.57× higher, respectively (Figure 4). Figure 4(a) further illustrates interactions involving macaque faces, cassette-player controls, and a crane's head, revealing coherent image regions without imposing spatial constraints (see Section 5.4 and Appendix E.1 for experimental details). SHAP-IQ reuses model evaluations to estimate multiple interaction scores, whereas our approach uses context variance to prioritize which coalitions merit further inspection. Accordingly, our experiments assess whether variance-guided selection can identify coalitions with substantial higher-order interaction effects within a limited search space.

Broader diagnostic validation (CW1). We also extended the exact tabular evaluation (Section 5.2, Appendix E.2) from California Housing to Bike Sharing and Adult Census. Variance-based selection captures substantially more next-order interaction energy than ranking by scalar mean effects on California Housing (56.7% versus 34.5%) and Bike Sharing (68.7% versus 55.7%). The improvement is smaller on Adult Census (72.4% versus 69.4%), where large scalar effects already track context variance more closely. This pattern shows that variance-guided inspection is especially informative when scalar effect magnitudes do not reflect the underlying context dependence. Appendix E.2 also compares the variance ranking with SII, BII, and Faith-Shap rankings.

Estimation reliability (Q1). We added supporting evaluations in Appendix E.4. On three exactly evaluated tabular datasets, the MSE of permutation-based variance and energy estimates decreases approximately at a 1/M1/M rate (Figure 7). For ImageNet and IMDB, where exact evaluation is impractical, we assessed the reproducibility of variance rankings using independent permutation samples. Spearman rank correlations consistently increase with the sampling budget (Table 3).

Thank you for the responses. The new algorithm and experiments address several of my concerns. But I'm still unsure how the proposed method compares with alternatives under the same evaluation budget. So I keep my score.

Reviewer_gtYROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The article addresses the problem that standard Shapley values lose context dependent contribution in different orders by averaging all marginal attributions. In order to fix this they want to define a form of variance on the contribution of a feature depending on the context. The example would be that if the contribution of an important feature was canceled out by averaging, we could somewhat expect this feature to have non-negligable variance.

Critical strengths
  1. They identify an interesting way to overcome the problem that Shapley values face with averaging out effects.
  2. The article takes an mathematical way to presenting their argument as to why the variance can characterize the context interactions
  3. The examples at the end of the article are nice.
Critical weaknesses
  1. Equation (2) introduces intraction effect without really giving any intuition on why it's defined that way. The previous sentences give some intuition for the case where set C consists of only two elements but from my understanding this intuition does not readily extend to sets of larger cardinality. An example or a better description of what interaction effect is supposed to capture which would justify its definition would be nice. This seems to be mostly the theme of the presentation
  2. The biggest weakness in my opinion is the presentation. The results and formulations are given before giving any context on why we need them. For example, thm 3.2, is an identity of Shapley values. This fact, and its interepretation (k=1, k=n) is given after the statement. Also P_c, is defined within the thm statement itself. As a results, it is difficult to tell what is being claimed and why it matters.
  3. Phrases like “context-wise effects,” “scalar summaries,” “higher-order structure compressed by…” and “higher-order inspection may be informative” recur on nearly every page. They often stand in for saying something concrete. And also are not defined rigorously.
  4. Examples come too late. The attention example is the first thing in the article that attempts to give a better understanding of all the concepts introduced.
  5. the overall structure of the article makes sense if it was executed well however I found myself entirely lost and after multiple rereads could finally understand the reasoning behind the structure.
Questions for authors
  1. Am I wrong in my understanding that Thm 4.2 is an instance of the p-biased fourier (efron-stein) decomposition?
3 comments

We thank the reviewer for recognizing the soundness of our analysis and for the constructive comments on presentation. Our work connects an order-wise representation of what Shapley averaging aggregates, an interpretation of the resulting scalar summaries, and a variance-based diagnostic of their limitations. To make the common motivation behind these components clearer, we have substantially revised the Introduction to present them as a progression from understanding what is averaged to identifying when further interaction analysis is warranted.

Motivation and presentation (W1–W5). The revised Introduction now grounds this progression in Figure 1, which contrasts similar Shapley values with markedly different context-dependent effects. It also explicitly links the illustration to Section 5.1, where the same cases are instantiated using an analytical attention model, making this connection apparent early in the paper. We further clarified the development across Sections 2–4. Section 2.2 introduces higher-order discrete derivatives recursively before Equation (2); Section 3.2 explains the purpose and special cases of the order-kk representation and introduces the context distribution pCp_C before Theorem 3.2; and Section 4 connects scalar expected effects to the context variation and higher-order interaction structure they may conceal.

Relation to p-biased Fourier/Efron–Stein (Q1). Yes. Theorem 4.2 builds on the standard p-biased Fourier/Efron–Stein decomposition. A key challenge in our setting is that feature-inclusion indicators under the Shapley context distribution are dependent, preventing its direct application. We address this through a Bernoulli-mixture representation, where conditioning on the shared inclusion probability QC=qQ_C=q yields independent Bernoulli(qq) indicators. This allows us to apply the standard orthogonal decomposition and identify its coefficients with the conditional expected higher-order interaction effects μC∪R(q)\mu_{C\cup R}(q). We have clarified this connection and cited O'Donnell (2014) in Appendix C.3. Separately, Theorem 4.3 establishes a bound relating full Shapley context variance to aggregate next-order interaction energy, providing the theoretical basis for our diagnostic.

Reviewer_gtYROpenReview ↗

response to authors

I thank the authors for their response. I understand the context of thm 4.2 and 4.3 better but my other comments still hold. Thank you.

Thank you for your follow-up. We are glad that our clarification of Theorems 4.2 and 4.3 was helpful.

We would also like to emphasize that the revised manuscript has been uploaded to OpenReview, with substantial changes addressing your presentation concerns. In particular, we revised the Introduction to clarify our motivation and contributions, improved the exposition and organization of Sections 2–4, and introduced a concrete motivating example in Figure 1, highlighting its quantitative findings early in the Introduction and providing its analytical derivation in Section 5.1.

We hope these revisions help address your remaining concerns. If you find any aspects of the revised manuscript still insufficient, we would appreciate the opportunity to understand and address them more directly.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response for Rebuttal

We thank all reviewers for their constructive feedback. In response, we have substantially revised the manuscript with two main goals: clarifying our theoretical contributions and strengthening the empirical evidence for their practical significance.

First, we refined the motivation and theoretical exposition to better communicate how our order-wise representation reveals the higher-order interaction structure hidden in Shapley explanations, and how context variance provides a theoretically justified diagnostic for further analysis.

Second, we substantially expanded the evaluations to address the reviewers' concerns about empirical support and practical utility. These additions provide broader evidence for the diagnostic and demonstrate how it can guide the inspection of substantial higher-order interactions.

We provide detailed explanations of these revisions in our individual responses.

AuthorsOpenReview ↗

Final Remarks on the Revised Manuscript

We thank the reviewers for their feedback and engagement. In response to their concerns, we have substantially expanded the empirical evaluation and introduced a concrete variance-guided procedure for selective higher-order interaction analysis. We appreciate the reviewers' recognition of these improvements and their suggestions for further strengthening the evaluation and practical development of this work.

As the discussion concludes, we would also like to emphasize the central contribution of this work. Beyond the observation that Shapley averaging can conceal interaction effects, our study characterizes the underlying higher-order interaction structure and establishes a theoretical basis for diagnosing when and why scalar summaries leave substantial context-dependent effects unresolved. In particular, we connect context variance to higher-order interaction energy and demonstrate how this relationship can guide selective interaction analysis across different models and datasets.

We believe that this combination of structural understanding, theoretical justification, and demonstrated diagnostic utility constitutes a meaningful contribution to interpreting scalar Shapley explanations and provides a foundation for further methodological advances in interaction-based explanations.

We sincerely appreciate the reviewers' and AC's time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.