On the Hidden Interaction Structure of Shapley Value Explanations

The Shapley value summarizes a feature's marginal contribution by averaging it over different contexts, but this scalar attribution does not reveal how the underlying effect varies with the surrounding features. We study the higher-order interaction structure compressed by this averaging. We derive an order-wise representation in which the ordinary Shapley value, at any order $k$, is expressed through lower-order empty-context effects and expected context-dependent order-$k$ effects. This makes explicit how higher-order interaction structure is aggregated into scalar Shapley summaries. For cardinal interaction indices, we further show that efficiency is equivalent to Shapley recovery under uniform member-wise allocation. We then ask when these scalar expectations adequately represent their underlying context-wise effects. We quantify the remaining variability by Shapley context variance. We show that conditional context variance decomposes into nonnegative squared contributions from higher-order interactions, and that full context variance implies a lower bound on the aggregate second moment of immediate next-order extensions. Thus, large context variance requires substantial higher-order effects, whereas small variance indicates that the scalar expectation closely represents the underlying context-wise effects. Analytical and empirical studies further show that similar scalar summaries can mask markedly different context dependence and that context variance identifies cases where higher-order inspection is informative.