acceptodds
Under review as a conference paper at ICLR 2027

Identifying Reliable and Unreliable Components of Shapley Values

Abstract

We propose a method to decompose Shapley values into reliable and unreliable components. First, we prove that, given a set of AND–OR interaction patterns modeled by the DNN, evenly allocating the utility of each interaction to the input units involved in the interaction yields the Shapley value. Moreover, interaction patterns can be categorized into reliable and unreliable interactions according to their transferability across different DNNs. In this way, we define the Shapley attributions allocated from reliable interaction patterns as the reliable Shapley components, which reflect the relatively browntrustworthy decision logic learned by the DNN, while the Shapley attributions allocated from unreliable interaction patterns are termed unreliable Shapley components. We prove that the decomposed reliable and unreliable Shapley components still satisfy the extended four Shapley axioms. brownExperiments demonstrate that the proposed reliable and unreliable Shapley components capture decision logic with markedly different representational qualities: (1) Reliable Shapley components are substantially more similar across LLMs and more robust to input perturbations than unreliable Shapley components. (2) Unreliable Shapley components tend to exhibit greater cancellation between positive and negative attributions, thereby contributing less to the LLM’s predictions. (3) Reliable Shapley components tend to concentrate more heavily on input units relevant to the prediction than unreliable Shapley components.

Primary area:
interpretability and explainable AI
Keywords:
Shapley Value, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PJF2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Shapley values can measure the contribution of individual input features to a model's prediction. This paper proposes to separate these contributions into reliable and unreliable components using AND-OR interactions between features. The authors express each Shapley value as the sum of contributions from interactions involving the corresponding feature. They prove that this reconstruction has a bounded approximation error. They then classify an interaction as reliable if it appears in both the original model and a reference model and has the same positive or negative effect on the prediction.

Critical strengths
  1. The reconstruction result provides a mathematical basis for separating attributions according to interaction properties.
  2. The empirical evaluation is extensive in terms of language models.
Critical weaknesses
  1. The method first represents a model's predictions using AND-OR interactions. However, Appendix F.1 states that the optimization used to extract these interactions can have multiple optimal solutions. These solutions may produce different interactions while representing the same model predictions. Since reliability is assigned to individual interactions, the resulting reliable Shapley values may depend on which solution is selected. The paper does not establish whether these components remain stable across different optimal solutions. The theoretical error bound only controls the sum of the reliable and unreliable components.
  2. The experiments do not establish whether cross-model agreement is responsible for the observed improvements. The authors show that reliable components are more stable and assign more importance to relevant tokens. However, Figure 5 also shows that reliable and unreliable interactions differ in the number of features they involve. This difference may partly explain the results. Comparing interactions with similar sizes and contribution magnitudes would help determine whether agreement across models provides an additional benefit.
Questions for authors
  1. Appendix F.1 acknowledges that the interaction-extraction problem may have multiple optimal solutions. Can these solutions produce different reliable and unreliable Shapley components for the same model? Could you evaluate how much the components change across different optimal solutions, or establish conditions under which they remain unchanged?
  2. Figure 5 shows that reliable and unreliable interactions have different distributions across interaction orders. Could you compare two methods for selecting interactions? Method A uses your proposed criterion. It selects interactions that also appear in a reference model with the same contribution sign. Method B does not use a reference model. It selects the same number of interactions while matching their orders and contribution magnitudes to those selected by Method A. Could you then compare the resulting Shapley components using the metrics already reported in the paper? This would help determine whether agreement across models provides benefits beyond simply selecting interactions with these characteristics.
Reviewer_N336OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper decomposes Shapley values into reliable and unreliable components by allocating AND–OR interaction utilities to input features. Reliability is defined by matching interaction type, feature subset, and sign with a reference model. Experiments cover 14 LLMs on WikiText and finance data, showing that the reliable components have greater cross-model and consistency perturbation stability. The method may be useful for assessing attribution explanations.

Critical strengths
  1. Separating attribution into components with different stability and relevance is a reasonable research question.

  2. The allocation formula correctly recovers the Shapley value of the logical model, with a bounded error relative to the original model. The experiments examine several relevant properties and use distinct reference models for the two targets in cross-model comparisons.

Critical weaknesses
  1. Appendix F.1 acknowledges that sparse extraction can have multiple global optima, but Algorithm 1 does not specify how one is selected. Even exact representations can differ: for binary inputs, z1 + z2 = (z1 and z2) + (z1 or z2). Both representations have two terms and the same interaction L_1 norm, but disjoint supports. Comparing identical functions using the same representation makes all terms reliable; using these different representations makes all terms unreliable. The Shapley values are unchanged, but the two components change. The paper does not establish whether its extraction procedure avoids such ambiguity or whether the reported components are stable across alternative optimal solutions.

  2. The stability and human-relevance results support the proposed reliability interpretation, but its dependence on the reference model and salience threshold remains untested. The specific target–reference assignments are also unclear. Comparisons across reference choices, together with baselines matched for interaction count, order, and magnitude, would help determine how much cross-model matching adds beyond selecting simpler or stronger interactions.

3.The subsequent removal of small interactions can introduce additional error. Please report the maximum absolute reconstruction error over all evaluated masked inputs after removing small interactions, so that the practical accuracy guaranteed by the 2ϵ attribution bound can be assessed.

Questions for authors
  1. Appendix B limits each analysis to ten selected words or phrases for computational reasons. What evidence supports extending the findings to attribution over full inputs? Please clarify how unselected tokens are handled and whether the main findings are robust to different selections.
Reviewer_jdFMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors argue that a single Shapley value cannot tell which part of an attribution reflects trustworthy model behaviour. So, they propose to decompose each Shapley value in a reliable and an unreliable component.

The paper has 2 main contributions: First, they build on the AND-OR interaction framework from previous studies, which approximates the output of a model with a small set of AND and OR interactions between input units. The authors show that a Shapley value can be rebuilt from these interactions the model encodes. They prove that if the utility of each interaction is hsared equally between units it involes, the share of each unit, in total recovers its Shapley value, up to a small error of 2 ϵ\epsilon.

Second, they use this view to split each Shapley value into a reliable and unreliable component. AN interaction is reliable if a second LLM uses it too. Otherwise it is unreliable. For each unit, adding up the shares from the reliable interaction gives its reliable component, and adding up its shares from unreliable interaction gives its unreliable component. This sum back to the Shapley value and satisfy the extended versions of the efficiency, symmetry, dummy and additivity axioms.

The experiments use 14 open source LLMs (from 1.5B to 14B params) on WikiText and AdaptLLM finance dataset. They analyze 10 input units per sample.

The findings are that compared with the unreliable components, the reliable cases are more similar between different LLMs, more stable when noise is added, are resposible for most of the predictions, and focused on words that human annotators marked as releveant.

Critical strengths

The paper has several strengths such as:

  1. The research question to ask about which part of a Shapley attribution can be trusted is well motivated.
  2. The theoretical part is clear and carefully presented. It connects the AND-OR interactions to Shapley values. It makes approximation errors explicit through /epsilon/epsilon-bounds. The authors provide full proofs.
  3. The decomposition is simple to follow and intuitive. Once the interactions are extracted, each Shapley value spits into two components, and using agreements with other LLMs as a signal of reliability is an interesting idea.
  4. 14 different LLMs of different sizes, two datasets (general and finance), and four complementary analyses that point in the same direction.
  5. Using distinct reference LLMs for the two target models in Sec. 3.1 avoids an obvious source of inflated similarity.
  6. The paper is clearly written, Fig. 1 is a helpful illustration, and hardware, software versions and runtime are reported.
Critical weaknesses
  1. I wonder if Theorem 1 is a direct application of existing results or not.

The AND utilities in Eq. (18) are the Harsanyi dividends of the component game vandv^{\mathrm{and}}, and the identity ϕi=∑T∋iΔ(T)/∣T∣\phi_i = \sum_{T \ni i} \Delta(T)/|T| is classical (Shapley, 1953; Harsanyi, 1963).

The OR utilities in Eq. (19) appear to be the Harsanyi dividends of the dual game w(S)=vor(N)−vor(N∖S)w(S) = v^{\mathrm{or}}(N) - v^{\mathrm{or}}(N \setminus S), and since the Shapley value is self-dual, i.e., ϕi(w)=ϕi(vor)\phi_i(w) = \phi_i(v^{\mathrm{or}}), the OR case would follow as well. If so, the theoretical contribution is incremental, and this connection should be acknowledged.

  1. The reliable/unreliable components are not compared to plain Shapley values. For instance, if plain Shapley values are similar between different LLMs, it is unclear what the reliable component adds.

  2. The reliable/unreliable label depend on the reference LLM. The reference LLM used it is not reported.

  3. No sensitivity analysis is provided regarding the threshold /tau/tau. No variance or significance test are reported.

  4. No case study is shown for a decomposition that exposes a misleading Shapley value. In the case studies the reliable component dominated the Shapley value (but the plain value is not shown for comparision) and is hard to see what the decomposion adds. Also, it is not shown how the 10 words are chosen. In fig. 4(a) the words like "fortification", "surrender", or "commander are not considered to be analyzed, and these words could be related to the target token "reinforcements". Also, no example show the same sentence in 2 different LLMs.

  5. There is no reproducibility statement or code release.

  6. Minor: Fig. 4(c) is labeled Phi-4 in the plot but Llama-3-8B in the text. The “AI Flow” paragraph in App. D seems unrelated to the paper. The commercial product SymTrust citation in App. B seems unrelated too.

Questions for authors
  1. Could you clarify how Theorem 1 relates to the Harsanyi-dividend representation of the Shapley value? So, are the AND utilities the dividends of vandv^{and}? Does the OR case follow from the self-duality of the Shapley value?
  2. Could you report the same metrics for the plain Shapley value, so readers can compare and see what the decomposition adds?
  3. Can you show cases where the unreliable component is large for some unit, for instance, and the plain Shapley value would be misleading and the decomposition corrects it?
  4. How does the reliable/unreliable split change with a different reference LLM or threshold /tau/tau?
  5. What are teh reference LLMs used in sections 3.1-3.4? How were the 14 pairs in Table 1 selected, among all possible combinations of the considered LLMs?
  6. How are the 10 input units selected per sample?
  7. Is there any limitations of the method? How would it scale to longer inputs? Would approximate interaction estimators preserve the reliable/unreliable split?
  8. Do you plan to release the code so we can reproduce the results? or enough details to make it reproducible?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.