acceptodds
Under review as a conference paper at ICLR 2027

PullSHAP: Principled Attribution Pullback Through Learned Representations

Abstract

Learned representations create an attribution mismatch: predictions are made after transforming the original features, while explanations are often required in the original feature space. We study when Shapley attribution can be pulled back through a factorization . We first show that no generic local chain rule exists: two smooth representations can agree on neighborhoods of both the baseline and explicand, and hence have identical endpoint derivatives of every finite order, yet induce different composite input-feature Shapley values. Moreover, representation compression alone is insufficient; even a smooth scalar representation can realize any normalized coalition game on the relevant hybrid vertices. We then establish sufficient conditions for pullability. Mixed cross-feature curvature of the representation controls its deviation from an additive coalition surrogate, showing that nonlinearity alone need not cause representation-side coalition distortion. On this surrogate, a distinct obstruction is the discrepancy between Aumann–Shapley path allocation and Shapley–Shubik allocation. These results lead to \PullSHAP, which combines integrated representation contributions with integrated downstream sensitivity, together with an error decomposition separating representation distortion, downstream allocation mismatch, and numerical approximation. Controlled experiments confirm the predicted exact and failure regimes. Across 1,250 exact attribution instances nested within successful learned-representation runs, median relative- error is , with a non-negligible upper tail. Within dense MLP encoder tracks across three datasets, representation-side distortion is positively associated with exact attribution error. A separate exact-reference benchmark compares the differentiable approximation with established SHAP estimators while keeping black-box model-evaluation budgets distinct from gradient/Jacobian computation.

Primary area:
interpretability and explainable AI
Keywords:
Shapley values, feature attribution, learned representations, explainable AI, attribution pullback

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5NSDOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the theoretical conditions under which baseline shapley values calculated over an input feature space X\mathcal{X} can be recovered ("pulled back") through a factorized model F=f∘pF = f \circ p, where p:Rd→Rrp: \mathbb{R}^d \to \mathbb{R}^r is a learned intermediate representation and f:Rr→Rf: \mathbb{R}^r \to \mathbb{R} is a downstream predictor. The authors first prove two fundamental boundary results. To provide positive guarantees, the paper introduces PullSHAP, establishing that pullability is governed by two distinct obstructions, .i.e, representation-side coalition distortion and downstream allocation mismatch.

Critical strengths

The paper derives an error decomposition separating representation distortion, downstream allocation mismatch, and numerical quadrature error (Qj\mathcal{Q}_j), along with a conditional certificate template.

Critical weaknesses
  1. Limited novelty in the algorithmic formulation (PullSHAP-Fast). While the theoretical characterization of the pullback error is insightful, the practical algorithm proposed in Algorithm 1 (PULLSHAP-FAST) boils down to a decoupled, two-path numerical quadrature integration using Vector-Jacobian Products (VJPs). Mathematically, it simply computes ϕPull=Ax⊤g‾\phi^{\text{Pull}} = A_x^\top \overline{g}, where AxA_x is the integrated encoder Jacobian and g‾\overline{g} is the average downstream gradient. This is a natural, factorized extension of Integrated Gradients (IG). The distinction between decoupled integration (PullSHAP) and coupled integration (Composite IG) is heavily emphasized, but functionally, PULLSHAP-FAST remains a path-based heuristic gradient method that approximates SHAP without offering a fundamentally new algorithmic mechanism for handling non-additive, highly interacting deep representations.
  2. Limited scope and scalability of the experimental benchmark. The empirical evaluation relies heavily on severely low-dimensional and synthetic/reduced setups. Concretely, the exact-reference evaluation on real datasets (Breast Cancer and Adult) is restricted to 10-feature reduced models (d=10d=10) because exact SHAP requires exhaustive 2d2^d coalition enumerations. The full-dimensional versions (d=30d=30 for Breast Cancer, d=108d=108 for Adult) are omitted from the accuracy evaluation.The latent dimension is primarily fixed at r=4r=4.The practical SHAP estimator comparison in Section 6 (RQ4, Figure 3) evaluates only 20 explicands. As a result, it remains unverified whether PullSHAP scales accurately or maintains low error on high-dimensional, highly non-linear real-world representations (e.g., modern vision transformers, large language model latent states, or deep multi-layer MLPs with hundreds of features).
  3. Overly restrictive theoretical scope excluding modern deep architectures. The theoretical framework strongly depends on smoothness assumptions, and continuous differentiability for ff. This explicitly excludes piecewise linear activations (such as ReLU, LeakyReLU, or GELU approximations with sharp edges), decision tree ensembles, vector quantization layers, and discrete/attention-based representations. Given that a vast majority of modern deep learning architectures utilize non-smooth activation functions (like ReLU), the curvature bounds (Kp\mathcal{K}_p) and guarantees presented in the paper do not strictly apply to standard deep learning pipelines without smoothing modifications.
Questions for authors

Please refer to the weaknesses.

6 comments
AuthorsOpenReview ↗

Weakness 1 - Algorithmic Novelty Beyond Path Integration: PullSHAP-Gate Structured Coalition Correction

Weakness 1

Limited novelty in the algorithmic formulation (PullSHAP-Fast). While the theoretical characterization of the pullback error is insightful, the practical algorithm proposed in Algorithm 1 (PULLSHAP-FAST) boils down to a decoupled, two-path numerical quadrature integration using Vector-Jacobian Products (VJPs). Mathematically, it simply computes ϕPull=Ax⊤g‾{\phi }^{Pull}={A}_{x}^{\top }\overline{g}, where Ax{A}_{x} is the integrated encoder Jacobian and g‾\overline{g} is the average downstream gradient. This is a natural, factorized extension of Integrated Gradients (IG). The distinction between decoupled integration (PullSHAP) and coupled integration (Composite IG) is heavily emphasized, but functionally, PULLSHAP-FAST remains a path-based heuristic gradient method that approximates SHAP without offering a fundamentally new algorithmic mechanism for handling non-additive, highly interacting deep representations.

We agree that decoupled path quadrature and VJPs alone (PullSHAP-FAST) do not resolve discrete multi-feature interactions. In direct response, our revision introduces PullSHAP-Gate (Section 5.2, Theorem 8, Appendix H)—a fundamentally new, interaction-aware algorithmic mechanism that eliminates downstream allocation mismatch.

1. Downstream Allocation Obstruction (Theorem 4). In Section 5 (Theorem 4), our theory identifies an intrinsic limitation of pure path integration: the gap between continuous Aumann–Shapley path allocation and discrete Shapley–Shubik allocation on the representation surrogate h(S)=f(p(xS))h(S) = f(p(x_S)). For smooth surrogates, this mismatch satisfies: Ij(h)≤112∑i≠j(∥∂jiih∥∞+∥∂jjih∥∞).I_j(h) \le \frac{1}{12} \sum_{i \neq j} \left( \|\partial_{jii} h\|_\infty + \|\partial_{jji} h\|_\infty \right). No quadrature refinement can eliminate this term because 1D path integrals cannot capture multi-feature coalition interactions.

2. PullSHAP-Gate: Discrete Coalition Correction (Section 5.2). To eliminate downstream mismatch without heuristic paths, we introduce PullSHAP-Gate for structured hinge-sum downstream heads: f(z)=c+b⊤z+∑m=1Mβm[wm⊤z+θm]+.f(z) = c + b^\top z + \sum_{m=1}^M \beta_m [w_m^\top z + \theta_m]_+. PullSHAP-Gate operates via a three-stage pipeline:

  1. Representation Atoms: Integrate encoder coordinates along the path into affine basis vectors ai=∫01Dp(γ(t))i dta_i = \int_0^1 D p(\gamma(t))_i \, dt.
  2. Gate Classification: Classify each hinge gate mm as stable-active (allocating linearly), stable-inactive (zero), or switching.
  3. Meet-in-the-Middle Oracle: For switching gates with effective support k=∣supp(wm⊤Ax)∣k = |\mathrm{supp}(w_m^\top A_x)|, exact discrete Shapley weights are computed via cardinality-stratified meet-in-the-middle subset sums in arithmetic complexity: O(k32⌈(k−1)/2⌉),O\left(k^3 2^{\lceil(k-1)/2\rceil}\right), yielding a quadratic-to-square-root speedup over naive O(k2k)O(k 2^k) enumeration.
  4. Support Truncation (Proposition 9, Appendix H): For large supports k>kmax⁡k > k_{\max}, endpoint-preserving thresholding truncates small coordinates while preserving exact completeness, with error bound ∣ej∣≤12∣βm∣τ|e_j| \le \frac{1}{2} |\beta_m| \tau.

3. Theoretical Guarantees (Theorem 8). In Theorem 8 (Section 5.2), we prove that full-support Gate attribution exactly recovers the discrete Shapley–Shubik allocation of the representation surrogate: ϕGate=ϕSS(h).\phi^{\mathrm{Gate}} = \phi^{\mathrm{SS}}(h). Consequently, downstream allocation mismatch is identically zero (Ij(h)=0I_j(h) = 0), and total attribution error is bounded solely by representation distortion: ∣ϕjGate−ϕj⋆∣≤Dp,j.|\phi_j^{\mathrm{Gate}} - \phi_j^\star| \le D_{p,j}.

4. Empirical Validation (Table 10).

  • Surrogate Agreement: Across 9,000 exact-track ReLU/LeakyReLU explicands (Table 10, Appendix K), full-support Gate matches brute-force 2d2^d surrogate Shapley values to numerical precision (max absolute L1L_1 discrepancy 2.56×10−142.56 \times 10^{-14}).
  • Fidelity Gains: Gate strictly reduces median relative error on Diabetes (18.55% →\to 16.72%) and Breast Cancer (7.37% →\to 6.86%).
  • Signed Cancellation: On Adult, Gate yields neutral median error (9.81%). In Appendix H, we analyze counterexamples showing that when representation distortion and allocation mismatch have opposite signs, eliminating downstream mismatch removes beneficial cancellation.

PullSHAP-Gate thus provides a genuinely discrete, interaction-aware mechanism that directly resolves the limitations of path-based approximations.

AuthorsOpenReview ↗

Weakness 2 - Experimental Scope, Representation Width (), and Computational Scalability

Weakness 2

Limited scope and scalability of the experimental benchmark. The empirical evaluation relies heavily on severely low-dimensional and synthetic/reduced setups. Concretely, the exact-reference evaluation on real datasets (Breast Cancer and Adult) is restricted to 10-feature reduced models (d=10d=10) because exact SHAP requires exhaustive 2d{2}^{d} coalition enumerations. The full-dimensional versions (d=30d=30 for Breast Cancer, d=108d=108 for Adult) are omitted from the accuracy evaluation.The latent dimension is primarily fixed at r=4r=4.The practical SHAP estimator comparison in Section 6 (RQ4, Figure 3) evaluates only 20 explicands. As a result, it remains unverified whether PullSHAP scales accurately or maintains low error on high-dimensional, highly non-linear real-world representations (e.g., modern vision transformers, large language model latent states, or deep multi-layer MLPs with hundreds of features).

We have comprehensively expanded the experimental benchmark from 12 models / 240 evaluations to 330 trained models and 14,700 model–explicand evaluations (Table 1, Section 6; Table 7, Appendix K).

1. Massive Expansion of Benchmark Scope (Table 1 & Table 7)

The revised experimental suite encompasses:

  • 330 trained neural network models (up from 12 in the original submission).
  • 14,700 detailed model–explicand evaluations (up from 240).
  • 13,500 exact-reference evaluations on reduced benchmarks (d=10d=10) across depths 3, 5, 8 and 10 random seeds.
  • 1,200 full-dimensional evaluations on unreduced Breast Cancer (d=30d=30) and Adult (d=108d=108) against 8,192-pair antithetic references and 65,536-pair two-stream closure checks (Table 13, Appendix K).

2. Systematic Latent Width Sweep: r∈{4,16,64,128}r \in \{4, 16, 64, 128\} (Table 16)

We conducted a comprehensive sweep over representation widths r∈{4,16,64,128}r \in \{4, 16, 64, 128\} across 40 model configurations and 800 attribution evaluations (Table 16, Appendix L):

  • On Breast Cancer, attribution fidelity improves monotonically with width: median relative error decreases from 19.82% (r=4r=4) to 17.20% (r=128r=128).
  • On Adult, optimal fidelity is achieved at intermediate width r=16r=16 (9.64%), with slight deterioration at r=128r=128 (10.51%).
  • All 29 flagged paths across 22 explicands were escalated to GL256–GL2048 with complete numerical convergence (<10−11< 10^{-11} residual), confirming that aggregate width trends are stable and immune to integration artifacts.

3. Affine Embedding Invariance (Proposition 12)

To understand why representation width alone does not govern pullability, we proved Affine Representation-Embedding Invariance (Proposition 12, Appendix J): if an encoder is expanded by an affine embedding pembed(x)=Mp(x)+cp_{\mathrm{embed}}(x) = M p(x) + c with left-invertible MM, PullSHAP attributions are mathematically invariant: ϕPull(f∘p)=ϕPull((f∘M†)∘pembed).\phi^{\mathrm{Pull}}(f \circ p) = \phi^{\mathrm{Pull}}((f \circ M^\dagger) \circ p_{\mathrm{embed}}). Controlled padding experiments expanding width from 16 to 64 and 128 verified exact numerical invariance across all 60 test instances. Nominal width rr is therefore not an intrinsic measure of attribution difficulty.

4. Modern Vision Architectures: ResNet-18 and ViT-B/16 (Table 17)

On ImageNet 49-group core games (Figure 1(b), Section 6; Table 17, Appendix L), sample-based factorization selection improves composite IG on 12/12 ResNet-18 images (median error 236.23% →\to 158.98%) and 11/12 ViT-B/16 images (129.27% →\to 98.49%).

5. Matched-Budget Runtime and Peak GPU Memory Scalability (Tables 23 & 24)

  • Matched-Runtime Trade-offs (Table 23, Appendix M): Under an equal wall-clock budget of 150 ms on Adult, paired permutation sampling achieves 0.337% median relative error versus 0.454% for latent cubic proxy correction. We explicitly state that sample-efficiency gains do not guarantee wall-clock dominance under optimized parallel execution.
  • Peak GPU Memory Footprint (Table 24, Appendix M): Conversely, representation-space surrogate construction provides dramatic memory savings. On 50 Adult explicands, median incremental peak GPU allocation is 11.14 MB for the latent proxy versus 351.87 MB for the input-space proxy—a 31.6-fold reduction in incremental GPU memory allocation.

These additions establish empirical behavior across a broad spectrum of dimensions, widths, architectures, and hardware resource profiles.

AuthorsOpenReview ↗

Weakness 3 - Theoretical Extension Beyond Smooth Networks: Continuous Piecewise-Affine (ReLU/LeakyReLU) Framework

Weakness 3

Overly restrictive theoretical scope excluding modern deep architectures. The theoretical framework strongly depends on smoothness assumptions, and continuous differentiability for ff. This explicitly excludes piecewise linear activations (such as ReLU, LeakyReLU, or GELU approximations with sharp edges), decision tree ensembles, vector quantization layers, and discrete/attention-based representations. Given that a vast majority of modern deep learning architectures utilize non-smooth activation functions (like ReLU), the curvature bounds (Kp\mathcal{K}_{p}) and guarantees presented in the paper do not strictly apply to standard deep learning pipelines without smoothing modifications.

We have developed a comprehensive non-smooth theoretical framework that directly covers continuous piecewise-affine (CPWL) networks—including ReLU and LeakyReLU activations—without surrogate smoothing.

1. Why Almost-Everywhere Classical Derivatives Fail

We first clarify why naively applying classical curvature formulas to almost-everywhere derivatives is mathematically unsound. Consider the two-feature function:

q(u1,u2)=[u1+u2−1]+.q(u_1, u_2) = [u_1 + u_2 - 1]_+.

Its classical mixed second derivative ∂2q/∂u1∂u2\partial^2 q / \partial u_1 \partial u_2 is zero almost everywhere on R2\mathbb{R}^2. Yet for baseline (0,0)(0,0) and explicand (1,1)(1,1), the representation distortion for coalition S={1}S=\{1\} is:

Dq({1})=∣[1+0−1]+−[0+0−1]+−∫011⋅dt∣=∣0−0−1/2∣=12.D_q(\{1\}) = \left| [1+0-1]_+ - [0+0-1]_+ - \int_0^1 1 \cdot dt \right| = |0 - 0 - 1/2| = \frac{1}{2}.

Relying on almost-everywhere derivatives would predict zero distortion, completely missing the boundary-crossing effect.

2. Path-Compatible Selected Jacobians and Variation Bounds (Theorem 3)

To resolve this, we construct path-compatible selected Jacobians for finite continuous piecewise-affine representations (Section 4, Theorem 3; Appendix B). Let P\mathcal{P} denote a finite polyhedral complex partitioning the domain into open cells where pp is affine. For any straight-line segment γ(t)=x′+t(x−x′)\gamma(t) = x' + t(x - x'), the intersection with cell boundaries occurs at a finite set of parameter values {t1,…,tK}\{t_1, \dots, t_K\}.

We define the coordinatewise directional Jacobian variation VikV_{ik} across all boundary transitions and regular intervals:

Vik=∫01∣ddt[Dp(γ(t))]ik∣dt.V_{ik} = \int_0^1 \left| \frac{d}{dt} [D p(\gamma(t))]_{ik} \right| dt.

In Theorem 3 (Section 4), we prove the non-smooth representation distortion bound:

Dp(S)≤∑i∈S,k∉SVik.D_p(S) \le \sum_{i \in S, k \notin S} V_{ik}.

When the field decomposes into regular curvature and discrete boundary normal jumps Jik=∑m∣Δm[Dp]ik∣J_{ik} = \sum_m |\Delta_m [Dp]_{ik}|, this refines to:

Dp(S)≤∑i∈S,k∉S(12κikreg+Jik).D_p(S) \le \sum_{i \in S, k \notin S} \left( \frac{1}{2} \kappa_{ik}^{\mathrm{reg}} + J_{ik} \right).

The regular coefficient 1/21/2 and jump coefficient 11 are proven sharp (Appendix B).

3. Downstream Allocation Mismatch Under Non-Smooth Representations (Theorem 4)

We extend the downstream allocation mismatch bound Ij(h)I_j(h) without requiring classical third derivatives (Section 5, Theorem 4). Using coordinate interpolation and mixed rectangular defects on compatible gradient selections, the bound applies directly to continuous piecewise-affine predictors.

4. Non-Smooth Empirical Validation (Tables 10, 11, 12)

We validated this non-smooth framework across 220 trained ReLU and LeakyReLU models evaluated on 9,800 explicands (Tables 10–12, Appendix K). The theoretical bounds strictly hold, and sample-selected PullSHAP maintains its performance advantage without smoothing the underlying networks.

5. Clear Delineation of Architectural Scope (Appendix J)

Finally, we clearly delineate the architectural scope of the continuous-path framework:

  • Smooth Architectures: Smooth networks with GELU, Swish, Softmax attention, and stabilized LayerNorm satisfy the C2/C3C^2 / C^3 assumptions of the original framework.
  • Continuous Piecewise-Affine (CPWL): ReLU and LeakyReLU networks satisfy our new non-smooth framework via path-compatible Jacobian variations.
  • Discontinuous Architectures (Explicit Limitation): Step encoders, hard vector-quantization (VQ-VAE), and discrete decision trees contain jump discontinuities. In Appendix J, we provide an explicit counterexample proving that continuous path integration cannot detect interior jump discontinuities occurring off the evaluation path. These models remain an explicit limitation of the path-based attribution paradigm.

This rigorous non-smooth theory and empirical validation directly extend PullSHAP's guarantees to standard ReLU/LeakyReLU deep learning architectures.

Reviewer_vKhbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper suggests that model can be interpret as F=f∘pF=f\circ p, where pp is the learned representation, investigates when attribution recovers the fixed baseline Shapley values of the composite model with respect to the input features. The proposed PullSHAP integrates and multiplies the encoder Jacobian and the downstream gradient along the path separately.

Critical strengths
  • The problem is well formulated and defines the single-baseline semantics precisely. The proofs are correct to me.
Critical weaknesses
  • The method differs from composite IG only on a non-trivial split, and for deep nets, the choice of split layer seems to be arbitrary. The paper also stated that the results depend on the factorization, but it does not provide a rigorious principle for choosing the split, and does not provide experiments to compare the split layers. Evidence that an interior split will reduce the total error should be provided as well.
  • The practical case seems not supported. The Breast cancer and Adult datasets are retrained on 10 features instead of evaluated at full dd against high-budget sampling reference. The paper is motivated by modern learned representations, such as CNN and Transformer architectures, which have much greater dd, and the paper does not provide experiments for those.
Questions for authors
  • How to choose the layer of split and how sensitive the split is?
6 comments
AuthorsOpenReview ↗

Weakness 1 and Question 1 - Principled Split Selection, Conditional Certificates, and All-Layer Sensitivity

Weakness 1

The method differs from composite IG only on a non-trivial split, and for deep nets, the choice of split layer seems to be arbitrary. The paper also stated that the results depend on the factorization, but it does not provide a rigorious principle for choosing the split, and does not provide experiments to compare the split layers. Evidence that an interior split will reduce the total error should be provided as well.

Question 1

How to choose the layer of split and how sensitive the split is?

We provide a complete theoretical and empirical answer to how split layers behave, how sensitive they are, and how beneficial interior factorizations can be rigorously certified and practically selected.

1. Structural Unification: Composite IG as a Boundary Case (Proposition 6). We formalize the relationship between PullSHAP and composite IG (Section 5, Proposition 6). For any composite predictor F=fℓ∘pℓF = f_\ell \circ p_\ell, under exact integration:

  • Setting (p,f)=(Id,F)(p, f) = (Id, F) evaluates the trivial input factorization and recovers composite IG: ϕPull=ϕIG\phi^{\mathrm{Pull}} = \phi^{\mathrm{IG}}.
  • Setting (p,f)=(F,Id)(p, f) = (F, Id) evaluates the scalar output factorization and likewise recovers composite IG.

Thus, composite IG is not a separate competing method, but the boundary member of the parameterized PullSHAP family.

2. Operational Certificate of Strict Improvement (Corollary 5). In Section 5 (Theorem 4), we establish feature-wise error bounds separating representation distortion from downstream allocation mismatch:

∥aℓ−ϕ⋆∥1≤12LfKp+∑jIj(h)≡Uℓ\|a_\ell - \phi^\star\|_1 \le \frac{1}{2} L_f \mathcal{K}_p + \sum_j I_j(h) \equiv U_\ell

Because signed representation and allocation errors can partially cancel, minimizing unsigned bounds UℓU_\ell does not guarantee minimal actual error. However, we prove an operational certificate (Corollary 5, Section 5):

∣aℓ−b∣1>2Uℓ  ⟹  ∣aℓ−ϕ⋆∣1<∣b−ϕ⋆∣1|a_\ell - b|_1 > 2U_\ell \implies |a_\ell - \phi^\star|_1 < |b - \phi^\star|_1

where bb denotes composite IG. Whenever this condition holds, the interior split is mathematically guaranteed to achieve strictly lower actual error than IG, with certified improvement at least ∥aℓ−b∥1−2Uℓ\|a_\ell - b\|_1 - 2U_\ell.

3. Exact Analytical Improvement Frontier (Appendix C, Proposition 7). On the parameterized family pλ(u)=(u12+λu1u2,u2)p_\lambda(u) = (u_1^2 + \lambda u_1 u_2, u_2) with f(z)=z1z2f(z) = z_1 z_2, the exact errors are EIG=∣1−λ∣/3E_{\mathrm{IG}} = |1-\lambda|/3 and EPull=λ/2E_{\mathrm{Pull}} = \lambda/2. Interior PullSHAP is strictly superior to composite IG if and only if 0≤λ<2/50 \le \lambda < 2/5. This proves mathematically that an interior split strictly reduces total error across an entire continuum of non-zero cross-feature interactions.

4. Comprehensive All-Layer Empirical Evaluation and Selection (Appendices K & L). To evaluate layerwise sensitivity, we evaluated all candidate interior splits across 90 smooth exact-track models covering depths 3, 5, and 8 across 10 training seeds (4,500 held-out evaluations; Table 8, Appendix K).

  • The percentage of explicands admitting a strictly better interior split increases with network depth: on Adult, from 68.6% at depth 3 to 76.6% at depth 5 and 81.0% at depth 8.
  • On Breast Cancer, the proportion increases from 51.2% (depth 3) to 68.4% (depth 8).

We introduced a practical sample-based selector that evaluates candidate splits using independent coalition samples without accessing the ground-truth oracle (Table 9, Appendix K):

  • Diabetes: 17.80% improvements vs. 0.33% degradations relative to IG.
  • Breast Cancer: 55.80% improvements vs. 5.53% degradations.
  • Adult: 74.53% improvements vs. 0.80% degradations.
  • On Adult, the mean paired reduction in relative error is 1.945 percentage points (bootstrap 95% CI [1.708, 2.193]).

5. Numerical Refinement and Stability. To verify numerical stability across quadrature orders, all 29 flagged paths across 22 explicands in the width study were escalated to GL256–GL2048 (Table 13, Appendix K). All paths satisfy strict completeness and successive-order convergence diagnostics (<10−11< 10^{-11}). Only one explicand changed its sample-selected split (from IG to L3), and its 2.38 pp advantage over IG remained invariant through GL2048. All 12 previously unresolved deep-network IG cases were resolved at GL1024.

These results confirm that interior splits systematically improve attribution accuracy, that beneficial splits become more prevalent in deeper networks, and that sample-based selection reliably captures these improvements.

AuthorsOpenReview ↗

Weakness 2 - Full-Dimensional Evaluation () and Large-Scale Reference Sampling

Weakness 2

The practical case seems not supported. The Breast cancer and Adult datasets are retrained on 10 features instead of evaluated at full dd against high-budget sampling reference.

We have fully validated PullSHAP on the unreduced, full-dimensional feature spaces: Breast Cancer (d=30d=30) and Adult (d=108d=108), spanning 60 trained models and 1,200 model–explicand evaluations (Table 1, Section 6; Table 7, Appendix K).

1. Full-Dimensional Benchmark Scope (Table 1 & Table 7)

In the revised manuscript, we evaluate models trained on the complete feature sets:

  • Full Breast Cancer: d=30d=30 input features.
  • Full Adult: d=108d=108 one-hot encoded input features.

The full-dimensional evaluation encompasses 60 trained neural networks across smooth (Tanh) and non-smooth (ReLU, LeakyReLU) architectures with 10 random seeds, totaling 1,200 detailed model–explicand evaluations (Table 1, Section 6; Table 7, Appendix K).

2. High-Budget Reference Protocol and Two-Stream Closure (Table 13)

Because exhaustive 2d2^d evaluation is computationally infeasible for d=30d=30 (10910^9 coalitions) and d=108d=108 (103210^{32} coalitions), we established a rigorous high-budget reference protocol:

  1. Primary References: Generated using 8,192 antithetic permutation pairs (16,384 model passes per explicand).
  2. Two-Stream Closure Study (Table 13, Appendix K): On 150 locked cases, we constructed two independent sampling streams of 32,768 antithetic pairs each (65,536 pairs total). The median absolute L1L_1 discrepancy between the independent streams is 0.00310.0031, confirming high reference stability. We report all reference uncertainties explicitly and do not conflate Monte Carlo estimates with exact combinatorial Shapley values.

3. Quantitative Results on Full Dimensions (Table 7)

Across the full-dimensional benchmarks, sample-based factorization selection consistently improves upon composite IG:

  • Full Breast Cancer (d=30d=30, Smooth): Median relative error decreases from 22.70% (composite IG) to 19.23% (sample-selected PullSHAP).
  • Full Adult (d=108d=108, Smooth): Median relative error decreases from 10.19% (composite IG) to 9.81% (sample-selected PullSHAP).
  • Across both datasets, the proportion of explicands exhibiting strict improvement over composite IG under sample-based selection exceeds 70%.

These additions confirm that the theoretical and empirical benefits of attribution pullback persist on realistic, full-dimensional tabular problems.

AuthorsOpenReview ↗

Weakness 3 - Modern Learned Representations: Pretrained CNN (ResNet-18) and Transformer (ViT-B/16)

Weakness 3

The paper is motivated by modern learned representations, such as CNN and Transformer architectures, which have much greater dd, and the paper does not provide experiments for those.

We definitively demonstrate that the principles of attribution pullback generalize to modern deep representations, including pretrained ResNet-18 (convolutional residual hierarchy) and ViT-B/16 (patch-based vision transformer) on ImageNet (Section 6, Figure 1(b); Appendix J).

1. Grouped Shapley Formulation on Modern Vision Encoders (Appendix J)

Individual pixel-level Shapley evaluation on ImageNet (224×224×3=150,528224 \times 224 \times 3 = 150,528 features) is computationally intractable for coalition sampling. We therefore evaluate fixed-baseline grouped Shapley games (Section 6, Figure 1(b); Appendix J):

  • Input images are partitioned into a core grid of 49 spatial superpixel groups (7×77 \times 7).
  • We evaluate standard pretrained ImageNet models: ResNet-18 (convolutional residual hierarchy) and ViT-B/16 (patch-based multi-head self-attention transformer).
  • For ResNet-18, candidate interior factorizations correspond to intermediate residual stages (layer1 through layer4).
  • For ViT-B/16, candidate factorizations correspond to transformer encoder blocks (block3, block6, block9, block11).
  • References are computed using high-budget antithetic permutation sampling on the 49-group game.

2. Experimental Results on ResNet-18 and ViT-B/16 (Table 17)

In the completed 49-group vision evaluation (Table 17, Appendix L), sample-based factorization selection substantially improves upon composite IG:

  • ResNet-18: Sample-based selection improves upon composite IG on 12 out of 12 images (100%). The median relative error decreases from 236.23% for composite IG down to 158.98% for PullSHAP.
  • ViT-B/16: Sample-based selection improves upon composite IG on 11 out of 12 images (91.7%). The median relative error decreases from 129.27% for composite IG down to 98.49% for PullSHAP.

3. Transparent Delineation of Vision Limitations (Appendix L, Table 18)

While these experiments confirm that factorization selection operates effectively on modern deep vision backbones, we explicitly emphasize the following boundaries in Section 6 and Appendix L:

  1. Grouped vs. Pixel Games: Grouped superpixel attributions do not constitute individual pixel-level Shapley recovery.
  2. Magnitude of Residual Error: Although PullSHAP reduces error by over 77 percentage points on ResNet-18 and 30 percentage points on ViT-B/16, the remaining median errors (158.98% and 98.49%) remain substantial. Gradient-based pullback approximations do not match the accuracy of brute-force Monte Carlo permutation sampling when compute budgets are unrestricted.
  3. Quadrature Resolution Sensitivity: High-resolution vision representations exhibit measurable sensitivity across Gauss–Legendre quadrature orders (Table 18, Appendix L), indicating that path integration requires fine discretization in deep latent spaces.

The revised manuscript thus demonstrates that the principles of attribution pullback extend directly to modern CNNs and Transformers, while maintaining rigorous scientific honesty regarding the remaining challenges in high-dimensional vision representations.

Reviewer_SBdmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the problem of attempting to recover attributions to an input space when attributions to the latent space are available. The paper 1) identifies error bounds due to curvature caused by feature interaction, 2) introduces a method, PULLSHAP, which minimizes the full attribution error, and 3) experiments testing the cause of error in attributions.

Critical strengths
  • Rigorous presentation
  • Experiments validate that PULLSHAP reduces recovery error
  • Bounds on recovery error are given
Critical weaknesses
  • To the reviewer's knowledge, there are very few instances where a practitioner has access to ff but not to pp. Accurate attributions require access to both, but when is this a problem/who is this a problem for?
  • Experiments show that PULLLSHAP reduces error, but the error is still significant. Is the amount of error reduction useful for anything?
Questions for authors

Please see weaknesses section

5 comments
AuthorsOpenReview ↗

Weakness 1 - Motivation, Access Model, and Problem Relevance

Weakness 1 — Motivation, access model, and problem relevance

To the reviewer's knowledge, there are very few instances where a practitioner has access to ff but not to pp. Accurate attributions require access to both, but when is this a problem/who is this a problem for?

We thank the reviewer for this important question. PullSHAP assumes access to both pp and ff in F=f∘pF=f\circ p. Our objective is to determine when internal model structure improves attribution fidelity to a fixed input-space Shapley target, rather than to overcome restricted model access.

1. Problem relevance under full model access (Sections 1–2).

Consider a practitioner auditing a predictor built from a pretrained feature extractor and a downstream prediction head. Although predictions depend on latent features, the practitioner needs attributions to original input variables, such as measured covariates or image regions. Direct end-to-end Shapley estimation is available, but it does not explicitly exploit the encoder–head factorization. PullSHAP uses integrated representation contributions and downstream sensitivity to approximate the same fixed-baseline target. Algorithm 1 implements this construction using VJPs.

2. Structural limits and positive guarantees (Theorems 1–4).

Theorem 1 proves that even identical endpoint germs of smooth representations can yield different composite Shapley values with a common downstream predictor. Theorem 2 proves that, when all input coordinates differ from the baseline, a smooth scalar representation can realize any normalized coalition game. Thus, endpoint-local information and latent compression alone cannot guarantee exact pullback. These results concern structural recovery, not the impossibility or computational complexity of exact SHAP with full model access.

Theorems 3–4 provide the corresponding positive theory: sharp coalition-distortion bounds and a decomposition separating representation, downstream allocation, and numerical errors.

3. Exact analytical improvement frontier (Proposition 7; Appendix C).

Consider the family

pλ(u)=(u12+λu1u2,u2),f(z)=z1z2,p_\lambda(u)=(u_1^2+\lambda u_1u_2,u_2),\qquad f(z)=z_1z_2,

with baseline (0,0)(0,0), explicand (1,1)(1,1), and λ≥0\lambda\geq0. The exact Shapley vector and the two path-based allocations are

ϕ⋆=(1+λ2,1+λ2),\phi^\star=\left(\frac{1+\lambda}{2},\frac{1+\lambda}{2}\right),

ϕIG=(2+λ3,1+2λ3),\phi^{\rm IG}=\left(\frac{2+\lambda}{3},\frac{1+2\lambda}{3}\right),

ϕPull=(12+λ4,12+3λ4).\phi^{\rm Pull}=\left(\frac12+\frac{\lambda}{4},\frac12+\frac{3\lambda}{4}\right).

Their exact absolute ℓ1\ell_1 errors are

EIG=∣1−λ∣3,EPull=λ2.E_{\rm IG}=\frac{|1-\lambda|}{3},\qquad E_{\rm Pull}=\frac{\lambda}{2}.

Thus, interior PullSHAP strictly improves on composite IG if and only if 0≤λ<2/50\leq\lambda<2/5. Moreover, using the sound error bound U=λ/2U=\lambda/2 derived in Appendix C, the certified improvement margin is

C=∥ϕPull−ϕIG∥1−2U=13−5λ6.C=\|\phi^{\rm Pull}-\phi^{\rm IG}\|_1-2U=\frac13-\frac{5\lambda}{6}.

The certificate recovers the entire strict-improvement regime of this family. The comparison holds for the same predictor and the same Shapley target at each λ\lambda.

4. Empirical relevance (Section 6; Appendix K).

On 1,500 exact-reference smooth Adult evaluations, SampleSelect improves on composite IG in 74.53% of cases, with 0.80% degradations. The mean relative-error reduction is 1.945 percentage points (95% model-cluster bootstrap CI [1.708, 2.193]). Selection uses independent permutation samples, not the exact evaluation oracle. The expanded full-dimensional and pretrained vision studies further establish factorization sensitivity beyond reduced-feature models.

In summary, the revised manuscript clarifies the full-access setting and establishes when internal factorization can improve fidelity, both analytically and empirically. Computational costs are evaluated separately; we do not assume that exploiting internal structure always improves accuracy or runtime.

AuthorsOpenReview ↗

Weakness 2 - Practical Decision Utility and Budgeted Feature Inspection

Weakness 2 — Decision utility and practical significance of error reduction

Experiments show that PULLLSHAP reduces error, but the error is still significant. Is the amount of error reduction useful for anything?

We thank the reviewer for asking about the practical meaning of attribution fidelity. The revision addresses this question by distinguishing two objectives: capturing Shapley importance under a limited feature-inspection budget, and maximizing actual prediction changes under feature deletion or insertion. We provide guarantees and empirical evidence for both, while identifying their different limitations.

1. Sharp guarantee for budgeted feature inspection (Proposition 10; Appendix J).

When an analyst can inspect only kk features, a natural Shapley-based objective is to capture as much absolute reference importance as possible. Let Sk⋆S_k^\star maximize ∑j∈S∣ϕj⋆∣\sum_{j\in S}|\phi_j^\star| over sets of size kk, and let Sk(a)S_k(a) select the kk largest magnitudes of an approximate attribution vector aa. Proposition 10 proves

0≤Rk(a):=∑j∈Sk⋆∣ϕj⋆∣−∑j∈Sk(a)∣ϕj⋆∣≤∥a−ϕ⋆∥1.0\leq R_k(a):=\sum_{j\in S_k^\star}|\phi_j^\star|-\sum_{j\in S_k(a)}|\phi_j^\star|\leq\|a-\phi^\star\|_1.

The coefficient 1 is sharp. Thus, smaller attribution error gives a tighter upper bound on the loss of captured Shapley importance. This is a concrete guarantee for budgeted inspection, not a claim that every reduction in attribution error improves the selected feature set. Appendix J also establishes sign and top-kk stability under sound coordinate-error bounds.

2. Measured importance-capture gains (Table 14; Appendix L).

We evaluate importance-capture regret across 4,500 smooth exact-reference model–explicand evaluations from three datasets. On Adult (d=10d=10, 1,500 evaluations), for k=3k=3:

  • Composite IG: mean normalized capture regret 0.3118%.
  • SampleSelect: mean normalized capture regret 0.2344%.
  • Mean paired reduction: 0.0773 percentage points, with a 95% model-cluster bootstrap CI of [0.0365, 0.1234].

This provides empirical evidence of reduced decision regret for this Shapley-based objective. The absolute gain is modest, but the result quantifies how fidelity improvements can translate into better feature inspection. We do not extrapolate it to all datasets or to clinical or financial decision outcomes.

3. Signed prediction interventions: guarantees and limitations (Proposition 11; Appendix J; Table 15, Appendix L).

For the fixed-baseline coalition game vv, define signed deletion and insertion by

D(S)=v([d])−v([d]∖S),I(S)=v(S)−v(∅).D(S)=v([d])-v([d]\setminus S),\qquad I(S)=v(S)-v(\varnothing).

Let the global coalition nonadditivity residual be

ϵ=max⁡S⊆[d]∣v(S)−v(∅)−∑j∈Sϕj⋆∣.\epsilon=\max_{S\subseteq[d]}\left|v(S)-v(\varnothing)-\sum_{j\in S}\phi_j^\star\right|.

Proposition 11 proves that selecting the kk largest signed entries of aa incurs regret, relative to the optimal size-kk subset for either objective, bounded by

Rk,M(a)≤∥a−ϕ⋆∥1+2ϵ,M∈{D,I}.R_{k,M}(a)\leq\|a-\phi^\star\|_1+2\epsilon,\qquad M\in\{D,I\}.

The additional term captures coalition nonadditivity. Consequently, even exact Shapley attribution need not optimize signed deletion or insertion; improved Shapley fidelity alone cannot guarantee better prediction perturbations.

We conducted 1,650 predetermined checkpoint-based perturbation replays. Among the 1,350 exact-reference cases, exact SHAP top-3 ranking is suboptimal for signed deletion in 514 cases (38.07%) and for signed insertion in 565 cases (41.85%). Empirical intervention gains are mixed, consistent with the distinction in Proposition 11. These interventions are baseline substitutions, not causal interventions.

Conclusion. The revision gives a sharp link between attribution fidelity and Shapley-based inspection regret, quantifies a measurable improvement on Adult, and explains why intervention utility requires additional control of coalition nonadditivity. This answers the reviewer's concern without treating smaller attribution error as a universal guarantee of practical utility.

We sincerely thank the reviewer once again for the insightful and constructive comments, which have enabled us to significantly improve the manuscript. We hope that the newly added theoretical results, experimental evidence, and clarifications on problem relevance and decision utility fully address your concerns. We would greatly appreciate your favorable reconsideration of our revised work and welcome any additional feedback during the remainder of the discussion period.

Thank you again for your time and thoughtful evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.