- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the theoretical conditions under which baseline shapley values calculated over an input feature space can be recovered ("pulled back") through a factorized model , where is a learned intermediate representation and is a downstream predictor. The authors first prove two fundamental boundary results. To provide positive guarantees, the paper introduces PullSHAP, establishing that pullability is governed by two distinct obstructions, .i.e, representation-side coalition distortion and downstream allocation mismatch.
Critical strengths
The paper derives an error decomposition separating representation distortion, downstream allocation mismatch, and numerical quadrature error (), along with a conditional certificate template.
Critical weaknesses
- Limited novelty in the algorithmic formulation (PullSHAP-Fast). While the theoretical characterization of the pullback error is insightful, the practical algorithm proposed in Algorithm 1 (PULLSHAP-FAST) boils down to a decoupled, two-path numerical quadrature integration using Vector-Jacobian Products (VJPs). Mathematically, it simply computes , where is the integrated encoder Jacobian and is the average downstream gradient. This is a natural, factorized extension of Integrated Gradients (IG). The distinction between decoupled integration (PullSHAP) and coupled integration (Composite IG) is heavily emphasized, but functionally, PULLSHAP-FAST remains a path-based heuristic gradient method that approximates SHAP without offering a fundamentally new algorithmic mechanism for handling non-additive, highly interacting deep representations.
- Limited scope and scalability of the experimental benchmark. The empirical evaluation relies heavily on severely low-dimensional and synthetic/reduced setups. Concretely, the exact-reference evaluation on real datasets (Breast Cancer and Adult) is restricted to 10-feature reduced models () because exact SHAP requires exhaustive coalition enumerations. The full-dimensional versions ( for Breast Cancer, for Adult) are omitted from the accuracy evaluation.The latent dimension is primarily fixed at .The practical SHAP estimator comparison in Section 6 (RQ4, Figure 3) evaluates only 20 explicands. As a result, it remains unverified whether PullSHAP scales accurately or maintains low error on high-dimensional, highly non-linear real-world representations (e.g., modern vision transformers, large language model latent states, or deep multi-layer MLPs with hundreds of features).
- Overly restrictive theoretical scope excluding modern deep architectures. The theoretical framework strongly depends on smoothness assumptions, and continuous differentiability for . This explicitly excludes piecewise linear activations (such as ReLU, LeakyReLU, or GELU approximations with sharp edges), decision tree ensembles, vector quantization layers, and discrete/attention-based representations. Given that a vast majority of modern deep learning architectures utilize non-smooth activation functions (like ReLU), the curvature bounds () and guarantees presented in the paper do not strictly apply to standard deep learning pipelines without smoothing modifications.
Questions for authors
Please refer to the weaknesses.
6 comments
Weakness 2 - Experimental Scope, Representation Width (), and Computational Scalability
Weakness 2
Limited scope and scalability of the experimental benchmark. The empirical evaluation relies heavily on severely low-dimensional and synthetic/reduced setups. Concretely, the exact-reference evaluation on real datasets (Breast Cancer and Adult) is restricted to 10-feature reduced models () because exact SHAP requires exhaustive coalition enumerations. The full-dimensional versions ( for Breast Cancer, for Adult) are omitted from the accuracy evaluation.The latent dimension is primarily fixed at .The practical SHAP estimator comparison in Section 6 (RQ4, Figure 3) evaluates only 20 explicands. As a result, it remains unverified whether PullSHAP scales accurately or maintains low error on high-dimensional, highly non-linear real-world representations (e.g., modern vision transformers, large language model latent states, or deep multi-layer MLPs with hundreds of features).
We have comprehensively expanded the experimental benchmark from 12 models / 240 evaluations to 330 trained models and 14,700 model–explicand evaluations (Table 1, Section 6; Table 7, Appendix K).
1. Massive Expansion of Benchmark Scope (Table 1 & Table 7)
The revised experimental suite encompasses:
- 330 trained neural network models (up from 12 in the original submission).
- 14,700 detailed model–explicand evaluations (up from 240).
- 13,500 exact-reference evaluations on reduced benchmarks () across depths 3, 5, 8 and 10 random seeds.
- 1,200 full-dimensional evaluations on unreduced Breast Cancer () and Adult () against 8,192-pair antithetic references and 65,536-pair two-stream closure checks (Table 13, Appendix K).
2. Systematic Latent Width Sweep: (Table 16)
We conducted a comprehensive sweep over representation widths across 40 model configurations and 800 attribution evaluations (Table 16, Appendix L):
- On Breast Cancer, attribution fidelity improves monotonically with width: median relative error decreases from 19.82% () to 17.20% ().
- On Adult, optimal fidelity is achieved at intermediate width (9.64%), with slight deterioration at (10.51%).
- All 29 flagged paths across 22 explicands were escalated to GL256–GL2048 with complete numerical convergence ( residual), confirming that aggregate width trends are stable and immune to integration artifacts.
3. Affine Embedding Invariance (Proposition 12)
To understand why representation width alone does not govern pullability, we proved Affine Representation-Embedding Invariance (Proposition 12, Appendix J): if an encoder is expanded by an affine embedding with left-invertible , PullSHAP attributions are mathematically invariant: Controlled padding experiments expanding width from 16 to 64 and 128 verified exact numerical invariance across all 60 test instances. Nominal width is therefore not an intrinsic measure of attribution difficulty.
4. Modern Vision Architectures: ResNet-18 and ViT-B/16 (Table 17)
On ImageNet 49-group core games (Figure 1(b), Section 6; Table 17, Appendix L), sample-based factorization selection improves composite IG on 12/12 ResNet-18 images (median error 236.23% 158.98%) and 11/12 ViT-B/16 images (129.27% 98.49%).
5. Matched-Budget Runtime and Peak GPU Memory Scalability (Tables 23 & 24)
- Matched-Runtime Trade-offs (Table 23, Appendix M): Under an equal wall-clock budget of 150 ms on Adult, paired permutation sampling achieves 0.337% median relative error versus 0.454% for latent cubic proxy correction. We explicitly state that sample-efficiency gains do not guarantee wall-clock dominance under optimized parallel execution.
- Peak GPU Memory Footprint (Table 24, Appendix M): Conversely, representation-space surrogate construction provides dramatic memory savings. On 50 Adult explicands, median incremental peak GPU allocation is 11.14 MB for the latent proxy versus 351.87 MB for the input-space proxy—a 31.6-fold reduction in incremental GPU memory allocation.
These additions establish empirical behavior across a broad spectrum of dimensions, widths, architectures, and hardware resource profiles.
Weakness 3 - Theoretical Extension Beyond Smooth Networks: Continuous Piecewise-Affine (ReLU/LeakyReLU) Framework
Weakness 3
Overly restrictive theoretical scope excluding modern deep architectures. The theoretical framework strongly depends on smoothness assumptions, and continuous differentiability for . This explicitly excludes piecewise linear activations (such as ReLU, LeakyReLU, or GELU approximations with sharp edges), decision tree ensembles, vector quantization layers, and discrete/attention-based representations. Given that a vast majority of modern deep learning architectures utilize non-smooth activation functions (like ReLU), the curvature bounds () and guarantees presented in the paper do not strictly apply to standard deep learning pipelines without smoothing modifications.
We have developed a comprehensive non-smooth theoretical framework that directly covers continuous piecewise-affine (CPWL) networks—including ReLU and LeakyReLU activations—without surrogate smoothing.
1. Why Almost-Everywhere Classical Derivatives Fail
We first clarify why naively applying classical curvature formulas to almost-everywhere derivatives is mathematically unsound. Consider the two-feature function:
Its classical mixed second derivative is zero almost everywhere on . Yet for baseline and explicand , the representation distortion for coalition is:
Relying on almost-everywhere derivatives would predict zero distortion, completely missing the boundary-crossing effect.
2. Path-Compatible Selected Jacobians and Variation Bounds (Theorem 3)
To resolve this, we construct path-compatible selected Jacobians for finite continuous piecewise-affine representations (Section 4, Theorem 3; Appendix B). Let denote a finite polyhedral complex partitioning the domain into open cells where is affine. For any straight-line segment , the intersection with cell boundaries occurs at a finite set of parameter values .
We define the coordinatewise directional Jacobian variation across all boundary transitions and regular intervals:
In Theorem 3 (Section 4), we prove the non-smooth representation distortion bound:
When the field decomposes into regular curvature and discrete boundary normal jumps , this refines to:
The regular coefficient and jump coefficient are proven sharp (Appendix B).
3. Downstream Allocation Mismatch Under Non-Smooth Representations (Theorem 4)
We extend the downstream allocation mismatch bound without requiring classical third derivatives (Section 5, Theorem 4). Using coordinate interpolation and mixed rectangular defects on compatible gradient selections, the bound applies directly to continuous piecewise-affine predictors.
4. Non-Smooth Empirical Validation (Tables 10, 11, 12)
We validated this non-smooth framework across 220 trained ReLU and LeakyReLU models evaluated on 9,800 explicands (Tables 10–12, Appendix K). The theoretical bounds strictly hold, and sample-selected PullSHAP maintains its performance advantage without smoothing the underlying networks.
5. Clear Delineation of Architectural Scope (Appendix J)
Finally, we clearly delineate the architectural scope of the continuous-path framework:
- Smooth Architectures: Smooth networks with GELU, Swish, Softmax attention, and stabilized LayerNorm satisfy the assumptions of the original framework.
- Continuous Piecewise-Affine (CPWL): ReLU and LeakyReLU networks satisfy our new non-smooth framework via path-compatible Jacobian variations.
- Discontinuous Architectures (Explicit Limitation): Step encoders, hard vector-quantization (VQ-VAE), and discrete decision trees contain jump discontinuities. In Appendix J, we provide an explicit counterexample proving that continuous path integration cannot detect interior jump discontinuities occurring off the evaluation path. These models remain an explicit limitation of the path-based attribution paradigm.
This rigorous non-smooth theory and empirical validation directly extend PullSHAP's guarantees to standard ReLU/LeakyReLU deep learning architectures.
Weakness 1 - Algorithmic Novelty Beyond Path Integration: PullSHAP-Gate Structured Coalition Correction
Weakness 1
We agree that decoupled path quadrature and VJPs alone (PullSHAP-FAST) do not resolve discrete multi-feature interactions. In direct response, our revision introduces PullSHAP-Gate (Section 5.2, Theorem 8, Appendix H)—a fundamentally new, interaction-aware algorithmic mechanism that eliminates downstream allocation mismatch.
1. Downstream Allocation Obstruction (Theorem 4). In Section 5 (Theorem 4), our theory identifies an intrinsic limitation of pure path integration: the gap between continuous Aumann–Shapley path allocation and discrete Shapley–Shubik allocation on the representation surrogate h(S)=f(p(xS)). For smooth surrogates, this mismatch satisfies: Ij(h)≤121∑i=j(∥∂jiih∥∞+∥∂jjih∥∞). No quadrature refinement can eliminate this term because 1D path integrals cannot capture multi-feature coalition interactions.
2. PullSHAP-Gate: Discrete Coalition Correction (Section 5.2). To eliminate downstream mismatch without heuristic paths, we introduce PullSHAP-Gate for structured hinge-sum downstream heads: f(z)=c+b⊤z+∑m=1Mβm[wm⊤z+θm]+. PullSHAP-Gate operates via a three-stage pipeline:
3. Theoretical Guarantees (Theorem 8). In Theorem 8 (Section 5.2), we prove that full-support Gate attribution exactly recovers the discrete Shapley–Shubik allocation of the representation surrogate: ϕGate=ϕSS(h). Consequently, downstream allocation mismatch is identically zero (Ij(h)=0), and total attribution error is bounded solely by representation distortion: ∣ϕjGate−ϕj⋆∣≤Dp,j.
4. Empirical Validation (Table 10).
PullSHAP-Gate thus provides a genuinely discrete, interaction-aware mechanism that directly resolves the limitations of path-based approximations.