- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies limited-query Shapley regression for differentiable models with expensive coalition evaluations, especially implicit GNNs. It uses gradients at empty/full masks to choose one weighted cubic basis function e3(w⊙y), estimates its coefficient from sampled coalition values under efficiency, and regularizes the remaining cubic space. A theorem characterizes intrinsic multilinear-game endpoint information modulo quadratic additions; constrained partial regression yields exactness for a restricted quadratic-plus-supplied-direction class. Four graph-classification datasets, twenty trained models, balanced node-feature groups, and matched controls support an empirical benefit from assigning sensitivity weights to the correct groups.
Critical strengths
- The distinction between choosing a representation direction and estimating its coefficient is useful and clearly implemented. Model-derived information selects a column before reading interior coalition responses; the responses still determine the coefficient.
- The theory carefully distinguishes the model's fractional-mask extension from the game's intrinsic multilinear extension. I checked the endpoint-quotient argument in Appendix A and the constrained regression/exactness arguments in Appendix C and found their stated conditions important and plausible. Estimability of the attribution, rather than full coefficient rank, is handled explicitly.
- The controls are well designed. Uniform, group-size, label-permuted, and no-supplied-block controls share masks, values, and normalization. Averaging all 105 fixed-point-free involutions preserves the multiset of weights while removing their correct assignment and avoids dependence on arbitrary group labels.
- The paper reports a predeclared balanced estimand, hierarchical uncertainty, full cell-level results, leave-one-dataset-out analyses, and an official OddSHAP comparison. Appendix D quantifies derivative overhead rather than calling gradients free. Exact games at g=8 provide a strong evaluation reference.
- The evidence goes beyond the main aggregate: balanced feature groups address group-size confounding, the residual-angle diagnostics show genuinely different directions, and noisy-response experiments are explicitly distinguished from derivative-noise robustness.
Critical weaknesses
- The sensitivity-specific benefit is smaller and more heterogeneous than the main quadratic-baseline headline suggests ( Sec. 5, Appendix F). The full procedure improves the balanced error by 5.22-fold over quadratic regression, while the no-supplied-block cubic-ridge control is 1.35-fold worse than the full method and uniform weights are 1.29-fold worse. Thus the total improvement combines cubic regularization and sensitivity selection. IMDB-BINARY strongly enlarges the separation and interval upper tails; without it, sensitivity-specific control ratios are about 1.14–1.29. These remain useful gains, but the abstract/main text should separate them from the total gain and explain dataset heterogeneity.
- Practical scalability and equal-total-cost advantage remain incompletely demonstrated ( Sec. 5, Appendices D–E). All experiments use eight groups. This enables exact references but leaves unanswered how the selected direction and cubic matrices behave as group count increases. Also, the experiments computed gradients in forward mode: Appendix D reports 15.5 coalition-equivalent units for both endpoints versus 2.07 for the verified adjoint alternative. Comparisons match Boolean queries, not complete compute or wall time. The distinction is disclosed, but a concise total-cost comparison, including regression and baselines spending the saved derivative budget on additional coalitions, would strengthen practical claims.
- The negative theorem and exactness guarantee need consistently bounded interpretation (Theorems 1–3). Theorem 1 concerns endpoint-only, quadratic-invariant rules, not every estimator using the available interior coalition sample. It does not establish that game information in general cannot choose a useful cubic term. The exactness result concerns v in G_<=2 plus span{q}; it is not a general approximation-error guarantee for real implicit-network games. State these restrictions in the headline motivation as carefully as in the formal sections.
- Why endpoint magnitudes predict a useful Boolean interaction direction remains mostly empirical (Sec. 3, Appendix F). Fractional-mask derivatives depend on the extension and can change while all Boolean coalition values stay fixed. The paper recognizes this; it means the selected direction is an inductive bias tied to the mask parameterization rather than information determined by the Shapley game. The existing exact games could support a diagnostic relating selected directions to actual dividends or attribution corrections. Appendix F.5 also finds no clear advantage from using both endpoints over the empty endpoint alone, which deserves main-text visibility.
- Comparative and uncertainty scope could be sharper. ControlSHAP and regression-adjusted Monte Carlo are discussed but not evaluated; [ControlSHAP, Goldwasser and Hooker] (https://arxiv.org/abs/2310.07672) is particularly relevant to whether derivatives buy more accurate attribution at matched cost. OddSHAP changes native sampler and estimator jointly and is secondary evidence, as the appendix appropriately notes. Four datasets limit population-level generalization despite 100,000 bootstrap draws; the four aggregate intervals are marginal, without simultaneous correction. These qualifications should remain visible when describing all comparisons as supported.
Questions for authors
- Please distinguish the total gain over quadratic regression from sensitivity-specific gains, and report the quadratic-baseline aggregate with IMDB-BINARY removed. Is there an existing diagnostic explaining that dataset's especially large control separation?
- What is the practical total-cost comparison for the actually used forward-mode implementation and for the verified adjoint alternative, including regression time? Are results with the corresponding extra coalition budget already recorded?
- Can existing exact games quantify alignment with relevant cubic dividends or attribution corrections, and does that explain when the sensitivity prior helps? Please also summarize the empty-endpoint control.
- Are there existing larger-group or ControlSHAP comparisons, or should the claims explicitly remain scoped to g=8 implicit-model benchmarks? These are clarification questions, not a demand for a complete new benchmark suite during rebuttal.
4 comments
Response to Reviewer MeJr (1/3)
Thank you for checking the arguments of Appendices A and C so closely and for your precise questions. We are sorry to have kept you waiting. Answering the reviewers' questions called for new proofs and experiments, which we have only now completed. Your follow-up asked why first-order sensitivities should indicate third-order interactions, and we answer that first. The theorems that you checked are unchanged in content, and new and substantively changed text is marked in red in the revised paper.
Theoretical justification for first-order sensitivities. First-order sensitivities point to the relevant third-order interactions when interactions grow out of main effects, as the effect-heredity principle of experimental design assumes. The revision states this assumption, proves that it holds in a class of single-index models, bounds the error on any game by the remaining misalignment and the dividends of degree four and higher, and measures that misalignment on the exact games of our models. At our budgets no estimator can be accurate for every game (Appendix C.1), so any guarantee of accuracy must rest on an assumption about the game, and we state ours explicitly.
Theorem 1 (Section 2.3) shows that an assumption is needed. Because the regression fits every degree-two term directly, a rule that looks for the cubic direction should be unaffected by adding a degree-two game. Such a rule, reading only endpoint data, depends on the game through a single number, so for it cannot return the cubic direction of every game, and if it treats the groups alike it returns the symmetric direction of our uniform-weight control. An instance-specific choice must therefore give up this invariance and tie the cubic part to information beyond that number, such as the magnitudes of the individual endpoint sensitivities, which distinguish the groups.
The assumption we make is an established principle of experimental design. By effect heredity, an interaction tends to matter only when its main effects do (Hamada and Wu, 1992; Chipman, 1996), and the prior of Joseph (2006) ties the scale of a three-factor interaction to the product of the scales of its three main effects. We read the sensitivity magnitudes as these scales and give each triple the product of its members' weights (Sections 1 and 3.2). Product weights also help when they enter the ridge penalty instead of one direction. Heredity ridge, which uses them in this way, improves on cubic ridge by a factor of and on uniform weights by (Appendix F.13).
The new Appendix B.4 proves the identification in a class of models and bounds the error outside it. Take a single-index model, where the mask enters only through a weighted sum with positive weights and a smooth link whose derivative is nonzero at one end at least. Proposition B.4 shows that the normalized endpoint magnitudes recover the normalized weights of the sum and that the cubic dividend of each triple is the product of its three weights times an average of the link's third derivative. The cubic direction built from these magnitudes, before the small uniform share is mixed in, therefore coincides with the game's cubic part when this derivative is a nonzero constant, and when it keeps one sign with absolute value between and , the sine of the angle between the two is at most . For every game, on designs where the quadratic stage is exact and the selected regressor has a nonzero residual, Corollary B.1 bounds the error of the unpenalized partial regression by a constant times the size of what the fitted model leaves out, namely the component of the game's cubic part orthogonal to the selected direction and the dividends of degree four and higher. The constant depends only on the design and the selected regressor.
The proposition concerns a class of models, and it makes a prediction that can be checked without any Shapley value. The two endpoint gradients of a single-index model are parallel, and on our models the median of over targets is on IMDB-BINARY, on MUTAG, on NCI1 and on PROTEINS.
As you suggested, we used the exact games to check how well the selected direction matches the game's cubic part (Appendix F.8). We measure the alignment by the squared cosine between the centered Shapley vectors of the two. Within every dataset it is positively rank-correlated with the per-target gain over uniform weights, with Spearman correlations between and at , and in each dataset the median gain grows from the least to the most aligned third of the targets. With the thirds formed within each dataset and the targets of the four datasets pooled, the median gains are , and , so the least aligned third shows no gain. At the correlation is again positive within every dataset, and the most aligned third has median gain .
Response to Reviewer MeJr (2/3)
Total and sensitivity-specific gains. The revised abstract first states the gain over second-order PolySHAP and then reports that every matched control has higher error, times ours when no direction is singled out and times when the sensitivity weights are permuted across groups. Section 5.2 splits the total gain of into a factor of from keeping a regularized cubic block, which cubic ridge already achieves, and a factor of from the selected direction with its own penalty. Without IMDB-BINARY, the total gain over second-order PolySHAP is , with interval , of which cubic ridge accounts for a factor of and the selected direction for (Appendix F.1). The four control gains then lie between and , and Section 5.2 reports this range.
An existing diagnostic accounts for the large separation on IMDB-BINARY. This dataset's groups differ most in size, and its sensitivity magnitudes grow almost linearly with group size, with a median over targets of for the Pearson correlation across groups. Uniform and permuted weights ignore or misassign these sizes, and as Section 5.2 now states, group-size weights already account for most of our gain over uniform weights there, improving on them by a factor of . Beyond group size the sensitivities improve by a factor of on IMDB-BINARY and by factors between and within every dataset, so the sensitivity-specific gain over group size is similar across the four datasets.
Total cost. In adjoint mode the two endpoint solves and the overhead of the regression over the quadratic estimator together cost about coalition solves at , timed on the same machine, and the forward mode that we ran corresponds to about sixteen (Appendix D.3). The two modes compute the same endpoint gradient, agreeing to a relative difference of on all targets at , so the estimator can use either. The adjoint mode solves one system per endpoint at any , and on our graph-classification models the two endpoint solves cost about two cold-start coalition solves.
We therefore give each competing method two or four extra coalitions, two for the adjoint solves and four also allowing for the regression time (Appendix F.11). Our estimator stays ahead of second-order PolySHAP and OddSHAP with either number, in all model-seed-budget cells. Even with sixteen extra coalitions, the forward-mode count, it stays ahead of each in of the cells. Cubic ridge, uniform and group-size weights run the same regression as ours, so two extra coalitions put them at about equal total cost. Our gains over them then become , and , with intervals that include one, and with the sixteen extra coalitions of the forward-mode count these three controls overtake our estimator. The most direct comparison at equal cost uses weights that need no solves at all. With weights from the game's own boundary marginals, read from the singleton and complement rows that our designs already contain, our estimator gains , and over the same three controls on exactly the same coalitions.
Mask extension. We agree that the selected direction is an inductive bias, and the revision states it as effect heredity. As you note, Theorem 1 constrains only rules that are unaffected by adding degree-two games. Once that invariance is given up, the game's own information can select a useful term, and the boundary-marginal weights above do exactly this. The boundary marginals and are the endpoint gradients of the game's own multilinear extension, so their magnitudes are determined by the Shapley game, and with weights from them the estimator is within four percent of the model-gradient version on MUTAG, PROTEINS and NCI1 at . The direction is then chosen from the singleton and complement rows, and its coefficient is still estimated from the residual of the whole sample. Model gradients remain the default because they work whether or not the design contains the singleton and complement coalitions, with two solves at any , and on IMDB-BINARY they halve the error.
The empty-endpoint control. Weights built from the empty-mask gradient alone give times the balanced error of the two-endpoint rule at , with interval , and times at (Appendix F.5), and Section 5.2 now reports this in the main text. The agreement is consistent with Proposition B.4, since the two endpoint gradients of a single-index model are parallel. In practice, one adjoint solve at the empty mask gives nearly the same accuracy as two.
Response to Reviewer MeJr (3/3)
Scope of the theorems. The abstract and introduction now state Theorem 1 for rules that read only endpoint data and are unaffected by adding degree-two games, and the introduction states the exactness of Theorems 2 and 3 only for a quadratic game plus a multiple of the selected regressor. Outside that class, Appendix C.1 now shows that at our budgets every estimator that reads the sampled values and the endpoint data of a mask extension has unbounded error over all games, so a guarantee for an arbitrary game has to depend on the game, as the error bound of Corollary B.1 does. Theorem 2 holds for any supplied regressor, and Appendix C proves Theorem 3 for weights from model gradients and from boundary marginals alike.
Larger group counts and ControlSHAP. The revised paper adds both. The study with groups reuses the twenty models, with explained graphs each, at five budgets from to . The cubic model stays underdetermined at every budget, with at most interior rows for its free coefficients, and the complete game of coalitions gives the exact Shapley value (Appendices E.7 and F.15). The gain over second-order PolySHAP is , against at . Our gains over the permutation, group-size and cubic-ridge controls are , and , with intervals above one, and our gain over uniform weights is , with interval , which includes one. The cubic matrices remain well conditioned (Table F.23). GCV leaves the coefficient of the selected regressor unshrunk in to percent of the fits. The median compact condition number of the penalized remaining block, over fits with a finite selected penalty, stays between and across the budgets. The per-target computation does not depend on the number of cubic terms (Appendix C.6).
Appendix F.14 shows that the partial-regression step is a control-variate correction with a least-squares coefficient. On designs where the quadratic stage is exact, a Taylor surrogate of degree at most two leaves the quadratic estimator unchanged, so we adapted ControlSHAP with the cubic Hermite interpolant of the same endpoint data. With a scalar coefficient its correction then moves the attribution along the same vector as our partial-regression step, the exact attribution of the selected regressor minus that of its quadratic fit, and only the coefficient differs. Given the same gradients and coalition budget, the best variant of ControlSHAP gains over second-order PolySHAP and our partial regression alone gains . Adding the ridge on the remaining cubic directions, our full estimator gains over the best ControlSHAP variant, with interval . If the coefficient is instead the one that the Hermite interpolant fixes from the endpoint data, the error is times that of second-order PolySHAP, so on these models calibrating the coefficient on coalition values is much more accurate. RegressionMSR, the implementation of regression-adjusted Monte Carlo in shapiq, has higher error than ours on each of the seven datasets of the paper.
Uncertainty and the scope of the comparisons. Section 5.1 now describes the four intervals as marginal. All four gains exceed one simultaneously in percent of the bootstrap draws, a descriptive frequency that is not a classical -value. We agree that the OddSHAP comparison is secondary evidence, since it changes the sampler and the estimator jointly, and Appendix E.6 treats it as such. With our design drawn from a paired sampler uniform over coalition sizes, like OddSHAP's own, our estimator stays ahead in all model-seed cells at every budget of the study. We also agree that four datasets limit how far the intervals generalize. The hierarchical bootstrap resamples datasets at its top level, but with four of them it can only give a coarse interval. Further evidence comes from three citation networks and five benchmark games of shapiq built on tabular, image and language models, and from the study with twelve groups, which repeats the primary comparison on the same models with a different grouping.
We hope that these answers address your questions, and we would be glad to discuss any of them further. Thank you again for the effort and expertise that you have devoted to our work in this review. The diagnostic that you suggested is now Appendix F.8, and your questions on cost and scope shaped Sections 5.2 and 5.5.
follow-up and reconsideration
My main concern regarding the lack of theoretical justification for why first-order endpoint sensitivities should identify relevant third-order interactions remains unaddressed. Since the proposed method relies on this assumption, I am revising my rating from 3 (weak acceptance) to 2 (weak rejection).