- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the explanation multiplicity that arises from prediction-preserving perturbations of the class-wise scores of multi-class classifiers. The main focus is on classifiers used as surrogates for clustering models, although the core contributions are not specific to clustering. The authors argue that, to make explanations less sensitive to these prediction-preserving perturbations, one should explain standardised scores instead of the raw model output.
Critical strengths
- The paper investigates a topic that I think is understudied in the explainability literature: what quantity should be explained for multi-class classifiers? Existing work either explain the logits, the constrast, or the softmax whil rarely considering the implications of these choices Proposition 3 demonstrates that explaining the scores s, the constrast, or the softmax probalities all attribute importance to the dummy variable, putting these common practices into question.
- The proof are correct and well-written. As a result it is easy to verify their validity.
Critical weaknesses
-
Some important citations are missing. Notably the prior studies of [1] and [2] are missing from the related work. [1] previously demonstrated that additive model interpretations are underspecified because of the translation orbits (they did not consider scaling the scores by a(x)). Since interventional shapley values will coincide with the centered shape functions of additive models, the current paper is actually a generalization of [1] for model with interactions. [2] previously demonstated that gradient-based explanations of deep neural networks are arbitrary when explaining the logits because of the translation orbits .
-
While the theoretical section of the paper is solid, the experiments are hard to follow. Notably, based on Figure 2 from the appendix, I was expecting the bulk of the results to demonstrate that explaining H instead of the score, contrast, or softmax leads to reduces multiplicity across models types and boostrap refits. From my understanding, only Table 8 from the appendix shows this results ( on two datasets. ). Why were these results put in appendix if they are the most aligned with the paper premise? The results in the main paper do not show the merits of explaining H so the message of the paper gets diluted.
[1]Zhang, X., Tan, S., Koch, P., Lou, Y., Chajewska, U., & Caruana, R. (2019, July). Axiomatic interpretability for multiclass additive models. In Proceedings of the 25th ACM SIGKDD international conference on Knowledge Discovery & Data Mining (pp. 226-234).
[2]Srinivas, S., & Fleuret, F. (2020). Rethinking the role of gradient-based attribution methods for model interpretability. arXiv preprint arXiv:2006.09128.
Questions for authors
The problem the paper studies is the ambiguity of explaining multiclass classifiers. So why does the paper focus so much on clustering surrogates? I think to proposed approach should work out of the box the any multiclass classifier (including deep learning)?
Can you explain in more details how you compute the least square coefficients in section 7.1? Is it a linear model of x used to predict the difference between the centered f and g at any sample x_i?
4 comments
CW2: We agree with this presentation concern and appreciate the concrete connection you draw to Figure 2. Readers should be able to assess the proposed target alongside the theoretical motivation, using comparable experiments and target-specific refit baselines. The main text now reports the probability//hard-label comparison under the full protocol on all six datasets, with refit baselines for each target (Table 4, §6.2), showing both where improves agreement and where its benefit is limited. It supersedes the two-dataset comparison you mention (former Table 8, now Table 14).
- Across families. raises the median cross-family similarity from to .
- Across refits. The refit baseline is essentially unchanged ( vs. ).
- What limits the gain. Most of the remaining multiplicity disappears only when the decisions at SHAP's hybrid points are forced to agree, and any target that reflects the decision inherits that disagreement: the hard label still gives .
- Where works. When the two surrogates' decisions are forced to agree at every input, reduces the median gap from to (for the binary dataset, reduces to the decision, so there the reduction holds by construction).
To make room, the multiplier diagnostics (former §7) were moved to Appendix F, and legacy or mixed-protocol analyses are either labelled as such or removed.
Q1: The theory holds for any multiclass score map, including deep networks, and the reversal also holds for gradient-based attributions (Integrated Gradients, gradients, gradient×input; reply to ZFdK). Clustering is the motivating case because the partition has no scores of its own: the surrogate's score representation is added by the analyst, whereas supervised probabilities target . The workflow is also used in practice, including at ICLR (Manduchi et al., 2022, App. I; further examples in our reply to Ewbc, CW1).
Q2: No model of x is fit. At each input x, a(x) is a scalar regression through the origin across the K class coordinates,
the pointwise least-squares fit of the centered vector gc(x) on fc(x). It is evaluated by calling both models at x, including at hybrid points. The equation now appears in the definition in the main text.
Response
Thank you for your detailed answers. I am satisfied with the updated manuscript and will increase my score to a 3. The paper now positions itself correctly relative to prior work that investigated translation orbits for additive models and neural networks. Also, Table 4 shows extended results regarding the H function across all datasets. This figure demonstrates that a large portion of disagreements are due to model extrapolation (as expected from prior work). What is really novel is that disagreements remain even after aligning the predictions over all inputs. These remaining disagreements are further reduced by explaining H instead of the output probabilities. These empirical results strengthen the message of the paper.
My only issue with the manuscript is that Table 4 is referenced in section 4.3 (the theoretical section) and not in the experimental section, which is odd. Table 4 should be referenced in section 6.2.
I now understand why the focus on clustering is relevant : classification has a natural target , wheras numerical scores at 100% arbitrary in clustering. This exacerbates prediction-preserving perturbations. Still, I think the discussion section could clarify that the insights from this work may extend to classification. Indeed, although it is natural to explain the conditional probabilities for classification, the probabilities returned by the model are not necessarily calibrated. One could calibrate uncertainties with temperature scaling but doing so change model interactions in unpredictable ways, especially when extrapolating the data.
CW1: We thank the reviewer for recognizing the strength of the theoretical section and for pointing us to two directly relevant references. These connections help position the contribution more precisely: the additive ambiguity is established in prior work, and our Proposition 2 should be presented explicitly in that context. The main reversal result concerns the additional ambiguity introduced by an input-dependent positive multiplier, which preserves decisions while changing contrasts. We also appreciate your observation that the experimental presentation should make the role of H more visible; the responses below address both the relationship to prior theory and the alignment between the theoretical claims and the empirical evaluation.
Both are now discussed in §3. Both concern the additive symmetry of the softmax: adding g(x) to every logit leaves the probabilities unchanged. Zhang et al. show that multiclass GAM shape functions are therefore underspecified and resolve this by post-processing. Srinivas & Fleuret show that logit gradients are arbitrary for the same reason. Our Prop. 2 is the Shapley-value form of this symmetry, and we now say so; since interventional Shapley values of a GAM are its centered shape functions, Prop. 2 reduces to [1] in that case.
We would distinguish the main result from a generalization of [1] to interactions. The reversal uses the multiplicative part a(x), which:
The new Prop. 4 (§4.3) characterizes what removes it: H is the maximal invariant of the positive affine orbit, so every pointwise target invariant to it (one computed from x and s(x)) is a function of H. H must be computed in the space where the transformation acts. Applied to logits, a(x) is an input-dependent temperature, removed by H of the logits; it changes the probabilities nonlinearly, so H of the probabilities generally does not remove it. Our experiments explain H of probability outputs, which is invariant to positive affine changes of the probabilities, not to temperature scaling of logits (§4.3).