- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper observed that "removing an edge" in a message-passing GNN is ambiguous: structural deletion (T_S) removes the edge and recomputes every edge-set-dependent quantity (degree normalization in GCN, attention softmax in GAT), whereas message-channel suppression (T_M) keeps the edge and only zeros the transmitted message. The authors 1) formalized the two interventions in one message-passing template and gave a sufficient condition for their equivalence, 2) pointed out that the choice enters twice, once when an explainer perturbs the model to build an edge ranking and once when the evaluator perturbs it to compute deletion fidelity, 3) cross the two choices in a 2x2 construction-by-evaluation design with contrasts C_S=F_SS-F_MS,C_M=F_MM-F_SM and interaction D=C_S+C_M.
Critical strengths
The motivation is clear, and the paper pined down an ambiguity that prior work left unresolved. Deletion fidelity was reported throughout the GNN-explanation literature with the word "removed" and nothing more, and the paper showed that this hides two different interventions, structural deletion and message-channel suppression, which produce different input to the model whenever the architecture uses degree normalization or attention. This paper was the first to make the choice itself explicit and to note that it is made twice, once by the explainer and once by the evaluator.
Critical weaknesses
W1) The headline "home advantage" claim was supported only for edge IG and may be a property of IG's construction rather than of fidelity. Signed IG along the T_S gate path attributed f_S (1)-f_S (0); evaluating with T_S retraces that same path, thus yielding that r_S scoring higher under T_S is close to what the completeness identity already guarantees. The paper did not test an explainer without a path identity. When it switches to GNNExplainer, matched superiority drops from 9/11 to 4/11 and the signs of C_S,C_M scatter across all four quadrants (BA-2Motifs: +1.38/-0.32; BA-MultiShapes: -0.14/-0.71); however, the abstract and conclusion lead with the 9/11 figure. The authors attributed the GNNExplainer result to restart instability, but no experiment separated "noise" from "IG-specific effect". Without a perturbation-free or converged explainer in the crossing, the strong claim should be stated as IG-specific; as written, it overreaches. This is my main reason for not recommending acceptance.
W2) The empirical panel is narrow and the main text carried little of the evidence; combined with a near-trivial Proposition 1, the contribution rests on a thin base. Proposition 1 followed in a few lines from the definitions (adding zero to a sum, coefficients that do not depend on E); it is necessary to justify the GIN control but does not carry novelty on its own. The paper's weight therefore falls on the empirical finding, and that finding was established on 8 GCN cells and 3 GATv2 cells, the latter all molecular; thus, architecture and domain cannot be separated (acknowledged in Section 8). Mean aggregation, which violates the other condition, had no trained-model cell. More broadly, three backbones are too few for a claim about message-passing GNNs in general: GraphSAGE with mean aggregation (the paper’s own witness for the zero-absorption failure), the original GAT, and attention-based or transformer-style architectures (e.g. GraphGPS, Graphormer) were all absent, and GATv2 itself appears in only three cells. Since Proposition 1 predicted that any architecture failing either condition can diverge, the natural test of the paper’s generality is to run the same crossing on several such architectures; as it stands, the empirical claim was established on one degree-normalized model and one attention model.
Questions for authors
-
State explicitly which datasets GATv2 was run on and reconcile "22 cells" with nine datasets x three backbones. Were the nine GIN controls run including Tox21-GIN, which Appendix B says was excluded by the health check?
-
Can the crossing be extended to additional backbones that violate Proposition 1’s conditions, in particular GraphSAGE-mean (zero-absorption failure) and at least one more attention or transformer-style model, and can GATv2 be run on the synthetic datasets as well? The current panel of 8 GCN + 3 molecular GATv2 cells cannot distinguish an architecture effect from a dataset effect.
-
Can you run the construction-by-evaluation crossing with an explainer that does not perturb the model while constructing its ranking (e.g. attention weights, gradient x input, GNN-LRP), or with a mask-based explainer run to a certified convergence budget? If matched superiority persists there, the strong claim is general; if not, it is an IG property and should be stated as such.
-
Report depth-level (K = 2, …, 5) C_S,C_M and D for all 11 edge-IG cells, as done for BA-2Motifs-GCN. How many other cells change sign across depth?
-
Figure 1 used c_uv=1/|N(v)| and called it degree-normalized; GCN uses symmetric normalization with self-loops. Is the toy meant as a mean-aggregation example? If so, label it that way.
-
How does the framework relate to the removal-semantics axis of Covert, Lundberg and Lee (2021) and to GraphFramEx's hard/soft masking? A sentence positioning T_S/T_M against these would clarify what is new.
7 comments
Response to M8Cq (2/2): architectures, depth and positioning
This comment continues our reply.
W2 and Q2: architectures and domains. Occlusion now covers GCN, GATv2, GAT, GraphSAGE-mean and TransformerConv on the same five datasets: BACE, MUTAG, Mutagenicity, PROTEINS and Graph-Twitter. Matched superiority occurs in 20/25 combinations, respectively 4/5, 5/5, 5/5, 3/5 and 3/5. This tests mean aggregation and additional attention layers, although not GraphGPS or Graphormer.
We also tested GCN, GATv2, GAT and TransformerConv on all four synthetic datasets with degree features. Occlusion gives 7/16, including failures on all four TransformerConv cells. On the twelve GCN/GATv2/GAT cells, signed edge IG gives 12/12. Matched superiority therefore extends to additional architectures, but does not hold for every explainer on those architectures: on the same synthetic tasks, occlusion fails in cells where edge IG succeeds. The complete grids, including negative and inconclusive contrasts, are in Tables 13 and 14.
On the 28 additional configurations evaluated with both edge IG and occlusion, matched superiority occurs in 28/28 and 22/28, respectively, including two Tox21 retrains with class-weighted loss. In two of these cells, some trained models fail the IG completeness check, which requires an aggregate relative error below 5% on eight validation graphs: 9 of 40 in D&D–GATv2 and 2 of 40 in the Tox21–GATv2 retrain. Our protocol certifies an edge-IG cell only when every one of its models passes this check, so these two cells, counted on the remaining models, are not certified. On the other 26 cells, edge IG gives 26/26 (Appendix D.3). Edge IG has no result on Reddit-Binary–GAT, where the attributions are not finite at the required quadrature power. GraphSAGE on the four synthetic datasets fails the model-health check, because, with constant node features, mean aggregation cannot distinguish graphs without isolated nodes that have the same number of nodes. Table 7 lists every cell, with the reason whenever a cell is not counted.
On GraphSAGE and TransformerConv, we report the crossed comparison using occlusion, which applies removal and suppression directly. Edge IG and GNNExplainer approximate structural removal with a continuous gate on each edge. On these backbones, scaling down all incoming gates together leaves the normalized aggregate unchanged until the gates reach zero, where it can jump. Gradients along the continuous part do not capture that jump, so this relaxation can miss the effect of actually removing the selected edges, and we do not use edge IG or GNNExplainer on these backbones.
Q4: depth. For , original BA-2Motifs–GCN changes sign in and , and Synth-Dist–GCN in only. The other nine edge-IG cells retain positive contrast estimates at every depth; eight retain positive lower bounds throughout. We distinguish sign changes from loss of support (Table 15).
Q1, Q5 and Q6: accounting and positioning. The original design contains 22 dataset–architecture configurations: nine with GIN, nine with GCN and four with GATv2. The 11 core cells are eight GCN and three GATv2. GATv2 covers BACE, MUTAG, Mutagenicity and Tox21. We corrected the account of the original Tox21 models in Appendix B: they predominantly predict the majority class and were not all excluded before attribution. The extension includes separately retrained GCN and GATv2 models with class-weighted loss. The original Tox21–GIN remains an algebraic control regardless of predictive accuracy.
We have corrected the Figure 1 caption to identify a mean-style coefficient without self-loops, rather than GCN normalization. Covert, Lundberg and Lee already identify removal semantics as a dimension of explanation methods. GraphFramEx organizes evaluation around user needs and necessity versus sufficiency. Our distinction specifies the computation performed when a mask is applied: even a binary mask can remove edges and recompute context, or suppress messages while retaining that context. The contribution is the equivalence criterion and the construction-by-evaluation experiment for these two interventions.
Additional unresolved points
Thank you for the response and the additional experiments. However, several major points remain unresolved yet.
W1 / Q3. My concern was not that completeness guarantees the deletion ordering, but that constructing and evaluating along the same path makes matched superiority close to mechanical; the counterexample does not address this. The revision partly concedes the point: Section 7 states that matched superiority is more frequent when the explainer’s objective and the evaluator’s reading ask the same question, and Appendix D.3 notes that occlusion’s single-arc advantage holds by construction. The only construction without such alignment, gate saliency, gives 5/11. I therefore read the evidence as narrowing the claim from “IG-specific” to “specific to deletion-aligned constructions”. The abstract, however, still says that the advantage “also occurs with occlusion and signed gate saliency” without the 5/11 count or this qualification. Please state it there and in the conclusion. In addition, the convergence I asked for remains uncertified, as Section 8 acknowledges, and the 0/11 case is defended in the reply with D > 0, a weaker criterion.
W2 / Q2. I appreciate the added backbones, which I had proposed as a test of generality. The results, however, point the other way: the backbones that most clearly violate the conditions of Proposition 1 show the weakest matched superiority (GraphSAGE 3/5; TransformerConv 3/5 and 0/4 in Tables 13 and 14). The conditions therefore indicate when the two interventions differ, but not when the home advantage appears, which the conclusion now calls an open question. These backbones are also evaluated only with occlusion, which inherits the concern in W1. Finally, degree features are held fixed under structural removal (Section 3), so the input no longer matches the perturbed graph in the Reddit-Binary and degree-feature synthetic cells; and GraphSAGE was not run on the synthetic datasets with degree features, without a stated reason.
Response to M8Cq (1/2): evidence beyond IG
Please also see our two-part “General response” for the motivation, new results and changes to the revised PDF.
The new experiments show matched superiority beyond IG and beyond GCN and molecular GATv2, with failures that delimit its scope. The revised abstract states this dependence.
W1 and Q3: completeness and evidence beyond IG. IG varies all gates together; deletion visits binary masks selected by a ranking. Completeness constrains the sum of attributions, not the ordering of deletion effects. For f(g1,g2)=g1+2g2+6g12g2, straight-line IG assigns scores 5 and 4 with exact completeness, but deleting the corresponding gates decreases the output by 7 and 8. Even the first deletion need not follow the IG ordering.
We tested finite occlusion and signed gate saliency on the same 11 cells. Saliency is gradient × input with respect to the edge gates, evaluated with all gates at one. It uses no integration path or finite perturbation, but remains specific to each gate parameterization.
We also ran GNNExplainer with two objectives. Its default objective, which we call sufficiency, optimizes a mask whose retained edges preserve the prediction. The necessity objective optimizes a small mask whose soft removal lowers the probability of the predicted class, so it is aligned with deletion. This variant tests the role of the objective within GNNExplainer and does not add an independent explainer.
The outcome is fidelity on the target logit, read by deletion or by retention, which perturbs the complement of the selected edges. A cell is counted when both CS and CM have positive one-sided 95% lower bounds, clustering by training seed. These are nominal cell-level results, not a simultaneous guarantee across cells. Table 8 reports a sensitivity analysis with seed-level sign-flip tests and Benjamini–Yekutieli correction across 154 comparisons.
Occlusion is optimal for its own single-edge deletion by construction, but not for joint deletions. It still gives 9/11 from 10% deletion onward. Thus matched superiority is not exclusive to IG, but is not universal across explainers.
For the sufficiency objective at learning rate 0.1, the 0/11 result concerns deletion. Matched superiority requires both within-intervention advantages to be positive. Their sum, D=CS+CM, has a nominal 95% interval above zero in 5/11 core cells and in 26/29 additional configurations evaluated with this GNNExplainer setting (Table 7). Thus, an overall matching interaction can remain even when one construction does not win under its own intervention. Under retention, which perturbs the complement of the selected edges, the same masks meet the nominal criterion in 7/11 cells, and 2/11 remain supported in the sensitivity analysis. This sensitivity analysis uses discrete seed-level tests and corrects across 154 comparisons, which limits its power. However, a comparison that does not pass is inconclusive under that rule, rather than evidence of no effect. The gate-saliency experiment addresses the gradient × input alternative in Q3. These are fixed-budget mask results. For the sufficiency objective at learning rate 0.1, the median shares of loss decrease occurring in the last fifth of the 100 epochs are 0.9% under suppression and 1.3% under removal. Yet the median Spearman correlation between two restarts on the same graph is 0.35 and 0.52 (median over graphs within each cell, then over the 11 cells), and at most 0.62 in every cell except Reddit-Binary–GCN. Limited late-stage loss decrease therefore does not establish ranking convergence. Changing the objective nevertheless changes matched superiority from 0/11 to 9/11 under deletion at the same learning rate and budget. We do not attribute this difference to restart instability alone. The independent evidence beyond IG comes from occlusion and gate saliency.
The second part of this reply addresses architectures and domains (W2, Q2), depth (Q4), and accounting and positioning (Q1, Q5, Q6).