- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper proposes GraphVMP, an intrinsically interpretable variational GNN that aims to ground explanations in structural concepts discovered from data. It first introduces an unsupervised grouping model to learn a shared dictionary of subgraph templates without class labels or motif annotations, and uses their graphlet histograms as structural references. GraphVMP then aligns dedicated structural latent representations with these references through an MMD objective, while retaining residual latent channels for prediction. Node-level explanations are obtained by projecting structural latent activations back to the input graph.
The paper also introduces MotifCausalShift, a synthetic benchmark designed to evaluate prediction and explanation under controlled spurious motif–label correlations and structural distribution shifts. Experiments demonstrate improved explanation localization and OOD robustness compared with the model without MMD alignment, while maintaining competitive prediction accuracy on several standard graph benchmarks.
Critical strengths
S1: The paper introduces a grouping model that discovers shared subgraph templates without class labels or motif annotations, providing an interesting approach to constructing structural references for graph explanations. On MotifCausalShift, the model successfully recovers all four planted training motifs and isolates an unseen Star motif without a corresponding learned template. These results suggest that meaningful structural concepts can be discovered without explicit motif supervision.
S2: GraphVMP provides a systematic framework for aligning dedicated structural latent representations with discovered graphlet descriptors through MMD. The proposed MotifCausalShift benchmark is also useful for evaluating explanations under controlled structural shifts. In the primary stress test, GraphVMP improves node-level explanation ROC-AUC from 55.4% to 85.0% over the baseline without MMD, while achieving 97.0% classification accuracy. The improvement in localization persists after causal-motif replacement, providing encouraging evidence for the effectiveness of structural alignment.
Critical weaknesses
W1: GraphVMP aligns structural latent representations with graphlet descriptors through MMD, but this distribution-level alignment does not necessarily establish graph-specific correspondence between the latents and their underlying motifs. More importantly, the explanations are generated from the structural latents, while the classifier predicts from the full latent representation, including unconstrained residual channels. Consequently, the model could potentially achieve good structural localization while relying primarily on the residual channels for prediction. The paper does not provide sufficient evidence that the structural latents actually drive the predictive decisions, which is important for supporting its claim of intrinsic interpretability.
W2: The MMD targets are constructed from all detected templates, including spurious motifs, without specifying which structures are causally relevant. It is therefore unclear what mechanism encourages the model to prioritize causal motifs when both causal and spurious motifs are predictive during training. Furthermore, in Experiment 3, the spurious motifs are replaced at test time by an unseen Star motif, while the causal motifs remain familiar. The high explanation ROC-AUC may partly reflect the distinction between previously learned and unseen structural patterns, rather than the ability to distinguish causal from spurious motifs. Additional controlled comparisons are needed to separate these explanations and substantiate the claimed causal-subgraph localization.
Questions for authors
Q1: In Eq. (9), node-level explanations are computed from the structural latents, while classification uses the full latent representation. Could the authors provide evidence that predictions actually depend on the structural channels? For example, comparing structural-only and residual-only prediction, or intervening on the structural latent channels, would help establish whether the explanations faithfully reflect the model's predictive behavior.
Q2: In Experiment 1, causal and spurious motifs are perfectly correlated with the labels during training, while MMD alignment does not explicitly distinguish between them. Could the authors clarify which inductive biases explain GraphVMP's preference for the causal motifs? Additionally, for Experiment 3, could the authors evaluate test graphs that retain both familiar causal and familiar spurious motifs, and report their explanation scores separately? This would help determine whether the improvement comes from causal discrimination or from differences in motif familiarity.
Q3: The number of templates (M) and maximum subgraph size (k_{\max}) are important hyperparameters for the unsupervised grouping model, but their experimental values are not reported. Could the authors provide the settings for each dataset and analyze their sensitivity? In particular, since the number and size of meaningful motifs are generally unknown in real-world graphs, it would be helpful to understand how these choices affect template discovery, prediction accuracy, and explanation quality.
Q4: Section 4.2 evaluates template discovery only on the synthetic MotifCausalShift dataset. Could the authors provide qualitative or quantitative results on molecular datasets with known substructures, such as Mutagenicity, Benzene, AlkaneCarbonyl, or FluorideCarbonyl? Visualizing learned templates and comparing them with known functional groups would help demonstrate whether the proposed grouping method can discover meaningful structures in more realistic settings.
Q5: Table 2 reports single accuracy values for Experiments 1 and 2 without standard deviations or the number of runs. Given the substantial variance observed in Experiment 3, could the authors report results across multiple random seeds? Additionally, the main OOD comparison in Table 3 includes only DIR and the model without MMD. Comparisons with other representative intrinsic explanation methods, such as GSAT, CIGA, or TopInG, would help establish the relative effectiveness of GraphVMP.
Q6: A few details would benefit from clarification:
- The metric (p_{\text{keep}}) in Table 4 is not explicitly defined.
- The results in Table 3 and Appendix Tables 8–9 are not fully consistent, including the reported means and standard deviations for Experiment 3.
- The compared baselines are not systematically introduced in Section 4.1, and PiGNN and KerGNN in Table 5 appear to lack corresponding references.
- Additional details on the construction of graph-specific MMD targets, particularly how the detector handles multiple motif instances or unseen structures, would improve reproducibility.
3 comments
Thank you for the detailed rebuttal and substantial revisions. I appreciate the authors' efforts to address my concerns.
The additional channel-level analyses, motif interventions, familiar-distractor evaluations, and ablation studies have significantly strengthened the empirical support for the proposed approach. Most of my original concerns have been adequately addressed. Accordingly, I have increased my Soundness score from 2 to 3.
However, I maintain my overall rating of 2 (Weak Reject), primarily because the main experimental results and the newly introduced theoretical and ablation analyses rely on different model configurations. This leaves some uncertainty about the extent to which the stronger interpretability claims are supported by the primary results. The distinction between structural grounding and causal identification also warrants careful framing.
I appreciate the improvements and do not consider additional experiments necessary at this stage. Clearer alignment between the claims and existing evidence would further strengthen the paper.
Thank you for acknowledging the improvements and increasing the soundness score.
One configuration Thank you for raising this. We reran all the experiments and now all Exp. 3 results, in the main text and the appendix come from a single configuration. We hope this resolves your concern, and we would be grateful if you would consider updating your score.
Thanks for invaluable review.
W1 (Grounding). (i) Tabs. 2–6’s classifier reads residuals (only 0.2% of absolute logit-margin contribution); causal channels give −7 to −8 Grid/+10 to +11 Wheel logits (Tab. 6). Revised GraphVMP bars residuals: margins exactly sum template-channel contributions plus bias (§3.2); reopening them yields seed-dependent decisions (App. E). (ii) Deleting a motif leaves ≤2.5% of its channel contribution at all sizes (Tab. 13), returning both classes to chance (P=49%/52%; Tab. 15). Each channel responds to its motif in each graph.
W2 (Causal preference/familiarity). All-motif MMD targets do not specify causality. Predictive fit alone cannot identify causal roles under perfect training co-occurrence (§3.1). Exp. 1 supplies causal-motif descriptors as a preliminary sanity check. Exp. 2/3 target both types, differing in training reliability: causal motifs always set labels; spurious ones are balanced in Exp. 2, 80% label-aligned in Exp. 3. MMD grounds structure; classifier training determines use. Tab. 4’s lower rows keep both familiar motif types. Localization still improves over no MMD (69.8–70.1% vs 56.1–56.2%). Familiarity alone cannot explain this, though gains weaken vs Star. Other interventions: spurious deletion retains 98.1% accuracy; spurious/causal swaps flip 5.3%/88.1% of predictions (Tab. 5). This supports preferential, not exclusive, causal dependence.
Q1. See W1(i) and the residual-readout ablation (App. H/Tab. 16).
Q2 (Bias/familiarity). See W2. Exp. 1’s supplied causal information differs from Exp. 2/3’s all-motif targets.
Q3. MotifCausalShift Stage 1: Kg=6, dictionary capacity M=8, k_\max=10; Stage 2 keeps four templates (Q6(d)). App. J/Tab. 17 retrains discovery: three seeds/setting. All four motifs are recovered at k_\max=10,M=6,8,12,16 and M=8,k_\max=12,15. At k_\max=8, every seed misses nine-node Grid. These test dictionary recovery. App. G tests downstream dictionary/detection errors: deleting a causal template may preserve accuracy yet raise spurious channels’ absolute share (22.7% without Grid). Thus, dictionary coverage matters for explanations even at high accuracy.
Q4. Tab. 1(b) reports held-out Benzene/Mutagenicity grouping. Recall/precision: Benzene 0.85/0.77; Mutagenicity 0.97/0.31 for NO₂/NH₂, often with the attached ring. App. I/Fig. 6 shows learned Benzene templates/held-out groups. All three seeds recover a six-membered-ring template. Discovery is topological: Stage 1 omits atom types; heterocycles also match. These dictionary results differ from Mutagenicity grouping, which does not establish isolated functional-group templates.
Q5. Tab. 2 gives eight-seed mean ± SD for Exp. 1/2. Tab. 3 adds CIGA/TopInG alongside DIR. CIGA has comparable localization (85.1±7.8% vs GraphVMP’s 85.8±6.8%) but chance accuracy (50.5±1.3%). TopInG is worse on both metrics.
Q6. (a) We rename pkeep original-label retention: the fraction of edited graphs predicted as the original label. P (Tabs. 13–16) instead is mean predicted original-label probability. (b) Tab. 3 uses Tabs. 11–12’s eight runs: accuracy/ROC-AUC 97.5±0.9%/85.8±6.8% with MMD, 91.0±9.9%/55.0±18.6% without. (c) §4.1 introduces baselines, cites PiGNN/KerGNN, and sources reported values. (d) App. K details detection/targets. For MotifCausalShift:
We keep the four learned templates most often recovered as exact Stage 1 training groups, label-free. Binarized templates use exact induced VF2 matching, no timeout/truncation. Copies need ≤b=4 external edges (generator’s largest planted boundary); all distinct copies at the minimum accepted boundary remain. All planted training copies are recovered exactly, no false positives. At b=6/8, recall is 1.00; only House/Cycle give false positives: ≤0.8%/6.7%, respectively.
Presence targets sm(G)∈{0,1} ignore counts: repeats/overlaps do not increase them. Deletion training (§3.4) independently deletes each detected copy with probability 1/2. A copy survives only if all its nodes do. MotifCausalShift’s planted copies are disjoint: ≤one/template/graph.
Unmatched structures add no template-presence entry. No detections gives zero references. Detection builds training targets/deletion copies; inference needs no detection. Unseen Star has no dictionary entry; test interventions assess its effect.
H∈N6×4 columns are the four templates’ 6D graphlet descriptors. Each graph has six references:
tj(G)=(Hj1s1(G),…,Hj4s4(G))/45∈R4,j=1,…,6.
Divisor 45, the templates’ largest descriptor entry, scales to [0,1]. These are compared to six structural posterior means μis∈R4.
Eq. (3)’s biased estimator compares six means/six references per graph, then batch-averages. The kernel is
k(a,b)=∑α∈{0.1,1,10,50,100}exp(−∥a−b∥2/(2α2)).
Matching does not pool graphs’ references.