- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates the limitations of explicit Information Bottleneck (IB) objectives in GNN explainers and proposes BottleneckMLP, a lightweight module combining MLP-based compression with importance-aware Gaussian noise. The authors theoretically characterize the gap between variational IB surrogates and their intended information objectives, and empirically demonstrate improvements across multiple explainers and graph learning tasks.
Critical strengths
-
The analysis of explicit IB objectives, particularly the decomposition involving total correlation and marginal mismatch, provides valuable insights into the limitations of existing regularization approaches.
-
BottleneckMLP can be integrated into different GNN explanation frameworks with relatively limited architectural modifications.
-
The experiments cover multiple explainers, datasets, and tasks, with additional analyses of representation dynamics, information compression, and component ablations.
Critical weaknesses
-
The mutual information reduction guarantee assumes a fixed Gaussian noise channel, while the actual noise scale depends on input-dependent importance scores. The applicability of the theoretical result to the implemented mechanism is not fully established.
-
The analysis primarily characterizes information compression in representations or selection masks, which does not directly guarantee faithful explanatory subgraphs.
-
Although the overall improvements are encouraging, some configurations exhibit performance degradation, and the gains from MLP capacity versus importance-aware compression are not entirely disentangled.
Questions for authors
-
Can the theoretical MI reduction guarantee be extended to input-dependent importance-scaled noise? How does the analysis of mask information relate to the information retained by the selected subgraph?
-
Can the authors further clarify how representation compression translates into improved explanation faithfulness, beyond the observed empirical correlation?
-
Are the pretrained prediction models kept frozen in all post-hoc experiments? Could the authors discuss the negative results on certain backbones and explainers?
4 comments
Response to Reviewer KFVe part 2 of 2
(Response to reviewer KFVe part 2/2)
W3 (Negative results and capacity versus importance-aware compression)
Concerning capacity, we would like to refer the reviewer to our parameter-matched controls in Appendix AA and AB. The noise component does not add any trainable parameters over the variant with the MLP-only, so the ablations hold the parameter capacity fixed while only varying the noise mechanism. The shuffled importance experiment shows that when the architecture, the parameter count, and the noise distribution are kept the same, and only the alignment between the noise and the importances was changed, we see the drop in accuracy and AUC-ROC performance.
Concerning the negative results, we have collected them in the table below. They do indeed give meaningful insights when taken together, and we appreciate the reviewer for encouraging this analysis. We have included the table and discussion in Appendix AE.
| Setting | Metric(s) affected | Magnitude |
|---|---|---|
| ProxyExplainer | BA-2Motifs AUC; MUTAG AUC; Fluoride Carbonyl Fid- | 0.940-->0.779; 0.847-->0.822; 0.409 --> 0.633 |
| GOAt | BA-2Motifs Fid- (GCN and GIN); BA-2Motifs Fid+ (GCN) | 0.001 --> 0.056 / 0.070; 0.542 --> 0.509 |
| V-InFor | MUTAG Mean Probability; BA3 Prob-N | 0.968-->0.712; -0.0003 |
| DIR | Causal accuracy; test MRR | below original on all splits |
| PGExplainer | TreeCycles AUC | -0.3% |
| GNNExplainer | all metrics | inconclusive: all variants within 0.493--0.510 |
| PNA backbone | MUTAG explanation AUC | 0.993-->0.965 |
We have determined possible mechanisms that account for these negative results, and this relates to the discussion in Q1 of Reviewer NDFd connecting bottleneck width, noise scales, and convergence results.
-
The ceiling effects where the baseline is already near-optimal. Proposition 2 caps the achievable separation at . Where the base explainer is already at or near that ceiling the noise does not offer benefit and may lead to some degradation. This covers GOAt on BA-2Motifs (baseline Fid$^-$ of is essentially perfect and cannot be improved), ProxyExplainer on BA-2Motifs (baseline AUC ), and PNA on MUTAG (baseline AUC ). We note the converse holds in the same table: GOAt-GIN on BA-2Motifs, where Fid$^+$ was not at ceiling, improves .
-
Latent noise was already injected. ProxyExplainer generates proxy graphs with a Graph VAE and V-InFoR denoises through a variational latent, so both already inject noise before our module adds the noise component. The composite importance signal-to-noise ratio then falls to a level which is unproductive, as in (1.).
-
Degenerate embedding geometry. Three of the failures above are on BA-2Motifs, which we find has only distinct embedding vectors across test nodes because its node features are constant and embeddings are a deterministic function of local structure. A near-degenerate makes ill-scaled at any fixed , and is also why the entropy-power lower bound of Proposition 2 is vacuous there.
-
Objective mismatch, in the DIR case. DIR's invariance penalty is not a compression term, but rather, it enforces distributional invariance across environments, which our module does not provide. BottleneckMLP recovers what ablating that penalty loses, but it should not be expected to reproduce causal identification, and causal accuracy and test MRR are precisely the metrics that measure it. We interpret this more as a scope limit and not as a method negative result.
GNNExplainer is separate from all four because with every variant within – AUC, the setting does not discriminate between any methods and we draw no conclusion from it in either direction.
Q3 (Are the pretrained models frozen in the post-hoc experiments)
Thank you for pointing out that we did not mention this explicitly. We clarify that BottleneckMLP operates on the encoder’s embeddings, so in order for it to work in the post-hoc method pipeline, we need to retrain the base model. In these settings,(PGExplainer, GNNExplainer, ProxyExplainer, ReFine, V-InFoR), we retrain the classifier with the module in place and compare against the retrained baseline under identical settings. The comparison is matched, but it is not a post-hoc explanation of a fixed pretrained model in the strict sense. We would like to highlight that our method is mostly aimed at the ante-hoc setting, but we show that it operates well in the post-hoc setting as well for generalizability.
Thank you for the detailed rebuttal. I appreciate the additional ablations, theoretical clarifications, and discussion of negative results, which address most of my original concerns.
Two points would benefit from more precise wording: (1) the conditional MI reduction does not necessarily guarantee unconditional compression, and (2) the post-hoc results involve retraining the predictive models rather than explaining fixed pretrained models.
I believe these remaining issues can be addressed through clearer theoretical statements and appropriate qualification of the claims, without additional experiments.
Overall, the rebuttal has strengthened the paper, and I maintain my weak-accept rating (3).
Response to Reviewer KFVe
We thank the reviewer for their continued discussion, and for acknowledging that the paper has become stronger throughout the rebuttal. We greatly appreciate the reviewer for their contribution to improving our work.
We would like to address the reviewer’s two remaining concerns. The reviewer is absolutely correct that the conditional MI prediction does not necessarily guarantee unconditional compression. To this end, we have made the following adjustments to the manuscript, which are now marked in orange for distinction from our previous edits.
-
We have updated the abstract from “We show that the variational objectives used by existing explainers are invariant to the joint structure of the selected subgraph, and that the module's importance-scaled noise provably reduces input dependence while bounding predictive leakage” to “We show that the variational objectives used by existing explainers are invariant to the joint structure of the selected subgraph, and that the module's importance-scaled noise provably reduces the conditional input dependence of unimportant-node representations, given the importance scores, while bounding conditional predictive leakage through those representations.”
-
We have updated contribution point 2 in the introduction from “We prove that the noise component monotonically reduces , that label-relevant leakage through unimportant nodes is bounded by that same term, and that retained information is graded by importance,” to “We prove that the noise component monotonically reduces , that label-relevant leakage is bounded by the corresponding conditional compression term, and that retained information is graded by importance. We characterize the residual unconditional information carried by the input-dependent importance scores.”
-
In Section 4.2.1, we update “Theorem 1 shows it reduces monotonically, strictly under conditional non-redundancy,” to “Theorem 1 shows that, conditional on the importance scores , increasing the noise applied to unimportant nodes monotonically reduces , strictly under conditional non-redundancy. Because the importance scores are themselves input-dependent, this conditional reduction does not in general imply monotonic reduction of unconditional mutual information. We characterize this residual dependence explicitly in Remark 1”.
-
We also updated Proposition 2 to have made the conditioning explicit in the mutual-information, covariance, and entropy-power terms, consistent with the proof. No additional changes were necessary.
We also agree that our post-hoc experiments should not be interpreted as evaluations on unchanged, frozen pretrained predictors. Although the underlying methods (e.g., PGExplainer, GNNExplainer, V-InFoR, ReFine, and ProxyExplainer) are post-hoc explainers, our experiments incorporate BottleneckMLP into the corresponding trainable representation-learning pipelines, involving modification and retraining of the predictive models. We therefore establish compatibility with post-hoc explainer architectures and improvements within the resulting modified pipelines, rather than improvements in explanations of a fixed pretrained predictor.
We have added “We would like to be clear that the post-hoc designation refers to the underlying explainer methodology, not to a frozen-predictor evaluation protocol. In our BottleneckMLP variants, the module is incorporated into the relevant trainable representation pipeline. Thus, the reported comparisons evaluate modified and retrained pipelines rather than holding the original pretrained predictive model fixed” to Appendix O for clarification.
We appreciate the reviewer’s careful reading and positive assessment. These revisions sharpen the scope of our theoretical and empirical claims without changing the underlying method or experimental results. We are grateful that the reviewer recognizes the strengthened contribution of the paper.
Response to Reviewer KFVe part 1 of 2
(Response to reviewer KFVe part 1/2)
We thank the reviewer for the careful reading and positive soundness assessment. We are glad the decomposition involving total correlation and marginal mismatch provided valuable insight, and that the module's ease of integration and the breadth of the analyses were recognized. We are particularly grateful for the identification of the input-dependent noise issue, which we agree is the most substantive technical concern raised on this submission and that we correct below. We have marked all updates in blue text within the manuscript.
W1/ Q1a. (Input-dependent importance-scaled noise)
The reviewer raises a good point that the theory assumes a fixed noise level, but ours depends on the input. The fix for this is minor. Theorem 1's proof treats the noise as independent of X. One step uses h(Zi∣X)=2dlog(2πeti), which is valid only when ti is a constant. Ours is not because ti=σ/αi, and αi comes from the embeddings.
Zi now gives information about X in two ways. It informs about the signal fi(X), which is intended, and it also informs of how noisy it is. A noisier Zi means the node had lower importance, and this is something which the theorem never accounted for. Conditioning separates the two. To make the necessary updates to the Theorem, we now fix α=a. Then every ti=σ/ai is just a number, the noise is independent of X again, and Theorem 1 applies unchanged. The chain rule splits the quantity to match:
I(X;Zunimp)=leak via the noise levelI(α;Zunimp)+Theorem 1 appliesI(X;Zunimp∣α)
The second term is the one Theorem 1 covers, derivative and strictness condition included. The first is the leak the reviewer identifies, and it is at most H(α). Corollary 2 already has this shape, where H(M) is the cost of the mask revealing its own length. In this case H(α) is the cost of the noise revealing its own scale. We have updated Theorem 1 accordingly, and have added the above discussion as the remark following the theorem. We have also updated the proof of Theorem 1 and added the conditioning to Lemma 1 and Proposition 2 (with no effect on their theoretical outcome or proof). There were no other downstream effects caused by this correction.
Q1b. (How mask information is related to information in the explanatory subgraph)
Gs is determined by (G,s^), so conditional on G, these two carry the same information. s^ is the quantity that the regularizer in GSAT bounds, and this is why we cover this in Proposition 1. We have added a paragraph to the beginning of Section 4 that explains each term and how we use it such that this is clear from the start. We have also added these three terms to Table 6 in Appendix A with our list of symbols and notations for clarity. This clearly distinguishes the selection mask, I(G;s^), the embedding I(X;Z~), and the explanatory subgraph I(G;Gs). We note that Gs is determined by (G,s^), so conditional on G, the mask and the subgraph carry the same information, and s^ is the quantity GSAT's regularizer actually bounds (as is stated in Proposition 1).
Theorem 1 claim 1 provides the exact derivative ∂I/∂ti=−mmse(1/ti∣Z−i)/2ti2, so I(X;Zunimp) strictly decreases. This is not merely an upper bound. The part that is only a bound is the transfer to I(X;Gs) via the DPI. We have indirect empirical evidence that I(G;GS) does fall because the sparsity results (Appendix U) and the Fid$^+$ improvements both indicate more concentrated selections. We also see that the visualized explanatory subgraphs via BottleneckMLP are indeed the correctly identified ground truth motifs for BA-2Motifs and Mutag, and this is the exact mode where the bottleneck objectives are respectively optimized. We also have that Theorem 1 claim 3 propagates, since I(X;Z~unimp)→0 as miniti→∞ forces the unimportant nodes’ contribution to GS to vanish, and this is a valid guarantee.