- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TopoSHAP, a cooperative game-theoretic framework for explaining the predictions of topological neural networks and the contributions of their neighborhood modules. TopoSHAP-cell treats cells of different ranks as players, constructs coalitions through masking, and computes Shapley attributions, while using a separate greedy elimination procedure to generate compact explanations. TopoSHAP-neighbor evaluates the contributions of neighborhood modules and generates candidate architectures using masked validation performance after five epochs of training the full model. The authors evaluate explanation quality on GraphXAI, lifted topological datasets, and MANTRA, and assess computational cost and predictive performance through architecture-selection experiments across multiple datasets.
Critical strengths
- The paper extends the objects of analysis to cells of different ranks and neighborhood modules within the network. It explicitly defines masking, neighborhood reconstruction, and coalition evaluation procedures, supporting different model families, including GCCN and HOPSE.
- The paper connects explanation analysis with model design and provides a relatively extensive empirical evaluation.
Critical weaknesses
- FORGE already generates cell-level explanation masks as an intermediate step, whereas this paper primarily emphasizes that its final outputs are node-level attributions, without sufficiently discussing the overlap and differences between the two approaches to higher-order explanations. Furthermore, the main graph-explanation experiments lack comparisons with recent, directly relevant methods such as FORGE and GraphSHAP-IQ. Consequently, the current results do not sufficiently establish TopoSHAP’s additional contributions in explanatory capabilities, explanation quality, and computational efficiency.
- Cells of different ranks exhibit structural dependencies and overlapping information. Treating them as independently maskable players therefore affects how the resulting attributions should be interpreted. In Table 4, the relative performance of detachment and feature zero-filling reverses across backbone models, further indicating substantial sensitivity to the choice of intervention.
- For architecture selection, the practical pipeline relies on masked scores after five training epochs. However, the optimal-architecture recovery results in the exhaustive evaluation use oracle scores obtained from fully trained architectures, which do not directly validate the reliability of the early-stage scoring mechanism.
- Qualitative case studies of explanation quality are limited. Although Figure 3 presents attribution distributions on MANTRA, the paper primarily evaluates explanation quality through quantitative metrics and lacks visualizations that map the explanations back onto the corresponding input structures.
Questions for authors
- FORGE already introduces intermediate cell-level explanation masks. Could the authors clarify the overlap and differences between TopoSHAP and this mechanism?
- How closely do the masked scores after five training epochs agree with the performance of candidate architectures after continued training? On the existing exhaustive benchmark, how often does the practical low-cost pipeline include the optimal architecture among its candidates, and how does its final selection performance compare with the oracle?
- Table 4 shows that, on Benzene, replacing the default detachment strategy with feature zero-filling increases the explanation score for GCN from 0.377 to 0.648, but decreases the score for GIN from 0.969 to 0.569. A similar reversal occurs on Mutagenicity. Could the authors explain this phenomenon and discuss its implications for the applicability of the default masking strategy and the conclusions drawn from the explanations?
- The paper primarily compares against established methods such as GNNExplainer (2019), PGExplainer (2020), and SubgraphX (2021), while omitting more recent relevant approaches already discussed in the paper. Could the authors include comparisons with recent relevant methods?
2 comments
Thank you for the detailed responses and additional experiments. The direct comparison with FORGE and the continued training experiments strengthen the empirical evaluation and address some of my concerns.
However, some issues remain insufficiently resolved. First, reproducing the GCN/GIN reversal confirms the method’s sensitivity to the choice of intervention, but its cause remains unexplained—at least, I could not readily find such an explanation in the revised manuscript (because the authors did not mark the revisions in colored text). In addition, Figure 6 provides useful qualitative examples, but its readability still needs improvement. Although the color meanings are stated, the text is too small. A color bar with numerical ticks would help readers interpret the results.
Overall, I appreciate the authors’ efforts to improve the paper, but these additions are not sufficient to change my overall assessment. I will therefore maintain my original score.
FORGE Comparison, Continued Training, and Visual Explanations
We thank the reviewer for their helpful feedback. These clarifications and additional experiments strengthen our original contribution: extending explainability to TNNs.
1 What does TopoSHAP add beyond FORGE? | W1 Q1 Q4
We acknowledge that while FORGE does use intermediate cell masks in order to explain graph-level outputs; TopoSHAP markedly and definitively extends beyond FORGE (Sec. 2):
Following your request, we add two direct comparisons and an exact Shapley reference:
2 Why does the way we remove cells matter? | W2 Q3
We agree that removal choices affect explanations. Separately masking cells does not assume statistical independence: attribution reflects model behavior across combinations of cells (Sec. 4.1). Zero-fill retains connections, while detachment removes cells from computation, changing information flow.
We reproduce the reported GIN/GCN reversal after retraining: zero-fill improves greedy motif recovery for GCN but reduces it for GIN on both Benzene and Mutagenicity (three seeds; Tables 15–16).
An explanation must therefore specify its intervention. Detachment measures reliance on supplied cells; remove-and-re-lift rebuilds higher-order inputs after base-node removal (Sec. 4.1; App. F.2).
3 Do scores after five epochs predict final performance? | W3 Q2
We ran 198 continued-training experiments: six architectures formed by removing one neighborhood at a time, across 11 settings and three seeds. Each resumes from its five-epoch checkpoint (App. F.9).
Selected models achieve similar mean performance to continuing the full model, often retaining fewer neighborhoods:
Can early scores identify the better candidate? They correctly order the two candidates in 12/24 untied classification runs and 1/3 ZINC runs. Our pipeline trains both further and chooses using continued validation scores. The correlations cover only six architectures chosen using early scores.
How often do we find the benchmark optimum? Against the 64 architectures trained independently from scratch, our two candidates include one within 1σ of the best in 22/33 runs, including the exact best in 5/33. The final selected architecture meets this tolerance in 16/33. Here, σ is the median standard deviation across training seeds for the 64 architectures. In this benchmark, selected architectures average 1.01 percentage points below the best classification accuracy and 0.0180 higher ZINC MAE (Table 39).
The submission already includes continued training and validation-based selection (Sec. 6.1, Table 9).
4 Visualizing explanations on inputs | W4
We add Figure 6 (p. 33) to complement MANTRA: signed atom/bond/ring attributions, greedy selections and annotated motifs on Benzene and Mutagenicity. It shows the best and worst structure-only GCCN cases among ten inspected molecules per dataset (Sec. 5.2), with interpretation in App. F.1.