- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes to use GNNs with handcrafted weights and known ground-truth explanations to evaluate GNN explainers. If an explainer gives an unexpected explanation, it is usually not clear whethe this comes from a the explainer itself or whether actually the model learned something unexpected. Using the proposed approach, unexpected explanations can be attributed solely to the explainer, since the model-ground truth is known. The results show that the readout function used in the GNNs can have a huge impact on the performance of the explainers, especially on that of Integrated Gradients.
Critical strengths
The paper proposes to use GNNs with handcrafted weights and known ground-truth explanations to evaluate GNN explainers. Explaining model decisions gets increasingly relevant as they are used more and more in practice, and also for understanding what is actually relevant for certain tasks. The paper well written and easy to follow.
Critical weaknesses
None
Questions for authors
Q1: In the paper, you essentially evaluate a subgraph search task. Do you think the explainers performance will be similar on real-world tasks?
Q2: There are tables and figures referenced that ate not in the (main) paper, for example S6. Could you add a pointer to where these can be found?
Q3: Table 2 only shows whether the explainer achieved perfect score or not. It would be more interesting to see the actual numbers. Can you please provide them?
3 comments
I thank the authors for addressing my concerns. Having read also the other reviews and answers to them, my score remains unchanged.
Thank you!
We thank the reviewer for their time, for reading the full discussion, and for their continued support of WOMBAT.
We thank the reviewer for their review and for recognizing our contributions. We are pleased to hear that our paper is well-written and easy to follow.
Q1: Performance transfer to real-world tasks. WOMBAT cannot assert similarity of performance of explainers on other GNN architectures (which likely would be used in real-world tasks). We observe that preserving our architecture and merely changing weights can alter XAI performance (while the underlying models detect the same motifs), and change of architecture alters these results as well. WOMBAT should mainly be seen as a controlled environment in which we can analyse XAI method’s performance.
As for the applicability of subgraph search tasks in real-world tasks, chemists have been defining subgraph detection tasks as simple proxies for many properties. For example, pan-assay interference compounds (PAINS) are molecules that often produce false positive results in high-throughput screening [F]. A set of molecular patterns was proposed to quickly identify problematic structures. Similarly, some subgraph alerts were defined for mutagenicity and other toxicity-related endpoints [G]. It is also worth noting that in many molecular prediction tasks, tree-based models on molecular fingerprints remain competitive baselines [H], which are essentially sequences of decisions on extracted molecular subgraphs (fingerprint bits).
[F] Baell, Jonathan B., and Georgina A. Holloway. "New substructure filters for removal of pan assay interference compounds (PAINS) from screening libraries and for their exclusion in bioassays." Journal of medicinal chemistry 53.7 (2010): 2719-2740.
[G] Sushko, Iurii, et al. "ToxAlerts: a web server of structural alerts for toxic chemicals and compounds with potential adverse reactions." Journal of chemical information and modeling 52.8 (2012): 2310.
[H] Jiang, Dejun, et al. "Could graph neural networks learn better molecular representation for drug discovery? A comparison study of descriptor-based and graph-based models." Journal of cheminformatics 13.1 (2021): 12.
Q2: Figure references. We thank the reviewer for reporting this issue. These figures are in the Appendix (e.g. Figure S6 on page 27). In Adobe Acrobat PDF Reader, they are properly marked, and one can navigate to them. In general, clicking on the figure number should navigate to the appropriate figure. This could be an issue with a specific PDF reader; however – could the Reviewer try a different one to see if a problem persists? If the problem still occurs, we would be happy to hear more, reproduce the issue on our end, and fix it if possible.
Q3: AUROC table. The AUROC table was in the Appendix; we’ve moved it to the main text to make it more visible (it’s in blue in the revised manuscript). We thank the reviewer for this suggestion, as we think that the paper’s readability will benefit from this change.