- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper is among the first works studying Mechanistic Interpretability for GNNs, for the specific case of Neural Algorithmic Reasoning (NAR). The authors adapt standard attribution method previously introduced for other neural networks, to the case of GNNs, and then introduce a simple sensitivity analysis that is shown to be competitive performance wrt previous methods. In terms of evaluation, the paper focuses on two uses cases: extracting a circuit for the Bellman-Ford algorithm and on the CLRS NAR benchmark. The types of insights the authors can provide with their method are, for instance, for detecting shortcut learning.
Critical strengths
-
Mech. Int. for GNNs is understudied, and this paper provides two clear use cases where Mech. Int. can be useful. In particular, finding model shortcuts can be its most useful application
-
The motivation is solid and the paper is well written
-
I'd love to see more work on Mech. Int. for GNNs, nonetheless, the work has some pitfalls described below
Critical weaknesses
-
W1: There are some notation issues:
- In Eq. 5 and 6, is evaluated over the same quantity, hence it should be zero. The authors are probably referring to the two to two different graphs, and I recommend that the authors be explicit here
- In Eq. 6, WEIGHTGRAD takes as arguments the tuple , and disregards even tough, to my understanding is evaluated over them. I would recommend using the same notation as EAP.
-
W2: The authors claim that WEIGHTGRAD being simpler and cheaper. Why is that? Can you show a comparison of its runtime with respect to, for instance, EAP?
-
W3: In Section 5.2, the authors present the discovered circuits as fixed-size computational subgraphs. However, to my understanding, attribution patching gives soft importance scores, and it is therefore not clear the selection criteria to go from soft to a discrete selection of the most important components.
-
W4: Perhaps the most relevant critique I have is that the discovered circuits presented, e.g., in Figure 1b are difficult to understand. It is not clear to me how the circuit should be read, and it seems that the circuit contains some undefined notation: and have a superscript that conflicts with that introduced in Eq. 8
-
W5: Despite offering a new perspective on graph XAI, the paper lacks a discussion of relevant related work on more classical forms of XAI for GNNs.
Questions for authors
-
Q1: Can you please be more precise and explicit in defining what a neuron is? Is it the input to a layer, the output, the activation value, otìr a single weight rather than a full weight matrix of a layer?
-
Q2: You say that MINAR
exposes shortcut learning which is invisible even to our large out-of-distribution test set.
Nonetheless, the shortcut you later present can indeed be discovered by testing on OOD graphs, i..e, testing on reachable nodes that are far away more than hops. Why are the two statements not in contradiction?
3 comments
Answer to rebuttal
Thanks for your clarifications. I have increased my score from 2 to 3
Thank you for your consideration! Your feedback has helped us clarify several points, which we believe has strengthened the paper.
Response to Reviewer gn8Z
We thank the reviewer for their time and careful attention to our work. We agree that mechanistic interpretability is underexplored for GNNs, and we are pleased you found our paper well-motivated and clear. Below we hope to answer your questions:
W1. We thank the reviewer for their careful reading. We fix the notational issues in our revision.
We would also like to point out a subtle change we have made in Eq. (5): while circuit discovery in LLMs multiplies the difference in an edge’s input activations by the gradient with respect to its output, a first-order Taylor expansion at the neuron level requires an additional factor of Wi,j, as the gradient taken with respect to ψj will not capture how Wi,j scales the activation of ψi. Rerunning our experiments with this adjustment, our results have improved: We identify an additional component of the BFS shortcut, and both EAP and EAP-IG achieve slightly higher characterization scores on SALSA-CLRS.
W2. Thank you for this question. WeightGrad is simpler than EAP in that it avoids the overhead of collecting clean and corrupted activations from hidden layers, but a timing study actually shows that this is negligible in wall-clock time. We clarify this in the revision, removing the claim that it is cheaper.
W3. As we state in lines 200-201, we use the greedy algorithm from Hanna et al. (2024) to select a discrete circuit based on the soft importance scores. This greedy algorithm is described in detail in Appendix A and compared to alternative strategies.
W4. The circuits in Figures 1b and 2 use the neuron notation introduced in line 235. Each node in the circuit represents a neuron (see our response to Q1. below) and each edge represents a connection between two neurons, i.e. a model weight.
W5. This is a good point, and we thank the reviewer for the suggestion. The revised manuscript contains further discussion on classical explainability methods for GNNs.
The main difference we would like to highlight is that traditional GNN explanation focuses on identifying important subgraphs or node features from the input graph. Where intermediate computations are explored, they typically concern node or edge in the message-passing computation graph, which is distinct from the model computation graph we consider here.
Q1. To be precise, we take neuron to mean a single feedforward hidden unit, and we use its scalar activation values in Equation (5). Each weight Wi,j corresponds to a computation edge between neurons ψi and ψj.
Q2. It is true that a different set of OOD graphs would successfully identify failure modes for the shortcut in Section 5.3. Our claim is that the shortcut is invisible to our OOD test set, i.e. to the specific set of graphs that we use. While this is a subtle distinction, we note that without knowing the nature of the shortcut, there is no way to predict what shortcut might arise and to design a suitable OOD test set a priori. The only real way to verify how the learned shortcut works is to use MINAR to extract a circuit which is tractable to analyze.