- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework that constructs a computation graph from the internal computations of transformer-based models, where nodes correspond to selected attention heads and FFN neurons, and edges represent forward interactions between these components. A GNN is then trained on the graphs to classify properties, serving as a graph-based probe. To interpret the GNN predictions, the authors further apply post-hoc graph explanation tools, including GNNExplainer and gSpan-based motif discovery, to extract class-discriminative subgraphs and recurring interaction patterns. These motifs are subsequently treated as concepts and combined into logical rules that summarize the structural patterns associated with the GNN’s predictions.
Critical strengths
The paper is well-organized and easy to follow, and provides extensive empirical evaulation across datasets and models.
Critical weaknesses
- My main concern is about the novelty and the interpretability. Prior work has already introduced attribution graphs for analyzing Transformer computations [1]. The GNN explanation and rule extraction techniques employed in this paper are also largely based on existing methods. Also, although the extracted logical rules achieve high fidelity to the GNN predictions, their practical utility remains unclear as they lack human-understandable semantic interpretations. More importantly, while existing GNN explainability tools can identify subgraphs relevant to GNN predictions, the effect on the original transformer-based model is not evaluated. It would be better to conduct some intervention experiments on the identified components and interactions to validate their causal relevance to the original model.
- The MLP baseline generally outperforms both GCN and GAT on the proposed graph representations (Table 1), but the paper does not justify the necessity of GNNs. Also, given the stronger performance of the MLP baseline, it would be valuable to investigate whether its predictions can also be explained using existing model-agnostic interpretability methods.
[1] Ameisen et al. (2025). Circuit Tracing: Revealing Computational Graphs in Language Models.
Questions for authors
Most of my questions have already been mentioned in the weaknesses above. One additional question is whether the authors have considered applying standard linear or MLP probes directly to the intermediate hidden states of the original transformer-based models. How would their performance compare with the proposed graph-based classifiers?
3 comments
Response by Authors (Part 2)
W2: MLP baseline and necessity of GNN
Answer: We agree that the MLP matches or outperforms the message-passing models. We see this as a validation of our graph construction as the MLP used the same node features we construct. Its high accuracy shows that this construction step, which is part of our proposed method, captures behavior-specific information from the Transformer. However, classification accuracy is not the goal of the graph classifier. We use it as a probe that we can explain. A Transformer has thousands of heads and neurons, and from its output alone we cannot see which of them recur across many inputs of the same behavior. Representing the computation as a graph lets us use graph explanation methods.
To test whether these interactions matter, we randomly rewire the graph edges (same edge count and node degrees), retrain the GAT, and repeat the explanation, mining, and rule pipeline.
| Setting | GAT acc (orig / shuffled) | Motif (orig / shuffled) | Rule fidelity (orig / shuffled) |
|---|---|---|---|
| GoT-Cities | 0.88 / 0.90 | 0.46 / 0.15 | 0.80 / 0.56 |
| HC3 | 0.93 / 0.94 | 0.79 / 0.18 | 0.97 / 0.67 |
| WildJailbreak | 0.92 / 0.92 | 0.76 / 0.18 | 0.99 / 0.58 |
Classification accuracy remains similar after rewiring, but the motifs lose their class-specificity and rule fidelity drops substantially. We agree that model-agnostic methods can also explain MLP predictions. However, such explanations attribute the prediction to individual node features and do not directly represent interactions between Transformer components, which are the focus of GRAPH2RULE. To provide a component-level comparison, we additionally construct node-based rules using individual attention heads and FFN blocks without the GNN. Our interaction-based rules match or outperform these rules in four of six settings, with the largest gains on GoT-Cities and WildJailbreak with Llama-1B (0.85 vs. 0.70 and 0.87 vs. 0.72, respectively).
Revision: These results have been added to Section 5.1 and Appendix A.4.3 (Tables 9 and 10) in the revised version.
Q. Linear and MLP probes
Answer: Following the reviewer's suggestion, we trained linear and MLP probes on the intermediate hidden states of the frozen Transformer. The probes achieve higher accuracy, indicating that behavioral information is decodable from the full hidden-state representations. Part of this comes directly from the input tokens: on WildJailbreak and HC3, the token embeddings alone (layer 0) already reach 0.90 and 0.80. However, we deliberately exclude token identities and hidden states to investigate whether this information can also be recovered from individual Transformer components and their interactions. Despite using this restricted representation, our graph classifiers achieve strong performance. More importantly, while probes show that behavioral information is decodable, they do not identify which heads, neurons, or interactions are involved. GRAPH2RULE instead focuses on discovering recurring component interactions and expressing them as compact Boolean rules.
| Setting | Linear probe | MLP probe | Embeddings only (layer 0) | GAT (ours) |
|---|---|---|---|---|
| GoT-Cities, Llama-1B | 0.99 | 0.98 (0.98–0.99) | 0.50 | 0.92 (0.88–0.94) |
| GoT-Cities, Llama-3B | 1.00 | 1.00 (1.00–1.00) | 0.50 | 0.99 (0.99–0.99) |
| HC3, Llama-1B | 1.00 | 0.99 (0.99–1.00) | 0.80 | 0.94 (0.93–0.95) |
| HC3, Llama-3B | 1.00 | 1.00 (1.00–1.00) | 0.81 | 0.97 (0.96–0.98) |
| WildJailbreak, Llama-1B | 0.99 | 0.99 (0.99–0.99) | 0.90 | 0.92 (0.91–0.93) |
| WildJailbreak, Llama-3B | 0.99 | 0.99 (0.98–0.99) | 0.90 | 0.92 (0.92–0.92) |
Revision: These results have been added to Appendix A.4.3 (Table 12) in the revised version.
We hope these clarifications and additional experiments address your concerns, and we would be happy to answer any further questions.
Waiting for your feedback
Dear Reviewer TUCf,
Thanks for your constructive review and effort. We are approaching the end of the discussion period and would greatly appreciate your feedback. We hope our rebuttal has addressed your concerns. Please let us know if there are any other questions. We are eagerly awaiting your response.
Regards,
Authors
Response by Authors (Part 1)
We sincerely thank you for your careful review and constructive suggestions.
Answer:
Comparison with Attribution Graphs [1, 2]:
[1] Ameisen, et al., "Circuit Tracing: Revealing Computational Graphs in Language Models", Transformer Circuits, 2025.
[2] Dunefsky, Jacob, Philippe Chlenski, and Neel Nanda. "Transcoders find interpretable llm feature circuits." Advances in Neural Information Processing Systems 37 (2024): 24375-24410.
Intervention on the original Transformer. To test whether the identified components matter for the Transformer itself, we performed an intervention experiment on GoT-Cities with Llama-1B. We zeroed out the components of the five most class-discriminative motifs during the forward pass and asked the model to judge each test statement as true or false with a fixed prompt, reading the next-token probabilities of "True" and "False". As a control, we zeroed the same number of randomly selected components of the same type from the same layers, repeated with 20 random draws. Suppressing the motif components changes 71% of the model's true/false answers and reduces its accuracy from 0.68 to chance level (0.52), whereas suppressing random components changes only 3% of the answers on average and leaves accuracy unchanged (0.69); none of the 20 random draws produces a comparable effect. This shows that the components identified by our motifs are causally relevant to the Transformer's own behavior.
Revision: We have added a discussion of the attribution graph paper [1] and the transcoder paper [2] to the Related Work section of the revised manuscript. Intervention results have been added to Appendix A.4.3 (Table 13) in the revised version.