acceptodds
Under review as a conference paper at ICLR 2027

Graph2Rule: Learning Behavioral Rules from Transformer Computation Graphs

Abstract

Large language models (LLMs) exhibit distinct internal computation patterns across different input types, yet these patterns remain difficult to characterize due to complex interactions among distributed attention heads and feed-forward neurons. Our goal is to understand whether these component interactions contain behavior-specific patterns and whether these patterns can be expressed in an interpretable form. To achieve this, we propose GRAPH2RULE, a framework to represent the computation of a pretrained transformer as a directed weighted graph. In this graph, nodes represent selected attention heads and feed-forward neurons, while edges represent their forward interactions. We begin by testing whether these computation graphs contain sufficient information to distinguish behavioral classes without access to the original input text. Across three datasets and multiple transformer backbones, our graph-based classifiers achieve strong performance. Importantly, the graph representation and GNN classifier enable us to analyze the structure underlying these predictions via GNN explainability techniques. Specifically, we extract sample-level explanation subgraphs, mine recurring class-discriminative motifs from these subgraphs, and convert these motifs into interpretable logical rules. Our results show that transformer component interactions contain structured behavioral information that can be represented as graphs and translated into compact and human-interpretable symbolic rules.

Primary area:
interpretability and explainable AI
Keywords:
Transformer interpretability, computation graphs, graph neural networks, subgraph mining, behavioral classification, logical rule discovery

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TUCfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework that constructs a computation graph from the internal computations of transformer-based models, where nodes correspond to selected attention heads and FFN neurons, and edges represent forward interactions between these components. A GNN is then trained on the graphs to classify properties, serving as a graph-based probe. To interpret the GNN predictions, the authors further apply post-hoc graph explanation tools, including GNNExplainer and gSpan-based motif discovery, to extract class-discriminative subgraphs and recurring interaction patterns. These motifs are subsequently treated as concepts and combined into logical rules that summarize the structural patterns associated with the GNN’s predictions.

Critical strengths

The paper is well-organized and easy to follow, and provides extensive empirical evaulation across datasets and models.

Critical weaknesses
  1. My main concern is about the novelty and the interpretability. Prior work has already introduced attribution graphs for analyzing Transformer computations [1]. The GNN explanation and rule extraction techniques employed in this paper are also largely based on existing methods. Also, although the extracted logical rules achieve high fidelity to the GNN predictions, their practical utility remains unclear as they lack human-understandable semantic interpretations. More importantly, while existing GNN explainability tools can identify subgraphs relevant to GNN predictions, the effect on the original transformer-based model is not evaluated. It would be better to conduct some intervention experiments on the identified components and interactions to validate their causal relevance to the original model.
  2. The MLP baseline generally outperforms both GCN and GAT on the proposed graph representations (Table 1), but the paper does not justify the necessity of GNNs. Also, given the stronger performance of the MLP baseline, it would be valuable to investigate whether its predictions can also be explained using existing model-agnostic interpretability methods.

[1] Ameisen et al. (2025). Circuit Tracing: Revealing Computational Graphs in Language Models.

Questions for authors

Most of my questions have already been mentioned in the weaknesses above. One additional question is whether the authors have considered applying standard linear or MLP probes directly to the intermediate hidden states of the original transformer-based models. How would their performance compare with the proposed graph-based classifiers?

3 comments
AuthorsOpenReview ↗

Response by Authors (Part 1)

We sincerely thank you for your careful review and constructive suggestions.

W1: Prior work and Intervention Experiments .

Answer:

Comparison with Attribution Graphs [1, 2]:

  • Direct Modeling of the Original Computation Graph vs. Replacement Models: Attribution graphs operate on a replacement model (built with cross-layer transcoders) rather than the target LLM itself. This introduces two key limitations: (1) the replacement model is only an approximation and does not perfectly replicate the behavior of the original model, and (2) more importantly, the resulting attribution graph characterizes the computational pathways of the replacement model rather than the native computation graph of the original LLM. Consequently, translating graph insights into direct causal interventions on the original model is not straightforward. In contrast, GRAPH2RULE extracts graphs directly from the native attention heads and FFN neurons of the frozen target LLM.
  • Unified Local and Global Interpretability: GRAPH2RULE provides both local (prompt-specific explanations via instance-level GNNExplainer subgraphs) and global (dataset- or domain-level explanations via frequent subgraph mining and symbolic Boolean rules) interpretability. By contrast, circuit tracing [1] mostly focuses on producing locally valid, prompt-specific attribution graphs.
  • Replacement-Model Pre-Training vs. Task-Specific Graph Learning: Constructing attribution graphs requires pre-training a replacement model (the cross-layer transcoder) over broad corpora to mimic the behavior of the original LLM, which can incur substantial upfront computational cost. By comparison, GRAPH2RULE avoids training an architectural surrogate of the LLM altogether, instead training a task-specific spatial GNN directly over single-pass activation graphs extracted from the target behavioral dataset.

[1] Ameisen, et al., "Circuit Tracing: Revealing Computational Graphs in Language Models", Transformer Circuits, 2025.

[2] Dunefsky, Jacob, Philippe Chlenski, and Neel Nanda. "Transcoders find interpretable llm feature circuits." Advances in Neural Information Processing Systems 37 (2024): 24375-24410.

Intervention on the original Transformer. To test whether the identified components matter for the Transformer itself, we performed an intervention experiment on GoT-Cities with Llama-1B. We zeroed out the components of the five most class-discriminative motifs during the forward pass and asked the model to judge each test statement as true or false with a fixed prompt, reading the next-token probabilities of "True" and "False". As a control, we zeroed the same number of randomly selected components of the same type from the same layers, repeated with 20 random draws. Suppressing the motif components changes 71% of the model's true/false answers and reduces its accuracy from 0.68 to chance level (0.52), whereas suppressing random components changes only 3% of the answers on average and leaves accuracy unchanged (0.69); none of the 20 random draws produces a comparable effect. This shows that the components identified by our motifs are causally relevant to the Transformer's own behavior.

Revision: We have added a discussion of the attribution graph paper [1] and the transcoder paper [2] to the Related Work section of the revised manuscript. Intervention results have been added to Appendix A.4.3 (Table 13) in the revised version.

AuthorsOpenReview ↗

Response by Authors (Part 2)

W2: MLP baseline and necessity of GNN

Answer: We agree that the MLP matches or outperforms the message-passing models. We see this as a validation of our graph construction as the MLP used the same node features we construct. Its high accuracy shows that this construction step, which is part of our proposed method, captures behavior-specific information from the Transformer. However, classification accuracy is not the goal of the graph classifier. We use it as a probe that we can explain. A Transformer has thousands of heads and neurons, and from its output alone we cannot see which of them recur across many inputs of the same behavior. Representing the computation as a graph lets us use graph explanation methods.

To test whether these interactions matter, we randomly rewire the graph edges (same edge count and node degrees), retrain the GAT, and repeat the explanation, mining, and rule pipeline.

SettingGAT acc (orig / shuffled)Motif ∣Δ∣\lvert\Delta\rvert (orig / shuffled)Rule fidelity (orig / shuffled)
GoT-Cities0.88 / 0.900.46 / 0.150.80 / 0.56
HC30.93 / 0.940.79 / 0.180.97 / 0.67
WildJailbreak0.92 / 0.920.76 / 0.180.99 / 0.58

Classification accuracy remains similar after rewiring, but the motifs lose their class-specificity and rule fidelity drops substantially. We agree that model-agnostic methods can also explain MLP predictions. However, such explanations attribute the prediction to individual node features and do not directly represent interactions between Transformer components, which are the focus of GRAPH2RULE. To provide a component-level comparison, we additionally construct node-based rules using individual attention heads and FFN blocks without the GNN. Our interaction-based rules match or outperform these rules in four of six settings, with the largest gains on GoT-Cities and WildJailbreak with Llama-1B (0.85 vs. 0.70 and 0.87 vs. 0.72, respectively).

Revision: These results have been added to Section 5.1 and Appendix A.4.3 (Tables 9 and 10) in the revised version.

Q. Linear and MLP probes

Answer: Following the reviewer's suggestion, we trained linear and MLP probes on the intermediate hidden states of the frozen Transformer. The probes achieve higher accuracy, indicating that behavioral information is decodable from the full hidden-state representations. Part of this comes directly from the input tokens: on WildJailbreak and HC3, the token embeddings alone (layer 0) already reach 0.90 and 0.80. However, we deliberately exclude token identities and hidden states to investigate whether this information can also be recovered from individual Transformer components and their interactions. Despite using this restricted representation, our graph classifiers achieve strong performance. More importantly, while probes show that behavioral information is decodable, they do not identify which heads, neurons, or interactions are involved. GRAPH2RULE instead focuses on discovering recurring component interactions and expressing them as compact Boolean rules.

SettingLinear probeMLP probeEmbeddings only (layer 0)GAT (ours)
GoT-Cities, Llama-1B0.990.98 (0.98–0.99)0.500.92 (0.88–0.94)
GoT-Cities, Llama-3B1.001.00 (1.00–1.00)0.500.99 (0.99–0.99)
HC3, Llama-1B1.000.99 (0.99–1.00)0.800.94 (0.93–0.95)
HC3, Llama-3B1.001.00 (1.00–1.00)0.810.97 (0.96–0.98)
WildJailbreak, Llama-1B0.990.99 (0.99–0.99)0.900.92 (0.91–0.93)
WildJailbreak, Llama-3B0.990.99 (0.98–0.99)0.900.92 (0.92–0.92)

Revision: These results have been added to Appendix A.4.3 (Table 12) in the revised version.

We hope these clarifications and additional experiments address your concerns, and we would be happy to answer any further questions.

AuthorsOpenReview ↗

Waiting for your feedback

Dear Reviewer TUCf,

Thanks for your constructive review and effort. We are approaching the end of the discussion period and would greatly appreciate your feedback. We hope our rebuttal has addressed your concerns. Please let us know if there are any other questions. We are eagerly awaiting your response.

Regards,

Authors

Reviewer_CyRyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Graph2Rule, a framework for extracting symbolic rules from graph representations of frozen Transformer computations. Selected attention heads and FFN neurons form graph nodes, with input-dependent features and parameter-derived edge weights. A GNN is trained to classify these graphs, after which GNNExplainer extracts local explanation subgraphs. Recurring class-discriminative motifs are mined and used as binary concepts for a shallow decision tree, yielding compact Boolean rules that approximate the GNN’s predictions. Experiments cover harmful-prompt detection, statement truthfulness, and human-versus-ChatGPT text classification across multiple Transformer backbones, reporting classification performance, rule complexity, and test fidelity to the auxiliary GNN.

Critical strengths
  • The framework connects local graph explanations to compact global Boolean rules, making recurring patterns easier to inspect than individual explanation subgraphs.

  • Experiments across three tasks and multiple Transformer backbones show that rules using only a few motif concepts can reproduce the auxiliary GNN’s predictions with reasonably high test fidelity.

  • Results over four independent runs and analyses of motif stability provide evidence beyond selected qualitative examples.

Critical weaknesses
  • The connection between the extracted rules and the original Transformer’s behavior remains insufficiently validated. The reported fidelity measures agreement with an auxiliary GNN trained to predict dataset labels, rather than faithfulness to the Transformer’s outputs. For example, distinguishing harmful from benign prompts does not establish which components cause refusal. Without interventions on the identified Transformer components, the motifs primarily describe patterns used by the probe, leaving their role in the original model unclear.

  • The contribution of graph interactions is not established. In Table 1, the MLP operating on the same node features generally matches or outperforms the message-passing models. On HC3 with Llama-1B, for example, MLP achieves 0.98 accuracy compared with 0.94 for GAT. Better classification is not a prerequisite for better explanations, but the paper needs evidence that interaction motifs reveal information beyond component-level statistics. Comparisons with node-based rules and shuffled-edge controls would help establish this.

  • The semantic meaning of the motif concepts is unclear. FFN neurons are grouped into blocks of 1,024 according to their indices, without evidence that neighboring indices correspond to shared functions. Consequently, the rules are compact but remain expressed through layer, head, and neuron-block identifiers. The paper does not sufficiently establish what these concepts compute or whether the findings remain stable under alternative neuron groupings, limiting the interpretability of the resulting rules.

Questions for authors
  • Does applying a learned rule to a new input require running GNNExplainer again to obtain the concept vector? Please clarify the test-time procedure and its computational cost, since the decision tree itself accounts for only the final step.

  • For GoT-Cities, are the true and false statements about the same city kept in the same split? A city-disjoint evaluation would help determine whether the discovered rules generalize beyond entities encountered during training.

  • Table 2 shows substantial variation in rule fidelity across runs, despite relatively stable classification performance. What drives this variation? The stability analysis identifies recurring layer pairs, but do the actual selected heads, motifs, and Boolean rules also agree across runs?

2 comments
AuthorsOpenReview ↗

Response by Authors (Part 1)

We sincerely thank you for your careful review and constructive suggestions.

W1. Interventions on the identified Transformer components.

Answer: To test whether the components identified by our motifs matter for the Transformer itself, we performed an intervention experiment on GoT-Cities with Llama-1B. We selected the five most class-discriminative motifs and the components they contain. For each test statement, we prompted the Transformer with a fixed question ("Is the following statement true or false? Answer with one word: True or False.") and, instead of generating text, read the next-token probabilities of "True" and "False"; the more likely of the two is the model's answer. We did this once without intervention and once with the motif components set to zero during the forward pass. As a control, we zeroed the same number of randomly selected components of the same type from the same layers, repeated with 20 random draws. We report three measures: answer flips, the fraction of statements whose answer changes compared with no intervention; P(correct), the probability the model assigns to the correct answer, normalized over "True" and "False" and averaged over statements; and accuracy, the fraction of statements answered correctly.

ConditionAnswer flipsP(correct)Accuracy
No intervention–0.710.68
Motif components zeroed0.710.520.52
Random components zeroed (mean, range over 20 draws)0.03 (0.00–0.09)0.71 (0.69–0.73)0.69 (0.62–0.77)

Suppressing the motif components changes 71% of the model's true/false answers and reduces its accuracy to chance level, whereas suppressing the same number of random components in the same layers leaves its behavior almost unchanged; none of the 20 random draws produces a comparable effect. This shows that the components identified by our motifs are causally relevant to the Transformer's own truth judgment.

Revision: These results have been added to Appendix A.4.3 (Table 13) in the revised version.

W2. Node-based rules and shuffled-edge controls.

Answer: We agree that the MLP matches or outperforms the message-passing models. We see this as a validation of our graph construction as the MLP used the same node features we construct. However, classification accuracy is not the goal of the graph classifier. We use it as a probe that we can explain. As suggested, we conduct two additional experiments.

Node-based rules. We build rules without the GNN: concepts are attention heads and FFN blocks taken directly from the raw graphs, selected by the same class contrast ∣Δ∣|\Delta|, with the same top-15 concepts and depth-3 tree. Test accuracy:

SettingNode-based rulesOur rules: mean (range over 4 seeds)
GoT-Cities, Llama-1B0.700.85 (0.75–0.91)
GoT-Cities, Llama-3B0.900.93 (0.80–0.99)
HC3, Llama-1B0.890.89 (0.86–0.92)
HC3, Llama-3B0.910.86 (0.78–0.93)
WildJailbreak, Llama-1B0.720.87 (0.76–0.92)
WildJailbreak, Llama-3B0.830.82 (0.72–0.89)

Our rules are comparable or more accurate in most settings, and unlike node-based rules they specify which interactions between components separate the classes.

Shuffled-edge control. We retrained the GAT on graphs whose edges are randomly rewired, and applied the same explanation, mining, and rule pipeline.

SettingGAT acc (orig / shuffled)Motif ∣Δ∣\lvert\Delta\rvert (orig / shuffled)Rule fidelity (orig / shuffled)
GoT-Cities0.88 / 0.900.46 / 0.150.80 / 0.56
HC30.93 / 0.940.79 / 0.180.97 / 0.67
WildJailbreak0.92 / 0.920.76 / 0.180.99 / 0.58

Rewiring the edges does not reduce classification accuracy. However, the explanations of the shuffled model lose their class-specificity: the mean ∣Δ∣\lvert\Delta\rvert of the top-15 motifs drops from 0.46–0.79 to 0.15–0.18, and rule fidelity drops by 0.24–0.41.

Revision: These results have been added to Section 5.1 and Appendix A.4.3 (Tables 9 and 10) in the revised version.

AuthorsOpenReview ↗

Response by Authors (Part 2)

W3. The semantic meaning, FFN neurons blocks and alternative neuron groupings.

Answer: We agree that 1,024-index blocks have no functional meaning; they serve only to coarsen 8,192 neurons per layer so that motifs can recur across prompts. To test whether our findings depend on this choice, we re-mined the same explanations under alternative groupings and refitted the rules:

SettingBlocks 256Blocks 512Blocks 1024 (paper)Blocks 2048Random groups of 1024
GoT-Cities, Llama-1B0.830.790.800.910.75
GoT-Cities, Llama-3B0.800.820.800.910.82
HC3, Llama-1B0.990.990.990.990.99
HC3, Llama-3B0.800.800.810.810.81
WildJailbreak, Llama-1B0.980.990.991.000.98
WildJailbreak, Llama-3B0.920.880.870.880.85

Fidelity remains stable across block sizes. Moreover, randomly grouping neurons into blocks of the same size gives similar fidelity, showing that our results do not rely on neighboring indices sharing a function.

Revision: These results have been added to Section 4.3 and Appendix A.4.3 (Table 11) in the revised version.

Q1. Test-time procedure and computational cost

Answer: Yes. For a new prompt, we (i) run the frozen transformer once and build its computation graph, (ii) run the GNN, (iii) run GNNExplainer to obtain the explanation subgraph, and (iv) match the 15 rule concepts and evaluate the depth-3 tree. Motif mining and rule learning are done once and are not repeated.

Median test-time cost per-prompt on HC3 and WildJailbreak on GAT:

StepHC3 1BHC3 3BWJB 1BWJB 3B
Transformer forward25 ms42 ms22 ms35 ms
Graph construction2.31 s5.94 s2.07 s5.53 s
Graph loading0.17 s0.31 s0.17 s0.31 s
GNN forward2 ms3 ms2 ms3 ms
GNNExplainer1.23 s1.26 s1.15 s1.26 s
Concept matching + tree26 ms49 ms27 ms49 ms
Total3.8 s7.6 s3.4 s7.2 s

GNNExplainer adds about 1.2 s per prompt, and a full prediction takes 3.4–7.6 s on one GH200 GPU.

Revision: These results have been added to Appendix A.3.3 (Table 5) in the revised version.

Q2. For GoT-Cities split.

Answer: Yes. The GoT-Cities split is already city-disjoint. We verified that the training set contains 588 unique cities and the test set contains 147 unique cities, with zero overlap between the two sets; consequently, none of the 294 test statements contains a city seen during training. Thus, the reported GoT-Cities results already evaluate generalization to unseen city entities.

Q3. Table 2 variation and Stability.

Answer: The variation comes from how class-specific the mined motifs of a run are. We analyzed the variation across all 24 runs (6 settings × 4 seeds). Rule fidelity is strongly correlated with the class contrast ∣Δ∣|\Delta| of the most discriminative motif (Spearman ρ=0.87\rho=0.87, p<10−7p<10^{-7} ). In all six settings, the seed with the weakest motif contrast also has the lowest fidelity. These results indicate that rule fidelity is closely associated with how class-specific the discovered motifs are.

SettingSeed 1Seed 2Seed 3Seed 4
GoT-Cities, Llama-1B0.800.920.970.98
GoT-Cities, Llama-3B0.801.000.961.00
HC3, Llama-1B0.990.950.900.91
HC3, Llama-3B0.810.940.950.79
WildJailbreak, Llama-1B0.991.000.840.97
WildJailbreak, Llama-3B0.870.930.970.75

Agreement across seeds. Although the exact combinations of motifs in the rules vary across seeds, several underlying Transformer components consistently recur. For example, on WildJailbreak with Llama-1B, H-L15-14 appears in all four runs, while F-L14-b5 and H-L15-23 appear in three. These results show that independently trained GNNs can identify overlapping component-level patterns even when their final symbolic rules differ.

Revision: This finding has been added to Section 5.4 in the revised version.

We hope these clarifications and additional experiments address your concerns, and we would be happy to answer any further questions.

Reviewer_yiK8OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a method that takes a transformer trained for Boolean classification, and extracts from it conjunctions of propositional literals, where each proposition represents a specific pattern of activations and attentions within certain layers and neurons of the transformer, which aim to provide a sufficient reason for the transformer's classification. The method is based on three stages: first, extracting "relevant" circuits from the transformers, represented as graphs; second, using a GNN-classifier and GNN-explainer to extract subgraphs of those circuits that play a primary role in the prediction, and third, use an ILP-like approach to distill general rules (Boolean Horn rules where the body are the aforementioned propositions and the head is a "yes" or "no" symbol) from the subgraphs. The experiments consider a few common benchmarks and transformer models, and show that the extracted rules fit well the GNN classifier behaviour.

Critical strengths

Clarity. The approach is very well explained - some initial intuitions are provided first during the introduction, so that when the full technical explanation is given, the reader is already familiar with the high-level picture, so they can focus on understanding the details.

Technical quality: the proposed approach, although purely empirical (i.e. offering no formal guarantees of correspondence), is reasonable: the transformer circuit is selected via most salient elements, and the explanation and rule mining approaches are well-understood and suitable for the task.

Novelty: as far as I am aware, this is the first approach that distills the internal circuits of transformers into general rules; most work with circuits is always input-specific, whereas the extracted "rules" here are general and apply independently of any input.

Critical weaknesses

Clarity/Technical quality: (important, but fixable) Section 3.2 is not very clear because it mixes up problem formulation with an explanation of the selected approach. For example, in "Problem 3: rule discovery" it would be helpful to be more explicit about the division between the assumptions and the problem: first please define what a "rule" is and how it is evaluated (i.e. with respect to a set of selected "motifs"), and then explain the problem of finding a rule that minimises error with respect to your chosen evaluation method. There are also a few gaps in the definitions: see also my rebuttal points below.

Technical quality: the extracted rules seem to explain well the GNN classifier. However, did you apply the rules directly on the transformer (e.g. check the transformer's weights to see if the rules are triggered or not)?

Significance: (major) the significance of the extracted rules is unclear. The experiments show that they approximate well the GNN classifier, but there is an open question about whether they approximate well the backbone transformer. Furthermore, the introduction motivates the rules as being "human-readable", however, they seem to be conjunctions of literals of sets of features describing certain activation patterns within the transformer. In other words, the signature of the rules (their vocabulary) is very distant from the human-readable, high-level and domain-specific concepts that one would need to use in order to achieve understanding of the transformer. Roughly speaking, a rule might say "if attention head X focuses on neuron Y... then predict positive" but users will be more interested in rules whose signature matches either the input vocabulary or other relevant high-level, human-understandable concepts. I believe that the paper should argue more for the utility of the extracted rules, and be more upfront about its limitation.

Questions for authors
  1. Line 171: please clarify what is c. Is this an index iterating over every node of the graph?

  2. Line 176: please clarify also what is S_i^exp. It is, in general, unclear what a "motif" is or how can it be contained in a sample. Is a motif a subgraph and a sample a graph? Are these defined for the whole graph or for each node independently?

  3. Line 176: please explain here the language you use to express the rules.

  4. Line 228: please clarify what is a "valid key position". Do you mean non-zero? What are "the remaining probabilities"? Similarly, what is a "valid token position"? - I suspect this means that you exclude the token with maximum attention value and then renormalise the remaining attention values so that they sum to one. If so, please explain this clearly; if not, please explain what you mean.

  5. Can you explain the intuition behind equation (1)? What is it based on?

  6. Line 238: what are K_h and K_f? Are these hyper-parameters fixed by you, determining how much you retain during graph construction? If so, how do you choose them in practice?

  7. Line 240: node features. The meaning of these features and the reason why you select these is unclear. Can you clarify this?

  8. Line 253: please explain the reason behind your choice of edge weights. What are they meant to represent? Why did you choose these specific equations?

  9. Line 308: what is "class-c"? Is this referring to the Boolean classes 1 and 0?

  10. Line 340: how exactly are paths simplified into Boolean rules? Would you, for example, include negated concepts to identify specific paths in the decision tree?

  11. Line 360: what experiment, precisely, is being run here? This only says "the classification performance", suggesting this is a straightforward application of the transformer models to the datasets. However, the discussion implies that we are drawing some conclusions about the selected transformer graph, which makes me think something else is going on. Please clarify.

  12. Do the extracted rules approximate well the backbone transformer? How robust are they?

2 comments
AuthorsOpenReview ↗

Response by Authors (Part 1)

We sincerely thank you for your careful review and constructive suggestions.

W1. Clarity/Technical quality: Section 3.2 clarification.

Answer: We agree that the rule definition and optimization objective should be separated. We have revised Section 3.2 to first define the motifs, their binary activations, and the Boolean rule, and then formulate the objective of finding a compact rule.

Revision: This has been clarified in Section 3.2 (Problem 3) in the revised version.

W2. Technical quality: the extracted rules and transformer's weights.

Answer: The rules cannot be checked against the Transformer’s weights alone, since these weights are fixed across inputs, whereas the rule conditions depend on prompt-specific interaction motifs. For a new prompt, we construct its computation graph from the frozen Transformer, obtain the GNN explanation, and check which learned motifs are present before evaluating the Boolean rule.

W3. Significance: Relation to Transformer and Human-readability.

Answer: Relation to the Transformer. We acknowledge that the rules are extracted from the GNN, so fidelity alone does not establish their causal relevance to the original Transformer. We address this with two experiments. First, in a shuffled-edge control, we rewire the graph edges and retrain the GAT. Classification accuracy remains similar, but rule fidelity drops from 0.80–0.99 to 0.56–0.67, showing that the discovered rules depend on the meaningful connectivity of the constructed graphs. Second, we intervened on the original Transformer: zeroing the components of the five most discriminative motifs on GoT-Cities with Llama-1B flips 71% of the model’s true/false answers and reduces its accuracy to chance, while zeroing the same number of random components from the same layers has almost no effect (3% flips).

Human-readability. Our rules describe recurring interactions between Transformer components. Their utility is in reducing thousands of heads and neurons to a few class-discriminative interactions. For example, on WildJailbreak with Llama-1B, F-L14-b5 → H-L15-23 appears in 85% of benign and 0% of harmful explanation graphs. We agree that “human-readable” may suggest input-level semantic concepts. Our rules are interpretable at the level of Transformer components: they are compact Boolean formulas over identifiable attention heads, FFN neuron groups, and their interactions. We have therefore revised the wording to “interpretable symbolic rules over Transformer components.”

Revision: These results have been added to Appendix A.4.3 (Tables 10 and 13), and the wording has been revised in the Abstract, Section 4.3, and the Conclusion in the revised version.

Q1. Line 171: what is c.

Answer: No. cc is not an index over graph nodes. It denotes the class label. Since our tasks are binary, c∈{0,1}c \in \lbrace 0,1 \rbrace. The quantity fc(P)f_c(P) measures how frequently motif PP appears among samples belonging to class cc.

Revision: This has been clarified in Section 3.2 (Problem 2) in the revised version.

Q2. Line 176: please clarification of SiexpS_i^{\mathrm{exp}}.

Answer: A motif PP is a recurring subgraph pattern, while SiexpS_i^{\mathrm{exp}} is the explanation graph obtained for sample ii using GNNExplainer. Each sample is one prompt, represented by one graph. Thus, P⊆SiexpP\subseteq S_i^{\mathrm{exp}} means that motif PP occurs as a subgraph of the explanation graph for sample ii. The motifs are defined over the graph structure, not independently for individual nodes.

Revision: This has been clarified in Section 3.2 (Problem 3) in the revised version.

Q3. Line 176: rule language explanation.

Answer: The rules are Boolean expressions over the discovered motifs. Each retained motif defines a binary concept CjC_j, indicating whether that motif is present for a particular sample. The depth-3 decision tree then combines these binary concepts through presence or absence conditions. A rule is a formula in disjunctive normal form: OR , AND and NOT.

Revision: This has been clarified in Section 3.2 (Problem 3) in the revised version.

Q4. Line 228: "valid key position" clarification.

Answer: A valid position refers to a non-padding token position. We first remove all padding positions. We then exclude the attention assigned to the first valid key position, which often acts as an attention sink, and renormalize the remaining attention probabilities of each query so that they sum to one. We also skip the first query position, which has no preceding context. The entropy-based concentration score is computed on these renormalized distributions. We do not remove the token receiving the maximum attention.

AuthorsOpenReview ↗

Response by Authors (Part 2)

Q5. Intuition behind equation (1)?

Answer: Equation (1) measures how concentrated the attention distribution of a head is for a particular input. A nearly uniform attention distribution has high entropy and therefore low concentration, while an attention distribution focused on a small number of token positions has lower entropy and higher concentration. We use this to identify heads with more concentrated prompt-dependent attention patterns.

Q6. Clarification of K_h and K_f?

Answer: KhK_h and KfK_f are fixed graph-construction hyperparameters specifying how many attention heads and FFN neurons are retained from each layer. We use Kh=4K_h=4 and Kf=64K_f=64 throughout all experiments. Their purpose is to obtain a sparse graph representation that keeps the most active components of each layer.

Q7. Line 240: The meaning node features.

Answer: Each node is described by its activation profile over the prompt: its strength (attention concentration for heads and mean absolute activation for neurons), its maximum activation, where in the prompt it is active (normalized center of mass, with a 32-dimensional sinusoidal encoding), the fraction of its activation on the last 30% of tokens, and the position of its peak. These features capture both how strongly or selectively a component responds to the input and where that response occurs.

Q8. Line 253: Choice of edge weights.

Answer: The connections are a design choice motivated by information flow through the Transformer residual stream. We connect retained components within the same and adjacent layers to obtain a sparse graph. Edge weights are computed from the frozen Transformer parameters by composing the relevant source-output and destination-input projections. Each weight measures how strongly the output of the source component is coupled to the input of the destination component, and is therefore independent of the prompt.

Q9. Line 308: Clarification of "class-c".

Answer: Yes, c denotes the behavioral class, with c∈{0,1}c \in \lbrace 0,1 \rbrace in all three datasets. For example, for GoT-Cities, the two classes correspond to false and true statements.

Q10. Line 340: Paths of Boolean rules

Answer: Yes, negated concepts are included. A root-to-leaf path corresponds directly to a conjunction of Boolean conditions. If the path takes the branch corresponding to Cj=1C_j=1, the rule includes CjC_j; if it takes the branch corresponding to Cj=0C_j=0, the rule includes the negated condition ¬Cj\neg C_j. The rule for a class is the disjunction of all paths that end in a leaf predicting that class.

Q11. Line 360: experiment clarification.

Answer: The Transformer itself is frozen and is not being trained as a classifier in this experiment. For each input, we first construct its Transformer-derived computation graph using the selected heads and FFN components, their node features, and the structural edges between them. We then train MLP, GCN, and GAT classifiers to predict the dataset label from these graph-derived representations. The purpose of this experiment is to test whether the constructed representation retains sufficient behavior-specific information to distinguish the classes. The subsequent explanation and rule-discovery analyses are then performed on the trained GAT classifier.

Q12. Backbone transformer approximation and robustness

Answer: We assess the robustness of the extracted rules in three ways. First, across four training seeds, the exact rules differ, but several underlying Transformer components recur. To understand the variation in fidelity, we analyzed all 24 runs (6 settings × 4 seeds) and compared rule fidelity with the class contrast ∣Δ∣|\Delta| of the most discriminative motif. We find a strong correlation (Spearman ρ=0.87\rho=0.87 ), and in all six settings, the seed with the weakest motif contrast also has the lowest fidelity. This suggests that lower fidelity is associated with less class-specific motifs. Second, the results are not strongly dependent on FFN neuron grouping: fidelity remains broadly similar for block sizes from 256 to 2048, including random groupings. Third, GoT-Cities uses a city-disjoint split, so the rules are evaluated on cities unseen during training.

Regarding the relation to the backbone Transformer, please see our response to W3: the rules depend on the Transformer-derived graph structure (shuffled-edge control), and the components they identify are causally relevant to the Transformer’s own behavior (intervention experiment).

Revision: These results have been added to Section 5.4 and Appendix A.4.3 (Tables 10, 11, and 13) in the revised version.

We hope these clarifications and additional experiments address your concerns, and we would be happy to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Reviewer CyRy

We thank the reviewer for the thoughtful follow-up and for raising the score. We are glad that the intervention experiment, the shuffled-edge control, the alternative grouping analysis, and the city-disjoint split clarification addressed the main concerns.

We acknowledge the remaining limitations. Our intervention suppresses all motif components together and shows that they are important for the model's answers. Isolating the effect of individual motif connections and testing the specific Boolean combinations in the rules, for example with path patching, is beyond the scope of this work. Similarly, the FFN neuron blocks are structural aggregation units used for motif mining and do not carry established functional semantics. We will state these points explicitly in the Limitations section of the final version and adjust the wording of our claims accordingly.

Thank you again for your constructive feedback, which has helped us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.