acceptodds
Under review as a conference paper at ICLR 2027

CircuitFACT: Circuit Discovery for Mechanistic Interpretability of Graph Neural Networks

Abstract

Mechanistic interpretability seeks to understand neural networks by identifying the internal computational mechanisms underlying their behavior, often represented as circuits. However, recent circuit discovery methods have focused predominantly on transformer language models, and their application to graph neural networks (GNNs) remains largely unexplored. We propose CircuitFACT, a circuit discovery method for GNNs built on computation trees, which explicitly represent the message-passing dependencies involved in a GNN prediction. CircuitFACT first constructs these computation trees, attributes their components using Shapley-based attribution, and trims the resulting structures to obtain compact circuits. These circuits provide a structured view of the computations supporting a prediction and can be related back to the corresponding elements of the input graph.

Primary area:
interpretability and explainable AI
Keywords:
GNN, mechanistic interpretability, circuit discovery, interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cgUAOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to perform circuit discovery on GNNs by proposing CircuitFACT. For a GNN computation on a particular graph, CircuitFACT unrolls the GNN's computation tree on that instance at the neuron resolution, then uses Shapley values to attribute the computation tree components. The Shapley values are thresholded and the tree is pruned to produce a concise circuit. The authors compare CircuitFACT against a zero-ablated activation patching, finding that CircuitFACT is more reliably able to identify circuits which meet a particular significance threshold.

Critical strengths
  • It is interesting that CircuitFACT unrolls both the GNN's internal computation graph and the message-passing computation tree on individual input graphs. This allows for visualization of how a circuit responds to individual nodes, such as Figure 4.
  • Using Shapley values rather than activation patching or attribution patching machinery is an interesting difference from existing circuit discovery
Critical weaknesses
  • It is not clear to me why the median is chosen as the "active" cutoff in Eq. (4) rather than a constant cutoff of, say, 0. It seems conceivable that a neuron with a positive activation (but below the median) should still be considered, particularly when there is significant class imbalance or rare node types.

  • I think the pp-value estimation in Section 5.1 needs further justification, particularly if it is being used to exclude certain circuits from the comparison in Table 1. It seems that the Shapley value is better than zero-ablation mostly with respect to this pp-value, and I think justifying the pp-values is necessary to appreciate the comparison between Shapley attribution and zero-ablation.

Questions for authors

Questions

  1. Can the authors provide further justification for the median cutoff to consider a neuron "active"?

  2. Why threshold circuits based in the estimated pp-value in Section 5.1? What are the best pp-values achieved by the unreported zero-ablation circuits?

  3. Can the authors provide any runtime analysis (either theoretical or wall-clock) for CircuitFACT? Even with Monte Carlo sampling, Shapley values can be inefficient to estimate; this is part of the original motivation for EAP-style attribution patching, which only requires one backwards pass (or a few, for EAP-IG).

Clarifications

  1. Section 5.2 (lns. 461-463) asserts that the zero-ablated circuit retains connections that are uninformative because they contain negative evidence for the predicted class. But isn't this information relevant to the prediction by helping characterize why the confidence isn't higher? Classically, one of the values of SHAP is that it can provide both positive and negative feature contributions, so it seems strange to me to discard this information.

  2. Why is the τ\tau for zero ablation restricted to {0.7,0.8}\{0.7, 0.8\}?

5 comments

Median cutoff (Eq. 4). We want to retain, for each neuron, only the trees on which it fires strongly relative to its own behaviour. The median provides such a relative, per-neuron criterion: it selects the upper half of each neuron's activation distribution, regardless of its scale, which varies across neurons, layers and architectures. A fixed cutoff at 0 does not have this property: it would treat any positive activation as relevant, so neurons that are weakly but almost always positive would be considered active on nearly every tree, inflating the candidate sets and the cost of the Shapley games. We prefer the median to the mean because activation distributions are typically skewed, with many values close to zero, and the mean is sensitive to this skew and to a few large activations. Regarding rare node types, activity at the input layer is not based on the median: an input feature is active whenever the corresponding atom type is present (Sec. 4), so rare atom types are never filtered out.

p-values (Sec. 5.1). Ablation and activation are strong interventions: setting many neurons to zero or to a high value can change the prediction regardless of which neurons are chosen. The test rules out this possibility by comparing each circuit with K = 50 random sets with the same number of neurons per layer; a circuit is significant only if its effect exceeds that of almost all random sets, i.e., if its effect is due to the specific neurons it contains and not to the size of the intervention. This is a standard Monte Carlo permutation test; the +1 correction (Phipson & Smyth, 2010) avoids zero p-values. The advantage of Shapley does not depend on this filter: the results for all extracted circuits, without any significance filtering, are reported in the "All" rows of Tables 3–6 in the appendix and show the same trend (e.g., on MUTAG, class-0 ablation of 92.3% vs. 26.7% for GAT and 90.3% vs. 9.1% for GIN).

τ for zero-ablation. The values {0.7, 0.8} were selected by a grid search over 0.3 to 0.9, in which they gave the best results for zero-ablation; lower values yielded larger and less effective circuits. We will report the full grid in the appendix.

Runtime. Each Shapley game is local: the value νt(S)\nu_t(S) is the activation of the target tt computed from the activations of the layer below, so each evaluation is a single-layer computation rather than a full forward pass. Estimating the contributions for one target on one computation tree therefore costs M⋅∣Pt∣M \cdot |P_t| such evaluations, where ∣Pt∣|P_t| is bounded by the layer width (32 in our models) and is further reduced by the activity filter of Eq. 4. The total cost grows linearly with the number of distinct computation trees and with the number of targets visited during backward tracing. We agree that this is more expensive than EAP, which needs a single backward pass, but Shapley values measure the effect of actual interventions rather than a first-order approximation, which can be inaccurate when activations saturate. Wall-clock times for all configurations are reported below and will be added to the revision.

Negative evidence (Sec. 5.2). We agree that negative contributions are informative, and CircuitFACT does not discard them. Negative evidence is captured at the level of circuits: some circuits support the predicted class and others oppose it (Sec. 5.1, where the latter are tested with the activation intervention), and the prediction results from the combination of these excitatory and inhibitory circuits; the projections in Fig. 4 show both positive (red) and negative (blue) contributions. Within a single circuit, however, all elements should agree in sign, since a circuit is meant to describe one coherent mechanism that pushes the prediction in one direction. Our point in Sec. 5.2 is that the Shapley circuit satisfies this property, whereas the zero-ablation circuit mixes connected neurons with opposing contributions within the same circuit, which makes it a less coherent mechanism. We will clarify this sentence in the revision.

Running times and machine

DatasetGATGCNGIN
MUTAG31 s (τ=0.5)56 s (τ=0.3)29 s (τ=0.3)
AIDS10 min 40 s (τ=0.5)15 min 22 s (τ=0.6)7 min 19 s (τ=0.3)
NCI110 min 21 s (τ=0.5)34 min 01 s (τ=0.7)22 min 12 s (τ=0.3)
Mutagenicity18 min 19 s (τ=0.4)37 min 40 s (τ=0.3)11 min 49 s (τ=0.5)

Total for the 12 runs: 2 h 49 min.

ComputerApple MacBook Pro (Mac17,9)
ChipApple M5 Pro, 18-core CPU (6 Super + 12 Performance cores)
Memory48 GB unified memory
OSmacOS 26.5.1 (build 25F80)
Compute deviceCPU only (no CUDA; Apple GPU/MPS not used by the code)
SoftwarePython 3.10.21, PyTorch 1.13.1 (18 CPU threads), PyTorch Geometric 2.5.2, NumPy 1.26.4

Models (training included) and all extraction and verification runs used this machine and this CPU setup.

I thank the authors for their response, which has clarified several points. A few of my questions remain:

p-values

Thank you for this clarification. One part of my question has not been answered: what pp-values are achieved by the non-significant circuits? I understand the trend overall is that Shapley values are more efficient than zero-ablation, but I still find it difficult to evaluate the difference between the two without the full context of their significance. If zero-ablation, for example, routinely identifies circuits that are near chance, then the advantage is very clear; if, on the other hand, zero-ablation circuits are only barely deemed insignificant because the cutoff is p=0.05p=0.05 (a standard but ultimately somewhat arbitrary choice) then the tradeoff is more nuanced.

Median cutoff

Thank you for clarifying. It makes sense that the cutoff should be selective to keep the Shapley games manageable, and that the median is more robust than the mean in this setting. I would still be interested to see an comparison of this choice at different quantiles or by using the mean, although this is more of a suggestion: I recognize it may be difficult to evaluate this comprehensively in the time remaining, and I weight the first question more heavily.

AuthorsOpenReview ↗

Distribution of p-values for all circuits

We thank the reviewer for reminding us of this question, which we did not address in our previous response, and we apologize for the oversight. To answer it, we re-ran the significance test on all circuits of the five unseeded runs, for both methods, and report the full distribution of p-values below.

Each circuit is compared with 50 random neuron sets of the same size through two tests, ablation and activation; its p-value is the smaller of the two, and the circuit is significant if p ≤ 0.05. Circuits is the total number of circuits, Sig. (p ≤ 0.05) the percentage of significant ones, and Non-sig. the number of non-significant ones. For the non-significant circuits, Median p (non-sig.) reports their median p-value, and the columns 0.05 < p ≤ 0.10, 0.10 < p ≤ 0.25, 0.25 < p ≤ 0.50 and p > 0.50 show how they are distributed across p-value ranges. A median close to 0.05, or a high value in 0.05 < p ≤ 0.10, would mean that these circuits narrowly miss the threshold, while higher values mean that they are far from it.

The table shows that the zero-ablation circuits that are not significant do not miss the threshold by a small margin: most of them have p-values far above 0.05. The advantage of Shapley therefore does not depend on the choice of the 0.05 threshold, and it still holds with a looser threshold such as 0.10.

Dataset / ModelMethodCircuitsSig. (p ≤ 0.05)Non-sig.Median p (non-sig.)0.05 < p ≤ 0.100.10 < p ≤ 0.250.25 < p ≤ 0.50p > 0.50
MUTAG GCNShapley12757%540.2031%26%20%22%
MUTAG GCNZero-abl.12415%1060.3811%25%31%32%
MUTAG GINShapley14759%600.5310%15%23%52%
MUTAG GINZero-abl.15130%1060.3914%19%29%38%
MUTAG GATShapley15051%740.2722%27%24%27%
MUTAG GATZero-abl.14627%1070.2423%28%23%25%
Mutagenicity GCNShapley14737%930.2030%30%19%20%
Mutagenicity GCNZero-abl.14412%1260.2613%36%29%22%
Mutagenicity GINShapley15421%1210.3610%30%31%29%
Mutagenicity GINZero-abl.15316%1280.3111%25%37%27%
Mutagenicity GATShapley15635%1020.2224%38%20%19%
Mutagenicity GATZero-abl.14427%1050.2818%21%37%24%
NCI1 GCNShapley14338%890.1826%36%17%21%
NCI1 GCNZero-abl.14417%1200.2918%27%24%32%
NCI1 GINShapley15317%1270.348%27%38%28%
NCI1 GINZero-abl.15419%1250.2818%26%33%23%
NCI1 GATShapley15727%1150.2021%33%24%22%
NCI1 GATZero-abl.14819%1200.2815%28%38%19%
AIDS GCNShapley14966%500.1248%28%8%16%
AIDS GCNZero-abl.15136%970.2219%37%30%14%
AIDS GINShapley14741%860.2327%27%19%28%
AIDS GINZero-abl.14830%1030.3220%21%36%22%
AIDS GATShapley16053%750.1439%29%15%17%
AIDS GATZero-abl.12717%1060.2721%26%35%18%
AllShapley179042%10460.2423%30%23%25%
AllZero-abl.173422%13490.3017%27%32%25%
Reviewer_4DjAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a circuit discovery method for message-passing graph neural networks (GNNs). The method is based on computation trees, i.e., the LL-hop neighborhood graphs of vertices in an LL-layer GNN, and Shapley values. The Shapley values estimate the informativeness of neurons corresponding to specific vertices and are used to prune the computation graph. The method is empirically evaluated on four graph classification benchmark datasets.

Critical strengths

Novel and model-agnostic approach for explaining GNN predictions: The paper proposes a novel approach to identify substructures that might help explain the prediction of message-passing graph neural networks. The approach is model-agnostic for any architecture that follows a message-passing paradigm.

This method seems like it could useful for explaining the predictions of GNNs in certain domains (e.g., a chemist/biologist might be able to gain a better understanding of what their predictor does while also being able to judge the domain-specific soundness of the discovered circuits).

Critical weaknesses

Scope of experiments and experimental setup: The scope of the experiments is rather limited; the method is only applied to four small-scale binary classification datasets and for three GNN architectures. It would be helpful to understand how the method performs for, e.g., node classification, or for more expressive GNN architectures. Despite the paper's introduction describing discovered circuits to be useful only if a human can recognize certain behavior, the only qualitative evaluation is one single molecule in one of the datasets. There is also no comparison at all to other explanation methods; even if it is not possible to compare the proposed approach to other circuit discovery methods, it would substantially strengthen the contribution to show that the proposed approach is as good as or better than existing explanation methods.

I have some concerns about the soundness of the experimental setup. As the paper does not offer sufficient details to understand how the experiments were conducted (i.e., how many runs, how many epochs, details on the architectures, etc.), I checked the code. In gnn.py there is a manual seed set in the initialization of each GNN architecture. Does this mean that there was only one run conducted per model per dataset?

Given that the usefulness of the proposed method can only be evaluated empirically and that it is currently not clear to me whether the experiments are even reproducible if initialized differently, I lean toward rejection.

Clarity: While the paper is generally easy to follow, I needed several passes to (hopefully) understand how the proposed method for circuit discovery works. It would be helpful to clearly define all necessary concepts and steps; for instance, GNNs are never formally defined and some variables appear without having been explicitly defined (e.g., wc,jw_{c,j} in Section 5.2).

Minor weaknesses and remarks (do not affect my score, but could be helpful for the authors):

  • Figure 2: Should it be u1,u2,u4u_1, u_2, u_4 instead of u1,u3,u2u_1, u_3, u_2? And why does the computation tree not include another blue node as child in the right child of the middle layer?
  • Line 212: Eq. equation 1
  • Line 247: XX not bold anymore
  • Line 253: TT instead of T\mathcal{T}
  • Eq. (2): CrCr instead of CrC_r
  • Consider using \citet and \citep where appropriate
  • The text in Figure 3 is too small
  • The text mentions ''convolutional'' layers several times, what type of convolution?
  • Consider explaining your experimental setup including the datasets better (what is the task in Mutagenicity, for instance?)
  • Consider explaining your theoretical setting/circuit discovery method more rigorously
Questions for authors
  1. Given a root neuron, how is the computation tree candidate set determined?
  2. How many runs are conducted per dataset/architecture combination, and how do results differ across different runs/initializations?
  3. Could you elaborate on the choice of datasets and GNN architectures? Do you have any results on multi-class graph classification or node classification tasks?
  4. Are the obtained "explanations" (circuits) comparable to other explainability methods or fundamentally different?
6 comments

Experimental setup. GCN, GAT and GIN are the most widely used message-passing architectures, and MUTAG, NCI1 and Mutagenicity are standard benchmarks in GNN explainability (e.g., Ying et al., 2019; Armgaan et al., 2024), together with AIDS. Regarding more expressive architectures, GIN attains the upper bound of what message-passing GNNs can do: no standard MPNN can be more powerful than 1-WL (Xu et al., 2019), so our experiments already include the most expressive model of this family. Since CircuitFACT extracts circuits from the message-passing layers, before the readout and the prediction head (Sec. 4), it is not tied to graph classification and can be directly extended to node classification and other graph learning tasks. Regarding multi-class classification, although we experiment on binary classification datasets, the models have a softmax output over the two classes, therefore the approach is actually already suited for multi-class classification.

Reproducibility. The fixed seed in gnn.py was only meant to make results exactly reproducible. We repeated the whole pipeline five times per configuration without a fixed seed: new random initialization, new computation trees, circuits extracted with the τ of Table 2 (not re-tuned), and K = 50 newly sampled random controls (3 layers, hidden size 32; same train/test split in all runs). We report this comparison in our response to Reviewer oQVj.

Clarity. We apologize for the lack of clarity. We will add a formal definition of message-passing GNNs in Sec. 3 and define every variable where it first appears.

Given a root neuron, how is the computation tree candidate set determined? We take the unique computation trees of all training nodes, and a neuron’s candidate set contains the trees on which its activation is above its median (Eq. 4);

AuthorsOpenReview ↗

Comparison with post-hoc (1/2)

Regarding the comparison with post-hoc. Circuits differ fundamentally from post-hoc explanations, as they describe internal computations rather than input relevance. Since it was also raised by other reviewers we propose a comparison between post-hoc relevance and the projection of active circuits on the input graph. We take each significant circuit of the mutagenic class and project its contributions onto the atoms of every molecule on which it is active. From that projection we build an atom mask at five sparsity levels, where sparsity is the fraction of atoms removed. To compare explanations of equal size, each post-hoc method selects the same number of atoms as the CircuitFACT mask on the same molecule.

Metrics. Fid+ is the drop in the probability of the predicted class when the masked atoms are removed. Higher is better, because it means the mask contains what the model relies on. Fid− is the drop when only the masked atoms are kept. Lower is better, because it means the mask alone is enough to support the prediction. For each circuit, we average the scores over the molecules on which it is active. The tables report mean ± std across circuits.

This comparison evaluates how circuits project onto the input, not the circuits themselves. The causal role of the circuits is tested directly by the interventions in Sec. 5.1.

AuthorsOpenReview ↗

Comparison with post-hoc (2/2)

GCN

ThresholdSparsityMethodFid+ ↑Fid− ↓
0.10.15CircuitFACT0.505 ± 0.010-0.048 ± 0.011
0.15GNNExplainer0.517 ± 0.0020.045 ± 0.012
0.16SubgraphX0.540 ± 0.008-0.068 ± 0.006
0.15IG0.551 ± 0.0000.108 ± 0.010
0.30.33CircuitFACT0.553 ± 0.0380.050 ± 0.021
0.33GNNExplainer0.539 ± 0.0340.122 ± 0.021
0.34SubgraphX0.579 ± 0.016-0.036 ± 0.001
0.33IG0.646 ± 0.0140.327 ± 0.005
0.50.52CircuitFACT0.540 ± 0.0450.180 ± 0.003
0.52GNNExplainer0.538 ± 0.0040.230 ± 0.006
0.52SubgraphX0.537 ± 0.0100.127 ± 0.027
0.52IG0.638 ± 0.0170.560 ± 0.013
0.70.68CircuitFACT0.425 ± 0.0280.341 ± 0.032
0.68GNNExplainer0.465 ± 0.0040.410 ± 0.013
0.68SubgraphX0.462 ± 0.0140.272 ± 0.059
0.68IG0.509 ± 0.0130.532 ± 0.009
0.90.80CircuitFACT0.264 ± 0.0240.438 ± 0.023
0.80GNNExplainer0.232 ± 0.0020.460 ± 0.028
0.80SubgraphX0.324 ± 0.0060.410 ± 0.053
0.80IG0.313 ± 0.0030.454 ± 0.015

GIN

ThresholdSparsityMethodFid+ ↑Fid− ↓
0.10.36CircuitFACT0.495 ± 0.0260.045 ± 0.046
0.36GNNExplainer0.542 ± 0.0050.167 ± 0.080
0.37SubgraphX0.440 ± 0.0480.049 ± 0.071
0.36IG0.542 ± 0.0140.188 ± 0.054
0.30.48CircuitFACT0.511 ± 0.0670.069 ± 0.004
0.48GNNExplainer0.558 ± 0.0140.224 ± 0.025
0.50SubgraphX0.453 ± 0.0780.075 ± 0.053
0.48IG0.571 ± 0.0500.239 ± 0.043
0.50.62CircuitFACT0.429 ± 0.0530.116 ± 0.041
0.62GNNExplainer0.537 ± 0.0140.331 ± 0.017
0.64SubgraphX0.403 ± 0.0530.127 ± 0.034
0.62IG0.565 ± 0.0590.316 ± 0.018
0.70.72CircuitFACT0.384 ± 0.0050.184 ± 0.091
0.72GNNExplainer0.488 ± 0.0110.378 ± 0.030
0.73SubgraphX0.340 ± 0.0020.202 ± 0.012
0.72IG0.497 ± 0.0110.353 ± 0.022
0.90.80CircuitFACT0.294 ± 0.0730.259 ± 0.094
0.80GNNExplainer0.360 ± 0.1000.403 ± 0.030
0.81SubgraphX0.264 ± 0.0480.296 ± 0.098
0.80IG0.407 ± 0.0790.375 ± 0.047

GAT

ThresholdSparsityMethodFid+ ↑Fid− ↓
0.10.44CircuitFACT0.412 ± 0.0970.198 ± 0.101
0.44GNNExplainer0.703 ± 0.0290.278 ± 0.114
0.48SubgraphX0.568 ± 0.0450.124 ± 0.071
0.44IG0.270 ± 0.0930.600 ± 0.094
0.30.53CircuitFACT0.408 ± 0.1210.202 ± 0.104
0.53GNNExplainer0.725 ± 0.0210.309 ± 0.082
0.59SubgraphX0.558 ± 0.0580.156 ± 0.054
0.53IG0.230 ± 0.0530.654 ± 0.027
0.50.63CircuitFACT0.372 ± 0.1130.235 ± 0.096
0.63GNNExplainer0.717 ± 0.0470.380 ± 0.019
0.69SubgraphX0.506 ± 0.0410.212 ± 0.029
0.63IG0.214 ± 0.0350.664 ± 0.023
0.70.70CircuitFACT0.328 ± 0.0740.277 ± 0.051
0.70GNNExplainer0.643 ± 0.0480.427 ± 0.039
0.75SubgraphX0.451 ± 0.0510.263 ± 0.035
0.70IG0.218 ± 0.0410.642 ± 0.021
0.90.74CircuitFACT0.302 ± 0.0430.309 ± 0.031
0.74GNNExplainer0.578 ± 0.0780.431 ± 0.047
0.79SubgraphX0.404 ± 0.0720.296 ± 0.056
0.74IG0.219 ± 0.0470.622 ± 0.025

Results. CircuitFACT obtains a lower Fid+ than the post-hoc methods. We expected this, because a circuit captures one mechanism and not the whole prediction. When the atoms of one circuit are removed, the evidence carried by the other circuits is still there. On Fid−, CircuitFACT is among the best methods. It has the lowest Fid− on GIN at every sparsity level and is second only to SubgraphX on GCN and GAT. It also has a lower Fid− than GNNExplainer and IG in all 15 settings (3 architectures × 5 sparsity levels). This means the input pattern associated with a single circuit is enough on its own to largely preserve the prediction. That agrees with the intervention experiments in Sec. 5.1. We will add this comparison to Sec. 5.2 of the revision.

Reviewer_oQVjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a novel circuit discovery method for Mechanistic Interpretability for GNNs. This means authors seek to find a subgraph of the GNNs' computational graph that is the most accountable for the prediction. Experiments are run on MUTAG, NCI1, AIDS and Mutagenicity with GCN, GAT and GIN. Authors also provide a qualitative example projecting circuit attributions onto a mutagenic molecule.

Critical strengths
  • Mechanistic Interpretability is a timely contribution

  • Using shapely values make the approach more principled

Critical weaknesses
  • It's not clear the kind of insights that circuit could give to practitioners. Authors fall short in showing a practical application of the discovered circuit, or better, in showing how that could be useful for users, other than the extraction of local relevance scores, which is more asking a post hoc explainer rather than a Mechanistic Interpretability tool

  • Regarding Section 5.2 there is no comparison of this approach with standard post hoc explainers to assess the quality of the provided input level explanations, and the evaluation based purely on the resembrance of the explanation with the intended ground truth patterns like NO2 Is already known to be potentially misleading

  • The experimental comparison is weak. The claim that existing methods are not directly applicable”conflicts with the paper’s citation of MINAR, which alsos propose baselines for circuit discovery for GNNs. The authors should at least compare with these.

Questions for authors
  • Why do you believe MINAR Is not a valid competitor for you analysis?
7 comments

Insights for practitioners, and circuits vs. post-hoc explanation

We thank the reviewer for raising this point, and we agree the practical value of circuits should be stated more explicitly.

Circuits and post-hoc explainers answer different questions. A local relevance score says which parts of this input drove this prediction. A circuit says which internal components, and how they interact, implement a behavior across inputs. In Fig. 4, for example, the question is not only "which atoms make this molecule mutagenic?" but "how does the model combine message-passing information across layers to detect the mutagenic motif?". Relevance scores describe the input, whereas a circuit describes the computation. Regarding the usefulness to practitioners, we see three concrete uses.

  1. Validating that the explanation is causal. In Sec. 5.1, intervening on circuit neurons changes the prediction significantly more than intervening on random neurons. This confirms that the discovered circuit is what the model actually uses, rather than a plausible-looking story. Input-level scores don't test this.
  2. Localized model editing. Because the behavior is concentrated in few neurons, it can in principle be suppressed or amplified directly, without retraining the whole network. Sec. 5.1 supports the necessary condition (the neurons are causally relevant). We have not yet shown error correction end to end, and we will state this as a limitation and a direction for future work.
  3. Auditing and debugging. Circuits expose how a motif is detected, so a practitioner can check whether the model relies on an intended mechanism or a shortcut, and compare mechanisms across models or training runs.

We will add a paragraph to Sec. 4 to better highlight such distinction.

Comparison with other post-hoc and mechanistic baselines (MINAR)

We thank the reviewer for this suggestion; however, MINAR was designed for a different setting. For this reason, we decided not to include it as a baseline in the first place. However, to clear any doubts, we have adapted it to our setting, and we now provide the comparison. A few clarifications on the setup are needed first. MINAR does not seek to identify circuits for different behaviors but rather one single circuit for each task, whereas we analyse graph classification, so we adapted it to extract one circuit per class. MINAR builds each circuit from the KK top-scoring edges, and the paper does not specify how KK should be chosen. We therefore ran MINAR's published code (EAP-IG, 20 steps) with both of its circuit constructions (greedy and longest-path) over K∈{10,25,50,100,250,500,1000}K \in \{10, 25, 50, 100, 250, 500, 1000\}, and kept the single KK, shared by both classes and all five runs, with the most significant results in MINAR's own tests. Since this choice is made on the results themselves, it favours MINAR. MINAR cannot be applied to GAT: its circuits are defined over the fixed weights connecting neurons, whereas in GAT messages are additionally weighted by attention coefficients computed from the input. We therefore ran it on GIN and GCN.

H Abl. / H Act.: harmonic mean of the class-0 and class-1 prediction-change rates (%) under ablation and activation, averaged over 5 runs. All: every extracted circuit; Sig.: only circuits with p ≤ 0.05 ("–": none for at least one class). Sig. runs: (run, class) pairs with at least one significant circuit, out of 10. G = greedy, LP = longest-path, followed by KK. Best value per configuration in bold.

DatasetModelMethodAll: H Abl.All: H Act.Sig.: H Abl.Sig.: H Act.Sig. runs
MUTAGGATCircuitFACT72.771.493.891.610/10
Zero-abl.63.059.386.685.28/10
GCNCircuitFACT72.970.784.589.810/10
Zero-abl.26.339.3100.078.15/10
MINAR (G, 250)13.61.5––0/10
GINCircuitFACT58.967.181.685.110/10
Zero-abl.12.336.673.030.08/10
MINAR (G, 1000)0.039.9––3/10
NCI1GATCircuitFACT47.054.858.082.410/10
Zero-abl.28.943.742.261.98/10
GCNCircuitFACT66.261.581.685.310/10
Zero-abl.43.350.078.063.210/10
MINAR (G, 500)53.639.0––1/10
GINCircuitFACT53.338.979.251.27/10
Zero-abl.44.335.667.548.210/10
MINAR (G, 1000)20.843.170.785.82/10
AIDSGATCircuitFACT46.559.263.669.710/10
Zero-abl.6.137.319.859.99/10
GCNCircuitFACT45.464.556.373.710/10
Zero-abl.10.053.722.267.810/10
MINAR (LP, 25)12.22.848.72.23/10
GINCircuitFACT45.353.361.669.59/10
Zero-abl.32.049.929.370.29/10
MINAR (LP, 100)7.912.4––1/10
MutagenicityGATCircuitFACT54.665.061.578.010/10
Zero-abl.38.851.477.467.010/10
GCNCircuitFACT64.169.383.785.510/10
Zero-abl.47.860.288.777.68/10
MINAR (LP, 10)27.117.5––3/10
GINCircuitFACT37.650.652.964.910/10
Zero-abl.45.449.178.560.110/10
MINAR (G, 1000)53.370.5––1/10

Without any significance filter, CircuitFACT obtains the highest harmonic mean in 11 of 12 configurations for ablation and 10 of 12 for activation. MINAR's circuits are significant in only 14 of 80 (run, class) pairs, against 76 of 80 for CircuitFACT on the same models. Note also that MINAR's circuits are on average much larger than individual CircuitFACT circuits: larger interventions mechanically produce larger prediction changes, which explains why some non-significant MINAR circuits reach high activation rates (e.g., Mutagenicity GIN).

AuthorsOpenReview ↗

AI Statement

Thank you for pointing this out. Our AI use statement was missing from the submitted PDF because of a compilation error in our LaTeX source. We only noticed this thanks to the review. This was an oversight on our part, and we apologize for it. The statement is reproduced below and will been added to the revised PDF:

In this work, we used generative AI tools for implementing parts of our methods and experimental code (AI-assisted code writing for selected components of the CircuitFACT pipeline and the experiment and plotting scripts), for literature search, to help identify related work, and for editing the paper to improve readability and grammar. We have not used generative AI tools for generating synthetic datasets, developing theoretical models or conceptual frameworks, formulating or proving mathematical claims, proposing or refining hypotheses, designing the methodology or experiments, cleaning or reformatting datasets, supporting qualitative data analysis, assisting with translation, or interpreting results; the research ideas, method design, and analysis are the authors’ own. We have reviewed all AI-assisted work. AI-assisted code was read, tested, and verified for correctness by the authors, all references were checked against the original sources, and LLM-suggested text edits were checked to ensure that they preserve the intended meaning and do not introduce unsupported claims. We take responsibility for the final content of this work, including its text, claims, and code.

Comments on the paper

1 comment

Dear Reviewers,

Please take a moment to review the authors’ rebuttal, assess whether your concerns have been adequately addressed, and actively participate in the discussion. Please update your assessment if needed. Thank you for your time and effort!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.