acceptodds
Under review as a conference paper at ICLR 2027

Can Circuit Discovery Be Learned? A Graph Neural Network Approach

Abstract

Circuit discovery aims to find the sparse subnetworks responsible for specific behaviors in language models. Existing automated methods either approximate the effect of interventions to first order, as in attribution patching, or optimize a new set of mask parameters for every task, as in Edge Pruning. In both cases, nothing reusable is learned about how to find circuits. We take a different approach and train a graph neural network to predict circuits directly from activations, extending recent work that trains models to interpret model internals. The network operates on the transformer's computation graph, takes clean and counterfactual activations as input, and assigns a score to each edge. Keeping the highest-scoring edges gives a circuit at any target sparsity in a single forward pass, without attribution-derived features or test-time optimization. Scorers trained on individual tasks outperform the previous state-of-the-art methods EAP and EAP-IG on integrated circuit performance (CPR) on the MIB Benchmark. We then train a scorer jointly across multiple tasks and this scorer successfully infers the task and its circuit from the activations alone, remaining competitive against task specific GNN. We see this as the first step toward general-purpose learned circuit discovery.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Circuit Discovery, Graph Neural Networks, Geometric Deep Learning, Transformer Interpretability, Circuit Faithfulness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PJ2gOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a GNN-based learning framework to predict the activation circuit within language models. The main contribution of the paper is a general-purpose scorer for circuit prediction.

Critical strengths

The paper addresses a critical and challenging task in explainable AI. I appreciate the authors' efforts in this direction.

Critical weaknesses
  1. The novelty of the method is a big concern. The authors state: We formulate the task of discovering a circuit as a learning problem and propose to use graph neural networks (GNNs) to recover high-quality circuits from the activations of the residual streams of a language model. Utilizing GNNs for circuit discovery seems to be a weak contribution, though there is no existing work utilizing GNNs for circuit discovery. But it is unclear what the challenge is that prevents this step. GNNs have been proposed for decades, yet no work has utilized GNNs for circuit discovery. So what is the most challenging obstacle? I found the methodology section hard to follow, and I can only find one sentence: We train a graph neural network (GNN) to predict circuits from activations; a circuit is naturally a graph in which an edge’s importance depends on the components around it. What is the main contribution of this approach? Is the activation-based prediction a big contribution?

  2. The writing is messy and seems to be mainly AI-generated. The introduction lacks a clear contribution statement, and the methodology is difficult to follow.

  3. The learning machenism that enables GNN to generalize to general circuit-finding tasks is unclear. The authors state their contribution as the first step towards general-purpose circuit discovery, which is promising. But I can not clearly identify the technical or experimental evidence to support this statement.

Questions for authors
  1. I would appreciate it if the authors could clarify the challenges that prevent GNN-based circuit discovery and how they are resolved.
  2. I would appreciate it if the authors could clearly introduce how they design the training machenism that enables the GNN to learn general circuit discovery ability.
  3. Can the authors clearly point out which experimental results demonstrate the generalization ability of their methods?
4 comments

We thank the reviewer for engaging with the paper and for the kind words about the problem it addresses. It has made us more aware of the need to convey our contributions more effectively.

Weakness 1 and Question 1:

The reviewer is right that the usage of GNN is not the key contribution. The contribution is that circuit discovery is treated as something a model can learn, rather than something optimized or searched from scratch for every task. This is very different from the other methods, and we have now clarified this in Section 2.2 and 2.4 in discussing background work.

The GNN is simply the architecture we think that best fits the problem. A circuit's performance depends on the joint effect of its edges, which first-order attribution cannot capture (Section 2.2, item 2), and message passing lets each edge's score depend on its neighbors. The direction ablation in Appendix E shows the scorer makes use of this structure.

Weakness 2: writing and contribution statement.

The introduction now contains an explicit contribution statement and the methodology section has been restructured so that each stage of Figure 1 is more clearly laid out.

Weakness 3, Questions 2 and 3: the mechanism behind generalization, and the evidence.

The mechanism for generalization is the pipeline as a whole. Because the scorer is a function from activations to a circuit, trained through the frozen model, the same weights can serve any prompt whose activations it has learned to read. Other so-called learning methods only optimize per task. The evidence is given in Section 4.4 where for GPT2 and Qwen2.5 we show a single scorer trained on a variety of tasks finds more faithful circuits than EAP, EAP-IG and does comparable to GNNs that were trained for each task. We do acknowledge that performance on out-of-distribution tasks is modest in Appendix F. The claim the paper makes is that discovery can be learned and shared across the tasks it is trained on; general-purpose discovery across unseen tasks seems to require much more task diversity which we leave for future work. .

We hope the revised introduction and methodology make the contribution clearer, and we would be grateful if the reviewer would consider revising their score. We are happy to clarify anything further.

Dear Reviewer PJ2g, we wanted to check whether our response and the revised PDF address your concerns. In particular, the introduction now has an explicit contribution statement, Section 2.2 lays out the limitations of prior methods that motivate the approach, and Section 3 has been restructured with the full chain of equations (Eqs. 4–9) so that each stage of Figure 1 has its own paragraph; the evidence for generalization is laid out in Sec. 4.4 (Fig. 4) and App. F. If any of your concerns are not yet addressed, we would be glad to run further experiments and/or provide more clarification before the discussion period ends.

Reviewer_PJ2gOpenReview ↗

Response

Dear authors, Thanks very much for your response and clarification. I think your response and the modifications make the problem easier to understand. I would like to raise my score to 3.

Dear Reviewer PJ2g,

Thank you for revisiting the paper and for raising your score. We are glad the revised introduction and methodology section made the contribution and the problem setup easier to follow. Your comments helped us state the contribution more clearly, and we will carry these improvements into the final version.

Thank you again for your time and feedback.

Reviewer_9LiUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a GNN that predicts edge scores for transformer circuits from clean and counterfactual activations. It trains through differentiable interventions on a frozen language model, using soft masks during training and hard edge rankings during evaluation. Experiments cover four model families and 12 model and task pairs from MIB, alongside seven synthetic GPT-2 tasks. The method improves the proposed log-weighted circuit-performance metric over attribution baselines, and a separate matching objective improves circuit–model distance on most evaluated pairs.

Critical strengths
  1. The method offers a practical way to train circuit scorers without ground-truth circuit annotations or attribution-derived features. Evaluation across GPT-2, Qwen, Gemma, and Llama provides useful evidence that the approach extends beyond small, specially constructed transformers.
  2. The matching-objective results provide evidence beyond maximizing task performance: Table 2 reports lower circuit–model distance than EAP-IG on 9 of 12 model–task pairs. The connectivity analysis and comparison with known IOI components also help characterize the resulting circuits.
Critical weaknesses
  1. My major concern is that the comparisons do not sufficiently isolate the contribution of the GNN. The headline gains use Max-LD and a metric that rewards increasing the target logit difference beyond the full model’s value. These gains therefore do not directly establish better recovery of the original computation—a distinction also made by MIB. Appendix D partly addresses this through matching, but the paper still lacks comparisons with directly optimized masks under the same objective, an edge-wise MLP, or a scorer without message passing. Edge Pruning is evaluated only on GPT-2 and modified to remove node gates. Consequently, it remains unclear how much of the improvement comes from graph-based learning rather than the objective and optimization procedure.
  2. Joint training on seven known task families demonstrates multitask learning. Appendix F additionally tests unseen tasks, but the transferred scorer underperforms EAP-IG on five of seven tasks, including substantial failures on IOI and agreement. These results deserve discussion in the main text, which currently states that unseen-task evaluation has not been performed. Moreover, Section 4.1 averages scores over each task’s training split to obtain a fixed ranking; this does not directly validate discovering a circuit for an individual new prompt in one forward pass.
Questions for authors
  1. Do the gains persist against directly optimized masks and an edge-wise MLP using the same objective, sparsity schedule, and comparable compute? How much does removing message passing or activation features affect performance?
  2. What performance and latency are obtained when predicting circuits directly from individual held-out prompt pairs, without averaging scores over a task’s training split?
3 comments

We thank the reviewer for a precise and well-informed review. There were many helpful pieces of feedback that we tried to address below.

Weakness 1/Question 1: isolating the contribution of the GNN.

  • Objective and metric. We agree that CPR with Max-LD rewards overshoot and does not by itself establish recovery of the original computation. This is still a question up for debate so we also show that if we train our model to match the original model instead of just optimizing for task performance, we show good results on CMD objective. Table 2 reports the results on this different circuit discovery objective on all 12 MIB cells. We have also added per-prompt absolute error and positive shortfall (Tables 3-4, Appendix D).
  • Directly optimized masks under the same objective. We have now added Edge Pruning trained with our Max-LD objective on every model and task in addition to Edge Pruning with its native KL objective in Table 1. We still see our method out-performing by a significant margin, so the gain is not explained by the objective.

Weakness 2: unseen tasks and the single-forward-pass claim.

  • The main text was wrong to say that unseen-task evaluation had not been performed. That sentence has been replaced. Section 4.4 now clearly points to Appendix F where we discuss the leave-one-family-out results for both GPT-2 and the newly added Qwen2.5-0.5B. We do not evaluate circuits for individual prompts in this version and have removed wording that could suggest otherwise. The abstract now says that out-of-distribution generalization is modest. What the paper does establish is that a single scorer can serve many tasks at once which per-task methods cannot do by construction. Robust generalization will require a far larger collection of tasks with counterfactual pairs which we view as a future open problem. Question 2: Performance/Latency costs are now discussed in Appendix G which is referenced in Section 4.1’s implementation section.

We hope these new baselines and the clarifications resolve the reviewer's main concern, and we would be grateful if the reviewer would consider raising their score. We are happy to provide any further comparison that would help.

Thank you for the detailed response and additional experiments. The objective-matched comparisons and the clearer scope of the generalization claims address several of my concerns. Ablations isolating the contribution of message passing would still strengthen the paper. Overall, I remain supportive of acceptance and maintain my score of 3.

Dear Reviewer 9LiU, we wanted to check whether our response and the revised PDF address your concerns. In particular, Table 1 now compares against Edge Pruning trained with our own Max-LD objective on every model and task, alongside its native objective, with the scorer ahead on all 12 cells, and Tables 3–4 (App. D) add the per-prompt metrics that cannot be gamed by overshoot; the unseen-task results are now discussed in Sec. 4.4 and App. F, with Qwen2.5 added, and the single-forward-pass claim is clarified in Secs. 3 and 4.1. If any of your concerns are not yet addressed, we would be glad to run further experiments and/or provide more clarification before the discussion period ends.

Reviewer_okviOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. A GNN identifies sparse circuits from clean–counterfactual activation pairs which is trained per task or jointly across tasks.
  2. Authors report strong results on held-out prompts, but weaker generalization to unseen tasks than EAP-IG on 5/7 tasks.
  3. Log-weighted metrics sensibly emphasize small circuits however the agreement with known heads lacks baseline comparisons.
Critical strengths

Very strong results for Version (i) (GNN trained per task) on objective (a) (CPR_log).

Less strong, but still promising results for objective (b) (CMD_log) and Version (ii) (one GNN trained on multiple tasks).

Critical weaknesses

Version (ii) was only evaluated on GPT-2? Why not for the other LLMs as well? The performance for the out-of sample tasks is quite poor (but maybe good enough to keep some hope for future improvements in this direction). Lines 482-483 say “the shared scorer has not been tested on unseen tasks”, but the results are actually in appendix F. The abstract is formulated as if version (ii) is the core of this paper, but then the experimetnal results are incomplete for Version (ii). For the rebuttal you should add the missing settings for Version (ii). You should also apply Version (ii) to objective (b) (CMD_log).

Computational costs are not reported.

There are some problems with the evaluation metrics that should be fixed as I explain in the Questions For Authors.

These weaknesses seem fixable and I’m willing to raise my score if thy are fixed during the rebuttal.

Questions for authors
  1. Evaluation: Averaging logit gaps before computing CMD lets overshooting on some prompts compensate for failures on others. Please report metrics that apply absolute error (b*) or positive shortfall (c*) per prompt before averaging, including for existing trained models.

  2. Baselines: Compare against Edge Pruning trained with matched objectives to distinguish GNN gains from objective alignment. Specify its training objective, define “Warm-start Edge Pruning,” and discuss concurrent MAttr.

  3. Clarity: Give precise GNN update equations and clarify the “single forward pass” claim, since circuit selection averages over multiple prompt-pairs. Replace “fully connected” with the actual path-connectivity property measured.

  4. Missing evidence: Broader gap-preservation and multitask experiments, computational-cost comparisons, and acknowledgment of weak unseen-task performance in the main text would strengthen the case for acceptance.

4 comments

We thank the reviewer for a careful and constructive review. The specific requests have made the paper better and we have tried to address every point below.

Weakness 1: Clarifications and more experiments on the Generalization Experiments

  • We’ve added generalization experiments with Qwen2.5-0.5B over eight task families (Section 4.4, Figure 4b). Results trend-wise are roughly similar to GPT2 and the shared scorer performs near the task-specific scorer. Additional generalization results on Gemma-2 and Llama-3.1-8B was not feasible within the rebuttal window. The reviewer also suggested training with CMD log as well for the shared scorer experiments and we will be doing so but the experiments may not finish within the rebuttal window. We feel pretty confident in CMD showing similar trends to CPR, as it is only a difference in the circuit performance objective and based on the results on the task-specific results.
  • The reviewer is right that lines 482-483 contradicted Appendix F, and we apologize for the confusion. That sentence is gone and the conclusion now states that transfer to a family left out of training is still partial. We’ve tried to make it clear that the main-text experiments train on all N families and test on held-out examples of those families whereas Appendix F trains on N-1 families and tests on the excluded one. We agree that held-out-task transfer is weak with so few training families, which we believe to be future follow-up work. We’ve made our claims surrounding generalization much more modest.

Weakness 2: computational cost. We have added details about computational costs in Appendix G, which we reference in the Implementation section in 4.1. Even in the case where our method does not generalize well across a diverse suite of tasks, we can still train per task at comparable time to Edge Pruning. In the case that our scorer can generalize, obtaining circuits would be made fairly cheap.

Question 1: per-prompt metrics. We’ve added tables on absolute error (b*) or positive shortfall (c*) per prompt before averaging in Appendix F to go along with the CMD results. Our GNN still seems to out-perform the other baseline methods and we hope this alleviates the reviewer’s concern.

Question 2: baselines.

  • Edge Pruning is now reported on every model and task of Table 1, both with its native KL objective and trained with our Max-LD objective. We’ve also added Edge Pruning with the CMD objective in Appendix D. Under the matched objective our method still stays ahead on all 12 cells which as the reviewer mentioned separates what the learned scorer adds from what the objective adds.

  • "Warm-start Edge Pruning" was the label of an external MIB submission protocol that we copied. We’ve since reimplemented and reran all Edge Pruning numbers on our own with the normal objective and our Max-LD objective. The term has been removed.

  • MAttr is concurrent with our submission. Though it also does get impressive results, it is still per-task mask optimization: a stronger optimizer in the family of Edge Pruning, whereas our methodology can hopefully amortize discovery into a predictor that reads activations and can be shared across tasks. It does share a lot of the same components regarding training under a variety of sparsity levels and the way we make the objective differentiable. We will cite and discuss it in the final version, but as we have not yet rerun it under our protocol, we have not included discussions about it for now.

Question 3: clarity.

  • Section 3 now gives the full chain of equations (Eqs. 4-9): the activation descriptor, node and edge features, state initialization, the message, the node and edge update (new Eq. 8), and the readout.
  • The "single forward pass" claim is clarified in Section 3 and Section 4.1. One pass of the scorer scores one prompt pair. The task-level circuits we evaluate average these per-pair scores over the training split before ranking. We say a single forward pass since the other attribution-based methods also require a backward pass to obtain the circuit. In practice, however, due to our inefficient implementation, our GNN’s circuit inference speed for now does not have any advantage over the attribution based methods.
  • the phrase "Fully connected" has been replaced with more precise language.

We believe these changes address the points the reviewer listed as fixable, and we would be grateful if the reviewer would consider revising their score. We are glad to run any further check that would help.

Dear Reviewer okvi, we wanted to check whether our response and the revised PDF address your concerns. In particular, Edge Pruning is now reported under both its native objective and our Max-LD objective on all 12 model–task pairs (Table 1, Appendix D), the per-prompt absolute-error and positive-shortfall metrics you proposed are in Tables 3–4 (Appendix D), the shared scorer now covers Qwen2.5 (Sec. 4.4, Fig. 4b, App. F), computational cost is reported in Appendix G, and Section 3 gives the full update equations (Eqs. 4–9) with the single-forward-pass claim clarified. If any of your concerns are not yet addressed, we would be glad to run further experiments and/or provide more clarification before the discussion period ends.

Thank you very much for adding the requested experiments and changes to the paper! I think the paper has significantly improved now. The additional metrics b* and c* that never actively reward overshooting seem more reasonable to me. And the results seem to be quite stable under this change of metric. In the camera-ready version, this cross-metric evaluation would be very interesting (i.e., how well do models trained on each loss perform on all the other metrics, e.g., train on Max-LD and evaluate on b* and c* and all the other metrics). I think there is an overall lack of evidence in the whole community that these methods really find the circuits responsible for these tasks, rather than just circuits that optimize some arbitrary metrics. Do circuits that perform well in one of these metrics at least also perform well in these other reasonable metrics? The additional results on Qwen make the within-tasks results more trustworthy, while further evaluation on more models and datasets would be appreciated. On the other hand, these results emphasize that (at the current scale) generalization to other tasks doesn’t happen most of the time, which you transparently communicate in the updated version. However, the within-tasks results are still good. Removing the contradicting statement from the paper was also important. Given all these improvements my rating is now closer to 3 than 2.

Dear Reviewer okvi,

Thank you for revisiting the paper and for raising your score. We are glad the additional b*/c* metrics and the Qwen2.5 results addressed your concerns.

The cross-metric evaluation you suggest (training under each objective and evaluating on all of the metrics) is a good test of whether the circuits found are robust rather than tuned to a single metric, and we will include it in the camera-ready version, along with evaluation on additional models and datasets where feasible. We also agree that whether these methods recover the circuits responsible for a task, rather than circuits that optimize a given metric, remains an open question for the field, and we will say so in the discussion.

Thank you again for a careful and constructive review.

Comments on the paper

1 comment

We thank all the reviewers for their feedback. Here is a brief overview of what we worked on in our revision:

  • New added baselines for Edge Pruning in both CPR and CMD for all MIB tasks (Section 4.2, Appendix D)
  • Additional generalization experiments on Qwen2.5 (Section 4.4, Appendix F)
  • Computational cost comparisons of the different methods (Appendix G)
  • Clear discussion on limitations of prior work (Section 2.2)
  • Edits throughout for clarity and precision

For clarity, our revisions are marked in blue in the PDF. We hope our revision addresses all of the concerns the reviewers raised about our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.