- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a GNN-based learning framework to predict the activation circuit within language models. The main contribution of the paper is a general-purpose scorer for circuit prediction.
Critical strengths
The paper addresses a critical and challenging task in explainable AI. I appreciate the authors' efforts in this direction.
Critical weaknesses
-
The novelty of the method is a big concern. The authors state:
We formulate the task of discovering a circuit as a learning problem and propose to use graph neural networks (GNNs) to recover high-quality circuits from the activations of the residual streams of a language model.Utilizing GNNs for circuit discovery seems to be a weak contribution, though there is no existing work utilizing GNNs for circuit discovery. But it is unclear what the challenge is that prevents this step. GNNs have been proposed for decades, yet no work has utilized GNNs for circuit discovery. So what is the most challenging obstacle? I found the methodology section hard to follow, and I can only find one sentence:We train a graph neural network (GNN) to predict circuits from activations; a circuit is naturally a graph in which an edge’s importance depends on the components around it.What is the main contribution of this approach? Is the activation-based prediction a big contribution? -
The writing is messy and seems to be mainly AI-generated. The introduction lacks a clear contribution statement, and the methodology is difficult to follow.
-
The learning machenism that enables GNN to generalize to general circuit-finding tasks is unclear. The authors state their contribution as the first step towards general-purpose circuit discovery, which is promising. But I can not clearly identify the technical or experimental evidence to support this statement.
Questions for authors
- I would appreciate it if the authors could clarify the challenges that prevent GNN-based circuit discovery and how they are resolved.
- I would appreciate it if the authors could clearly introduce how they design the training machenism that enables the GNN to learn general circuit discovery ability.
- Can the authors clearly point out which experimental results demonstrate the generalization ability of their methods?
4 comments
Dear Reviewer PJ2g, we wanted to check whether our response and the revised PDF address your concerns. In particular, the introduction now has an explicit contribution statement, Section 2.2 lays out the limitations of prior methods that motivate the approach, and Section 3 has been restructured with the full chain of equations (Eqs. 4–9) so that each stage of Figure 1 has its own paragraph; the evidence for generalization is laid out in Sec. 4.4 (Fig. 4) and App. F. If any of your concerns are not yet addressed, we would be glad to run further experiments and/or provide more clarification before the discussion period ends.
Response
Dear authors, Thanks very much for your response and clarification. I think your response and the modifications make the problem easier to understand. I would like to raise my score to 3.
Dear Reviewer PJ2g,
Thank you for revisiting the paper and for raising your score. We are glad the revised introduction and methodology section made the contribution and the problem setup easier to follow. Your comments helped us state the contribution more clearly, and we will carry these improvements into the final version.
Thank you again for your time and feedback.
We thank the reviewer for engaging with the paper and for the kind words about the problem it addresses. It has made us more aware of the need to convey our contributions more effectively.
Weakness 1 and Question 1:
The reviewer is right that the usage of GNN is not the key contribution. The contribution is that circuit discovery is treated as something a model can learn, rather than something optimized or searched from scratch for every task. This is very different from the other methods, and we have now clarified this in Section 2.2 and 2.4 in discussing background work.
The GNN is simply the architecture we think that best fits the problem. A circuit's performance depends on the joint effect of its edges, which first-order attribution cannot capture (Section 2.2, item 2), and message passing lets each edge's score depend on its neighbors. The direction ablation in Appendix E shows the scorer makes use of this structure.
Weakness 2: writing and contribution statement.
The introduction now contains an explicit contribution statement and the methodology section has been restructured so that each stage of Figure 1 is more clearly laid out.
Weakness 3, Questions 2 and 3: the mechanism behind generalization, and the evidence.
The mechanism for generalization is the pipeline as a whole. Because the scorer is a function from activations to a circuit, trained through the frozen model, the same weights can serve any prompt whose activations it has learned to read. Other so-called learning methods only optimize per task. The evidence is given in Section 4.4 where for GPT2 and Qwen2.5 we show a single scorer trained on a variety of tasks finds more faithful circuits than EAP, EAP-IG and does comparable to GNNs that were trained for each task. We do acknowledge that performance on out-of-distribution tasks is modest in Appendix F. The claim the paper makes is that discovery can be learned and shared across the tasks it is trained on; general-purpose discovery across unseen tasks seems to require much more task diversity which we leave for future work. .
We hope the revised introduction and methodology make the contribution clearer, and we would be grateful if the reviewer would consider revising their score. We are happy to clarify anything further.