- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a framework which improves circuit discovery in transformers. Self-Discovering Circuit Models (SDCMs) is a self-supervised fine-tuning method which trains a pre-trained transformer jointly with a lightweight circuit predictor (a single affine layer with sigmoid on the final residual stream) so that each forward pass yields two things: next-token prediction and input-specific edge-level circuit. A training step has two passes: a clean one and one with all edges predicted to be outside the circuit patched via resampling, zero, or mean, and optimizes the language modeling loss, a KL-based faithfulness loss between clean and patched outputs, and a lagrangian constraint on average circuit density. Input-level scores are aggregated in logit space into task-level edge ranking from forward passes alone. The contributions are (i) addressing both the cost of post-hoc search and OOD nature of patching by finetuning the model on patched passes, (ii) efficient extraction of circuits that are more faithful than post-hoc ones and transfer across patching schemes, (iii) a backbone on which standard post-hoc algorithms recover better circuits, even under counterfactual patching and (iv) the use of SDCMs as "twin models" that supply circuits. Validation is on GPT2-Small and Qwen2.5-0.5B models across tasks (IOI, SVA, Greater-Than, MCQA), comparing against EAP-IG (activation and input interpolation variants) on the same pretrained model fine-tuned on the same data but with language loss only. On Qwen2.5-0.5B using the area under faithfulness curve (AUC) as a metric, SDCM circuits outperform all EAP-IG circuits under all three patching schemes, with the largest gains under zero patching, at roughly 6-10 seconds vs. up to 4,700 seconds extraction time. GPT2 results show exception on Greater-Than under mean patching. By comparing models, EAP-IG on SDCM backbone finds more faithful circuits than if applied on standard-trained model. Lastly, language-modeling and task performance are largely preserved in SDCMs.
Critical strengths
-
S1: Proposing an inherently self-interpretable fine-tuning procedure by re-framing circuit discovery as a training-time objective. I think the "self-discovering" framing in which the model exposes its subgraph is new and valuable: (i) It targets the patching OOD problem in post-hoc circuit discovery. Proposing yet another post-hoc algorithm cannot fix this as the patching distortion effect is model-dependent, not algorithm-dependent. (ii) The empirical results also support this: circuits discovered by SDCM have larger gains in AUC the further the patching is from the data distribution (zero patching): Table 1 shows a gap of 20-30 AUC points under zero patching compared to 5-10 points under resample and mean patching.
-
S2: SDCM makes circuit extraction cheap, as it reduces circuit discovery to one forward pass per example, while producing more faithful circuits (6-10s per task on Qwen2.5-0.5B vs. up to 4,300s for EAP-IG-act). I think this is quite useful in downstream applications that require querying the circuit many times, which is currently too expensive.
Critical weaknesses
-
W1: It is unclear what the source of the AUC gains is. SDCM jointly changes both the circuit predictor and the backbone, so it is unclear how much of the improvement comes from fixing OOD patching vs. finding better circuits. In Sec. 4.3, SDCM circuits retain most of their advantage on the standard-trained model, which was never trained on patched passes (84.0 vs. 87.0 on IOI under zero patching). I think there is a missing ablation in which only the predictor is trained on a frozen standard-trained model, and compare it against the full SDCM, which would isolate the contribution of reshaping the backbone.
-
W2: SDCM requires fine-tuning, so its circuits explain a modified model, rather than the one we actually want to interpret. Training on such patched passes changes how the model behaves under patching, so the recovered circuits may partly reflect structure induced by the training. The twin-model experiment in Sec. 4.3 is meant to address this, but it only tests the standard-trained control model, which shares the SDCM checkpoint, data and training steps. Sec. 3.3 motivates a different use case: using the SDCM circuits to explain a model that cannot be modified. I think this is a missing evaluation, and the authors should evaluate SDCM circuits on the original pre-trained checkpoint, since this is the main practical use case.
Questions for authors
Clarification
- Q1: Have you trained only the predictor on a frozen backbone (e.g., the standard trained model) to separate the contribution of the predictor from that of reshaping the backbone?
- Q2: How do the existing SDCM circuits perform on the original pretrained checkpoint?
Future work
- Q3: Since steering is one of the most prominent downstream applications of circuit discovery, do interventions on SDCM circuits steer the model more precisely than interventions on EAP-IG (or any other post-hoc algorithm) circuits?
- Q4: Does the approach scale to more complex tasks? (entity-tracking, multi-hop recall, complex behavior like hallucinations, ...)
5 comments
Disentangling the predictor from the reshaped backbone
We thank the reviewer for suggesting this ablation, which we agree is important. We have added Appendix K, in which we train only the circuit predictor on the frozen pretrained GPT-2 checkpoint, from which the SDCM and the standard-trained model are initialized, under each of resample, zero, and mean patching. This is the natural control for joint training: the jointly trained SDCM starts from the same checkpoint, so the two differ only in whether the backbone is updated. Each predictor-only run matches the corresponding joint-training condition in predictor architecture and initialization, faithfulness objective, density schedule, training windows, and number of updates.
The jointly trained SDCM is trained from the same checkpoint, updating the predictor and the backbone together, and all circuits are evaluated on the frozen pretrained model: the predictor-only circuits, the circuits of the jointly trained SDCM (as in Appendix F.2), and EAP-IG. The table below reports the results (full-range AUC of normalized KL faithfulness, %). Each cell shows Pred. / Joint (the predictor trained alone / the jointly trained SDCM), with the higher value in bold; rows give the training scheme and columns the evaluation scheme, with EAP-IG-activations, extracted under the evaluation scheme, shown for reference.
| Training patching | Task | Resample (Pred. / Joint) | Zero (Pred. / Joint) | Mean (Pred. / Joint) |
|---|---|---|---|---|
| EAP-IG-act. | IOI | 78.6 | 49.7 | 84.4 |
| Greater-Than | 92.9 | 80.3 | 97.5 | |
| SVA | 94.2 | 69.1 | 93.2 | |
| Resample | IOI | 71.1 / 85.4 | 46.8 / 76.0 | 59.0 / 84.9 |
| Greater-Than | 85.3 / 95.8 | 56.5 / 88.5 | 66.0 / 91.9 | |
| SVA | 89.6 / 96.1 | 42.4 / 85.8 | 86.1 / 95.6 | |
| Zero | IOI | 72.6 / 81.1 | 76.0 / 85.5 | 75.1 / 84.3 |
| Greater-Than | 85.0 / 93.4 | 89.7 / 96.1 | 74.4 / 90.9 | |
| SVA | 88.3 / 94.5 | 89.5 / 95.4 | 89.9 / 93.9 | |
| Mean | IOI | 71.6 / 85.6 | 31.7 / 72.3 | 76.0 / 88.9 |
| Greater-Than | 84.0 / 94.7 | 57.5 / 88.9 | 51.1 / 88.0 | |
| SVA | 90.2 / 95.5 | 52.8 / 76.3 | 88.0 / 95.4 |
The results show that joint training is essential: the jointly trained SDCM outperforms the predictor trained alone in all 27 settings, by 16.8 AUC points on average and by up to 43.4 points, whereas the predictor trained alone falls below EAP-IG in 24 of the 27 settings. The same holds under top-1 agreement, where joint training improves over the predictor trained alone in all 27 settings, by 21.4 points on average.
Together with the transfer results of Section 4.3 and Appendix F, these findings suggest that the backbone and the predictor play different roles. In this experiment, updating the backbone together with the predictor is what allows SDCMs to discover faithful and concise circuits, yet the discovered circuits are not tied to the modified backbone: all of the joint-training results above are obtained on the original pretrained model, and the same circuits also transfer to the standard-trained model (Section 4.3). Joint training is therefore a way to find faithful circuits for the original model's computation, not just a way to reshape it.
Steering with SDCM circuits
We thank the reviewer for raising this point. We agree that steering is an important application, and it is one of several downstream uses enabled by the online nature of SDCM circuits: since an SDCM produces its circuit as part of its own forward pass, a circuit is available for every input at essentially no additional cost, which makes it practical to steer the model, monitor its computation during deployment, or flag inputs that rely on unexpected mechanisms. Evaluating steering properly also requires measuring side effects on other behaviors [1, 2], so we leave it to future work and will discuss these directions in the final version.
More complex tasks
Following the reviewer's suggestion, we have added Appendix O, which evaluates SDCMs on four harder tasks on Qwen2.5-0.5B: two-hop factual recall, in which the model must compose two stored relations to produce the capital or the official language of the country in which a given landmark is located, although the country itself is never named [3]; entity-attribute binding, in which three people are each associated with a city in context and the model must retrieve the city of the queried person [4]; and ARC-Easy, a multiple-choice science benchmark [5], in the format of MIB [6]. The first two correspond to two of the behaviors the reviewer mentions, multi-hop recall and entity tracking, the latter through binding.
Importantly, we use the SDCMs of the paper unchanged: they were trained only on general text and are applied to the new tasks without any retraining or task-specific adaptation. This is an advantage of task-agnostic training: a new behavior can be evaluated with just a forward pass per example.
Given the size of the model, these tasks are not solved perfectly, but all models perform them well above chance (85-88% on two-hop recall and 74-80% on binding, with chance at 50%, and 73-76% on ARC-Easy, with chance at 25%), and SDCM training preserves these abilities, with every SDCM within 2 points of the standard-trained model on every task. Since faithfulness is measured against the model's own outputs rather than the correct answers, the circuits capture the computation the model actually performs on each task, including on the examples it gets wrong.
The table below reports the results with all circuits evaluated on the standard-trained model, which never received SDCM training (AUC of normalized KL faithfulness, %), with the full results in Appendix O. Each column is an evaluation scheme, and each cell shows EAP-IG / SDCM, where SDCM is the SDCM trained with that same scheme and EAP-IG is the strongest circuit, over both EAP-IG variants and all three extraction schemes.
| Task | Eval: Resample | Eval: Zero | Eval: Mean |
|---|---|---|---|
| Two-hop (capital) | 88.7 / 95.6 | 66.4 / 95.3 | 90.9 / 94.6 |
| Two-hop (language) | 85.3 / 93.4 | 68.5 / 92.7 | 96.2 / 97.1 |
| Binding | 86.8 / 95.6 | 72.4 / 94.7 | 85.3 / 89.2 |
| ARC-Easy | 82.9 / 93.7 | 69.0 / 93.5 | 88.4 / 94.9 |
The conclusions of the paper carry over to all three of its main experiments:
-
Inherent SDCM circuits (Section 4.1). On the SDCMs themselves, the SDCM trained with the evaluation scheme outperforms the strongest EAP-IG circuit in all 12 settings under KL faithfulness, by 13.0 AUC points on average and by up to 30.1 points, and in all 12 under top-1 agreement.
-
Post-hoc counterfactual discovery (Section 4.2). Under each task's own counterfactual, post-hoc methods recover more faithful circuits from the SDCM than from the standard-trained model in 7 of 8 comparisons under KL faithfulness, and in all 8 under top-1 agreement.
-
SDCM circuits on the standard-trained model (Section 4.3, table above). SDCM circuits outperform the strongest EAP-IG circuit in all 12 settings under KL faithfulness, by 12.5 AUC points on average and by up to 28.9 points, and in 11 of 12 under top-1 agreement.
In the final version, we will also summarize these results in the main text.
Beyond these tasks, the input-level analysis of Appendix L, added following Reviewer 9QSq, evaluates circuits on 200 next-token predictions from general WikiText-2 text, where SDCM circuits again outperform per-input post-hoc circuits for both models and under every evaluation scheme.
[1] Circuit Component Reuse Across Tasks in Transformer Language Models (Merullo et al., ICLR 2024).
[2] Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (Marks et al., ICLR 2025).
[3] Linearity of Relation Decoding in Transformer Language Models (Hernandez et al., ICLR 2024).
[4] How Do Language Models Bind Entities in Context? (Feng and Steinhardt, ICLR 2024).
[5] Think You Have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge (Clark et al., arXiv 2018).
[6] MIB: A Mechanistic Interpretability Benchmark (Mueller et al., ICML 2025).
We thank the reviewer for their helpful and constructive feedback and for acknowledging the importance of our work. Following the reviewer's comments, we have added several new experiments to the revised manuscript: an evaluation of SDCM circuits on the original pretrained models (Appendix F.2), a predictor-only ablation trained on the frozen pretrained model (Appendix K), and an evaluation on harder tasks that require composing stored facts, binding attributes to entities in context, and answering science questions (Appendix O). We have also added pointers to these results in the relevant sections of the main text, and summarize the additions below.
Evaluating SDCM circuits on the original pretrained model
We fully agree that explaining a model that cannot be modified is the main practical use case motivated in Section 3.3. We have added this evaluation in Appendix F.2. We take the circuits produced by each SDCM, unchanged, and evaluate them on the original pretrained MIB models of GPT-2 and Qwen2.5-0.5B, from which all models are initialized and which received neither SDCM nor standard training. As in Table 2, EAP-IG circuits are extracted directly from each pretrained model under each patching scheme, and all circuits are evaluated on that model. For brevity, the table below shows the results for Qwen2.5-0.5B (full-range AUC of normalized KL faithfulness, %), with the full results for both models in Appendix F.2. Each column is an evaluation scheme, and each cell shows EAP-IG / SDCM, where SDCM is the SDCM trained with that same scheme and EAP-IG is the strongest circuit extracted from the pretrained model, over both EAP-IG variants and all three extraction schemes.
The results closely match those on the standard-trained models (Table 2 and Appendix F.1): SDCM circuits achieve higher AUC than the strongest post-hoc circuit in nearly all comparisons. The main exception, as on the standard-trained GPT-2, is Greater-Than under mean patching, where EAP-IG remains stronger, and the remaining two exceptions differ by at most 1.4 points. SDCM circuits therefore transfer to the model one actually wants to interpret, without modifying it. Regarding the concern that SDCM circuits explain a modified model, we also note that SDCM training largely preserves the model's behavior: held-out perplexity changes by at most 1.9%, IOI and SVA accuracy stays within one percentage point of the standard-trained model, and the Greater-Than score within 1.6 points (Appendix B.5). In the final version, we will also summarize these results in Section 4.3. We thank the reviewer for this valuable suggestion.