acceptodds
Under review as a conference paper at ICLR 2027

Self-Discovering Circuit Models

Abstract

*Circuit discovery* is a central task in mechanistic interpretability, aiming to identify concise sets of transformer components responsible for a given behavior using either task-agnostic patching (e.g., zero, mean, resampling) or task-specific counterfactual patching. Obtaining concise circuits via *post-hoc* discovery methods is computationally demanding, while faster approximations often produce suboptimal, less faithful circuits. Moreover, patching combines activations in ways the model never encounters during training, which can distort the recovered circuits. To mitigate these limitations, we propose a self-supervised fine-tuning approach yielding *Self-Discovering Circuit Models* (SDCMs): transformers trained under task-agnostic patching to produce input-specific circuits for their own predictions as an integral part of their output, while preserving comparable predictive performance. Aggregated across examples, these input-level circuits can form a single circuit for an entire behavior. Empirically, under task-agnostic patching, SDCMs discover circuits far more efficiently than post-hoc methods, and these circuits are substantially more faithful and concise, with a single circuit outperforming post-hoc circuits extracted separately for each patching scheme. Moreover, although counterfactual patching is undefined on general training text, our training reshapes the model parameters so that post-hoc methods recover more faithful and concise circuits from SDCMs than from conventionally trained models under such patching. Lastly, we find that SDCM circuits remain superior to post-hoc circuits even on a conventionally trained model that never received SDCM training, indicating that their advantage lies also in the circuits themselves, and not only in the trained model. This opens the door to using SDCMs as “twin models” that efficiently supply concise and faithful circuits for their conventionally trained counterparts.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 3 reviews, above 99% of the others.
Reviewer_gmzkOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a framework which improves circuit discovery in transformers. Self-Discovering Circuit Models (SDCMs) is a self-supervised fine-tuning method which trains a pre-trained transformer jointly with a lightweight circuit predictor (a single affine layer with sigmoid on the final residual stream) so that each forward pass yields two things: next-token prediction and input-specific edge-level circuit. A training step has two passes: a clean one and one with all edges predicted to be outside the circuit patched via resampling, zero, or mean, and optimizes the language modeling loss, a KL-based faithfulness loss between clean and patched outputs, and a lagrangian constraint on average circuit density. Input-level scores are aggregated in logit space into task-level edge ranking from forward passes alone. The contributions are (i) addressing both the cost of post-hoc search and OOD nature of patching by finetuning the model on patched passes, (ii) efficient extraction of circuits that are more faithful than post-hoc ones and transfer across patching schemes, (iii) a backbone on which standard post-hoc algorithms recover better circuits, even under counterfactual patching and (iv) the use of SDCMs as "twin models" that supply circuits. Validation is on GPT2-Small and Qwen2.5-0.5B models across tasks (IOI, SVA, Greater-Than, MCQA), comparing against EAP-IG (activation and input interpolation variants) on the same pretrained model fine-tuned on the same data but with language loss only. On Qwen2.5-0.5B using the area under faithfulness curve (AUC) as a metric, SDCM circuits outperform all EAP-IG circuits under all three patching schemes, with the largest gains under zero patching, at roughly 6-10 seconds vs. up to 4,700 seconds extraction time. GPT2 results show exception on Greater-Than under mean patching. By comparing models, EAP-IG on SDCM backbone finds more faithful circuits than if applied on standard-trained model. Lastly, language-modeling and task performance are largely preserved in SDCMs.

Critical strengths
  • S1: Proposing an inherently self-interpretable fine-tuning procedure by re-framing circuit discovery as a training-time objective. I think the "self-discovering" framing in which the model exposes its subgraph is new and valuable: (i) It targets the patching OOD problem in post-hoc circuit discovery. Proposing yet another post-hoc algorithm cannot fix this as the patching distortion effect is model-dependent, not algorithm-dependent. (ii) The empirical results also support this: circuits discovered by SDCM have larger gains in AUC the further the patching is from the data distribution (zero patching): Table 1 shows a gap of 20-30 AUC points under zero patching compared to 5-10 points under resample and mean patching.

  • S2: SDCM makes circuit extraction cheap, as it reduces circuit discovery to one forward pass per example, while producing more faithful circuits (6-10s per task on Qwen2.5-0.5B vs. up to 4,300s for EAP-IG-act). I think this is quite useful in downstream applications that require querying the circuit many times, which is currently too expensive.

Critical weaknesses
  • W1: It is unclear what the source of the AUC gains is. SDCM jointly changes both the circuit predictor and the backbone, so it is unclear how much of the improvement comes from fixing OOD patching vs. finding better circuits. In Sec. 4.3, SDCM circuits retain most of their advantage on the standard-trained model, which was never trained on patched passes (84.0 vs. 87.0 on IOI under zero patching). I think there is a missing ablation in which only the predictor is trained on a frozen standard-trained model, and compare it against the full SDCM, which would isolate the contribution of reshaping the backbone.

  • W2: SDCM requires fine-tuning, so its circuits explain a modified model, rather than the one we actually want to interpret. Training on such patched passes changes how the model behaves under patching, so the recovered circuits may partly reflect structure induced by the training. The twin-model experiment in Sec. 4.3 is meant to address this, but it only tests the standard-trained control model, which shares the SDCM checkpoint, data and training steps. Sec. 3.3 motivates a different use case: using the SDCM circuits to explain a model that cannot be modified. I think this is a missing evaluation, and the authors should evaluate SDCM circuits on the original pre-trained checkpoint, since this is the main practical use case.

Questions for authors

Clarification

  • Q1: Have you trained only the predictor on a frozen backbone (e.g., the standard trained model) to separate the contribution of the predictor from that of reshaping the backbone?
  • Q2: How do the existing SDCM circuits perform on the original pretrained checkpoint?

Future work

  • Q3: Since steering is one of the most prominent downstream applications of circuit discovery, do interventions on SDCM circuits steer the model more precisely than interventions on EAP-IG (or any other post-hoc algorithm) circuits?
  • Q4: Does the approach scale to more complex tasks? (entity-tracking, multi-hop recall, complex behavior like hallucinations, ...)
5 comments

We thank the reviewer for their helpful and constructive feedback and for acknowledging the importance of our work. Following the reviewer's comments, we have added several new experiments to the revised manuscript: an evaluation of SDCM circuits on the original pretrained models (Appendix F.2), a predictor-only ablation trained on the frozen pretrained model (Appendix K), and an evaluation on harder tasks that require composing stored facts, binding attributes to entities in context, and answering science questions (Appendix O). We have also added pointers to these results in the relevant sections of the main text, and summarize the additions below.

Evaluating SDCM circuits on the original pretrained model

We fully agree that explaining a model that cannot be modified is the main practical use case motivated in Section 3.3. We have added this evaluation in Appendix F.2. We take the circuits produced by each SDCM, unchanged, and evaluate them on the original pretrained MIB models of GPT-2 and Qwen2.5-0.5B, from which all models are initialized and which received neither SDCM nor standard training. As in Table 2, EAP-IG circuits are extracted directly from each pretrained model under each patching scheme, and all circuits are evaluated on that model. For brevity, the table below shows the results for Qwen2.5-0.5B (full-range AUC of normalized KL faithfulness, %), with the full results for both models in Appendix F.2. Each column is an evaluation scheme, and each cell shows EAP-IG / SDCM, where SDCM is the SDCM trained with that same scheme and EAP-IG is the strongest circuit extracted from the pretrained model, over both EAP-IG variants and all three extraction schemes.

TaskEval: ResampleEval: ZeroEval: Mean
IOI78.7 / 89.069.7 / 81.584.6 / 90.7
SVA92.8 / 96.577.4 / 94.491.7 / 95.7
MCQA83.6 / 93.362.5 / 93.990.3 / 95.0

The results closely match those on the standard-trained models (Table 2 and Appendix F.1): SDCM circuits achieve higher AUC than the strongest post-hoc circuit in nearly all comparisons. The main exception, as on the standard-trained GPT-2, is Greater-Than under mean patching, where EAP-IG remains stronger, and the remaining two exceptions differ by at most 1.4 points. SDCM circuits therefore transfer to the model one actually wants to interpret, without modifying it. Regarding the concern that SDCM circuits explain a modified model, we also note that SDCM training largely preserves the model's behavior: held-out perplexity changes by at most 1.9%, IOI and SVA accuracy stays within one percentage point of the standard-trained model, and the Greater-Than score within 1.6 points (Appendix B.5). In the final version, we will also summarize these results in Section 4.3. We thank the reviewer for this valuable suggestion.

Disentangling the predictor from the reshaped backbone

We thank the reviewer for suggesting this ablation, which we agree is important. We have added Appendix K, in which we train only the circuit predictor on the frozen pretrained GPT-2 checkpoint, from which the SDCM and the standard-trained model are initialized, under each of resample, zero, and mean patching. This is the natural control for joint training: the jointly trained SDCM starts from the same checkpoint, so the two differ only in whether the backbone is updated. Each predictor-only run matches the corresponding joint-training condition in predictor architecture and initialization, faithfulness objective, density schedule, training windows, and number of updates.

The jointly trained SDCM is trained from the same checkpoint, updating the predictor and the backbone together, and all circuits are evaluated on the frozen pretrained model: the predictor-only circuits, the circuits of the jointly trained SDCM (as in Appendix F.2), and EAP-IG. The table below reports the results (full-range AUC of normalized KL faithfulness, %). Each cell shows Pred. / Joint (the predictor trained alone / the jointly trained SDCM), with the higher value in bold; rows give the training scheme and columns the evaluation scheme, with EAP-IG-activations, extracted under the evaluation scheme, shown for reference.

Training patchingTaskResample (Pred. / Joint)Zero (Pred. / Joint)Mean (Pred. / Joint)
EAP-IG-act.IOI78.649.784.4
Greater-Than92.980.397.5
SVA94.269.193.2
ResampleIOI71.1 / 85.446.8 / 76.059.0 / 84.9
Greater-Than85.3 / 95.856.5 / 88.566.0 / 91.9
SVA89.6 / 96.142.4 / 85.886.1 / 95.6
ZeroIOI72.6 / 81.176.0 / 85.575.1 / 84.3
Greater-Than85.0 / 93.489.7 / 96.174.4 / 90.9
SVA88.3 / 94.589.5 / 95.489.9 / 93.9
MeanIOI71.6 / 85.631.7 / 72.376.0 / 88.9
Greater-Than84.0 / 94.757.5 / 88.951.1 / 88.0
SVA90.2 / 95.552.8 / 76.388.0 / 95.4

The results show that joint training is essential: the jointly trained SDCM outperforms the predictor trained alone in all 27 settings, by 16.8 AUC points on average and by up to 43.4 points, whereas the predictor trained alone falls below EAP-IG in 24 of the 27 settings. The same holds under top-1 agreement, where joint training improves over the predictor trained alone in all 27 settings, by 21.4 points on average.

Together with the transfer results of Section 4.3 and Appendix F, these findings suggest that the backbone and the predictor play different roles. In this experiment, updating the backbone together with the predictor is what allows SDCMs to discover faithful and concise circuits, yet the discovered circuits are not tied to the modified backbone: all of the joint-training results above are obtained on the original pretrained model, and the same circuits also transfer to the standard-trained model (Section 4.3). Joint training is therefore a way to find faithful circuits for the original model's computation, not just a way to reshape it.

Steering with SDCM circuits

We thank the reviewer for raising this point. We agree that steering is an important application, and it is one of several downstream uses enabled by the online nature of SDCM circuits: since an SDCM produces its circuit as part of its own forward pass, a circuit is available for every input at essentially no additional cost, which makes it practical to steer the model, monitor its computation during deployment, or flag inputs that rely on unexpected mechanisms. Evaluating steering properly also requires measuring side effects on other behaviors [1, 2], so we leave it to future work and will discuss these directions in the final version.

More complex tasks

Following the reviewer's suggestion, we have added Appendix O, which evaluates SDCMs on four harder tasks on Qwen2.5-0.5B: two-hop factual recall, in which the model must compose two stored relations to produce the capital or the official language of the country in which a given landmark is located, although the country itself is never named [3]; entity-attribute binding, in which three people are each associated with a city in context and the model must retrieve the city of the queried person [4]; and ARC-Easy, a multiple-choice science benchmark [5], in the format of MIB [6]. The first two correspond to two of the behaviors the reviewer mentions, multi-hop recall and entity tracking, the latter through binding.

Importantly, we use the SDCMs of the paper unchanged: they were trained only on general text and are applied to the new tasks without any retraining or task-specific adaptation. This is an advantage of task-agnostic training: a new behavior can be evaluated with just a forward pass per example.

Given the size of the model, these tasks are not solved perfectly, but all models perform them well above chance (85-88% on two-hop recall and 74-80% on binding, with chance at 50%, and 73-76% on ARC-Easy, with chance at 25%), and SDCM training preserves these abilities, with every SDCM within 2 points of the standard-trained model on every task. Since faithfulness is measured against the model's own outputs rather than the correct answers, the circuits capture the computation the model actually performs on each task, including on the examples it gets wrong.

The table below reports the results with all circuits evaluated on the standard-trained model, which never received SDCM training (AUC of normalized KL faithfulness, %), with the full results in Appendix O. Each column is an evaluation scheme, and each cell shows EAP-IG / SDCM, where SDCM is the SDCM trained with that same scheme and EAP-IG is the strongest circuit, over both EAP-IG variants and all three extraction schemes.

TaskEval: ResampleEval: ZeroEval: Mean
Two-hop (capital)88.7 / 95.666.4 / 95.390.9 / 94.6
Two-hop (language)85.3 / 93.468.5 / 92.796.2 / 97.1
Binding86.8 / 95.672.4 / 94.785.3 / 89.2
ARC-Easy82.9 / 93.769.0 / 93.588.4 / 94.9

The conclusions of the paper carry over to all three of its main experiments:

  • Inherent SDCM circuits (Section 4.1). On the SDCMs themselves, the SDCM trained with the evaluation scheme outperforms the strongest EAP-IG circuit in all 12 settings under KL faithfulness, by 13.0 AUC points on average and by up to 30.1 points, and in all 12 under top-1 agreement.

  • Post-hoc counterfactual discovery (Section 4.2). Under each task's own counterfactual, post-hoc methods recover more faithful circuits from the SDCM than from the standard-trained model in 7 of 8 comparisons under KL faithfulness, and in all 8 under top-1 agreement.

  • SDCM circuits on the standard-trained model (Section 4.3, table above). SDCM circuits outperform the strongest EAP-IG circuit in all 12 settings under KL faithfulness, by 12.5 AUC points on average and by up to 28.9 points, and in 11 of 12 under top-1 agreement.

In the final version, we will also summarize these results in the main text.

Beyond these tasks, the input-level analysis of Appendix L, added following Reviewer 9QSq, evaluates circuits on 200 next-token predictions from general WikiText-2 text, where SDCM circuits again outperform per-input post-hoc circuits for both models and under every evaluation scheme.

[1] Circuit Component Reuse Across Tasks in Transformer Language Models (Merullo et al., ICLR 2024).

[2] Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (Marks et al., ICLR 2025).

[3] Linearity of Relation Decoding in Transformer Language Models (Hernandez et al., ICLR 2024).

[4] How Do Language Models Bind Entities in Context? (Feng and Steinhardt, ICLR 2024).

[5] Think You Have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge (Clark et al., arXiv 2018).

[6] MIB: A Mechanistic Interpretability Benchmark (Mueller et al., ICML 2025).

Reviewer_9QSqOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Circuit discovery is normally post-hoc: search a fixed model for a small subgraph, then score it by patching the rest. The authors argue this is expensive and biased, since patching feeds the model activation combinations it never saw. They instead train the model to expose its own circuit: an LM is fine-tuned with a linear mask head that scores every edge from the final residual, while a second pass patches everything outside the predicted circuit, a KL term pulls that pass onto the clean prediction, and a density penalty holds the circuit at 3% of edges. Training uses only general text, with no task labels. Per-example scores are averaged into one task-level ranking and compared against EAP-IG on IOI, SVA, Greater-Than and MCQA. The ranking is more faithful at matched density under all three patch schemes, and transfers to the standard model.

Critical strengths
  1. Simple, and it works. One ranking beats per-scheme EAP-IG, including under patch schemes it never trained on (Qwen, zero patching: 87.0 / 96.5 / 94.0 against 56.5 / 76.4 / 69.6 at best). I did not expect a single mask head to transfer like that.

  2. The twin-model result is the useful part. Circuits evaluated unchanged on the model that never saw SDCM training still beat that model's own EAP-IG circuits.

  3. Extraction is cheap. Forward passes only, no gradients or reference inputs, one ranking for all three schemes: seconds against thousands of seconds, with error bars over ten runs. That is what makes it usable at task scale.

Critical weaknesses
  1. The metric is the training objective. L_faith is a KL between clean and patched predictions, and the reported faithfulness is that same KL, normalized and averaged. So the headline claim partly says the loss went down. Tables 10/11 handle the model-versus-method confound; nothing handles the metric one. No measure the model was not trained on is reported.

  2. Only sufficiency is tested. Nothing checks necessity by removing circuit edges from the clean pass, and routing computation through a small subgraph is the cheapest way to satisfy the objective. Table 2 argues against that but does not rule it out.

  3. Input-level circuits are never evaluated on their own. Every number is for a ranking aggregated over 100 examples. No per-example density spread, no mask agreement across inputs, no per-example faithfulness against per-example post-hoc circuits. If the masks are near-constant, aggregation is close to a no-op and this is a trained task-agnostic ranker, not a self-discovering one.

Questions for authors
  1. Can you report faithfulness under something not trained on, and give the empty-circuit KL per model? If these differ between SDCM and baseline, the normalization in Table 1 is not on one scale.

  2. Does deleting the circuit's edges from the unpatched pass break the prediction?

  3. Table 2 sits within about a point of Table 1, so the circuits lose almost nothing across models. How much of Fig. 3 is then a changed backbone? A compute-matched baseline would settle it; 15.0 vs 30.5 GPU-hours is only step-matched.

3 comments

We thank the reviewer for their helpful and constructive feedback and for acknowledging the importance of our work. Following the reviewer's comments, we have added several new experiments to the revised manuscript: an evaluation under additional faithfulness metrics, together with a normalization analysis (Appendix J), a direct analysis of input-level circuits (Appendix L), a compute-matched standard-training control (Appendix M), a predictor-only ablation on a frozen backbone (Appendix K), and a necessity analysis of the discovered circuits (Appendix N). We have also added pointers to these results in the relevant sections of the main text, and summarize the additions below.

The faithfulness metric coincides with the training objective

We thank the reviewer for raising this important point, and first clarify two aspects of the original setup. First, the baselines target the same metric: the post-hoc methods are extracted with KL attribution, and for every baseline, circuit extraction and evaluation use the same metric and patching scheme (Appendices C.4 and G), so the comparison does not favor SDCMs by construction. Second, the object that is trained differs substantially from the object that is evaluated: training optimizes the KL of input-level circuits at a single soft density of 3%, at random positions of general text, whereas evaluation uses task-level rankings aggregated over 100 examples, on held-out task prompts, at densities ranging from 0.1% to 100%.

Moreover, the advantage of SDCMs extends well beyond the training setting: it holds under patching schemes never seen during training (Section 4.1), carries over to post-hoc discovery on the trained backbone under counterfactual patching (Section 4.2), and persists on models that never received SDCM training (Section 4.3), indicating that the circuits are not tailored to the training objective.

Nonetheless, to further strengthen these findings, the new Appendix J re-evaluates all circuits of Sections 4.1–4.3, without retraining the models or re-extracting the circuits, under two additional faithfulness metrics that are not optimized during training: top-1 agreement between the circuit and the full model over the entire vocabulary [1, 2], and Jensen-Shannon (JS) divergence between their output distributions [3]. In each comparison below, the SDCM trained with the evaluation scheme is compared against the strongest post-hoc circuit for the same model, task, and evaluation scheme, as in the tables of this response. The conclusions of the paper hold under both metrics:

  • Inherent SDCM circuits (Section 4.1, Appendix J.1). SDCM circuits achieve higher AUC than the strongest post-hoc circuit in nearly all settings under both JS divergence (17 of 18) and top-1 agreement (16 of 18). On Qwen2.5-0.5B, every SDCM variant, including those trained under a different scheme than the one evaluated, outperforms every post-hoc circuit in all settings under both metrics. The only setting in which a post-hoc circuit leads under both metrics is GPT-2's Greater-Than under mean patching.

  • Post-hoc counterfactual discovery (Section 4.2, Appendix J.2). EAP-IG-activations recovers more faithful circuits from the SDCM than from the standard-trained model in five of the six settings under each metric, as under KL.

  • SDCM circuits on the standard-trained model (Section 4.3, Appendix J.3). SDCM circuits again achieve higher AUC in nearly all settings under both JS divergence (17 of 18) and top-1 agreement (16 of 18), with the same single exception, even though all circuits are evaluated on a model that never received SDCM training.

In the final version, we will also summarize these results in the main text.

Normalization and the empty-circuit KL

Appendix J.4 now reports the empty-circuit KL of every model under every patching scheme. Under task-agnostic patching, the SDCMs' values are very close to those of the standard-trained model (within 5% in 50 of 54 cases), whereas under counterfactual patching the differences are larger, ranging from 8% lower to 19% higher. To test whether these differences affect our conclusions, we re-normalized both models with the standard-trained model's empty-circuit value, keeping the circuits and AUC windows fixed. Every task-agnostic comparison that favored SDCMs remains favorable under all three metrics (KL, top-1 agreement, and JS). In the six counterfactual settings of Section 4.2, shared normalization preserves every favorable comparison, and increases the number of settings in which SDCM is favorable under KL from five to all six. The comparisons of Section 4.3 (Table 2 and Appendix F) were already on a single scale, since all circuits there are evaluated on the same model. We thank the reviewer for this careful question.

Necessity of the discovered circuits

The reviewer is correct that our objective, like the standard faithfulness criterion of Eq. 2, targets sufficiency. We have therefore added Appendix N, which tests necessity directly: we delete the circuit by patching its own edges while the rest of the model computes normally, and measure the normalized deletion damage, where 100 corresponds to full ablation [4]. We compare against random edge sets of the same size, drawn both from the full graph and from the circuit's complement using path sampling [5]. The table below summarizes the results at 1% density (normalized KL deletion damage, %, under resample patching, where higher means more necessary); results at 2% and 5% density and under top-1 agreement are reported in Appendix N.

Circuit (1% density)GPT-2 IOIGPT-2 Greater-ThanGPT-2 SVAQwen IOIQwen SVAQwen MCQA
SDCM (ours)96.293.496.5100.0100.399.8
EAP-IG-act.94.095.593.898.999.599.8
Random edges, full graph29.919.327.463.158.048.3
Random edges, outside the SDCM circuit5.53.44.33.80.91.1

To address the reviewer's concern about routing, we run this test on the standard-trained model, which was never trained to route its computation through SDCM circuits. The SDCM circuits are clearly necessary: deleting circuits of only 1% density already produces KL damage of 93.4 to 100.3 and top-1 agreement damage of 99.0 to 100.0, essentially equivalent to ablating the entire model, and on par with EAP-IG circuits of the same size. In contrast, deleting random edge sets of the same size from the full graph produces KL damage of only 19.3 to 63.1, and deleting edges outside the SDCM circuit produces only 0.8 to 7.2 across all tested densities. SDCM circuits are therefore necessary as well as sufficient for the prediction. Since this holds on a model that never received SDCM training, it cannot be explained by computation having been routed through the circuit during training.

Evaluating input-level circuits on their own

This is an excellent point, which we agree deserves a direct analysis. We have added Appendix L, which evaluates input-level circuits on 200 next-token predictions from WikiText-2, with contexts of 1 to 64 tokens. These inputs are general text rather than templated task prompts, so this analysis also tests the circuits beyond our benchmark tasks. The table below summarizes the per-input faithfulness (mean per-input AUC of normalized KL faithfulness, %), with full results in Appendix L.1. Each column is a model and evaluation scheme, and the rows compare the SDCM trained with that same scheme against the strongest per-input EAP-IG circuit, over both EAP-IG variants and all three extraction schemes.

MethodGPT-2, Eval: ResampleGPT-2, Eval: ZeroGPT-2, Eval: MeanQwen, Eval: ResampleQwen, Eval: ZeroQwen, Eval: Mean
Strongest EAP-IG86.868.281.291.167.788.6
SDCM (ours)94.994.193.197.296.396.7
  • Per-input faithfulness (Appendix L.1). Each circuit is extracted from a single input and evaluated on that same input, without any aggregation. Beyond the matched comparison in the table, all three SDCM variants achieve a higher mean per-input KL AUC than every EAP-IG variant, across both models and all three evaluation schemes, and the advantage also holds under top-1 agreement, where the matched SDCM leads the strongest EAP-IG result by 11.0 to 27.1 points.

  • Circuit sizes vary across inputs (Appendix L.2). Since only the average density is constrained during training, the predictor's own masks, which keep every edge with a gate score of at least 0.5, are free to grow or shrink with the input, and they do: median densities are 2.31–2.42% for GPT-2 and 1.44–1.60% for Qwen2.5-0.5B, while the 5th–95th percentile range spans, for example, 0.79% to 7.49% for the resample-trained Qwen2.5-0.5B, close to a tenfold range.

  • Masks are input- and task-dependent (Appendix L.3). These masks have mean pairwise Jaccard similarities of only 0.505 to 0.624 on WikiText-2, indicating substantial input-dependent variation. At matched sizes (3% density), inputs from the same task agree strongly (89.3–92.0%), whereas inputs from different tasks agree considerably less (67.3–77.2%), as do general-text inputs (68.1–78.3%). Agreement within a task exceeds agreement between tasks for every pair of tasks, in all 144 task-pair settings across both models, all three training schemes, and all eight tested densities.

Together, these results show that the predictor is not a single task-agnostic ranker. Its circuits are faithful for individual inputs and change in size and composition across inputs and tasks, so aggregation is far from a no-op.

The contribution of the backbone and a compute-matched baseline

We first note that Figure 3 measures whether the SDCM backbone makes post-hoc discovery more faithful under counterfactual patching, which is a separate question from whether the predictor's circuits transfer to other models (Tables 1 and 2).

Post-hoc discovery (Figure 3). We agree that our original setting was step-matched rather than compute-matched: each SDCM update runs two backbone forward passes, whereas standard training runs one. We have therefore added a compute-matched standard-training control (Appendix M), which matches the SDCM's training data and compute by using all sampled training windows twice for language-modeling updates, reshuffled into new minibatches, with all other settings unchanged.

Repeating the comparison of Section 4.2, the SDCM yields higher KL AUC than this compute-matched control in all twelve comparisons (six counterfactual settings, each with EAP-IG-activations and EAP-IG-inputs). Moreover, the gap favoring the SDCM is even larger than against the original standard-trained model in ten of the twelve comparisons. Additional language-modeling training therefore does not account for the gains in Figure 3, which we attribute to training on patched passes. In the final version, we will also add this comparison to Section 4.2.

The predictor's circuits. To further isolate the contribution of the backbone, we have also added Appendix K, in which we train only the circuit predictor on the frozen pretrained GPT-2 checkpoint, from which the SDCM and the standard-trained model are initialized, matching the joint-training condition in all other respects. This is the natural control for joint training: the jointly trained SDCM starts from the same checkpoint, so the two differ only in whether the backbone is updated.

All circuits are evaluated on the frozen pretrained model: the predictor-only circuits, the circuits of the jointly trained SDCM (as in Appendix F.2), and EAP-IG. The table below reports the results (full-range AUC of normalized KL faithfulness, %). Each cell shows Pred. / Joint (the predictor trained alone / the jointly trained SDCM), with the higher value in bold; rows give the training scheme and columns the evaluation scheme, with EAP-IG-activations, extracted under the evaluation scheme, shown for reference.

Training patchingTaskResample (Pred. / Joint)Zero (Pred. / Joint)Mean (Pred. / Joint)
EAP-IG-act.IOI78.649.784.4
Greater-Than92.980.397.5
SVA94.269.193.2
ResampleIOI71.1 / 85.446.8 / 76.059.0 / 84.9
Greater-Than85.3 / 95.856.5 / 88.566.0 / 91.9
SVA89.6 / 96.142.4 / 85.886.1 / 95.6
ZeroIOI72.6 / 81.176.0 / 85.575.1 / 84.3
Greater-Than85.0 / 93.489.7 / 96.174.4 / 90.9
SVA88.3 / 94.589.5 / 95.489.9 / 93.9
MeanIOI71.6 / 85.631.7 / 72.376.0 / 88.9
Greater-Than84.0 / 94.757.5 / 88.951.1 / 88.0
SVA90.2 / 95.552.8 / 76.388.0 / 95.4

Joint training outperforms the predictor trained alone in all 27 settings, by 16.8 AUC points on average, whereas the predictor trained alone falls below EAP-IG in 24 of the 27 settings. Thus, in this experiment, training the backbone jointly with the predictor is what allows SDCMs to find faithful circuits, yet the circuits it finds are not tied to the modified backbone, since all of the results above are obtained on the original pretrained model.

[1] Finding Transformer Circuits With Edge Pruning (Bhaskar et al., NeurIPS 2024).

[2] Position-aware Automatic Circuit Discovery (Haklay et al., ACL 2025).

[3] Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Models (Mondorf et al., ACL 2025).

[4] Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (Marks et al., ICLR 2025).

[5] Hypothesis Testing the Circuit Hypothesis in LLMs (Shi et al., NeurIPS 2024).

Reviewer_cofxOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Instead of performing circuit analysis on LLMs after they have been trained, SDCM jointly trains a circuit predictor and an LLM, with the goal of obtaining a model where circuit extraction is easier - either by the circuit predictor or by other circuit extraction methods on the resulting trained LLM. They demonstrate across two different models that the circuits are more easily extractable from their jointly trained model both with the circuit predictor and other circuit extraction methods. They evaluate across a sweep of circuit densities. They also show extraction time is faster with their circuit method.

Critical strengths

The empirical results are strong. The circuits are more easily extractable on the trained model and both parts of their system generalizes (the circuit predictor is good at extracting circuits from other models and the trained model can have circuits extracted from it more cleanly with other circuit extraction methods).

Critical weaknesses

The main weakness from a pragmatic safety point of view is that to get the full benefits of SDCM you have to convince them to use your training scheme and those maximizing for model capabilities may not be interested in that. That being said, even the circuit predictor provides better performance on models trained without it.

In terms of computational cost, I would like to see a clear analysis of the total number of FLOPs required to do the additional training SDCM requires vs normal training and using the post-hoc circuit discovery models. Presumably there is some number of circuit extractions you need to do before SDCM starts saving you compute? I realize that computational savings are not necessarily the main point of the paper but I still think its a useful comparison to have.

While there is already a substantial amount of work in this paper, I think it would be possible to also generate synthetic text data to do counterfactual patching during training with SDCM?

Questions for authors

Took me a minute to figure out what res. ios on Table 1. Wouldnt take up that much space to not truncate it?

Could counterfactual text inputs be generated synthetically and added to the training mix? I think there are reasons why this might not make sense to do but an explanation of why it was not done would be nice to add.

4 comments

We thank the reviewer for their helpful and constructive feedback and for acknowledging the importance of our work. Following the reviewer's comments, we have added a full FLOP and break-even analysis (Appendix G) and an explanation of why training does not use counterfactual patching (Section 6 and Appendix B.1). In addition, we have evaluated SDCM circuits on the original pretrained models (Appendix F.2) and on harder tasks (Appendix O), which further support our answers. We have also added pointers to these results in the relevant sections of the main text, and summarize the additions below.

Adopting SDCM training when maximizing model capabilities is the main focus

This is an important point, and we agree that obtaining the full benefits of SDCMs requires an additional training objective, which developers focused on maximizing capabilities may be reluctant to adopt. However, as the reviewer correctly notes, SDCM circuits remain superior even on models that never received SDCM training: on the standard-trained model, which shares the SDCM's data and training steps but uses the language-modeling objective alone, SDCM circuits outperform the model's own EAP-IG circuits across tasks and patching schemes.

To further support this point, we have now also evaluated the SDCM circuits, unchanged, on the original pretrained models of GPT-2 and Qwen2.5-0.5B, which never received any fine-tuning (Appendix F.2). There, too, SDCM circuits achieve higher AUC than the strongest post-hoc circuit extracted from the pretrained model itself in nearly all comparisons. We will also report these results in Section 4.3 of the final version.

The table below shows the results for Qwen2.5-0.5B (full-range AUC of normalized KL faithfulness, %), with the full results for both models in Appendix F.2. Each column is an evaluation scheme, and each cell shows EAP-IG / SDCM, where SDCM is the SDCM trained with that same scheme and EAP-IG is the strongest circuit extracted from the pretrained model, over both EAP-IG variants and all three extraction schemes.

TaskEval: ResampleEval: ZeroEval: Mean
IOI78.7 / 89.069.7 / 81.584.6 / 90.7
SVA92.8 / 96.577.4 / 94.491.7 / 95.7
MCQA83.6 / 93.362.5 / 93.990.3 / 95.0

This is exactly the twin-model setting of Section 4.3, so developers do not need to change their training at all: they can keep training and deploying their original model exactly as before, and use a separately fine-tuned SDCM solely as a companion that supplies circuits for it. Since the original model is never modified, adopting SDCMs in this way comes at no cost to its capabilities, and the SDCM can even be trained by a third party, starting from a released checkpoint.

Moreover, in settings where a practitioner chooses to use the SDCM itself, its predictive performance remains comparable to that of the standard-trained model (Appendix B.5). SDCM training is a short post-training stage of 40M tokens that retains the language-modeling loss throughout. We also observed that SDCM training can sometimes even improve predictive performance: mean-trained SDCMs never fall below the standard-trained model on any of the evaluated tasks, and slightly exceed it on several (Table 7). We find this effect intriguing, and believe that investigating whether SDCM training can also serve to improve predictive performance is an interesting direction for future work, which we discuss in Section 6.

The total computational cost and the break-even point

There is indeed such a break-even point, and it is reached quickly: the revised manuscript now provides a full FLOP analysis of both training and extraction, together with the resulting break-even points (Appendix G), which we summarize below. Before doing so, we would like to emphasize again that SDCMs offer this efficiency together with substantially better circuits: SDCM circuits are substantially more faithful than the post-hoc circuits at the same size, and accordingly smaller at the same faithfulness, and a single SDCM circuit generalizes across patching schemes, outperforming post-hoc circuits even when those are extracted under the very scheme being evaluated (Table 1).

Training cost (paid once, offline). Using the standard FLOP approximation [1], with the full derivation in Appendix G, SDCM training costs 2.20 times (GPT-2) and 2.33 times (Qwen2.5-0.5B) the FLOPs of standard fine-tuning on the same 40M tokens, since each training step runs both a clean and a patched pass. This is in line with the measured GPU-hour ratios of 2.04 and 2.40, and amounts to about 3.6 x 10^16 (GPT-2) and 1.6 x 10^17 (Qwen2.5-0.5B) additional FLOPs.

Extraction cost (paid every time a circuit is needed, online). An SDCM produces a circuit with a single forward pass per example, whereas EAP-IG-activations runs five integration steps, each with a forward and a backward pass, at the input and at the attention and MLP outputs of every layer, amounting to 125 forward and backward passes on GPT-2 and 245 on Qwen2.5-0.5B. SDCM extraction therefore requires more than 630 times fewer FLOPs than EAP-IG-activations on GPT-2, and more than 1,230 times fewer on Qwen2.5-0.5B. This gap grows with model depth, since EAP-IG-activations repeats its integration at every layer.

Break-even point. Combining the two, we count the number of task-level circuit extractions (each on 100 examples) after which the total compute of the SDCM, including its additional training, becomes lower than that of running EAP-IG-activations on a standard-trained model. This happens after 66, 218, and 37 extractions on Qwen2.5-0.5B's IOI, SVA, and MCQA, respectively, and after 106, 223, and 345 extractions on GPT-2's IOI, Greater-Than, and SVA. Since the same SDCM serves all tasks, extractions on any task count toward the same break-even point, and break-even points computed from the measured GPU times instead of FLOPs are similar (Appendix G). Moreover, if circuits are needed under several patching schemes, for example to verify that they are robust to the choice of ablation, break-even comes about three times sooner, since a single SDCM ranking serves all schemes, whereas EAP-IG must be rerun for each.

Finally, beyond total compute, the low cost of each extraction enables new uses of circuits. As Reviewer gmzk noted, cheap extraction makes circuits practical for downstream applications that need to query them many times. In particular, since an SDCM produces its circuit as part of its own forward pass, circuits can be obtained on the fly for every input while the model is running, which is out of reach when each extraction requires hundreds of forward and backward passes. We thank the reviewer for this suggestion, which we believe makes the practical benefits of SDCMs considerably clearer.

Generating synthetic counterfactual data for training

We agree that this choice deserves an explanation, which we have now added to the revised manuscript (Section 6 and Appendix B.1). Synthetic counterfactuals are possible in principle, but they face both a practical and a conceptual obstacle. Practically, every training window would need its own counterfactual: a fluent, minimally different input whose answer changes in a known way. Conceptually, a counterfactual specifies the task, whereas a general-text example supports many tasks. Consider, for example, the input "Mary was born in 1950 and John in 1972, so the older of the two is" (answer: "Mary"). Changing 1950 to 1990 flips the answer through the comparison of the years, as in Greater-Than, whereas swapping the two names flips it through which name is moved to the output, as in IOI. The same prediction thus admits counterfactuals for two different tasks, each yielding a different circuit, and general text does not determine which one to train on.

One could instead generate counterfactuals for the very tasks we evaluate, but this would expose the model to the evaluation contrast itself, much like training on the test set: gains under those counterfactuals could then no longer be attributed to SDCM training, and the comparison with post-hoc methods would no longer be fair. We therefore train SDCMs under task-agnostic patching.

Importantly, however, SDCM training still improves circuit discovery under counterfactual patching itself, through the trained backbone. Although counterfactuals are never used during training, standard post-hoc methods recover more faithful and more concise circuits from SDCMs than from conventionally trained models under real task counterfactuals (Section 4.2). For example, on Qwen2.5-0.5B's IOI, matching the standard model's faithfulness requires 1.7 times fewer edges on the SDCM (5,205 rather than 8,872), and overall, EAP-IG-activations achieves a higher AUC on SDCMs in five of the six counterfactual settings. This advantage also holds against the new compute-matched standard-training control of Appendix M, where SDCMs yield more faithful circuits in all six settings, with both EAP-IG-activations and EAP-IG-inputs. The same holds on the four harder tasks we have added in Appendix O (two-hop factual recall, entity binding, and ARC-Easy): under each task's own counterfactual, post-hoc methods recover more faithful circuits from the SDCM than from the standard-trained model in 15 of the 16 comparisons across both EAP-IG variants and both metrics. The benefits of SDCM training therefore extend to counterfactual patching, even though it cannot be used during training.

The "Res." header in Table 1

"Res." stands for resample patching. Since writing the scheme names in full would make Table 1 wider than the page, its caption now defines Res., Zero, and Mean, which also covers all subsequent tables that follow Table 1's conventions. We thank the reviewer for catching this.

[1] Scaling Laws for Neural Language Models (Kaplan et al., arXiv 2020).

thank you for your response. I maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.