- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Circuit Condensation, shifting mechanistic interpretability from discovering small circuits in a frozen model to actively making a model's behavior more concentrated and verifiable through post-training. It combines EAP-IG-guided edge pruning, LoRA-based self-distillation, and validation constraints to produce smaller causal circuits while attempting to preserve the original behavior.
I find this a meaningful research direction, supported by substantial experimental and causal analyses. However, several important questions remain, particularly whether smaller circuits imply better interpretability, how faithfully the adapted model preserves the original mechanism, and whether general capabilities are sufficiently retained.
Critical strengths
S1. A meaningful research perspective with genuine interpretability value.
I particularly appreciate the perspective of this work. Mechanistic interpretability has long faced a practical obstacle: discovered circuits often contain hundreds or thousands of edges, making exhaustive causal verification infeasible. So this work resonates with me.
After examining the method, I recognize that its individual ingredients are relatively straightforward: EAP-IG for selecting important causal edges, LoRA for adapting the remaining computation, and accuracy/perplexity constraints for preventing unacceptable behavioral degradation. Nevertheless, I personally believe this is a paper whose value lies primarily in its research formulation and causal evidence, rather than algorithmic complexity and combination. The underlying research question is meaningful and worth pursuing.
S2. Strong experimental design and insightful analyses.
I appreciate the breadth and organization of the experiments, especially the efforts to distinguish the effects of weight adaptation from those of circuit search. The frozen-weight controls, exhaustive subset and pairwise ablations, and IOI mechanistic interventions form a relatively convincing experimental chain.
Importantly, the paper goes beyond reporting circuit-size reductions and demonstrates what smaller circuits enable in terms of causal verification. I find these analyses particularly valuable.
Critical weaknesses
**W1. Does a smaller circuit necessarily imply better interpretability? **
While I appreciate the substantial reductions in circuit size, I think the paper could better distinguish structural compactness from mechanistic interpretability.
Reducing the number of retained edges clearly makes exhaustive verification more tractable, but it does not necessarily make the underlying computation easier for humans to understand. The surviving components may still perform complex operations, and post-training may redistribute computational complexity rather than genuinely simplify it.
I would therefore appreciate stronger evidence connecting the reported circuit-size reductions to improvements in mechanistic legibility, beyond the reduced cost of causal verification. [Q1]
W2. The boundary between model reshaping and explaining the original model remains important.
Circuit Condensation uses LoRA to make a target behavior depend on fewer edges in the adapted model. However, similar outputs do not necessarily imply identical internal mechanisms, as the same behavior can be implemented through different computational pathways.
I recognize that the authors have made a serious effort to address this issue in Section 5.4, and I appreciate the analyses of edge overlap, output agreement, KL divergence, and IOI mechanistic interventions. Nevertheless, these results primarily establish location overlap, behavioral similarity, and partial mechanistic correspondence, rather than full preservation of the original computation.
In particular, the reported 96.5% answer agreement should be interpreted carefully. On tasks such as IOI, induction, and several docstring settings, both models already achieve near-perfect accuracy, so high agreement may arise even when their internal computations differ.
I would welcome more direct evidence of which mechanisms are preserved, which are reorganized, and what this distinction means for the scientific objective of the paper. [Q2]
Questions for authors
I would appreciate the authors' clarification of the following questions. I find the research direction particularly interesting and hope these questions can lead to further insights into the proposed approach.
[Q1] Does circuit condensation improve mechanistic interpretability beyond structural compactness?
The paper uses retained edge count as its primary measure of circuit complexity. This is reasonable for exhaustive verification, but I believe it captures only one dimension of interpretability.
For instance, a 4-edge circuit may still contain several complex or polysemantic MLP blocks. Without further decomposition, such a circuit is not necessarily easier to understand than a somewhat larger circuit with more clearly organized functional components.
Moreover, LoRA updates span multiple attention/MLP projections, adding approximately 0.9%-1.9% of the original model's parameters (Appendix L). This raises the possibility that some computational complexity is redistributed into the surviving components.
I would particularly appreciate more insightful analyses or quantitative evidence addressing the following:
- Beyond edge count, how much does condensation simplify the actual mechanism, including the number of functionally relevant components and their interactions?
- Does post-training genuinely simplify the computation, or partly transfer complexity into the retained heads, MLPs, and adapter?
- Could the authors provide a more detailed mechanistic case study demonstrating what becomes easier to understand, rather than only easier to verify?
[Q2] Original-model mechanistic fidelity.
As discussed in W2, I appreciate the authors' efforts in Section 5.4, but I remain interested in how faithfully the condensed circuit represents the original model's computation.
[Q2.1] Counterfactual and out-of-distribution consistency.
Could the authors compare the original and condensed models on additional matched counterfactual or out-of-distribution inputs? This would help determine whether their behavioral agreement extends beyond the relatively constrained evaluation distributions.
[Q2.2] Direct comparison of causal mechanisms.
Could the authors measure how the causal effects, activation patterns, or information-routing behavior of corresponding edges change before and after condensation? Such evidence would help distinguish preservation of existing mechanisms from newly learned computational pathways.
[Q2.3] What is the ultimate scientific objective?
This is perhaps my most important conceptual question, and I would particularly appreciate the authors' perspective.
Is the goal to obtain a more faithful mechanistic explanation of the original model, or to construct a behaviorally similar but more interpretable adapted model?
I understand that the paper acknowledges this distinction. Nevertheless, I believe a sharper articulation of the intended scientific contribution would strengthen the work considerably, since these two objectives have different implications for mechanistic interpretability.
[Q3] Novelty relative to existing training-based interpretability approaches.
I have a minor question about the novelty positioning and would appreciate a more insightful comparison with two related studies:
- Draye et al. (2025), Intrinsically Interpretable Attention via Sparse Post-Training, which uses post-training to induce sparse attention connectivity and reports substantially smaller downstream circuits.
- Li et al. (2025), Fine-Tuning is Subgraph Search, which proposes Circuit-Tuning, iteratively constructing task-relevant subgraphs and updating model parameters.
I recognize important differences: Draye et al. primarily constrain attention connectivity rather than directly pruning behavior-specific inter-component edges, while Li et al. focus more on task adaptation and learning dynamics than preserving an original model through self-distillation.
Nevertheless, these works already explore the broader idea of reshaping model computation through training to obtain more interpretable subgraphs.
Could the authors clarify what fundamentally new insight Circuit Condensation establishes beyond these approaches? A more direct methodological comparison, and empirical evidence where the evaluation settings can be aligned fairly, would strengthen the novelty argument.
[Q4] Generalization to larger models.
The largest backbone evaluated in this paper has 4B parameters. I understand the substantial computational cost of repeated circuit analysis and post-training, but I would be more confident in the method's generality if the authors could include at least two larger models.
My concern is that larger models may exhibit more complex redundancy, distributed computation, and adaptation dynamics. It would be interesting to see whether the same condensation behavior and causal properties persist at a larger scale.
[Q5] Experimental questions and concerns.
I have several additional questions about the experimental methodology. Please correct me if I have misunderstood any implementation or evaluation details.
[Q5.1] Computational cost and practical benefit.
Appendix G.1 reports a median condensation runtime of 2.95 hours on one A100 GPU, reaching 17.5 hours in the largest settings.
While smaller circuits substantially reduce subsequent verification costs, the initial post-training overhead may outweigh these benefits when only a limited number of analyses are performed.
Could the authors provide a more explicit cost-benefit or break-even analysis, comparing the total cost of condensation and subsequent verification against conventional frozen-model circuit discovery? I am particularly interested in when the proposed approach becomes practically advantageous.
[Q5.2] Reliability of adapted attribution baselines.
EAP-IG and EAP-GP are adapted to the proposed gate space. The implementation audit in Appendix C reports a Spearman correlation of 0.545 between EAP-IG scores and exact single-edge effects on a 325-edge graph.
Although this provides some support for the implementation, it does not establish equivalence to the original attribution methods, particularly given the modified interpolation space.
Could the authors provide stronger implementation validation, or demonstrate that the main comparisons are robust to these methodological adaptations? This would help establish the fairness of the frozen-weight baselines.
[Q5.3] General-capability preservation under independent evaluations.
The controller imposes a 1.05 perplexity-ratio constraint on its capability probe. However, Appendix E.1 reports approximately 9% median perplexity degradation on WikiText-2, with a worst-case ratio around 1.74, alongside a 5-7 percentage-point decline on LAMBADA.
If the same 1.05 threshold were applied to WikiText-2, approximately 18-19 of the 32 settings would exceed it.
I understand that the controller and independent evaluations use different datasets, but this raises an important question: how reliably does the capability gate preserve general capabilities beyond its particular probe distribution?
Could the authors investigate whether stricter or more diverse capability probes change the final circuit sizes and the resulting interpretability-capability trade-off?
Overall assessment.
I find the research perspective and scientific motivation of this paper particularly compelling. The idea of making causal circuits easier to verify through post-training resonates with my view of where mechanistic interpretability research could become more useful.
My remaining concerns mainly involve whether structural condensation translates into deeper mechanistic understanding, how faithfully the resulting circuits preserve the original model's computation, and whether the observed benefits generalize beyond the current evaluation conditions.
I would be very grateful for the authors' responses, especially for insightful explanations and additional empirical evidence addressing these central questions. I remain open to revising my assessment based on how convincingly these concerns are resolved, and I look forward to a substantive discussion.
10 comments
Response to Reviewer HeHi (2/4): Q2.1–Q2.3
Q2.1. Counterfactual and out-of-distribution consistency.
To test whether the agreement extends beyond the original data distribution, we created held-out sets for all 32 cells, with 1,000 examples each. These use unseen names and five unseen templates for IOI, unseen nouns for agreement, unseen identifiers for docstring, and 2× longer contexts for induction (App. P.4).
| OOD accuracy (median) | Original | Adapted model | C1 circuit | Frozen circuit |
|---|---|---|---|---|
| All 32 cells | 0.98 | 0.98 | 0.85 | 0.91 |
| IOI | 1.00 | 1.00 | 0.96 | 0.96 |
| Agreement | 0.78 | 0.79 | 0.82 | 0.71 |
| Induction | 0.90 | 0.96 | 0.85 | 0.84 |
| Docstring | 1.00 | 1.00 | 0.76 | 0.98 |
The adapted model stays within 0.05 of the original in 31 of 32 cells. The C1 circuit does so in 19 of 32, the same count as the frozen circuit. C1 is better on agreement (all 8 cells) and worse on docstring, where three circuits fail on unseen identifiers (Gemma-3-1B 0.62, Gemma-3-4B 0.11, Qwen3-1.7B 0.25, against 0.98–1.00 for the frozen circuits).
Counterfactual inputs. We also scored each circuit under three corruption constructions it was never trained with (a different template, another example's prompt, unrelated text). C1 degrades about as much as the frozen circuit: 0.69 vs 0.65, 0.69 vs 0.67 and 0.24 vs 0.25 (medians, 32 cells). Neither circuit is corruption-independent (App. P.8).
Q2.2. Direct comparison of causal mechanisms before and after condensation.
We compared the original and adapted models on the same edge sets, separating the three measurements you ask about. Results are medians across 32 cells unless stated (App. P.1).
Causal effects. We measured both estimated edge importance and whether the retained edges can carry the behavior together. EAP-IG scores correlate at Spearman 0.86 before and after training. With the adapter removed, C1's edges reach only 0.34 accuracy on the original weights (full model: 0.98). Training therefore makes this set sufficient on its own. These tests compare attribution estimates and circuit-level sufficiency; they do not measure the exact change in each edge's individual causal effect.
Activation patterns. We compared the same components' output activations on the same clean inputs before and after training. Relative L2 change is 1.36 vs 0.76 for retained versus other components. The retained components therefore change more, even though their edges were already important in the original model.
Information routing. On IOI, we measured whether the retained heads still attend to the indirect object or the repeated subject. The before-and-after attention scores correlate at Pearson 0.93 (0.86–0.996 across 8 models). On GPT-2 (seed 11), 19 of the 27 C1 heads have roles in Wang et al., including all 3 Name Movers and all 4 S-Inhibition heads, and none changes its preferred target. This shows similar routing on IOI, though not identical computation.
Two more results show where the circuit comes from and where training acts. Median containment of C1's edges in the original model's frozen circuit is 100% (all edges in 17 cells, at least 90% in 26 of 32), and their median rank is in the top 1.0% of the original EAP-IG ranking. Also, 83% of the adapter's update norm lies in the 55% of modules C1 uses. So C1 is built from pathways the original model already relies on, and training makes them sufficient: the original model's own frozen circuit needs a median 6.4× more edges (App. P.1).
Q2.3. What is the ultimate scientific objective?
Our goal is your second option: an adapted model whose target behavior can be verified through a small circuit, with measured links to the original. Under the strict setting, median WikiText-2 perplexity rises by 0.8% across 32 cells; the full adapted model's answer margins correlate at 0.98 with the original across the 13 audited IOI and agreement cells (App. P.1, P.3). Understanding this model is useful in its own right, as in work on weight-sparse transformers (Gao et al., 2025) and sparse-attention models (Draye et al., 2025). We also test what it tells us about the original: which pathways were already important and how training makes them sufficient. The revised abstract and §1 state this goal, with the evidence in Q2.2 above.
Response to Reviewer HeHi (3/4): Q3–Q5.1
Q3. Novelty relative to Draye et al. and Li et al.
We agree that both papers already explore reshaping computation through training. Our contribution is more specific: concentrating one existing behavior into a circuit small enough for exhaustive causal tests, while measuring what changes relative to the original model.
- Draye et al. add a behavior-agnostic sparsity constraint on attention between token positions. We concentrate the inter-component graph of one named behavior. After condensing one behavior, frozen discovery finds circuits for it with 2–16× fewer edges, while the other three stay at a median 1.0× in all twelve off-target directions (§5.5).
- Li et al. grow a subgraph while a model learns a task under a task loss. We start from a model that already performs the behavior and distill it toward its own outputs under an accuracy and capability gate, so the goal is preservation.
- What we add beyond both is evidence about the resulting circuit: exhaustive subset and pair tests (§5.3), and a measurement of what the circuit keeps of the original model (§5.4 and the audit above).
A direct empirical comparison with Draye et al. needs their sparse-attention models; we found no released checkpoints. The closest analogue we can run, joint L0 training of gates and adapter on our graph, returns an admissible circuit in 12 of 32 cells, 5 of them smaller than C1 (App. F.3).
Q4. Generalization to larger models.
We added Llama-3.1-8B and Qwen2.5-7B on IOI, using the same protocol as the smaller models (App. P.2):
| Model | Graph edges | Frozen EAP-IG | C1, seed 11 | C1, seed 22 | C1, strict setting (Q5.3) |
|---|---|---|---|---|---|
| Llama-3.1-8B | 34,881 | 1,407 | 9 | 34 | 56 |
| Qwen2.5-7B | 23,577 | 1,942 | 116 | 112 | 101 |
Sealed-test accuracy is 0.96–0.98 for every C1 entry (EAP-IG: 0.94 and 0.95). The reduction holds at twice the size of our previous largest model, between 17× and 156×. The two Qwen seeds agree closely; on Llama-8B the stopping point varies with the seed (9 vs 34). The causal checks carry over as well: on both models, five random circuits of C1's size are at chance, and the edge-by-edge results for Llama-8B are under Q1(c). (Revised paper: App. P.2.)
Q5.1. Computational cost and practical benefit: when does the approach pay off?
We timed the complete frozen EAP-IG pipeline (ranking, budget sweep, test) on 15 cells and logged wall-clock and peak memory for condensation, one A100 each (App. P.6).
| Frozen EAP-IG | Condensation | |
|---|---|---|
| Runtime, median (range) | 10.8 min (1.7–33) | 2.95 h (0.33–17.5), paper setting, App. G.1 |
| Llama-8B / Qwen-7B runtime | 31 / 20 min | 24.3 / 9.9 h |
| Llama-8B / Qwen-7B peak memory | 47 / 41 GB | 30 / 25 GB |
Whether this pays back depends on the analysis that follows. The runtimes above are measured; the totals below are estimates. They include producing the circuit, assume 200 examples per circuit evaluation, and use the per-cell wall-clock of our strict-setting runs, which are slower (median 6.4 h), so they understate condensation's advantage.
- Attribution only: frozen discovery is cheaper (at the median, a condensation run takes about 16× as long).
- One exact drop-one pass (cost grows with k): condensation is cheaper end to end in 5 of 15 cells, those with large frozen circuits (IOI/Qwen3-4B: 31 h vs 78 h).
- All-pairs test (cost grows with k²): cheaper in 14 of 15 cells, usually by orders of magnitude (Llama-8B: an estimated 46 h vs about 14,000 h). The exception is IOI/Gemma-3-4B, whose frozen circuit already has 8 edges.
- Exhaustive subsets (2^k): feasible only after condensation (§5.3).
So condensation is worth its cost when the goal is exhaustive verification, and not when a ranked edge list is enough. (Revised paper: App. P.6.)
Response to Reviewer HeHi (1/4): W1–Q1
Thank you for distinguishing verifiability from understanding the computation. We address both below. New results use seed 11 unless stated; existing results cite their paper sections.
W1. Does a smaller circuit necessarily imply better interpretability?
Not by itself: edge count measures the size of the causal graph, not the complexity inside a head or an MLP. What a smaller circuit buys directly is verifiability: exhaustive subset tests become possible for the 19 circuits, where every matched frozen circuit is too large to enumerate, and all-pairs interaction tests (run on 48 circuits) would need a median 55× more ablations on matched frozen circuits (§5.3). The evidence that goes beyond edge count is under Q1.
W2. Does the 96.5% answer agreement tell us whether the internal computation is preserved?
Similar outputs alone do not establish the same mechanism. We therefore compared edge attributions, attention patterns, and activations before and after condensation (Q2.2). Because the 96.5% answer agreement saturates near ceiling, we also compared per-example answer margins, logit(answer) − logit(foil), with the original model. C1's answer-margin correlation with the original model is lower than the frozen circuit's (median Pearson 0.44 vs 0.69), although the frozen circuit has 6.4× more edges. The full adapted model scores 0.65 (App. P.1).
Q1. Does condensation improve mechanistic interpretability beyond structural compactness?
We address your three sub-questions separately.
(a) Components and interactions. We checked whether pruning edges also reduces the number of components involved. The median per-cell reduction in writer nodes is 5.0× (median counts: 54 for C1, 198 for the frozen circuit; App. P.1). The remaining components still interact: every circuit has interacting edge pairs, with a median 7.4 partners per edge (§5.3). What changes is that all pairs can now be tested.
(b) Is complexity transferred into the retained components and the adapter? Training concentrates updates in the retained components: their output changes are 1.9× as large as those of other nodes, and the adapter's norm concentrates in the modules the circuit uses (App. P.1). To test whether the reduction requires such large changes, we applied the strict setting in Q5.3. Retained-node changes fall 4.3× (relative L2 0.22 vs 0.96 on the same 13 cells), while circuits remain smaller than EAP-IG (App. P.1, P.3). Restricting adaptation to one component type also preserves small circuits: attention-only and MLP-only adapters give 30 and 20 edges on IOI/Llama-1B, against 18 for the full adapter and 741 for EAP-IG (App. P.7).
(c) A case study of what becomes easier to understand. On the newly added Llama-3.1-8B IOI experiment, frozen EAP-IG needs 1,407 edges. C1 has 9 edges, arranged along this route:
embed → MLP 0 → head 5.11 → MLP 8 → head 15.16 → heads 27.5, 27.20 → logitsHead 15.16 reaches layer 27 both directly and through MLP 19, which is the parallel branch.
From the final position, head 15.16 attends to the repeated subject (attention to S2: 0.42 in the original model, 0.91 after condensation). Heads 27.5 and 27.20 attend to the indirect object (0.20→0.80 and 0.21→0.84) and write to the logits. This matches the S-inhibition → name-mover structure of Wang et al., and every edge in it can be tested on its own. The same heads already prefer the same tokens in the original model. Removing any of the first four edges gives chance (0.52), removing head 27.20's output gives 0.68, and the remaining four edges are parallel branches that are individually redundant (0.94–0.96). Five random 9-edge circuits score 0.52. The three MLPs remain undecomposed, as the paper notes, but they form a far more tractable target for feature-level tools than the 1,407-edge frozen circuit. (Revised paper: App. P.1, P.2.)