- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces a method to identify self-repair components of transformer circuits. Self-repair is a well-known phenomenon (e.g. “the Hydra effect”) in LLMs where ablating one component changes the loss little as another component takes over. Specifically, this backup component does normally not contribute to the loss, but starts contributing when the primary component is removed. The method introduced in this paper (“COAX”) identifies backup components as those whose importance (measured as ablation effect) increases when the primary circuit is ablated.
Critical strengths
Backup components are an important mechanism in LLMs that should be explained by interpretability; this paper helps identify these components
The method is cheap and simple to reproduce.
The evaluation is rigorous; I specifically appreciated the freezing LayerNorm check (section 3.2)
Critical weaknesses
Major: It seems to me that the method can only deal with a particular class of self-repair mechanisms, but fails for some other causal dependencies. Consider for example (1): A circuit A → B → Output where there exist two redundant backup components B’ and B’’. I believe COAX would not identify these, right? (2): The same circuit A → B → Output with a single backup head B’ that still relies on A. The method tests whether ablating B’ on the model sans A and B has an effect, but because B’ relies on A it no longer has any effect. If these examples are correct, then the method seems quite limited to certain cases to me. (If it depends on the choice of S: how would practitioners use how to choose S?)
Major: It is not clear that COAX provides novelty and significant improvement over past work (Wang et al., Rushing & Nanda). The metrics in section 3 and Appendix C don’t show a big advantage for COAX.
Minor: The paper focuses on 2023-era component circuits where components are large parts of the network like attention heads. More recent work in interpretability focuses on circuits made up of neurons, or sparse autoencoder or crosscoder features. It’s not clear whether COAX scales to such circuits with many components.
Minor: Using AUROC as a metric isn’t very sensitive to false positives because the prevalence of backup heads is low (8 out of 141), precision (which is closer to 50%) may be the more honest metric.
Questions for authors
Major: Does COAX scale to circuits made up of thousands of neurons or sparse dictionary features?
Minor: It’s not clear to me why the Fisher measure is preferred over the KL divergence; KL divergence and CE loss are more commonly used in circuit analysis.
7 comments
Response to RNmd (2/4): What CoAx Adds
Part 2. What CoAx Adds (W2)
CoAx finds heads whose effects grow under a specified fault, then tests whether their responses help and whether those heads recover what the circuit misses. On GPT-Neo year comparison, freezing the CoAx set causes a larger accuracy drop than freezing the two-state AtP* set Part 2.1. We separate the gains from comparing both states and measuring finite effects, then test whether the verified heads complete the circuit Part 2.2.
Part 2.1. Repair Gains
Takeaway. In year comparison, the CoAx set supplies stronger compensation than the two-state AtP* set.
In GPT-Neo-1.3B year comparison (adapted from Hanna et al., 2023), the fault alone leaves 78.6% accuracy. With the primary head still removed, freezing CoAx's selection lowers accuracy to 0.8%.
Lower accuracy after freezing means the blocked response helped more. To check whether the heads already mattered, we measure I by ablating the set in the intact model and R by blocking only its response after the fault. Both measure losses in later-minus-non-later year probability mass on the same held-out inputs. The table reports three-run means.
| Selection | Intact task contribution I | Freeze effect R | Accuracy after freezing |
|---|---|---|---|
| CoAx signed | 0.9782 | +0.792 | 0.8% |
| Two-state AtP* | 0.9207 | +0.316 | 29.4% |
| Intact | 0.9771 | -0.285 | 94.1% |
| Conditional | 0.9771 | +0.531 | 12.4% |
| CoAx ratio | −0.0041 | −0.296 | 94.7% |
① Comparison with two-state AtP*. The CoAx set has a higher R in all three paired runs. Mean R is 2.5 times that of AtP*, with ratios of 2.4–2.7 across three bins covering all tested year thresholds Reviewer d7Tf, Part 1.1.
② Similar intact contributions, opposite repair effects. CoAx and Intact have nearly identical I. After the fault, blocking the CoAx response harms year prediction, while blocking the Intact response improves it. Heads that are equally important in the intact model can therefore differ as backups.
Part 2.2. Measurement and Validation
Takeaway. The two-state comparison isolates newly important heads; freezing and completion establish what those heads do.
Wang et al. (2023) identified IOI backups, and Rushing & Nanda (2024) explained how effects such as LayerNorm rescaling contribute to repair. CoAx starts from a specified fault. It measures which components gain influence, then tests whether those components help and account for a missing part of the circuit's response.
① Why subtract the intact effect? Scoring only the faulted model conflates heads that were already important with backups that become important after the fault. We keep the finite ablations and Fisher metric fixed. The only change is subtracting each head's intact effect. In IOI, this raises backup retrieval AUC from .758 to .941. Freezing under the same fault gives R = .027 for the faulted-state ranking and .899 for CoAx (submitted Figure 3(c), revised Figure 4(c)). Subtracting the intact effect changes which heads are selected, and their responses provide more verified repair.
Table 7 of the submission tests a separate question: does CoAx still outperform intact-only ranking when we replace zero ablation with other interventions? Across the four replacement schemes, CoAx achieves .941–.974 retrieval AUC, versus .603–.753 for intact-only ablation ranking.
② Two-state AtP* also compares both states. What does direct measurement add? For the metrics in section 3 and Appendix C, CoAx and two-state AtP* do achieve similar AUC on known IOI backups (.941 vs. .938).
In Pythia-410M induction, we hold the task score fixed. Among downstream heads, the two methods select the same heads where their estimates agree. Where they disagree, direct ablation raises mean single-head freeze loss by .023–.130 across five runs. We see no gain over the full pool. Thus calibration disagreement helps target direct measurement among heads that can respond. See Reviewer pi2G, Part 2.1 for all five pairs.
③ What can the heads verified by freezing explain? In the original IOI experiment, adding the CoAx set tested by freezing reduces the fault-response gap D from .474 to .236. In Pythia-1.4B induction, freezing the CoAx set lowers the task score in all five runs. Adding those same heads to each run's base circuit reduces mean D from .1512 to .0718. Together, the interventions show what the retrieval AUC cannot: the selected response helps the full model and accounts for part of what the base circuit misses.
The historical IOI direct-logit score illustrates the distinction. It ranks the known backups slightly better (.971 AUC), using the IO−S direction, but its top eight give a poorer match to the documented knockout (margin distance 1.33 vs. .99; Appendix C.1.5).
Response to RNmd (3/4): Neurons and Features
Part 3. Neurons and Features (W3 / Q1)
We first verify neuron-level repair in GPT-2 and Pythia Part 3.1, then report scans of over ten thousand candidates and show how cached residuals let us score a sparse dictionary at the final layer Part 3.2.
Part 3.1. Neuron-Level Repair
Takeaway. CoAx identifies beneficial neuron responses in both GPT-2 and Pythia.
For GPT-2 IOI at L10 and natural text at L11, freezing the neurons selected by CoAx causes a larger mean task loss than freezing the neurons selected by two-state AtP*. In Pythia, freezing the CoAx neurons hurts performance in all five runs. We scan 3,072 candidates in each GPT-2 setting and evaluate each setting three times. For Pythia-410M, we scan 4,096 candidates and evaluate five times. The table reports mean freeze effects ± SD. Positive values mean the blocked response helped; compare methods within each task.
| Method | GPT-2 IOI / L10 | GPT-2 natural text / L11 | Pythia-410M natural text |
|---|---|---|---|
| CoAx signed | +0.0872±0.013 | +0.0493±0.011 | +0.0124±0.0048 |
| CoAx ratio | +0.0617±0.0066 | +0.00233±0.0092 | +0.00072±0.0016 |
| Two-state AtP* | +0.0618±0.011 | +0.0166±0.013 | +0.00577±0.008 |
| Conditional | +0.0862±0.0083 | -0.00674±0.032 | +0.012±0.0084 |
| Intact | +0.0588±0.022 | -0.0588±0.023 | +0.0128±0.0081 |
For these two GPT-2 queries, freezing the CoAx sets hurts more than freezing controls matched on layer and intact ablation energy. The full layer comparison in Appendix E.5 shows that the strongest responses occur at different layers for the two tasks; for IOI at L11, the signed set's mean R is negative. In Pythia, signed and Intact have similar means, but CoAx exceeds two-state AtP* and its response remains beneficial with final normalization gain fixed. These controls distinguish helpful neuron responses from score increases alone.
We can also verify a single neuron. On 61 new articles, clamping GPT-2 neuron 2362 in L11 to its intact activation lowers the mean log probability of the correct next token by .0187. Its direct contribution to the correct token rises from .0143 to .0313 after the fault, consistent with the direction of the freezing result.
Part 3.2. Scale and Cost
Takeaway. Exact scans cover over ten thousand neurons and thousands of active SAE features.
For thousands of neurons, we benchmark up to 12,288 candidates. With U candidates, scoring needs one ablation per candidate in each state plus two reference forward passes: 2U+2 states in all. There is no need to enumerate candidate pairs as U grows. All runs below use 96 calibration inputs. Neuron times include scoring, baselines, and validation; SAE times include scoring and validation.
| Model / task / unit | Candidate pool | End-to-end time |
|---|---|---|
| GPT-2 / years / neurons | L9: 3,072 | 51.5 min |
| GPT-2 / years / neurons | L9–10: 6,144 | 110.2 min |
| GPT-2 / years / neurons | L8–11: 12,288 | 152.7 min |
| GPT-2 / final-layer SAE | 24,576 dictionary entries; 6,374–6,622 active | 287.5–294.8 s |
For sparse dictionary features, we score the 6.4–6.6k features active on calibration inputs in a 24,576-entry final-layer SAE. We can compute these scores exactly from cached residuals by recomputing only the output. The same two-state comparison applies to individually ablatable crosscoder features, but an intervention at an intermediate layer still requires running the layers that follow it.
Response to RNmd (1/4): Backup Structures and Fault Set Selection
Thank you for your incisive questions and for recognizing the importance of backup mechanisms, the simplicity and reproducibility of the method, and the rigor of the evaluation. We address backup structures and fault set selection
Part 1, what CoAx adds to prior workPart 2, finer-grained validation and scalePart 3, backup retrieval precisionPart 4, and choice of metricPart 5.Part 1. Backup Structures (W1)
The two counterexamples change different parts of the experiment. If backups are interchangeable, change what you ablate to test a candidate
Part 1.1. If a backup depends on A, choose a primary fault S that leaves the shared input in placePart 1.2. We then give a procedure for choosing SPart 1.3.Part 1.1. Redundant Backups
The single-unit score misses redundant backup components that fully substitute for one another. If B′ and B″ are interchangeable, removing either leaves the computation intact, so its score is zero.
① Ablate pairs to test pairwise redundancy. In a controlled experiment, we ablated all 2,016 pairs among 64 candidates. Only the pair containing both backups showed an increase in effect after the primary fault.
② Sample additional faults when there are more backups. Other backups can hide a candidate's effect. Within each sampled fault context, toggle the candidate while keeping the other heads fixed. This reveals its contribution when all its substitutes are masked. In 200 blind trials, we sampled 64 additional fault contexts per trial, masking each candidate with probability .75. Retrieval AUC rose from .50 to 1.00 for sets of 2, 4, and 8 interchangeable backups.
Part 1.2. Shared Upstream Inputs
If the backup still relies on A, set S = {B} and retain A to explain B's failure. Removing A as well cuts off an input to the backup.
In IOI, B is the primary name-movers and A the upstream S-inhibition heads. We measure the IOI-margin loss from ablating the same eight documented backups in five batches of 512 new inputs.
Removing A cuts the backups' effect by about 80%; replaying A restores it in all five batches. We also check replay across 28 fault and upstream-role combinations.
In a separate test, we removed only B and kept all eight backups in the computation. Freezing their activations at intact values reduced the logit margin by .557. The freeze test establishes that the backups help after B fails. The replay test shows why A belongs outside S for that question: the same backups depend on its input.
Part 1.3. Fault Set Selection
To answer how to choose S, start with the failure you want to explain. If the relevant primary components are documented, remove them. Otherwise, find components whose removal impairs the target behavior on a detection split. In either case, leave shared inputs intact if the question is how the circuit responds to a downstream failure. Then fix S, rank candidates on calibration inputs, and freeze their responses on held-out inputs. Section 2.1 explains what S specifies. Appendix A.5 gives the protocol, and Appendix E.2 shows the split in practice for the induction study.
In IOI, choosing the right primary heads matters more than listing them all. In the original Table 21, taking the top one, two, or three known primary heads by ablation energy gives backup retrieval AUCs of .94, .97, and .94. A separate fault-set control in Appendix C.2 gives an AUC of about .40 for incorrect fault sets matched on behavioral damage, Fisher displacement, and layer position.