- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper asks what a set of ablation experiments actually supports when someone reports a circuit explanation. It separates three claims: finding the groups of components whose removal causes failure, ruling out other such groups under the assumption that removing more components never restores behavior, and predicting the outcome of ablations that were not tested. It proposes a way to report which group sizes have been ruled out, and a fragility score for how badly an explanation could fail on untested ablations. It proves bounds on this score with and without extra assumptions. Experiments cover small transformers, constructed test cases, and the GPT-2 indirect object identification circuit.
Critical strengths
- A clear distinction between confirming an explanation under an assumption and validating it beyond that assumption.
- A correct counterexample showing that a natural shortcut, ignoring connections to untested cases, gives wrong answers.
- A clean result showing that limiting recovery along each removal sequence barely helps, while limiting it in total does.
Critical weaknesses
- The guarantee is never useful at realistic scale. For the paper’s own GPT-2 setting, about 26 heads and about 1,000 tested ablations, the fragility score is at least about 33,000. Combined with the observed recovery rate, the resulting error bound is far above 100%, so it says nothing. The bound becomes meaningful only if the model almost never recovers. The paper’s final advice, to test the explanation directly on fresh random ablations, does not rely on any of its theory.
- The warnings cannot tell good explanations from bad ones. Unless a large share of all possible ablations has been tested, the warning is always very large, whatever the explanation.
- No test on existing work. No published circuit claim is re-examined under its original setup. The GPT-2 experiment differs from the original study in its ablation method and number of prompts, and it only shows that the paper’s core assumption fails.
Questions for authors
- Can you give a realistic example where any guarantee from your framework is meaningful, without assuming the model’s structure is known in advance?
- Which published circuit claim would change if analyzed with your framework?
4 comments
Response to Reviewer Namr (2/2): W3 and Q2
Response to Reviewer Namr (2/2): W3 and Q2
W3. No published circuit is re-examined under its original setup. We now re-examine the IOI circuit of Wang et al. (2023) in Section 5.5, Table 5 and Appendix N.1. We use GPT-2 small, 300 prompts from their released templates, and their knockout, which replaces a head's output by its mean over same-template prompts with three distinct names. Three differences remain: their reference values use 100,000 prompts, our units are whole heads rather than head–position pairs, and the binary label is our own. A knockout is harmful when its per-prompt fractional loss of logit difference, clipped to [0, 1], averages more than 0.25. Our reference values are consistent with theirs: the clean logit difference is 3.66 (theirs 3.56), and knocking out the name movers lowers it by only 0.8% (their "only 5%").
With the seven published head classes as units, the response is exactly monotone at our main threshold: none of the 448 adjacent pairs among the 128 class knockouts is a recovery. One minimal harmful group is the name movers together with the backup name movers, which agrees with the published backup claim. This agreement is only a consistency check, because the backup name movers were identified after the name movers were knocked out. Our versions of the knockouts that Wang et al. report cannot show that this group is minimal, because none of the harmless ones includes the backup name movers. Two additional knockouts show it. The result depends on the threshold (Table 7). At threshold 0.10, eight class knockouts single out an explanation that is still wrong on 3 of the 128 class knockouts, because adding the negative name movers creates recoveries. At threshold 0.50, 4 of the 448 class pairs are recoveries.
With single heads as units, we sampled 512 random adjacent pairs, ablating each head with probability 0.2. In 349 of them the smaller mask is harmful, and 7 of these recover (2.0%, 95% interval 0.8% to 4.1%). Five of the seven add a published negative name mover. When the prompts of each template and name order are split in half, 5 of the 6 recoveries found on each half recur on the other (half to three quarters at our other loss thresholds). Under the flip metric of Section 5.4, which counts prompts whose preferred name changes, none recurs (Table 8).
A head-level discovery run shows what the report adds (Table 9). At ablation probability 0.5, discovery finds 11 harmful groups and its record contains no contradiction, so the explanation looks complete. Yet the report certifies no rank class, because the rank frontier is 1 while the largest group found has 6 heads. A direct check on 256 fresh masks bounds its error by 6.5%. At probability 0.2, the explanation found errs on 11.6% of fresh masks.
Q2. Which published claim would change? No published IOI claim is overturned, but the framework changes how such claims can be stated. First, the backup claim (their Section 3.4) survives at class level at our main threshold once two more knockouts are run, with the caveats above. Second, a head-level list of minimal harmful groups would not be an exact explanation, because the published negative heads create replicated recoveries, so its error must be stated by direct validation. No such list is published for IOI, so this is a caution for future claims. Third, Wang et al. (Section 4.1) report that random and class subsets "suggested" completeness, while a greedy search found an incompleteness of up to 3.09. Our head-level version of their circuit keeps 86.4% of the model's logit difference (theirs 87%). On 256 fresh random subsets, its incompleteness exceeds 20% of the logit difference for 9.8% of subsets (one-sided 95% upper limit 13.4%), and a six-step greedy search reaches 3.59. This variant is less complete than theirs under their class tests and our search is stronger, so the result does not bear directly on their position-specific circuit, which we did not re-test. It shows the form a completeness claim can take: a validated rate over a declared distribution of subsets, plus explicit counterexamples.
I thank the authors for their detailed response. After reading the rebuttal and the other reviews, I have decided to keep my original score.
Response to Reviewer Namr (follow-up)
Thank you for reading our response and the other reviews.
The revision answers each of your three weaknesses. For W1, Corollary 9 (Appendix L.3) states when a guarantee built from past ablations needs at least as many new tests as a direct check, so the paper now presents this vacuous guarantee as a negative result and recommends direct validation. For W2, Section 5.6 shows that a direct check with 128 draws flags explanations with error above 3% with an area under the ROC curve of 0.98, against 0.79 for the warning. For W3, Section 5.5 re-examines the GPT-2 indirect-object-identification circuit of Wang et al. (2023), and its published claim that backup heads take over as a group survives at our main threshold.
If a specific concern remains, we would be glad to address it before the discussion closes.
Response to Reviewer Namr (1/2): W1, Q1 and W2
Response to Reviewer Namr (1/2): W1, Q1 and W2
Thank you for a precise review.
W1. The guarantee is vacuous at realistic scale. We agree, and the revision now presents this as a negative result. For a record of our GPT-2 size (26 heads and 1,017 tested masks) and any monotone explanation consistent with it, Theorem 8 gives a fragility of at least 32,993 under uniform sampling. Our actual GPT-2 record (Section 5.4) contradicts monotonicity, so it supports no record-based guarantee.
We also agree that direct validation does not need our theory. Corollary 9 (Appendix L.3, summarized at the end of Section 4) proves when the record route, a guarantee built from the record, cannot do better. Suppose one wants to certify error at most ε with confidence 1 − δ. The record route estimates the recovery rate η from fresh adjacent pairs (two masks that differ by one head) and accepts when the fragility β times an upper confidence limit on η is at most ε. A direct check on fresh random masks that finds no wrong prediction needs L_dir = ⌈ln(1/δ)/(−ln(1−ε))⌉ masks, which is 99 for 3% error at 95% confidence. Whenever β ≥ 1, the record route needs at least L_dir pairs whatever it observes, and under uniform sampling β ≥ 1 until more than a third of all 2^n masks have been tested. For a consistent record of your example's size, it needs at least 3,294,610 pairs. The corollary is elementary (its direct-check bound is the exact rule of three) but makes the trade-off exact. It also shows that a structural assumption helps only if it bounds η without sampling (Eq. 11) or brings β below 1 (Eq. 12).
Our data agree (Section 5.6 and Appendix N.2, Table 12). With 128 new tests for each of 591 explanations of small transformers with 8 heads, the record route accepts 16, exactly those whose records already contain all 255 masks, while direct validation accepts 279.5 on average over 20 simulated repetitions. Among the 188 explanations with error above 3%, direct validation accepts 0.5 on average, as expected for per-explanation 95% limits, and the record route accepts none.
Q1. Can a guarantee be meaningful without known structure? For the fragility guarantee the answer is no, for the reason given under W1. At realistic scale the report offers three other things: a direct check that assumes nothing about the model's structure and whose sample size does not grow with the number of heads, a rank frontier that is explicitly conditional on monotonicity, and concrete counterexamples. Part 2 shows all three on the GPT-2 indirect-object-identification (IOI) circuit.
W2. Warnings do not separate good from bad explanations. We partly agree. At GPT-2 scale the warning's size is set by the record, as you say, and in the IOI report it exceeds 2×10^5. On the 591 small-transformer explanations (Section 5.6 and Appendix N.2, Table 11), the area under the receiver operating characteristic curve (AUROC) for flagging error above 3% is 0.79 for the warning, 0.85 for exact fragility, 0.98 for a 128-draw direct check, and 0.99 for the true recovery rate alone. The warning has some signal here, where many records cover a large share of the 255 masks, and it keeps that signal when records are split by size into thirds (AUROC 0.79 to 0.87). The recovery rate is what discriminates, but by Corollary 9 measuring it costs at least as many new tests as a direct check whenever the fragility is at least 1. As Section 3.2 states, the warning is not a behavioral guarantee; it shows what the record leaves open and names masks to test next.