acceptodds
Under review as a conference paper at ICLR 2027

Finding Is Not Certifying: Evidence Reports for Redundant Mechanisms

Abstract

Finding a model's minimal harmful ablation groups, ruling out additional groups under an assumption, and predicting untested interventions are different tasks. We develop evidence reports that keep these claims separate. A conditional rank frontier states which orders of additional monotone interactions have been excluded by the observations, without claiming to infer the model's true interaction order. Sharp bounds for restricted response classes show that a small record can identify an explanation under monotonicity while remaining fragile even to responses with one minimal harmful set and one recovery per intervention chain; a bound on the total number of descents changes the scaling. Fragility warnings describe compatible alternatives, not actual prediction error. We identify observed-only warnings with elementary proof completions and treat neighborhood optimization as optional rather than a default validation method. A 36-seed tiny-transformer study evaluates the actual finite-candidate routine against cheap matched controls, while overlapping-reference tests separate probability-counting, query, and optimization costs. A pretrained-model reanalysis distinguishes prompt uncertainty from mask-pair uncertainty. The resulting recommendation is to report conditional coverage, retain concrete contradiction witnesses, and use independent behavioral validation for acceptance.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, ablation studies, redundant mechanisms, certification, behavioral validation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_NamrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper asks what a set of ablation experiments actually supports when someone reports a circuit explanation. It separates three claims: finding the groups of components whose removal causes failure, ruling out other such groups under the assumption that removing more components never restores behavior, and predicting the outcome of ablations that were not tested. It proposes a way to report which group sizes have been ruled out, and a fragility score for how badly an explanation could fail on untested ablations. It proves bounds on this score with and without extra assumptions. Experiments cover small transformers, constructed test cases, and the GPT-2 indirect object identification circuit.

Critical strengths
  • A clear distinction between confirming an explanation under an assumption and validating it beyond that assumption.
  • A correct counterexample showing that a natural shortcut, ignoring connections to untested cases, gives wrong answers.
  • A clean result showing that limiting recovery along each removal sequence barely helps, while limiting it in total does.
Critical weaknesses
  • The guarantee is never useful at realistic scale. For the paper’s own GPT-2 setting, about 26 heads and about 1,000 tested ablations, the fragility score is at least about 33,000. Combined with the observed recovery rate, the resulting error bound is far above 100%, so it says nothing. The bound becomes meaningful only if the model almost never recovers. The paper’s final advice, to test the explanation directly on fresh random ablations, does not rely on any of its theory.
  • The warnings cannot tell good explanations from bad ones. Unless a large share of all possible ablations has been tested, the warning is always very large, whatever the explanation.
  • No test on existing work. No published circuit claim is re-examined under its original setup. The GPT-2 experiment differs from the original study in its ablation method and number of prompts, and it only shows that the paper’s core assumption fails.
Questions for authors
  1. Can you give a realistic example where any guarantee from your framework is meaningful, without assuming the model’s structure is known in advance?
  2. Which published circuit claim would change if analyzed with your framework?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Namr (1/2): W1, Q1 and W2

Response to Reviewer Namr (1/2): W1, Q1 and W2

Thank you for a precise review.

W1. The guarantee is vacuous at realistic scale. We agree, and the revision now presents this as a negative result. For a record of our GPT-2 size (26 heads and 1,017 tested masks) and any monotone explanation consistent with it, Theorem 8 gives a fragility of at least 32,993 under uniform sampling. Our actual GPT-2 record (Section 5.4) contradicts monotonicity, so it supports no record-based guarantee.

We also agree that direct validation does not need our theory. Corollary 9 (Appendix L.3, summarized at the end of Section 4) proves when the record route, a guarantee built from the record, cannot do better. Suppose one wants to certify error at most ε with confidence 1 − δ. The record route estimates the recovery rate η from fresh adjacent pairs (two masks that differ by one head) and accepts when the fragility β times an upper confidence limit on η is at most ε. A direct check on fresh random masks that finds no wrong prediction needs L_dir = ⌈ln(1/δ)/(−ln(1−ε))⌉ masks, which is 99 for 3% error at 95% confidence. Whenever β ≥ 1, the record route needs at least L_dir pairs whatever it observes, and under uniform sampling β ≥ 1 until more than a third of all 2^n masks have been tested. For a consistent record of your example's size, it needs at least 3,294,610 pairs. The corollary is elementary (its direct-check bound is the exact rule of three) but makes the trade-off exact. It also shows that a structural assumption helps only if it bounds η without sampling (Eq. 11) or brings β below 1 (Eq. 12).

Our data agree (Section 5.6 and Appendix N.2, Table 12). With 128 new tests for each of 591 explanations of small transformers with 8 heads, the record route accepts 16, exactly those whose records already contain all 255 masks, while direct validation accepts 279.5 on average over 20 simulated repetitions. Among the 188 explanations with error above 3%, direct validation accepts 0.5 on average, as expected for per-explanation 95% limits, and the record route accepts none.

Q1. Can a guarantee be meaningful without known structure? For the fragility guarantee the answer is no, for the reason given under W1. At realistic scale the report offers three other things: a direct check that assumes nothing about the model's structure and whose sample size does not grow with the number of heads, a rank frontier that is explicitly conditional on monotonicity, and concrete counterexamples. Part 2 shows all three on the GPT-2 indirect-object-identification (IOI) circuit.

W2. Warnings do not separate good from bad explanations. We partly agree. At GPT-2 scale the warning's size is set by the record, as you say, and in the IOI report it exceeds 2×10^5. On the 591 small-transformer explanations (Section 5.6 and Appendix N.2, Table 11), the area under the receiver operating characteristic curve (AUROC) for flagging error above 3% is 0.79 for the warning, 0.85 for exact fragility, 0.98 for a 128-draw direct check, and 0.99 for the true recovery rate alone. The warning has some signal here, where many records cover a large share of the 255 masks, and it keeps that signal when records are split by size into thirds (AUROC 0.79 to 0.87). The recovery rate is what discriminates, but by Corollary 9 measuring it costs at least as many new tests as a direct check whenever the fragility is at least 1. As Section 3.2 states, the warning is not a behavioral guarantee; it shows what the record leaves open and names masks to test next.

AuthorsOpenReview ↗

Response to Reviewer Namr (2/2): W3 and Q2

Response to Reviewer Namr (2/2): W3 and Q2

W3. No published circuit is re-examined under its original setup. We now re-examine the IOI circuit of Wang et al. (2023) in Section 5.5, Table 5 and Appendix N.1. We use GPT-2 small, 300 prompts from their released templates, and their knockout, which replaces a head's output by its mean over same-template prompts with three distinct names. Three differences remain: their reference values use 100,000 prompts, our units are whole heads rather than head–position pairs, and the binary label is our own. A knockout is harmful when its per-prompt fractional loss of logit difference, clipped to [0, 1], averages more than 0.25. Our reference values are consistent with theirs: the clean logit difference is 3.66 (theirs 3.56), and knocking out the name movers lowers it by only 0.8% (their "only 5%").

With the seven published head classes as units, the response is exactly monotone at our main threshold: none of the 448 adjacent pairs among the 128 class knockouts is a recovery. One minimal harmful group is the name movers together with the backup name movers, which agrees with the published backup claim. This agreement is only a consistency check, because the backup name movers were identified after the name movers were knocked out. Our versions of the knockouts that Wang et al. report cannot show that this group is minimal, because none of the harmless ones includes the backup name movers. Two additional knockouts show it. The result depends on the threshold (Table 7). At threshold 0.10, eight class knockouts single out an explanation that is still wrong on 3 of the 128 class knockouts, because adding the negative name movers creates recoveries. At threshold 0.50, 4 of the 448 class pairs are recoveries.

With single heads as units, we sampled 512 random adjacent pairs, ablating each head with probability 0.2. In 349 of them the smaller mask is harmful, and 7 of these recover (2.0%, 95% interval 0.8% to 4.1%). Five of the seven add a published negative name mover. When the prompts of each template and name order are split in half, 5 of the 6 recoveries found on each half recur on the other (half to three quarters at our other loss thresholds). Under the flip metric of Section 5.4, which counts prompts whose preferred name changes, none recurs (Table 8).

A head-level discovery run shows what the report adds (Table 9). At ablation probability 0.5, discovery finds 11 harmful groups and its record contains no contradiction, so the explanation looks complete. Yet the report certifies no rank class, because the rank frontier is 1 while the largest group found has 6 heads. A direct check on 256 fresh masks bounds its error by 6.5%. At probability 0.2, the explanation found errs on 11.6% of fresh masks.

Q2. Which published claim would change? No published IOI claim is overturned, but the framework changes how such claims can be stated. First, the backup claim (their Section 3.4) survives at class level at our main threshold once two more knockouts are run, with the caveats above. Second, a head-level list of minimal harmful groups would not be an exact explanation, because the published negative heads create replicated recoveries, so its error must be stated by direct validation. No such list is published for IOI, so this is a caution for future claims. Third, Wang et al. (Section 4.1) report that random and class subsets "suggested" completeness, while a greedy search found an incompleteness of up to 3.09. Our head-level version of their circuit keeps 86.4% of the model's logit difference (theirs 87%). On 256 fresh random subsets, its incompleteness exceeds 20% of the logit difference for 9.8% of subsets (one-sided 95% upper limit 13.4%), and a six-step greedy search reaches 3.59. This variant is less complete than theirs under their class tests and our search is stronger, so the result does not bear directly on their position-specific circuit, which we did not re-test. It shows the form a completeness claim can take: a validated rate over a declared distribution of subsets, plus explicit counterexamples.

I thank the authors for their detailed response. After reading the rebuttal and the other reviews, I have decided to keep my original score.

AuthorsOpenReview ↗

Response to Reviewer Namr (follow-up)

Thank you for reading our response and the other reviews.

The revision answers each of your three weaknesses. For W1, Corollary 9 (Appendix L.3) states when a guarantee built from past ablations needs at least as many new tests as a direct check, so the paper now presents this vacuous guarantee as a negative result and recommends direct validation. For W2, Section 5.6 shows that a direct check with 128 draws flags explanations with error above 3% with an area under the ROC curve of 0.98, against 0.79 for the warning. For W3, Section 5.5 re-examines the GPT-2 indirect-object-identification circuit of Wang et al. (2023), and its published claim that backup heads take over as a group survives at our main threshold.

If a specific concern remains, we would be glad to address it before the discussion closes.

Reviewer_Z6gVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

For a non-expert, the abstract is meaningless.

The rest of the paper is not much better. For example, what is a 'pathway', a word that is used exactly once in the paper's 27 pages. Also, what is an 'attainer', again, a word that is used once in the paper's 27 pages. Another example is 'rank frontier', and so on. The way the paper is written, it seems apparent that many more similar examples could be found.

The paper is extremely hard to read for a non-expert. I am unsure this would have to be the case. The writing is "mechanical", with no effort made to make it appealing to a human reader who is a non-expert. Overall, I am not convinced such works merit publication.

Critical strengths

I could not find any.

Critical weaknesses

The paper is extremely hard to relate with, namely for a non-expert.

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Response to Reviewer Z6gV

Response to Reviewer Z6gV

Thank you for naming specific terms. You are right that the submission used terms before defining them. The revised PDF fixes this in the places listed below.

The abstract is rewritten in plain language and now begins: "Circuit explanations are often supported by ablations: remove a group of components, such as attention heads, and check whether the model's behavior changes."

The terms you flagged. "Pathway" now reads "two heads that carry the same information" (Section 1), and "attainer" is replaced by "the worst-case response" (Sections 3 and 4). "Rank frontier" is defined in words in the abstract and in Section 1. It is the largest group size up to which the experiments rule out any other group whose removal breaks the behavior, assuming that removing more components never restores it.

All other core terms are defined, with their synonyms, in one Terminology paragraph early in Section 1, and Sections 2 and 3 explain each main symbol in words where it first appears.

Section 1 now includes a worked example. Heads 1 and 2 are redundant copies, and head 3 works against the correct answer. Removing head 1 or head 2 alone leaves the behavior intact, removing both breaks it, and removing all three can restore it, because the opposing head is also gone. Testing head 1, head 2 and both shows that the pair is a harmful group. It does not show whether other harmful groups exist, or whether the rule "the behavior fails exactly when heads 1 and 2 are both removed" predicts untested ablations such as removing all three.

The structure is easier to follow: Section 1 ends with a contribution list that points to sections, and Sections 2, 3 and 5 open by saying what they show.

Why the results matter. Claims that some heads are backups, or that a circuit is complete, usually rest on removal experiments. We show which of these claims such evidence supports. Wang et al. (2023) identified the heads in GPT-2 that complete sentences with the correct name and sorted them into classes. At our main cut-off for a broken behavior, removing whole classes gives the results that their account of backup heads predicts. This agreement is only a consistency check, because the backup heads were themselves found by removing other heads. When heads are removed individually rather than by class, however, removing one more head can restore a behavior that an earlier removal had broken. In 5 of the 7 such reversals we observed, the original authors had identified the added head as pushing against the correct answer. A list of harmful head groups therefore does not by itself predict untested ablations.

We would value your reading of the revised abstract and first two pages.

Reviewer_ZcBFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This submission seems incomplete (no clear and meaningful experimental results are given). At least the authors should rebalance the appendix and main body, e.g., moving important expr results to main body.

Critical strengths

N/A

Critical weaknesses

No clear and meaningful experimental results are given.

Questions for authors

N/A

1 comment
AuthorsOpenReview ↗

Response to Reviewer ZcBF

Response to Reviewer ZcBF

Thank you for the concrete suggestion to move the important experimental results from the appendix into the main text. We agree that the submitted main text read as incomplete: it ended a quarter of the way down page 8 without a conclusion, and its only result on a pretrained model was a 32-prompt measurement. The revised PDF uses all ten pages, moves results into the main text, and closes with a new Section 6 on practical use and limits.

The main text now reports three new results. Here an explanation lists groups of attention heads whose removal breaks a behavior, which we call harmful.

  • Section 5.5 and Table 5 test the circuit that Wang et al. (2023) found in GPT-2 for completing a sentence with the right name (indirect-object identification), using their method of removing heads and 300 prompts. At our main cut-off for a harmful removal, removing more of the seven head classes they define never restores the behavior, and the smallest harmful groups agree with their claim that some heads act as backups. This agreement is only a consistency check, because the backup heads were found by knocking out the heads they back up. When random sets of single heads are removed, 2.0% of harmful removals become harmless again when one more head is removed, mostly through heads that Wang et al. identified as working against the answer. A search for harmful groups that looks complete still errs, and a direct check on 256 fresh random ablations bounds its error by 6.5%.
  • Section 5.6 shows, on 591 explanations of 72 small transformers, that a direct check on fresh ablations flags explanations with error above 3% far better than our warnings about untested ablations. Its area under the receiver operating characteristic curve is 0.98, against 0.79 for the warning.
  • In Section 6, a direct check with 128 new tests accepts 275 of these explanations and none of the 188 whose error exceeds 3%.

We also moved or clarified several results. Section 5 opens with a results-at-a-glance list, and Section 5.1 describes the small-transformer setup and every method in Table 2. Figure 1 has moved from the appendix into Section 4, and the captions of Figure 1 and Tables 2, 3 and 5 now say what each one shows.

Some results were already in the submission. In Table 2, a cheap rule for choosing extra tests catches 57.9% of wrong explanations within eight tests and a costlier optimizer only 54.2%, so we recommend the cheap rule.

Appendix N gives full protocols and tables for every new experiment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

General response

We thank the reviewers. In the revised PDF, the main text uses all 10 pages, and all original numbering is unchanged.

  1. A new Corollary 9 (Appendix L.3, summarized at the end of Section 4) concerns uniformly random ablations. Unless more than a third of all ablations were already tested, a guarantee built from past ablations needs at least as many new tests as a direct check on fresh ones that finds no wrong prediction. We therefore recommend direct checks for accepting an explanation.
  2. A new Section 5.5 with Table 5 re-examines the circuit that Wang et al. (2023) found in GPT-2 small for indirect-object identification, using their own ablation method.
  3. A new Section 5.6 tests whether our warnings about untested ablations separate good explanations from bad ones.
  4. We rewrote the abstract, Section 1 and the openings of Sections 2, 3 and 5 for readers outside the subfield, moved Figure 1 into the main text, and added Section 6 on practical use and limits.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.