acceptodds
Under review as a conference paper at ICLR 2027

Aggregation Order in Activation Patching: Which Circuit Is Selected and What It Preserves

Abstract

Activation-patching evidence supports a mechanistic claim only as far as the measurement pipeline behind it can distinguish that claim from alternatives. Here we study one step of that pipeline: how effects across prompts are aggregated. A circuit is often called faithful when ablating everything else preserves a task score aggregated over prompts, but opposite-signed prompt effects can be combined before or after taking their magnitude. We isolate this aggregation order: within each comparison, the model, candidate universe, prompt set, circuit size and exhaustive search are held fixed, and we measure which circuit each order selects and what that circuit preserves on held-out prompts. Across twelve independently drawn discovery-and-evaluation runs per model from one template bank, in the published indirect-object identification (IOI) universes of GPT-2 small and Pythia-160m, under mean ablation, averaging first selects circuits with 1.73× and 3.65× the full-ablation-normalized held-out KL of those selected by per-prompt-absolute scoring (raw gaps 0.049 and 0.129 nats; the two models' ablation scopes differ). The gap is already present on the discovery prompts. On held-out prompts, most of the excess lies in answer-set probability mass and, on normalized KL, average-first selections were not established to differ from random subsets of the same size. In Pythia, average-first circuits keep the clean model's top token on 75.6% of prompts versus 89.7% for per-prompt-absolute circuits (a prespecified descriptive endpoint), despite similar binary task accuracy. Under counterfactual patching the ratios fall to 1.11× and 1.12×, and the registered operator interaction was confirmed on both models. A preregistered six-family study confirms the mean-ablation contrast, on the relative scale, in only two GPT-2 families, and a preregistered 8B layer-group extension missed its materiality gate. We propose neither a new discovery algorithm nor a universally preferable faithfulness score: changing only the aggregation order can change which circuit is selected and what it preserves, so faithfulness claims should state their aggregation and intervention protocol and evaluate directly the property they are intended to preserve.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, circuit discovery, faithfulness metrics, activation patching, evaluation methodology, preregistration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cdLfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how the faithfulness of a language model circuit is measured, that is, whether it preserves the model's behavior. When combining the results from multiple prompts, the absolute value can be taken either after averaging, allowing errors to cancel each other out, or before averaging. The authors show that changing only this order can lead to selecting a different circuit, one that preserves the model's actual behavior less effectively. The effect is clear in the main setting but weaker across other tasks, methods, and models. They conclude that research on circuits should clearly specify how faithfulness is measured.

Critical strengths

Strengths

  • The experimentation is rigorous and coherent. The authors present the results in the main experiment and then investigate whether the findings extend to other tasks and models. This provides a solid experimental evaluation of the proposed effect.

  • Although the effect is weaker in some of the additional experiments, I think the work provides valuable insights into the task. The experimental evaluation is particularly useful because it offers evidence about both the strengths and limitations of the observed effect, helping to better understand when it occurs and how it may be measured.

  • The paper addresses an interesting methodological issue in circuit discovery. Its findings highlight how seemingly minor choices in the measurement of faithfulness can affect the conclusions drawn from an analysis. This is an important consideration for future work in this area.

Critical weaknesses

Weaknesses

  • The introduction starts somewhat abruptly. A paragraph introducing the task and the main idea would help readers understand the problem before moving into the technical details.

  • The concepts could be introduced more clearly throughout the paper. Many acronyms are used (FIELD, STRICT, MASS, IMP, DISC-KL, CMD, EAP, AtP), along with statistical and preregistration-related terms ("blocks", "materiality gate", "equivalence margin", "delete-one-block jackknife"), which make the paper more difficult to read. The introduction could be improved in this regard.

  • Although the experimental results support the main finding, the effect becomes weaker in some of the additional settings: it is confirmed in only 2 of the 6 families in the scope study, does not exceed the preregistered threshold in the 8B model, and almost disappears with counterfactual patching. These results make it difficult to determine how broadly the observations generalize to real-world circuit discovery, with larger models, other tasks, or standard search methods. Nevertheless, the experiments provide useful insights into the conditions under which the effect is more or less pronounced.

  • As the authors acknowledge, some of the observations in this work have already been reported in previous studies. Although Section 7 summarizes some differences, I think these should be explained more explicitly, particularly earlier in the paper. The specific contribution of this work is not entirely clear until the reader reaches this section. I suggest moving Related Work to Section 2, immediately after the Introduction, and clarifying the incremental contribution of this work relative to previous studies.

Questions for authors

Questions for the Authors

  • Could the authors clarify the specific incremental contribution of this work compared with previous studies, particularly given that some of the observations have already been reported?

  • Given that the contribution of this work is primarily methodological, do the authors plan to release the analysis code in a public repository after acceptance?

1 comment

W4/Q1 — What this paper adds. Thank you for this suggestion. Related Work now follows the Introduction (§2), ending with “What is new here” and Table 2.

Prior studies showed that opposite-signed prompt effects can cancel. We do not claim first discovery of cancellation, sign effects in selection, or circuit instability. We ask what happens when only the order changes: average-first [FIELD] takes the magnitude after averaging signed prompt effects; per-prompt [STRICT] takes magnitudes before averaging. We isolate three decisions: scoring equal-size circuits, ranking edges at matched budgets, and accepting a size with ranking fixed. We measure selection and held-out behaviour.

WorkEarlier contribution
Scheurer et al.Averaged loss can hide a wrong causal-scrubbing hypothesis
AtP* (attribution patching)Both forms and cancellation in indirect-object identification (IOI); quantifying its false-negative contribution left beyond scope
Goldowsky-Dill et al.Difference-of-expectations metrics, including Wang et al. (arXiv v1)'s, can cancel
Miller et al.Mean of ratios versus ratio of means changes assessment of one fixed IOI circuit, without selection
Haklay et al.Token-position aggregation changes edge attribution patching (EAP) rankings; both variants sum absolute scores across examples
This paperChanges order at matched heads, prompts, budgets and interventions; tests selected circuits and held-out behaviour

The revision also tests edge ranking inside the Mechanistic Interpretability Benchmark (MIB) under counterfactual patching (W3). We recommend disclosing the order and checking claimed behaviour (§9), not always choosing one order.

W3 — Where does the effect generalise? We agree with the limits: the six-family test passed in 2/6 settings, Qwen3-8B did not pass its role-mean threshold, and head-set counterfactual gaps were small. The abstract summarises these limits and §8 details them. New tests clarify where the effect appears.

Your questionEvidence and limits
Counterfactual patchingIn MIB's released plain-EAP IOI pipeline, per-prompt ranking improved nine-budget mean KL by 0.0484 nats (GPT-2 small) and 0.589 (Qwen2.5-0.5B); both passed thresholds in 8/8 samples. Qwen top-token agreement gained 17.61 points (8/8); GPT-2 gained 2.29 but exceeded 2 points in 6/8, so did not pass. Head-set gaps were smaller (1.11×/1.12×; we do not rely on the registered equivalence verdict).
Standard searchThe new check runs plain EAP in MIB's released pipeline; whole-graph tests use EAP-IG (EAP with integrated gradients). Checked EAP/EAP-IG defaults rank average-first; MIB calls EAP-IG. Greedy forward/backward search was not tested.
Other tasksOn Pythia-1.4B subject–verb agreement, whole-graph EAP-IG under role-mean ablation (outputs replaced by means matched to token roles) gained 6.73 top-token points with per-prompt ranking, averaged over nine matched requested budgets; 24/24 replicates exceeded 2 points (test fixed in advance; pilot-informed). Whole-graph preservation passed in 7/32 tests, all role-mean; 0/16 counterfactual, none for GPT-2 IOI.
Larger modelsAbove 1.4B, only Qwen3-8B: role-mean KL ratio 1.243 versus required 1.25 (did not pass); counterfactual 1.001. Generality remains open.

Benchmark thresholds were fixed in advance but pilot-informed; pools were fixed, test prompts reused, and some low-budget Qwen average-first circuits became empty after pruning (General response; Appendix G.1). The effect is conditional: larger under mean/role-mean in several settings; under counterfactual patching, head-set gaps shrink, no whole-graph test passed, yet our MIB plain-EAP test found gains. A separate evaluation audit reversed no rankings among four reproduced MIB configurations; we claim no revised leaderboard, and earlier EAP-IG-inputs KL/top-token claims on GPT-2 remain unestablished (§7; §8; Appendices B.1, C, F, G, G.1).

W1/W2 — Clarity. Section 1 now introduces circuit discovery, activation patching and IOI through a Mary/John example, then shows cancellation with two circuits (+2/−2 versus +1/+1) before any formula. The main text uses “average-first” and “per-prompt” throughout and removes or expands other shorthand; the appendices keep some old labels, defined in Appendix A. Section 3 explains thresholds, delete-one intervals and evidence labels.

Q2 — Code. Yes. The anonymous supplement includes analysis code, preregistrations and amendments, hashed results, and nine claim ledgers with verification commands; eight run as shipped, while one needs unshipped unit arrays. An MIT-licensed, tested package implements both aggregation orders, cancellation checks and edge scoring, with a command to rerun selection and compare with the paper. Omitted inputs are identified by hashes. We will release the code and size-omitted raw discovery files publicly on publication.

Reviewer_CRMDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper claims that for circuit identification, activation patching is sensitive to the choice of aggregation method across prompts. In particular, averaging the magnitude of the logit difference and vs the signed difference can make a difference.

Critical strengths

I think the study of this prompt-level aggregation is novel in the literature.

Critical weaknesses

The paper is fairly badly written, as most of the text is AI generated. The introduction doesn't define the basic methodology of activation patching. I believe this won't be accessible to non-expert. The terminology of Average- first scoring (FIELD) and per-prompt-absolute scoring (STRICT) are pretty non-standard and seem confusing first.

In terms of the result, i think the main claim is pretty weak. In finding (i), it claims "average-first circuits have 1.73× and 3.65× [...] than per-prompt-absolute circuits". But it is unclear that the circuits identified by the 2 methods are directionally different. In fact I suspect you should roughly extract the same IOI circuit as the original paper, for the IOI setting.

The overall experimental setting is pretty limited here; only GPT 2 small, Pythia and using standard mean ablation.

Questions for authors

None

2 comments

W1 — Accessibility and terminology. We agree the introduction assumed too much. The rewritten first paragraph now defines the method: “They test a proposed circuit by activation patching: replacing internal activations with reference values, for example an attention head's average activation over reference prompts” (mean ablation). It then explains faithfulness and uses a Mary/John sentence to define the IOI task and logit difference.

The two scoring rules now follow an example before any formula. Circuit A changes the score by +2 on one prompt and −2 on another; circuit B by +1 on each. Average-first [FIELD] prefers A because the errors cancel; per-prompt [STRICT] prefers B because it measures their magnitudes before averaging. The main text uses these plain names thereafter; the appendices keep some old labels, defined in Appendix A. The unclear exposition was our responsibility, and we have addressed it throughout the main text.

W2 — Are the IOI circuits directionally different? Your expectation about familiar IOI heads is reasonable. The cited head-set experiment searches published heads with the rest of the model intact; it cannot establish a different whole-model IOI circuit. In whole-graph EAP-IG (edge attribution patching with integrated gradients) at GPT-2 IOI's 1% requested budget, both orders kept a mean 22.875 of Wang et al. (arXiv v1)'s 26 heads, including every Name Mover, negative Name Mover and S-inhibition head (descriptive).

But the cited head-set selections differ consistently. Average-first kept negative Name Movers 10.7 and 11.10 in 75.0% and 38.1% of selections, versus 95.2% and 69.0% per-prompt. Backup head 9.0 reversed direction: 36.9% versus 1.2% (chance: 50.0%). All three paired intervals exclude zero, with the same directions in 12 fresh replicates (retention exploratory). At equal size, swapping 11.10 for 9.0 passed a test fixed in advance (pilot-informed) in 22/24 fresh prompt sets; 10.7 did not (16/24).

Shared heads need not mean shared edges: same-order edge overlap exceeded cross-order overlap by more than 0.05 in 12/12 pairs in each of four whole-graph settings (GPT-2 IOI mean 0.149; fixed in advance, pilot-informed). The head-overlap contrast was 0.044 (descriptive); pruning and final sizes may contribute.

Nor do edge differences alone prove better behaviour. Under mean ablation at equal head-set sizes, average-first had 1.73× (GPT-2) and 3.65× (Pythia-160m) the normalised held-out KL divergence of per-prompt (tests passed). Original held-out top-token agreement was 92.6% versus 92.7% on GPT-2, but 75.6% versus 89.7% on Pythia (descriptive). Frozen Pythia average-first selections were 7.9 points lower on 20 new templates (test passed). We claim no different IOI algorithm (§4; §5; §6, Table 4; Appendix F, Tables 34–35; Appendices A, A.3, K).

W3 — How broadly does the effect hold? We agree that generality matters. The submission included counterfactual results, a six-family study and a Qwen3-8B extension, but our presentation buried them. The revised abstract summarises this scope and §8 details it.

We also tested a different model family and intervention in the released Mechanistic Interpretability Benchmark (MIB): Qwen2.5-0.5B and GPT-2 small with counterfactual IOI patching. Averaged over nine requested edge budgets, per-prompt ranking of plain-EAP edge scores improved KL by 0.589 nats on Qwen and 0.0484 on GPT-2; each exceeded its threshold in 8/8 random samples of 1,000 attribution prompts. Qwen top-token agreement improved 17.61 points (8/8); GPT-2 improved 2.29 points but cleared its 2-point threshold in 6/8, which did not meet the 8/8 rule. Thresholds were fixed in advance, though pilot-informed. Pools were fixed and public test prompts reused; some low-budget average-first Qwen circuits became empty after pruning (General response; Appendix G.1). Beyond head sets, whole-graph EAP-IG on Pythia-1.4B agreement under role-mean ablation (omitted outputs replaced by reference means matched by token role) gained 6.73 top-token points with per-prompt ranking, averaged over nine matched requested budgets, above 2 in 24/24 replicates (test fixed in advance, pilot-informed; §7).

The results also show where tests did not pass:

SettingResult
Published head sets, mean ablation1.73×/3.65× held-out KL; passed
Same, counterfactual1.11×/1.12× KL: smaller gaps; registered equivalence verdict not relied on
Six automatically built familiesPassed in 2/6; did not pass in 4
Qwen3-8B induction, layer groupsRole-mean 1.243 versus required 1.25: did not pass; counterfactual 1.001 (small-effect clause met)
Whole-graph EAP-IG7/32 preservation tests passed, all role-mean; 0/16 counterfactual; none for GPT-2 IOI

The effect depends on intervention and pipeline; large-model generality is not established (§5, Table 3; §8; Appendices B.1, C, F, G.1).

I acknowledge the response and would like to keep my original rating, as the response also appears as AI slop.

Reviewer_BLZoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies the difference in activation patching circuits caused by different aggregation orders in circuit metric: absoluate value of summation (FIELD) VS. summation of absolute values (STRICT). The experiments show that FIELD-based circuit discovery tends to lead to worse behavior preservation in most of the cases, while often reaching better performance when the circuit is evaluated by itself. The paper also investigates whether their conclusions can carry over to broader setup and reveals that the difference between the two aggregation order is mostly associated with the negative Name Mover heads. The paper further discusses the effect of these two orders on circuit size.

Critical strengths

(1) The paper scrutinizes an experimental setup that prior work has often glossed over, which can be of value for future work.

(2) The paper carried out extensive experiments and thoroughly discusses the implications of the two aggregation orders. The main arguments are well-supported and defensible.

Critical weaknesses

(1) The writing is not understandable. This applies to almost everywhere. The paper needs to be re-written. The story itself is straightforward and easy, but is overcomplicated by the phrasing. It took me too much time to read. The authors admitted to AI-assisted writing. I think AI is definitely overused. Too many terminologies are mentioned without proper definitions.

(2) While the problem itself is overlooked by prior work and thus is worth discussing, it lacks importance or methodological novelty. The results are not surprising as well: [A] Of course, one would expect the two aggregation orders to lead to different performance and circuits. Previous papers are also quite transparent about their choices between these two orders as well. [B] STRICT is a stricter criterion and thus leads to better behavioral preservation in most of the cases. The insight falls below the bar for an ICLR publication but is better suited for a blogpost/workshop, in my view.

Questions for authors

(1) I found the results regarding negative Name Movers (line 340) interesting. Do these heads exhibit distinctive patterns, compared with other heads, that could explain why keeping them active substantially reduces the FIELD–STRICT preservation gap? Either a quantitative characterization of these effects or a mechanistic explanation would strengthen this part.

(2) Can the authors identify a substantive mechanistic claim from prior work that changes when the aggregation order changes?

3 comments

W1 — Writing and clarity. You are right that the writing made a straightforward question needlessly hard to follow. We are sorry it cost you time. That was our responsibility, whatever tools we used, and we rewrote the main text section by section.

The abstract now begins: “Circuit discovery seeks small sets of model components that reproduce a language model's behaviour. Yet averaging errors across prompts can cancel opposite mistakes, making a circuit look faithful despite errors on individual prompts.” Page 1 then defines activation patching, mean ablation and IOI before showing the two-prompt example (+2/−2 versus +1/+1); the Introduction presents the findings in plain, labelled steps.

In the main text, “average-first” [FIELD] and “per-prompt” [STRICT] replace the old labels, shown once at their definitions. Shorthand such as CMD, materiality gate and delete-one-block jackknife is spelled out or removed there; the appendices keep some old notation, defined in Appendix A. Section 6 now leads with the answer to your negative-Name-Mover question.

W2 — Why this matters. We agree that cancellation is expected; we claim no new algorithm or identity. We measure when the choice changes which circuit is found and how faithfully it reproduces the model.

This matters in practice: checked EAP and EAP-IG defaults rank the magnitude of signed prompt means, and MIB uses EAP-IG. Users can get average-first ranking without choosing it (Table 2(b); Appendix J). In MIB's released pipeline run with plain EAP, changing that order changed the returned circuit's behaviour (W2[B]).

The effect extends beyond that benchmark. In whole-graph EAP-IG on Pythia-1.4B's agreement task, under role-mean ablation (outputs replaced by averages matched to token roles), changing only the ranking to per-prompt raised top-token agreement by 6.73 points, averaged over nine matched requested budgets; 24/24 replicates exceeded 2 points (threshold fixed in advance; pilot-informed). Under mean ablation, equal-size average-first head sets had 1.73× (GPT-2 small) and 3.65× (Pythia-160m) the normalised held-out KL of per-prompt sets. On 20 new templates, frozen Pythia average-first circuits matched the full model's top token 7.9 points less often than per-prompt circuits; both passed preregistered tests. The choice also changes how two published IOI circuits rank in faithfulness on average (Q2).

The limits are informative: 7/32 whole-graph preservation tests passed, all under role-mean ablation, and none of 16 counterfactual tests did. We recommend stating the order and intervention, reporting both errors, and checking behaviour on new prompts (§9)—not always using per-prompt.

W2[B] — Why this is empirical. We see the intuition: per-prompt error is the stricter test for any given circuit. It prompted us to test what changes when per-prompt and average-first rank the same edge scores at matched requested budgets; pruning may alter final sizes.

In the released counterfactual IOI pipeline with plain EAP, averaged over nine edge budgets, per-prompt improved KL by 0.0484 nats (GPT-2) and 0.589 nats (Qwen2.5-0.5B), both above fixed margins in 8/8 samples. Top-token agreement rose 17.61 points on Qwen (8/8 above 2 points), but 2.29 on GPT-2 (6/8 above 2; the 8/8 rule was not met). Thresholds were preregistered (pilot-informed); test prompts were reused.

The better-preserving order is not automatic: at GPT-2's 1% budget, average-first did better by 0.0456 nats KL and 10.42 top-token points (descriptive). Under GPT-2 mean ablation, discovery-KL selections had higher held-out per-prompt error than per-prompt selections (+0.082 normalised units) yet lower held-out KL (−0.022 nats; exploratory). Counterfactual head-set gaps were smaller (1.11×/1.12×). The practical issue is that a silent choice in a standard pipeline changes which circuit is returned and how it behaves (General response; Appendix G.1).

W2[A] — Expected differences and transparency. We agree that orders can select different circuits; Wang et al. (arXiv v1), Marks et al. and AtP* state their choices. The question is how much this matters, and when. AtP* discusses cancellation but leaves quantifying its contribution to false negatives “beyond the scope of this work”.

Across 15 selected comparisons, 7 explicitly state the order at an audited stage, 3 describe a prompt mean, 4 leave the reduction unstated (including EAP-IG and MIB), and EAP's paper and checked default code differ, on AtP*'s reading. In a wider pass, 27/57 entries leave it unstated at the audited stage; nine discovery stages use signed prompt means in code without stating so. PAHQ's formula takes magnitudes per prompt, while its default code compares signed means. These are stage-specific checks from a convenience sample, not a prevalence estimate or evidence that published results are wrong. Reporting the order takes one sentence; our experiments show when it matters (§2, Table 2; Appendix J).

Q1 — Negative Name Movers. Thank you for raising this. We quantify how keeping or omitting heads 11.10 and 10.7 changes the error scores and KL, without claiming to explain selection or a neural mechanism.

What the scores show. Average-first [FIELD] takes the magnitude after averaging signed errors across prompts; per-prompt [STRICT] averages their magnitudes. For average-first-selected circuits omitting 11.10, keeping it gives a mean signed error of −0.554 logits; omission shifts that mean by +0.532 logits, nearly cancelling it while individual errors remain. Omission lowers average-first error by 0.347 logits, yet raises per-prompt error by 0.123 logits and KL by 0.074 nats. This accompanies its lower retention: 38.1% for average-first versus 69.0% for per-prompt. For 10.7, the changes are −0.123, +0.206 logits and +0.081 nats; its omission effects varied in sign across prompts. Size-5 average-first circuits nearly balance positive/negative discovery-error contributions (0.329/0.330 logits), unlike per-prompt circuits (0.121/0.272). This score-level analysis is exploratory and post hoc (Appendix K).

Equal-size check. We compared keeping backup head 9.0 instead of each tested head. A pass required average-first error to decrease by more than 0.02 logits while per-prompt error increased by more than 0.05 logits and KL by more than 0.02 nats; thresholds were fixed in advance.

Head (Wang et al. (arXiv v1) class)Isolated ablation raises scoreKept: average-first/per-promptEqual-size exchange
11.10 (negative Name Mover)84.7%38.1% / 69.0%22/24; passed
10.7 (negative Name Mover)89.0%75.0% / 95.2%16/24; did not pass (KL: 24/24)
9.6 (Name Mover)79.3%32.1% / 26.2%0/24; control
9.0 (backup Name Mover)45.7%36.9% / 1.2%—

Limits. Isolated effects are prespecified descriptive; retention is exploratory. Isolated ablation sign is not distinctive: control head 9.6 raises the score on 79.3% of prompts. Comparisons involve unequal sizes and different selected circuits for each head; neither they nor the 9.6 control test between-head differences. The exchange was pilot-informed; 10.7 did not pass, nor did the joint restoration test for 10.7 and 11.10 (14/24; 17 required). McDougall et al. call these copy suppressors, but we did not test that mechanism. This is a context-dependent score pattern, not proof of why either rule selects a head or of a neural mechanism (§6, Table 4; Appendices A.3, I, K). The restriction result you cited (line 340) is now in Appendix A.2.

Q2 — What changes in a published IOI account? We do not show a different IOI algorithm. What changes is which published circuit looks more faithful on average and whether another head's route to the output is visible.

Circuit faithfulness. Wang et al. (arXiv v1) reported comparable faithfulness for full (26-head) and naive (13-head) GPT-2 small IOI circuits. With their released circuits and code on 12 fresh 300-prompt sets, error measures ranked them differently on average:

Naive minus fullMean [95% interval]Reading
Average-first error−0.064 [−0.090, −0.033] logitsFavours naive; descriptive
Per-prompt error+0.289 [0.263, 0.317] logitsFavours full; above preset 0.10 in 12/12, also 12/12 with a second reference-average method

KL (naive 0.981 versus full 0.467 nats) and top-token agreement (33.5% versus 66.1%) also favour full (descriptive). In our test, the full circuit retained 86.5% of mean logit difference, yet absolute error exceeded one logit on 48.6% of prompt draws and two on 15.5%; name preference reversed on 1.8% (post hoc, descriptive). Wang et al. also used interventions to study incompleteness. Our test was fixed in advance after an attempted score reproduction failed its preset check and exploratory results informed its direction; the revised setup check omitted the naive-score condition. We do not reproduce their scores. A preregistered knockout test of the full circuit did not pass its direct aggregation criterion (Appendix H.2).

An additional pathway. Head 9.8 is outside their full circuit. With their head-to-logits intervention but our prompts, reference averaging and cutoffs, its patching effects were +0.2207 logits (ABBA) and −0.1469 (BABA): toward the correct name in one order, against it in the other. Its signed mean was 0.0369 logits versus 0.2367 mean absolute per-prompt. Fixed cutoffs (below 0.10 average-first, above 0.20 per-prompt), plus reliability, order-dependence and a separate functional test, passed in 12/12 fresh sets (development-informed). That test exchanged contextual values at name positions with attention fixed; the adjacent-position control did not match attention mass or perturbation size. This supports an additional conditional route, not better circuit faithfulness, copying versus suppression, or a different IOI mechanism (§7; Appendices D.1–D.3).

AuthorsOpenReview ↗

Aggregation order and circuit fidelity

You are right that for a fixed circuit, average-first [FIELD] error can never exceed per-prompt [STRICT] error. When each order chose its own circuit, KL and MIB's faithfulness measure sometimes favoured different orders. We used the same edge scores and matched the nine requested budgets in MIB's released pipeline. Neither order used KL or agreement on the top token to rank edges in our preregistered, development-informed transfer tests.

Both claims passed on colour copying. Over nine budgets, per-prompt reduced mean KL by 0.544 nats on Qwen2.5-1.5B and 0.311 on 3B. The gain exceeded 0.02 nats in 24/24 draws at each size. MIB measures distance from the full model using faithfulness averaged over prompts. That measure favoured average-first in 16/24 of those same 1.5B draws. At 3B it favoured per-prompt in 19/24.

On IOI with plain EAP, all eight claims passed. With per-prompt, validation KL fell by 0.290 nats at 1.5B and 0.163 at 3B. Agreement on the top token rose by 8.31 to 9.03 points across both sizes and panels. Distance favoured per-prompt in 8/8 draws on each 1.5B panel and on 3B validation. On the 3B public test it favoured average-first in 8/8.

Our answer to Q2 stays the same. The table in our new comment to all reviewers covers colour copying with EAP-IG-inputs and IOI with plain EAP. Appendices G.4 and G.5 give the Qwen details.

Comments on the paper

3 comments

Question and revision. We appreciate the reviews' recognition of this choice and take responsibility for the unclear writing. Can averaging prompt errors change which circuit appears faithful, which is selected, and what it preserves?

Two-prompt example. Circuit A changes a score by +2 on one prompt and −2 on another; B by +1 on each. Lower error wins. Average-first [FIELD] takes the magnitude after averaging, so A's errors cancel to zero and A wins. Per-prompt [STRICT] averages magnitudes, so B wins (1 versus 2). Cancellation is known; we measure what it does to selection and behaviour.

Writing revision. The Introduction defines activation patching and IOI, then gives the example and findings. Related Work moved to §2; §3 defines evidence labels. §§4, 6 and 7 are new (head retention and added tests). Operating-point analysis moved to Appendix E, the restriction test to Appendix A.2. Scope and limits are in §§8 and 10; Appendices D.1–D.3 and F–K are new.

Core result. Both rules used the same published heads, size, prompts and intervention; only the order differed. Under mean ablation, average-first had 1.73× (GPT-2 small) and 3.65× (Pythia-160m) the normalised held-out KL divergence of per-prompt. On 20 new templates, frozen Pythia average-first circuits matched the full model's top token 7.9 points less often than per-prompt. Both passed preregistered tests; the 7.9-point gap reversed in one of five sentence frames. Under counterfactual patching, ratios fell to 1.11× (GPT-2) and 1.12× (Pythia). The registered procedure declared equivalence; given its limited calibration, we do not rely on that verdict (Appendices A.3, E).

Released-pipeline test. We used the released Mechanistic Interpretability Benchmark (MIB) with counterfactual IOI patching, including Qwen2.5-0.5B. Only averaging of plain EAP (gradient-based edge scores) changed. Eight random samples of 1,000 attribution prompts gave these per-prompt-ranking gains, averaged over nine edge budgets (means descriptive):

OutcomeMean gainThresholdSamples above
Qwen IOI KL0.589 nats0.02 nats8/8 — met
Qwen top-token agreement17.61 points2 points8/8 — met
GPT-2 IOI KL0.0484 nats0.02 nats8/8 — met
GPT-2 top-token agreement2.29 points2 points6/8 — not met

Thresholds and the 8/8 rule were fixed in advance (pilot-informed). Holm-adjusted p: 1/64 for each of the first three; 37/256 for the fourth. Both orders ranked the same scores at matched requested budgets; pruning could change final sizes.

Before registration. An earlier Qwen IOI circuit-model-distance criterion was not met; descriptive KL/top-token findings informed the new outcomes and direction. The GPT-2 pool rows and their KL/top-token comparisons had been inspected with EAP-IG-inputs (those claims not established); plain EAP had not run there.

Scope. Each sample drew 1,000 rows uniformly from fixed pools, not MIB's contiguous sets; public test prompts were reused. The tests ask how often a sample clears each threshold, given these pools, implementations and prompts—not new-prompt generalisation or whether the population-mean gain exceeds a threshold. A criterion not met does not imply equivalence. At GPT-2's 1% budget, average-first was better by 0.0456 nats KL and 10.42 top-token points (descriptive); claims concern nine-budget means.

Inside the circuits. An exploratory, post hoc diagnostic found pruning left average-first circuits with no edges (EMPTY) at Qwen's two smallest budgets (8/8 each), at 0.5% (3/8), and at GPT-2's smallest (8/8). Qwen per-prompt circuits were never EMPTY, with positive top-token gains at every budget in every sample. So low-budget gains mix ranking with pruning (Appendix G.1).

Where it holds—and does not. In whole-graph EAP-IG, 7/32 tests met criteria, all under role-mean ablation (averages matched by token role); 0/16 counterfactual tests did. The benchmark shows a counterfactual case where it appears. Effects depend on intervention and pipeline; neither order is always better (Appendix F).

Published circuits. Wang et al. (arXiv v1) reported comparable faithfulness for full (26-head) and naive (13-head) GPT-2 small IOI circuits. On 12 fresh 300-prompt sets using their code, the rules ranked them differently on average: per-prompt favoured full (primary: naive−full 0.289 logits; above preset 0.10 in 12/12 with two reference-average methods); average-first favoured naive (−0.064; descriptive). After our score-reproduction check failed, an exploratory comparison informed the test direction. We do not reproduce their scores or show a different IOI mechanism (Appendix D.2).

Correction and code. We corrected the submitted Appendix A's "external review" attribution to generative-AI review of sealed results; the AI-use statement describes the assistance. Anonymous code and verification materials are in the revised supplement.

AuthorsOpenReview ↗

Two new preregistered results: a new-template panel and the benchmark's recommended attribution method

Thank you all for the thoughtful reviews. Two completed, preregistered follow-ups strengthen the released-pipeline evidence: the same frozen circuits passed three of four tests again on a new-template validation panel, and all four tests passed using the benchmark's recommended attribution method, EAP-IG-inputs, on Qwen2.5-0.5B. Together, these results address practical relevance in a standard counterfactual-patching pipeline, reliance on reused public test prompts, and whether the Qwen result depends on plain EAP.

1. Same circuits, separate validation panel

We compared average-first [FIELD] and per-prompt [STRICT] rankings of the same plain-EAP IOI edge scores using MIB's released construction and evaluator. Circuits selected from eight random samples of 1,000 attribution prompts per model were evaluated without reselection on 1,000 validation prompts from nine templates absent from the training split, the attribution samples and the public test split.

At nine matched requested budgets, with margins of 0.02 nats (KL) and 2 points (top-token agreement), three of four registered claims passed in one Holm family.

ModelValidation-panel endpointMean gainSamples above marginHolm-adjusted pOutcome
Qwen2.5-0.5BKL reduction0.611 nats8/81/64passed
Qwen2.5-0.5BTop-token agreement15.52 points8/81/64passed
GPT-2 smallKL reduction0.0424 nats8/81/64passed
GPT-2 smallTop-token agreement2.12 points5/893/256did not pass

This is a "preregistered replication on a separate, new-template validation panel." On the earlier public test panel, the same three endpoints passed (Qwen KL 0.589 nats; Qwen top-token 17.61 points; GPT-2 KL 0.0484 nats), while GPT-2 top-token agreement (2.29 points, 6/8) did not pass there either. The circuits and their public-test outcomes were known before the validation-panel registration.

2. The benchmark's recommended attribution method

We replaced plain EAP with five-step EAP-IG-inputs on Qwen2.5-0.5B IOI. Only the eight attribution row-index samples were reused: edge scores and circuits were newly computed. We compared the same two aggregation orders using nine matched requested budgets and MIB's released construction and counterfactual evaluator.

Both fixed panels contained 1,000 prompts. All four claims passed, in a separate four-claim Holm family with the same margins and 8/8 rule.

PanelKL reductionKL samplesTop-token gainTop-token samplesHolm-adjusted p
Validation (primary)0.0571 nats8/84.47 points8/81/64 each
Public test0.0629 nats8/86.68 points8/81/64 each

This is a "preregistered, development-informed transfer test." The direction was known from plain EAP on these samples (public test: 0.589 nats, 17.61 points; 8/8 each) and an earlier descriptive EAP-IG-inputs run on other attribution sets (public test means: 0.0791 nats, 8.04 points). Pruning left no circuit empty for either order, so the prespecified analysis excluding budgets with empty circuits matches the table. Descriptively, its validation gains are smaller than plain EAP's; no between-method difference was tested.

Limits. Both tests were one-sided in the previously observed direction. Inference varies over attribution samples, conditional on fixed pools, panels, models and implementations; neither test establishes generalisation to a population of prompts or to attribution methods generally. An earlier benchmark reproduction evaluated other average-first Qwen circuits on the validation split, inspecting aggregate scores only; the public test panel had been used in earlier studies. The EAP-IG-inputs design was frozen before the validation-panel replication outcome was known, although that outcome was read before EAP-IG-inputs analysis. EAP-IG-inputs readouts used a fixed evaluation batch size of 10; conclusions are conditional on it. With plain EAP, pruning emptied some low-budget average-first circuits; this and the descriptive per-budget results are in the appendices. Appendices G.2–G.3 carry the full registered disclosures.

The revised PDF and supplement are uploaded, including code, preregistrations, ledgers and verified analyses. The plain-EAP replication appears in the abstract, page 2, §7 and conclusion; the EAP-IG-inputs test in §7; details are in Appendices G.2–G.3.

Further preregistered studies (larger models, a second task, another model family) are running or queued. We will report completed results regardless of direction and state which remain unfinished.

AuthorsOpenReview ↗

Qwen2.5-1.5B/3B and Pythia-1.4B: every registered claim passed in three further tests

Averaged over nine requested edge budgets, per-prompt [STRICT] ranking gave lower KL than average-first [FIELD] in all three follow-up tests. All registered claims passed under counterfactual patching in the released Mechanistic Interpretability Benchmark (MIB) pipeline. We added two larger sizes (Qwen2.5-1.5B and 3B), a second task (colour copying) and Pythia-1.4B from another family for indirect object identification (IOI).

Both orders rank the same edge scores. With average-first we average signed scores across prompts, then take the magnitude. With per-prompt we reverse those steps. For these preregistered, development-informed transfer tests, earlier results informed our choice of direction, measures and margins. For the two IOI tests we drew on IOI results from Qwen2.5-0.5B and GPT-2. For colour copying we drew only on the Qwen2.5-0.5B colour copying result.

The table averages gains for per-prompt over average-first across draws and nine requested edge budgets (0.1% to 50%). For colour copying, an exact one-sided binomial test asked whether KL gain exceeded 0.02 nats in more than half the draws, with Holm correction over both claims. The gain exceeded it in 24/24 draws at each size. For each IOI claim, the mean over nine budgets had to exceed its margin (0.02 nats or 2 percentage points) in 8/8 draws and the one-sided test had to pass Holm correction.

Task, method and modelKL reduction (nats)Gain in agreement on the clean top token (points)Registered claims
Colour copying, EAP-IG-inputs, Qwen2.5 (1.5B / 3B)0.544 / 0.3116.15 / 10.83 (descriptive)Both KL claims passed
IOI, plain EAP, Qwen2.5 (validation / public test)1.5B: 0.290 / 0.243; 3B: 0.163 / 0.09881.5B: 8.84 / 8.68; 3B: 9.03 / 8.31All eight passed
IOI, plain EAP, Pythia-1.4B (validation / public test)0.399 / 0.43617.17 / 16.63All four passed

On colour copying, average-first had lower KL at some budgets, by 1.263 nats at 10% on 1.5B. On Qwen IOI, average-first did better at 0.2% for both sizes and at 0.5% for 3B. Pruning left both orders with zero edges in some cases at the smallest budgets, with average-first empty in 13 of 72 draw and budget pairs and per-prompt in 16 at 1.5B, and each empty in 8 at 3B.

On Pythia, 87% of the KL gain came from the three smallest budgets. From smallest up, pruning left average-first with zero edges in 8, 7 and 5 of 8 draws. Before we froze the design, a compatibility check on development rows showed that pruning left average-first circuits empty at those three budgets while per-prompt circuits kept edges. We had already written the pass criteria, and before the freeze we computed no contrast between the orders (Appendix G.6). All Pythia per-prompt circuits kept edges. KL favoured per-prompt at every budget, but agreement was higher for average-first at 10% on validation (by 0.20 points) and at 5%, 10% and 50% on the public test (by 0.19, 1.26 and 0.64 points). In a descriptive analysis restricted to budgets where both orders kept edges, mean gains were 0.0844 / 0.0936 nats and 6.57 / 6.34 points (validation / public test), and median KL gains were 0.0322 / 0.0392 nats. Under that restriction, KL exceeded its margin in 7/8 draws on each panel and agreement exceeded its margin in 8/8. We did not match circuits on edges kept. Inference is over attribution draws with fixed prompt pools and panels.

Earlier, GPT-2 agreement on the top token did not pass on either panel, and gaps were smaller under counterfactual patching of the published head sets than under mean ablation. The original Qwen3-8B induction test did not pass, and we did not run the study added by a dated amendment (EAP-IG-inputs on IOI at Qwen2.5-1.5B and 3B). We report every completed follow-up study and have uploaded the revised PDF and supplement. The abstract and Section 7 report the results, with full registered disclosures for the three tests in Appendices G.4 to G.6 (Tables 42 to 44), and generality beyond these tests remains open.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.