acceptodds
Under review as a conference paper at ICLR 2027

Bucketing the Good Apples: A Method for Diagnosing and Improving Causal Abstraction

Abstract

We present a method for diagnosing interpretation in neural networks by identifying an input subspace where a proposed interpretation is highly faithful. Our method is particularly useful for causal-abstraction-style interpretability, where a high-level causal hypothesis is evaluated by interchange interventions. Rather than treating interchange intervention accuracy as a single global summary, we refine this framework by partitioning the input space into well-interpreted and under-interpreted regions according to pairwise interchange-intervention behavior. This turns causal abstraction from a purely global evaluation into a more diagnostic tool: it not only measures whether an interpretation works, but also reveals where it works, where it fails, and what distinguishes the two cases. This diagnostic view also provides practical heuristics for improving interpretations. By analyzing the structure of the well-interpreted and under-interpreted regions, we can identify missing distinctions in a high-level hypothesis, discover previously unmodeled intermediate variables, and combine complementary partial interpretations into a stronger one. We instantiate this idea as a simple four-step recipe and show that it yields informative error analyses across multiple causal abstraction settings. In a logic inference task, recursively applying the recipe recovers a high-level hypothesis from scratch. More broadly, our results suggest that partitioning the input space is a useful step toward more precise, constructive, and scalable mechanistic interpretability.

Primary area:
interpretability and explainable AI
Keywords:
causal abstraction, mechanistic interpretability, interchange intervention, causal models, hypothesis testing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_REiiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a methodology that serves as a diagnostic tool to identify regions (or buckets) of the input space where a causal interpretation is faithful, rather than relying on global causal interpretations. The authors propose using this methodology to analyze high-level hypotheses and identify previously unmodeled intermediate variables. The proposed pipeline consists of four steps: (1) specify the task and input space, (2) find a candidate alignment, (3) partition the input space based on interchangeability, and (4) generalize and characterize the resulting partition.

Critical strengths
  1. The paper addresses an important problem in global interpretability: an interpretation may not generalize across the entire input space.
  2. The authors evaluate their methodology on three different tasks: logical inference, entity binding, and entangled factual recall.
  3. The authors clearly acknowledge and discuss the two main limitations of their approach.
Critical weaknesses
  1. Contribution: The authors claim that their methodology can identify previously unmodeled intermediate variables, but it is unclear how this is achieved in practice. As I understand it, DAS finds candidate variables for interventions. Why is partial alignment sufficient, rather than full alignment? What is the motivation for this criterion, and how is the alignment threshold determined? How does the methodology identify previously unmodeled intermediate variables? The authors also state that DAS “can tell us that the hypothesis is imperfect, but not how to improve it.” However, it is unclear how the proposed methodology can support hypothesis refinement. Could the authors clarify how this refinement is performed in practice?

  2. Evaluation Metrics: Interchangeability appears to be an important metric in this paper, but its motivation and role in the evaluation require further explanation. How in-bucket and cross-bucket IIA are measured and interpreted? The authors should clarify how these metrics are computed and explain why they are important for evaluating the proposed methodology.

  3. Explanation of the Methodology and Results: Step 3 of the methodology requires further clarification. In Step 4, it is unclear why two classifiers are trained and what the purpose of each is. The authors should explain the motivation behind this design. I don’t understand what you mean here: “That internal features outperform hand-labeled features suggests the distinction in different buckets is fundamentally encoded within the model’s own representations.” I also do not understand how the proposed methodology was used to identify the reasons for alignment failure in the Entangled Factual Recall task and what evidence supports the conclusions.

Questions for authors

I have included some questions above.

Reviewer_YnkGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose a bucketing approach to identify faithful causal abstractions. While prior work uses a summative single score (IIA), the authors present a 4-step process to splitting the various inputs that ensure "Interchange-Consistentency" ("bucketing the input space"). They apply their process on three different examples and conclude that this improves faithfulness of the identified causal abstractions.

Critical strengths

I like the idea of moving away from a single IIA score in order to identify more clearly the causal abstractions one should be sure about, and which ones are less plausible. The article reads well and features helpful figures and aid the understanding of the proposed approach.

Critical weaknesses

I feel that the proposed diagnosis is not a valid test of local causal faithfulness. High IIA inside a bucket might also happen because the intervention changes nothing, as in the RAVEL language buckets; therefore it's a wrong diagnostic for faithfulness. Specifically, in RAVEL, the English bucket has very high within-bucket IIA because swapping LANGUAGE between two English cities leaves LANGUAGE unchanged.

The search also directly looks for dense groups of successful interventions, so finding a "good" group is partly built into the method. The leftover inputs are then treated as under-interpreted even though they may form another valid group. This is especially problematic because (Geiger et al. 2021) already used intervention-success graphs and cliques, and explicitly required at least one impactful edge to avoid trivial cases.

Novelty remains unclear to me. As the authors acknowledge, Pıslar et al. 2025 already studied different causal models applying to different parts of the input space and the tradeoff between coverage and IIA. The authors claim that they add an explicit procedure, but I wonder how much of this procedure is actually already set out in the experimental section of this prior work.

(Chan et al. 2022) -- see https://www.alignmentforum.org/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-for-rigorously-testing -- already tests mechanistic hypotheses by swapping activations that the hypothesis says should be equivalent, and uses failures to refine the hypothesis. Given this prior work, it is not clear what new causal idea this paper adds.

I also felt that the testing was somewhat selective. In the logic task, known variables are used to explain the buckets after they are found, so the method does not clearly discover them. In some experiments, graph nodes are used to build the buckets first and then split into classifier train/test sets, so the "test" examples were already used during discovery. The paper also does not show that the same diagnosis appears on a fresh graph sample, a new alignment run, different prompts, or another model.

Last but not least, the method searches for dense groups by design, but does not show how often similar groups appear after shuffling intervention outcomes or using irrelevant variables. Without that, it is hard to know whether the discovered structure is meaningful or just what the search algorithm tends to produce.

Questions for authors
  • Can the authors clarify the differences to the work by Pıslar et al. 2025 in terms of procedure? This prior work also includes experimental results, which somewhat implies a procedure. I could not directly discern the material difference.
  • Can the authors review Chan et al. 2022 and comment on the principal differences to their approach (and procedure)
  • Can the authors clarify how often similar groups appear (see last comment under weaknesses)
5 comments

Thank you for the careful review. Before answering Q1–Q3, we would like to clarify what the paper claims.

What the paper contributes.

The paper does not propose a new faithfulness score, and it does not claim that a high-IIA bucket certifies a faithful abstraction. Its starting point is that IIA alone cannot tell us whether a causal abstraction is faithful: an intermediate score says neither where an alignment holds nor why it fails. Given a model, a hypothesis and one alignment, our method diagnoses that alignment. It locates the inputs on which interchange is consistent, characterizes what separates them from the rest, extends that boundary to unseen inputs, and reads it as a heuristic for improving the hypothesis. Prior work stops at a number: global IIA, the size of the largest clique (Geiger et al., 2021), coverage at a faithfulness level (Pîslar et al., 2025), or performance recovered (Chan et al., 2022). We turn that number into an explanation.

Q1 (Pîslar et al., 2025).

Both works build a graph from pairwise interchange outcomes, but they solve different problems. Pîslar et al. start from a pre-specified set of candidate causal models and ask how to assign inputs among them so that the combined model is more faithful. Their procedure enumerates the whole input space (1,000 and 64 inputs), builds one graph per candidate, assigns inputs greedily under a faithfulness threshold, and reports how many inputs the candidates cover. It needs the alternative hypotheses in advance, and the resulting input cells serve to measure coverage. We start from a single alignment with no alternatives, and the partition itself is our object of study. Step 4 characterizes its boundary with labelled or SAE features, generalizes it to unseen inputs without further interventions, and uses it to derive what the hypothesis lacks; in the logic task this recovers o4, then o1 and o2, from the output variable alone. We also move from fine-tuned GPT-2 small with DAS to pretrained Gemma-2-2B and Llama-3.1-8B with full-vector patching and MDAS. The two approaches are complementary: our diagnosis can supply the candidate models that their combination requires.

Q2 (Chan et al., 2022).

Causal scrubbing tests a complete hypothesis by resampling every activation the hypothesis treats as equivalent and measuring how much performance survives. Its output is one aggregate number. It gives no verdict per input, so it cannot say where a hypothesis fails or which variable is missing; in the authors' case studies the hypothesis is revised by hand. It also runs only swaps that are predicted to leave behavior unchanged, whereas interchange interventions also test counterfactuals that must change the output. Our unit of analysis is the individual pair of inputs, and the structure of pairwise outcomes over the input space is what yields the diagnosis.

Q3 (shuffled outcomes, irrelevant variables).

We did not run these controls because the diagnosis is defined only relative to an alignment. Our method does not test whether structure exists; that is the role of alignment search and IIA in Step 2. It takes an alignment that is established but imperfect, and explains where it holds and where it fails. Shuffled outcomes no longer correspond to any alignment, and a causally irrelevant variable does not yield one: if alignment search stays at baseline, Step 2 stops, and there is nothing to diagnose. An analogy: a doctor cannot localize an illness in someone who is not ill; given a patient, the doctor can say what is healthy, what is not, and what treatment would help. Our method plays that role for a causal abstraction with an intermediate IIA. Within that scope, the buckets are not an artefact of the search: they align with variables the search never saw (Table 1), they are stable across density thresholds and seed budgets (Appendix A.1), and PCA + K-means on the same activations fails to recover them in two of four passes (Table 2). RAVEL makes the same point from the other side: where an alignment's signal is hollow, the diagnosis says so (see below).

Additional clarification: RAVEL.

RAVEL is in the paper as a case where the alignment fails while IIA still shows a non-trivial signal (13.9% held out). The diagnosis explains that signal: the interchange-consistent regions are single-language sets (English, Spanish), on which the counterfactual is the identity, and the subspace does not reliably carry LANGUAGE across languages (Section 4.3). We therefore agree with the reviewer's reading of these buckets. It is the conclusion of that experiment, and one that a scalar IIA cannot deliver. Entity binding and the logic o5 pass are different: there, within-bucket interchanges change the output as predicted (across query positions, and across values of o3). We will reword "well-interpreted" and the Fig. 5 caption so that a consistent bucket is not read as a faithful one.

Reviewer_YnkGOpenReview ↗

Clarified authors' intentions but does not resolve my main methodological concerns

I thank the authors for their detailed replies.

I believe the comparisons with Pîslar et al. (2025) and Chan et al. (2022) are partially, and reasonably well, answered. My remaining concern is that the authors explain technical differences from Chan et al. (2022) while understating that work's diagnostic value. In particular, they argue that causal scrubbing only produces an aggregate performance score and does not identify which variables are missing. However, causal scrubbing explicitly uses failures of hypothesis-driven interventions to inform the refinement of mechanistic hypotheses. The fact that these refinements are performed manually, rather than through an explicit bucketing procedure, does not mean that the earlier approach lacks diagnostic value. Therefore, while the authors establish procedural differences, they do not fully justify the claimed novelty of using intervention failures to diagnose and improve causal hypotheses.

The authors explicitly acknowledge my criticism of the RAVEL experiment: high interchange intervention accuracy (IIA) can arise from interventions that do not change the relevant causal variable. They now characterize their method as a diagnostic heuristic, rather than a test that establishes causal faithfulness, but, critically, do not introduce a general safeguard. They explicitly decline to run the requested controls to check how reliable the method is, which I find concerning. Changing the interpretation of the method does not, by itself, establish its validity.

Moreover, I think this partly undermines the strength and novelty of the contribution. My concern is that IIA-based bucketing could be misleadingly used as a proxy for causal faithfulness in future research. A method that can hide as much as it reveals, without adequate safeguards, is potentially a harmful contribution. What's the point of such a method if we cannot reliably interpret its results? I believe this type of exploratory work would be more suitable for a workshop.

Regarding the concern that high-density cliques are partly guaranteed by the search procedure, the authors defend their approach without establishing statistical significance or reproducibility across independent experimental runs. This makes the empirical findings, in my opinion, difficult to trust. Their procedure relies on sampled inputs and learned alignments, both of which introduce variability, yet they do not establish whether the same findings would emerge under different samples or training runs. This is somewhat like flipping a coin twice, getting tails both times, and concluding that future flips will always give tails. Without quantifying uncertainty and demonstrating that the observed patterns are reproducible, such results do not constitute a sufficiently sound basis for scientific conclusions.

Overall, while the rebuttal clarifies the authors' intentions, it does not resolve my main methodological and soundness concerns. I am now torn between leaving my score unchanged or even changing it to a strong reject. I do not mean to punish the authors for their detailed rebuttal, but instead feel that -- having now understood the authors' intended contributions better -- I am more sure that this is not suitable to the main conference track. But this might be better discussed among the reviewers afterwards, so for now I leave a "weak reject" as my recommendation...

AuthorsOpenReview ↗

Additional experiments following your suggestions (1/2)

Thank you for the detailed follow-up. Following your suggestions, we ran the controls you requested and added a safeguard against trivially high IIA. The new results support the paper's claims:

  • The buckets behind our main results lie far beyond what shuffled outcomes or an irrelevant alignment produce (Tables 1–2).
  • They remain faithful when only interventions that must change the output are counted (Table 2).
  • They replicate on three fresh samples and on unseen inputs, and the pipeline recovers o4, a variable absent from the initial hypothesis, every time (Table 3).

1. Shuffled outcomes

We shuffled the pairwise outcomes and re-ran the same search, either uniformly or keeping each input's number of successful pairs fixed.

Table 1. Largest bucket over all shuffles (50 / 200 / 200 / 1,000 per type). None reached the real size; p is empirical.

GraphReal bucketUniform shuffleDegree-preserving shufflep
Entity binding129729< 0.02
Logic o51421535< 0.005
Logic o41152480< 0.005
RAVEL3858< 0.001

2. A general safeguard

You noted that IIA inside a bucket can be high because the intervention changes nothing, and asked for a general safeguard. Following the impactful-edge requirement of Geiger et al. (2021) that you pointed to, we call a (base, source) pair impactful when the high-level model predicts that the intervention changes the base's output. A bucket with no impactful pairs is trivially consistent: the hypothesized variable takes a single value there. Note that the occurrence of a trivially consistent bucket does not necessarily mean a failure of the bucketing; it is a flag: its IIA alone does not show that the alignment transfers the variable, so the pass needs a further check on impactful pairs.

Table 2. IIA on impactful pairs.

BucketnIIAImpactful pairsIIA on impactful pairs (95% CI)
Entity binding, target1290.9962%0.985 (0.978–0.992)
Entity binding, 199 held-out prompts790.9075%0.883 (0.845–0.921)
Logic o5, target1420.9928%0.966 (0.944–0.984)
Logic o4, target1150.990%n/a
RAVEL, English / Spanish38 / 230.99 / 0.990% / 0%n/a
Control: random direction, o5 site1181.000%n/a

The first three rows carry our main claims. The RAVEL buckets are trivially consistent, as in our earlier reply. The last row is the irrelevant-alignment control you requested: with a random direction in place of the trained DAS direction, no intervention changes the output, so every bucket is trivially consistent. The impactful share therefore separates this case from a faithful alignment.

On the Logic o4 row. This row should not be read as a problem with the o4 bucketing. The trivially consistent bucket is itself a useful signal here. Under the hypothesis, all inputs with o4 = False should be interchangeable at this site. In fact the inputs with o1 = False are perfectly interchangeable (density 1.000 on the original and on three fresh samples, Section 3), while the full set with o4 = False is not (0.90–0.94). The site therefore distinguishes o1, a variable not yet in the hypothesis, and this is what led us to refine o4 into o1 and o2. The further check on impactful pairs confirms the refinement (Table 4, second comment).

3. Fresh inputs

We drew three new samples of about 200 inputs, disjoint from the original, and ran all pairwise interventions with the paper's alignment (about 39,000 per sample).

Table 3. Logic o5 pass (layer 7, position 77).

SampleBucket nIIA on impactful pairsRule found from token relations (accuracy)Classifier on unseen inputs: IIA on impactful pairs
Original1420.966t0≠t5 ∧ t2≠t4 (0.925)–
Fresh 11530.984same (0.975)0.995
Fresh 21510.985same (0.990)0.992
Fresh 31500.984same (0.975)0.985

Table 3 addresses three of your points:

  • Reproducibility. The same bucket is recovered on every fresh sample, with IIA 0.98 on impactful pairs.
  • Discovery. The rule is found by searching the 15 token-equality relations without access to o1–o3 (at most 0.73 after shuffling bucket labels). It is o4, which is absent from the initial hypothesis, so the pipeline finds the variable itself.
  • Held-out test. The classifier was trained on the original sample and applied to unseen inputs before any intervention.

The follow-up on the second logic pass (Table 4) and our response on Chan et al. (2022) are in a second comment.

Reviewer_kjHMOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper turns a common mech-interp metric (interchange intervention accuracy, IIA) into recursive diagnostic tool. The first key observation is that IIA is asymmetric: the authors propose a stronger symmetrized version (Interchange Consistency). The second key observation is that Interchange Consistency is not uniform across a domain. Bringing in a clever trick from graph theory (finding cliques in an undirected graph) provides a natural way to improve the causal abstraction: just subdivide the causal variable into two subvariables, based on the part of the input space in the clique (which has high interchange consistency by design) and part outside the clique (which had lower interchange consistency and IIA).

2 of the 3 validating experiments are just rediscovering known mechanistic interpretations, but (presumably) with less human-in-the-loop (The logical inference causal model was part of the original series of papers in which IIA was developed; the entity binding task's positional model was discovered more recently by Gur-Arieh et al 2026).

More compelling, the third experiment (which investigates why the MDAS method fails to find a convincing mechanistic interpretation for entangled factual recall) actually inches forward a diagnostic: that MDAS's attempted disentanglement works only for English or Spanish inputs, not any other languages.

Critical strengths
  1. The idea of looking for cliques in the interchange-consistent input graph was quite clever, in my opinion. (The deeper insight here being that if IIA isn't uniformly bad across inputs, then perhaps the sub-domains where it is good should be considered "sub-variables"). In particular, I appreciate how this turns the problem of "How to do diagnostics over an arbitrarily complicated input space (e.g. natural language, or images, or residual stream vectors etc)?" into a simple clique-finding problem over a finite graph! Quite the elegant problem simplification, at least in theory.
  2. I find the rediscoveries of existing interpretations in Section 4.1. and 4.2 to be particularly helpful for understanding the method, especially given that I was already familiar with the existing literature. Pedagogically, it mapped onto tasks I've already thought a lot about, in addition to being good sanity checks. (Even if they don't compel me to believe the method is good at finding novel improvements to mechanistic interpretations, they are a non-trivial bar to pass and I really appreciate their inclusion!)
Critical weaknesses

Of the weaknesses I found, most were cosmetic. Only one weakness that merits "supporting rejection":

The type of evidence I was expecting to support the title of "Improving Causal Abstraction" was the discovery of some novel mechanistic interpretation. This is notably absent: the only "discoveries" were rediscoveries of known interpretations (sections 4.1 and 4.2), and it's not clear how much this is a result of e.g. trying a lot of rediscoveries and finding two that happened to work.

The closest think to a novel interpretation is section 4.3's novel diagnostic that MDAS found two English/Spanish buckets and failed elsewhere. But that's at best a diagnostic, not an improved causal model!

However, I don't think this sinks the paper: the gap could be quickly filled by an experiment which then continues training MDAS, now with a higher penalty on non-English-nor-Spanish inputs. If this improves the 0.21 IIA for the "Other" bucket while not harming the English and spanish buckets, then that's solid evidence that this diagnostic method actually can be used to make novel improvements. If this experiment comes back negative, then we can only really claim that this "Specify Task -> Find Alighment -> Partition Inputs -> Generalize via Classification" recursion can rediscover known interpretations, not that it can "Improve Causal Abstraction" generally.

Questions for authors
  1. How much human guidance was required during the development to rediscover the known interpretations in sections 4.1 and 4.2? For instance, in line 322, you align o4 to position 78, layer 5, instead of position 81, layer 7. Was the choice to align at (78,5) rather than (81,7) a human-made decision? Or automatically decided? [This is a clarification request which affects the significance of the results in 4.1 and 4.2].

UPDATE: Raising my score to strong accept, conditional on 1. adding the "human input" scope clarification to the paper, and 2. adding the results of the experiment which uses this to continue training MDAS. (Note that either a positive or negative result of the MDAS would be interesting, so long as the authors appropriately narrow the claim to fit a negative result if applicable).

3 comments

Q1 (human guidance in Sections 4.1 and 4.2). We chose (78, 5) with a fixed rule, not a case-by-case judgment, and we will state the rule explicitly in the revision. As Step 2 notes (lines 199–203), a near-perfect alignment means the hypothesis is already adequate, so bucketing has nothing to diagnose. Such sites are typically where the variable has already been fully computed. We consider only sites where the variable cannot yet be fully computed, and among those we pick the one with the highest IIA.

For o4 = o1 ∧ o2, the full value is only available at the final token. That is why DAS finds a near-perfect alignment at (81, 7). Every site before position 81 has only partial information. Among them, (78, 5) has the highest IIA, so we align o4 there. The o5 pass uses the same rule, which gives (77, 7).

The human input is limited to two things: specifying the rule, and using the task’s token layout to determine which sites come before full information is available. The site is then chosen by IIA alone, with no reference to the interpretation we expected to recover. In Section 4.2, we made no site choice of our own. We use the same setup as Gur-Arieh et al. (2026), full-vector patching at the final query position at layer 15, so that our results are directly comparable to theirs.

MDAS follow-up. Following the reviewer’s suggestion, we are now running the proposed experiment. Starting from the trained MDAS subspace, we continue training with the Cause loss upweighted on pairs that involve cities whose language is neither English nor Spanish. We will report IIA when it is available.

Great, thank you! Specifying the rule explicitly in the revision (and including the clarification you just stated about what human input provides/doesn't provide) would address my question.

So long as you include that human-input-scope in the main body of the paper, and the results of the experiment testing whether this actually helps improve causal abstractions, I'm increasing my score to strong accept. (note that either a positive or negative result of said experiment is interesting, so long as you appropriately narrow the claim to fit the negative result if applicable).

Please do share the proposed human-input-scope phrasing, and the results of the experiment by the end of the discussion period, if at all possible.

Thanks for the feedback and acknowledgment! Here’s the new results we have:

Requested experiment. Following the reviewer's suggestion, we continued training MDAS with a higher penalty on the "Other" bucket. Concretely, each training pair that swaps between two cities with different languages, neither of them English nor Spanish, receives loss weight w while the rest pairs receive weight 1 (w = 1 is the original objective). We compare w = 5 against continuing training with w = 1 under identical data, order, and seeds (bucket size, with IIA in parentheses, averaged over seeds):

English bucketSpanish bucketOther bucket
Original alignment38 (0.99)23 (0.99)139 (0.21)
Continued training, w = 129.3 (0.99)24.3 (0.99)146.3 (0.19)
Continued training, w = 531.7 (0.99)23.3 (0.99)145.0 (0.18)

Reweighting had no effect on the Other bucket (w = 5 vs. w = 1: −0.001, −0.002, −0.009 across seeds), and continued training lowered it.

Clarification of Block Selection. We will add the following explanation at the beginning of Section 4.1 (after line 314):

In the recursive logic-discovery experiment, we select alignment sites using the task's token structure and IIA scores. We first restrict candidate sites to positions where not all information required to compute the target variable is available, as determined by the task's token order. Among these sites, we select the layer-position pair with the highest IIA. We exclude later sites where the variable can already be fully computed, as these typically yield near-perfect alignments that leave little room for diagnosis through bucketing. This procedure selects position 77, layer 7 for (o_5), and position 78, layer 5 for (o_4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.