- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a methodology that serves as a diagnostic tool to identify regions (or buckets) of the input space where a causal interpretation is faithful, rather than relying on global causal interpretations. The authors propose using this methodology to analyze high-level hypotheses and identify previously unmodeled intermediate variables. The proposed pipeline consists of four steps: (1) specify the task and input space, (2) find a candidate alignment, (3) partition the input space based on interchangeability, and (4) generalize and characterize the resulting partition.
Critical strengths
- The paper addresses an important problem in global interpretability: an interpretation may not generalize across the entire input space.
- The authors evaluate their methodology on three different tasks: logical inference, entity binding, and entangled factual recall.
- The authors clearly acknowledge and discuss the two main limitations of their approach.
Critical weaknesses
-
Contribution: The authors claim that their methodology can identify previously unmodeled intermediate variables, but it is unclear how this is achieved in practice. As I understand it, DAS finds candidate variables for interventions. Why is partial alignment sufficient, rather than full alignment? What is the motivation for this criterion, and how is the alignment threshold determined? How does the methodology identify previously unmodeled intermediate variables? The authors also state that DAS “can tell us that the hypothesis is imperfect, but not how to improve it.” However, it is unclear how the proposed methodology can support hypothesis refinement. Could the authors clarify how this refinement is performed in practice?
-
Evaluation Metrics: Interchangeability appears to be an important metric in this paper, but its motivation and role in the evaluation require further explanation. How in-bucket and cross-bucket IIA are measured and interpreted? The authors should clarify how these metrics are computed and explain why they are important for evaluating the proposed methodology.
-
Explanation of the Methodology and Results: Step 3 of the methodology requires further clarification. In Step 4, it is unclear why two classifiers are trained and what the purpose of each is. The authors should explain the motivation behind this design. I don’t understand what you mean here: “That internal features outperform hand-labeled features suggests the distinction in different buckets is fundamentally encoded within the model’s own representations.” I also do not understand how the proposed methodology was used to identify the reasons for alignment failure in the Entangled Factual Recall task and what evidence supports the conclusions.
Questions for authors
I have included some questions above.
Thank you for the careful review. Before answering Q1–Q3, we would like to clarify what the paper claims.
What the paper contributes.
The paper does not propose a new faithfulness score, and it does not claim that a high-IIA bucket certifies a faithful abstraction. Its starting point is that IIA alone cannot tell us whether a causal abstraction is faithful: an intermediate score says neither where an alignment holds nor why it fails. Given a model, a hypothesis and one alignment, our method diagnoses that alignment. It locates the inputs on which interchange is consistent, characterizes what separates them from the rest, extends that boundary to unseen inputs, and reads it as a heuristic for improving the hypothesis. Prior work stops at a number: global IIA, the size of the largest clique (Geiger et al., 2021), coverage at a faithfulness level (Pîslar et al., 2025), or performance recovered (Chan et al., 2022). We turn that number into an explanation.
Q1 (Pîslar et al., 2025).
Both works build a graph from pairwise interchange outcomes, but they solve different problems. Pîslar et al. start from a pre-specified set of candidate causal models and ask how to assign inputs among them so that the combined model is more faithful. Their procedure enumerates the whole input space (1,000 and 64 inputs), builds one graph per candidate, assigns inputs greedily under a faithfulness threshold, and reports how many inputs the candidates cover. It needs the alternative hypotheses in advance, and the resulting input cells serve to measure coverage. We start from a single alignment with no alternatives, and the partition itself is our object of study. Step 4 characterizes its boundary with labelled or SAE features, generalizes it to unseen inputs without further interventions, and uses it to derive what the hypothesis lacks; in the logic task this recovers o4, then o1 and o2, from the output variable alone. We also move from fine-tuned GPT-2 small with DAS to pretrained Gemma-2-2B and Llama-3.1-8B with full-vector patching and MDAS. The two approaches are complementary: our diagnosis can supply the candidate models that their combination requires.
Q2 (Chan et al., 2022).
Causal scrubbing tests a complete hypothesis by resampling every activation the hypothesis treats as equivalent and measuring how much performance survives. Its output is one aggregate number. It gives no verdict per input, so it cannot say where a hypothesis fails or which variable is missing; in the authors' case studies the hypothesis is revised by hand. It also runs only swaps that are predicted to leave behavior unchanged, whereas interchange interventions also test counterfactuals that must change the output. Our unit of analysis is the individual pair of inputs, and the structure of pairwise outcomes over the input space is what yields the diagnosis.
Q3 (shuffled outcomes, irrelevant variables).
We did not run these controls because the diagnosis is defined only relative to an alignment. Our method does not test whether structure exists; that is the role of alignment search and IIA in Step 2. It takes an alignment that is established but imperfect, and explains where it holds and where it fails. Shuffled outcomes no longer correspond to any alignment, and a causally irrelevant variable does not yield one: if alignment search stays at baseline, Step 2 stops, and there is nothing to diagnose. An analogy: a doctor cannot localize an illness in someone who is not ill; given a patient, the doctor can say what is healthy, what is not, and what treatment would help. Our method plays that role for a causal abstraction with an intermediate IIA. Within that scope, the buckets are not an artefact of the search: they align with variables the search never saw (Table 1), they are stable across density thresholds and seed budgets (Appendix A.1), and PCA + K-means on the same activations fails to recover them in two of four passes (Table 2). RAVEL makes the same point from the other side: where an alignment's signal is hollow, the diagnosis says so (see below).
Additional clarification: RAVEL.
RAVEL is in the paper as a case where the alignment fails while IIA still shows a non-trivial signal (13.9% held out). The diagnosis explains that signal: the interchange-consistent regions are single-language sets (English, Spanish), on which the counterfactual is the identity, and the subspace does not reliably carry LANGUAGE across languages (Section 4.3). We therefore agree with the reviewer's reading of these buckets. It is the conclusion of that experiment, and one that a scalar IIA cannot deliver. Entity binding and the logic o5 pass are different: there, within-bucket interchanges change the output as predicted (across query positions, and across values of o3). We will reword "well-interpreted" and the Fig. 5 caption so that a consistent bucket is not read as a faithful one.