acceptodds
Under review as a conference paper at ICLR 2027

Abstraction of Intermediate Representations for Mechanistic Interpretability via Shapley Indirect Effects

Abstract

Providing interpretable explanations of neural networks is a central challenge in modern machine learning. We study how semantically meaningful changes in input attributes propagate through intermediate representations to affect model outputs, using causal mediation analysis and interchange interventions. Explaining large systems may require abstracting many low-level components into a smaller number of meaningful mechanisms, motivating the abstraction of intermediate representations. We first introduce a Shapley value-based indirect effect that treats causally non-ordered mediators as players in a cooperative game, averages each mediator’s marginal contribution across coalitions, and satisfies the desired additive decomposition property. Using this definition, we formalize abstraction quality in terms of completeness, requiring each input attribute’s effect to be concentrated in a dominant high-level mediator, and distinctness, requiring different input attributes to be associated with different dominant mediators. We then propose a greedy algorithm for constructing the abstraction. Finally, experiments on GPT-2 small and Qwen3.5-9B-base show that clear abstractions emerge at early and intermediate layers, with weaker completeness at later layers.

Primary area:
causal reasoning
Keywords:
Abstraction, Indirect Effects, Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NSnvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a methodology for analyzing how changes in input features affect model outputs using Shapley value-based indirect effects. The method satisfies an additive decomposition property, whereby the contributions of individual inputs sum to the total effect on the output. The authors evaluate the methodology using GPT-2 and Qwen3.5-9B-base on two tasks: third-person singular -s agreement and gender bias.

Critical strengths
  • The authors evaluate their methodology at two levels of abstraction using two LLM models.
  • The authors provide implementation details that support the reproducibility of the experiments.
  • The paper is well written and includes examples that help clarify the proposed methodology.
Critical weaknesses
  • Contribution: The authors should clarify the main contributions of the paper. One concern is the similarity of the proposed method to the original SHAP (SHapley Additive exPlanations) method, which is well established in the XAI literature, along with its many variants. I understand that the main distinction may lie in the use of indirect effects, but it is completly clear how the proposed method differs from the original SHAP approach. What happens if the original SHAP strategy is applied to the proposed low-level and high-level mediator interventions? Would it yield the same results? I recommend that the authors work more in the list of contributions at the beginning of the paper. In my view, applying the proposed method to the two selected tasks demonstrates its evaluation and applicability, rather than constituting a contribution in itself.

  • Experiments and Complexity: The authors mention that contiguous blocks are used as low-level mediators due to computational constraints. However, it is unclear how they ensure that these blocks contribute to the same behavior. Could the authors provide evidence or further justification for this assumption? Additionally, I would recommend to specify the computational complexity (runtime or number of operations), especially for the representation abstraction algorithm, which the authors acknowledge is computationally expensive.

Questions for authors

I have included questions above.

3 comments

We thank the reviewer for their time and valuable feedback. Below, we reproduce your comments and respond to each in turn.

Contribution: The authors should clarify the main contributions of the paper. One concern is the similarity of the proposed method to the original SHAP (SHapley Additive exPlanations) method, which is well established in the XAI literature, along with its many variants. I understand that the main distinction may lie in the use of indirect effects, but it is completly clear how the proposed method differs from the original SHAP approach. What happens if the original SHAP strategy is applied to the proposed low-level and high-level mediator interventions? Would it yield the same results? I recommend that the authors work more in the list of contributions at the beginning of the paper. In my view, applying the proposed method to the two selected tasks demonstrates its evaluation and applicability, rather than constituting a contribution in itself.

Our response:

Relation to SHAP. Shapley value-based attribution methods share the same aggregation formula; they differ in the cooperative game they define, that is, the players, the value function and the baseline, and hence in the quantity they explain. SHAP is one such method: its players are input features, and it explains a prediction. In our game, the players are internal mediators, the value of a coalition is the outcome when its mediators take their treatment-induced states and the others their reference-induced states (averaged over contexts), and the quantity explained is the total effect of changing one input attribute.

Would SHAP give the same result? Yes, by design. SHAP applied to this game, explaining Mi(Xxk,U)M^i(\boldsymbol{X}_{x^k}, \boldsymbol{U}) with baseline Mi(Xx′k,U)M^i(\boldsymbol{X}_{x'^k}, \boldsymbol{U}) and averaged over P−k,UP_{-k,\boldsymbol{U}}, is Baseline Shapley and yields IESHAPML(k,i;xk,x′k)IE^{\mathcal{M}^L}_{\mathrm{SHAP}}(k, i; x^k, x'^k); over groups it yields IESHAPMH(k,j;xk,x′k)IE^{\mathcal{M}^H}_{\mathrm{SHAP}}(k, j; x^k, x'^k). We now state this in the Introduction. Our contribution is therefore not a new aggregation formula but (i) the mediation game, which decomposes the total effect over causally non-ordered, interacting mediators, where individual natural indirect effects do not add up to the total effect, and (ii) its use to define, construct and validate abstractions. Because mediators interact, the effect of a group differs from the sum of its members' effects, so abstractions must be evaluated with their own high-level games rather than by summing low-level scores; this is why simpler groupings of low-level effects fail (Appendix C.4).

Contributions. We agree that applying a method is evaluation, and we revised the list so that the third item states the empirical finding rather than the application:

  • We formulate a Shapley value-based indirect effect that decomposes the total effect of changing an input attribute over multiple causally non-ordered mediators.

  • We propose a framework that, unlike prior causal abstraction objectives, derives a high-level causal model by grouping mediators to separate the causal contributions of different input attributes: completeness and distinctness criteria and a greedy algorithm that constructs abstractions.

  • We show empirically, in two tasks on GPT-2 small and Qwen3.5-9B-base, that attribute-specific abstractions can be discovered without specifying a high-level model and validated by independent interventions, and that separation is clear at early and intermediate layers but weaker at later layers.

Thus, the novelty lies in the mediation game and in using it to derive and validate abstractions, not in the Shapley formula; the experiments contribute an empirical finding about how attribute-specific effects are organized across layers.

Experiments and Complexity: The authors mention that contiguous blocks are used as low-level mediators due to computational constraints. However, it is unclear how they ensure that these blocks contribute to the same behavior. Could the authors provide evidence or further justification for this assumption? Additionally, I would recommend to specify the computational complexity (runtime or number of operations), especially for the representation abstraction algorithm, which the authors acknowledge is computationally expensive.

Contiguous blocks. We do not assume that the coordinates in a block contribute to the same behavior, and our method does not need this. Blocks are units of intervention, not assumed mechanisms: Algorithm 1 groups them only by their measured causal contributions, and interventions on whole groups confirm the agreement abstractions (Appendix C.3). We use contiguous blocks because they encode no prior knowledge: the order of residual-stream coordinates carries no meaning, so a contiguous partition is a fixed partition chosen without looking at the data. A block that mixed attributes would contribute to leakage, so the low leakage at layers 0 and 5 (L(P)=0.093L(\mathcal{P}) = 0.093 and 0.0570.057 with D(P)=1D(\mathcal{P}) = 1) suggests that the blocks there are not substantially mixed. As direct evidence, we reran Algorithm 1 with random partitions of the coordinates into blocks (Appendix C.6). With ∣I∣=16|\mathcal{I}| = 16, every random partition yields distinct dominant groups, Disentangle above random groupings (p≤0.015p \le 0.015), and the same layer pattern: separation at layers 0 and 5 and weaker completeness at layer 10. At the paper's resolution, ∣I∣=32|\mathcal{I}| = 32, the layer pattern also holds across partitions (best leakage 0.027 and 0.057 at layers 0 and 5 vs. at least 0.280 at layer 10); a single run can depend on the partition, but the weaker runs are exactly those our criteria and interventional scores flag (Appendix C.6). Thus, our results do not rely on the coordinates of a block sharing a role, and the layer-wise conclusions do not rely on the contiguous partition.

Complexity. At the step with nn groups, Algorithm 1 scores (n2)\binom{n}{2} candidate merges with NN permutations of nn coalitions each, giving O(N∣I∣4)O(N|\mathcal{I}|^4) coalition evaluations and O(KBN∣I∣4)O(KBN|\mathcal{I}|^4) forward passes for KK attributes and BB reference pairs. With caching, the four-attribute gender task (∣I∣=16|\mathcal{I}| = 16) needed 3.1M forward passes and about 2 hours with GPT-2 small on one GPU. We also implemented a batched version of Algorithm 1 that returns identical partitions; on one A100 GPU it runs ∣I∣=32|\mathcal{I}| = 32 in 8–45 minutes per layer. The revised Section 5 and Appendix C.6 report this analysis and the runtimes. Thus, the cost is polynomial in ∣I∣|\mathcal{I}| and depends on the model only through the cost of one forward pass.

Thank you for your clarifications. They helped me better understand the methodology.

Reviewer_B8d3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for abstracting intermediate representations of neural networks into high-level mediators for mechanistic interpretability. The authors first introduce Shapley-based indirect effects to quantify how changes in input attributes propagate through different mediators to affect model outputs. They then define two criteria for evaluating an abstraction: completeness, which measures whether the effect of an input attribute is concentrated in one mediator group, and distinctness, which measures whether different input attributes are associated with different dominant groups. A greedy algorithm is proposed to merge low-level mediators into high-level groups by optimizing these criteria. Experiments on GPT-2 Small and Qwen3.5-9B-base examine grammatical feature processing and gender-related influence, showing that the effects of different input attributes can be more clearly separated at early and intermediate layers than at later layers.

Critical strengths
  1. The idea of using Shapley values to measure indirect effects and guide mediator grouping is interesting. It provides a reasonable way to account for interactions between mediators and construct high-level abstractions.

  2. The experimental findings are interesting, particularly the observation that different input attributes can be more clearly separated at early and intermediate layers than at later layers.

Critical weaknesses
  1. It is unclear whether the method generalizes to more complex settings with more input attributes. Both experiments consider only two attributes, while achieving completeness and distinctness may become more difficult as the number of attributes increases. Another concern is generalization across contexts. For example, the grammatical agreement experiment uses only four prompt templates. It would be useful to learn the grouping on one set of contexts and test whether it remains effective on unseen contexts.

  2. The greedy algorithm considers all possible pairwise merges at each step, and each candidate requires estimating Shapley indirect effects. This can be expensive, especially with more mediators. The experiments are limited to at most 32 mediators due to computational constraints. It is unclear how practical this method would be at a larger scale.

  3. The hidden representations are divided into contiguous coordinate blocks, but the motivation for this partition is unclear. If different attributes are already mixed within the same block, the proposed algorithm cannot separate them, since it only merges groups. This could also be an alternative explanation for the weaker completeness at later layers. The results may depend on the initial partition rather than the actual separability of the underlying mechanisms.

  4. The experiments are relatively simple, with only two tasks involving two predefined attributes and binary outputs. It is difficult to draw broader conclusions about mechanistic interpretability from these settings. Moreover, the evaluation mainly uses the same completeness and distinctness criteria that the algorithm optimizes, without sufficient independent validation of the learned abstractions.

  5. The paper lacks comparisons with simpler grouping methods, such as clustering based on conventional indirect effects. While Shapley values provide an additive decomposition, it is unclear whether they actually lead to better grouping. Given the computational overhead, such comparisons are important to justify the use of Shapley values.

Questions for authors

Please refer to weaknesses

6 comments

We thank the reviewer for their time and valuable feedback. Below, we reproduce your comments and respond to each in turn.

It is unclear whether the method generalizes to more complex settings with more input attributes. Both experiments consider only two attributes, while achieving completeness and distinctness may become more difficult as the number of attributes increases. Another concern is generalization across contexts. For example, the grammatical agreement experiment uses only four prompt templates. It would be useful to learn the grouping on one set of contexts and test whether it remains effective on unseen contexts.

Our response:

We thank the reviewer for these suggestions. Our framework applies to any number of input attributes, and we conducted two additional experiments to address both concerns.

Generalization to unseen contexts. We applied the abstractions learned on the grammatical agreement task (∣I∣=32|\mathcal{I}| = 32, Table 3) to four unseen prompt templates (e.g., "In the morning, he"), without re-running the abstraction algorithm (Appendix C.2). Each attribute keeps the same dominant group, and both our criteria and the interventional score Disentangle (Appendix C.3) are largely preserved:

LayerLL (training / unseen)DD (unseen)Disentangle (training / unseen)
00.093 / 0.07210.972 / 0.969
50.057 / 0.10110.937 / 0.890
100.396 / 0.40210.597 / 0.582

Thus, the learned abstractions generalize beyond the training templates: separation remains strong at layers 0 and 5, and the weaker completeness at layer 10 also persists.

Generalization to more attributes. We extended the gender task from two to four attributes by adding hobby and clothing cues (e.g., "The rugged engineer in a suit who loves football said that"); each reference pair changes one attribute while holding the others fixed (Appendix C.5). At layer 5 with ∣I∣=16|\mathcal{I}| = 16, Algorithm 1 identifies a distinct dominant group for each attribute (bold):

Attribute (total effect)Shapley IEs (H1,H2,H3,H4)(H^1, H^2, H^3, H^4)
Occupation (0.424)(0.062, 0.203, 0.106, 0.052)
Adjective (0.144)(0.036, 0.045, 0.019, 0.046)
Hobby (0.172)(0.022, 0.052, 0.076, 0.022)
Clothing (0.132)(0.046, 0.033, 0.026, 0.028)

All four attributes have distinct dominant groups (D(P)=1D(\mathcal{P}) = 1), which none of 200 random partitions with the same group sizes achieves, and this also holds on four unseen templates. Completeness, however, is limited: leakage (L(P)=0.603L(\mathcal{P}) = 0.603) is comparable to the random-partition median (0.595). Importantly, this experiment illustrates why both criteria are necessary: distinct dominant mediators do not imply that the attributes' causal effects are well isolated, and leakage detects this.

In summary, the abstractions generalize across contexts, and the framework identifies distinct dominant mediators for more than two attributes, while its completeness criterion shows how well their effects are isolated. The four-attribute result motivates finer initial partitions and more scalable search algorithms. These results are included in the revised paper (Appendix C.2 and C.5).

The greedy algorithm considers all possible pairwise merges at each step, and each candidate requires estimating Shapley indirect effects. This can be expensive, especially with more mediators. The experiments are limited to at most 32 mediators due to computational constraints. It is unclear how practical this method would be at a larger scale.

Our response:

Computational scalability. Algorithm 1 needs O(KBN∣I∣4)O(KBN|\mathcal{I}|^4) forward passes for KK attributes, BB reference pairs and NN permutations (now in Section 5), and caching reduces this substantially: the four-attribute gender task with ∣I∣=16|\mathcal{I}| = 16 needed 3.1M forward passes and about 2 hours with GPT-2 small on one GPU. Two points make the method practical for the analyses in the paper. First, it does not require individual neurons as mediators: with ∣I∣=32|\mathcal{I}| = 32 blocks it already achieves low leakage (L(P)=0.093L(\mathcal{P}) = 0.093 and 0.0570.057) with D(P)=1D(\mathcal{P}) = 1 at layers 0 and 5 (Table 3), and finer blocks do not change the layer-10 result. Second, we implemented a batched version of Algorithm 1 that evaluates all coalitions of a greedy step together and returns identical partitions (it reproduces Table 3); on one A100 GPU it runs ∣I∣=32|\mathcal{I}| = 32 in 8–45 minutes per layer. The cost depends on the model only through the cost of one forward pass. Thus, useful causal abstractions do not require neuron-level mediators, and Algorithm 1 is practical at the resolution our analyses need; much finer mediators remain future work.

The hidden representations are divided into contiguous coordinate blocks, but the motivation for this partition is unclear. If different attributes are already mixed within the same block, the proposed algorithm cannot separate them, since it only merges groups. This could also be an alternative explanation for the weaker completeness at later layers. The results may depend on the initial partition rather than the actual separability of the underlying mechanisms.

Our response: Contiguous blocks and the initial partition.

Motivation. Shapley IEs are defined over coalitions of mediators, and Algorithm 1 needs O(N∣I∣4)O(N|\mathcal{I}|^4) coalition evaluations, so the 768 coordinates of the residual stream must be grouped into a manageable number of low-level mediators. We use contiguous blocks because they encode no prior knowledge: the order of residual-stream coordinates carries no meaning (permuting the coordinates together with the weights leaves the model unchanged), so a contiguous partition is a fixed partition chosen without looking at the data. Blocks are units of intervention, not assumed mechanisms.

What depends on the initial partition. Only the resolution: Algorithm 1 merges blocks but never splits them, so the effects of two attributes carried by the same block cannot be separated. Such mixing is detectable: with D(P)=1D(\mathcal{P}) = 1, it contributes to leakage, and we test abstractions by interventions on whole groups (Appendices C.3 and C.6). The conclusion that could still depend on the partition is the layer pattern (clear separation at layers 0 and 5, weaker completeness at layer 10), for example if contiguous blocks mixed attributes more at layer 10.

Robustness experiments. We tested this by rerunning Algorithm 1 with random partitions of the coordinates into blocks, keeping everything else identical (Appendix C.6). With ∣I∣=16|\mathcal{I}| = 16 and three random partitions per layer, every partition gives D(P)=1D(\mathcal{P}) = 1, Disentangle above random groupings (p≤0.015p \le 0.015), and the same layer pattern (layer-10 leakage 0.375–0.386 vs. 0.395 for contiguous blocks). At the paper's resolution, ∣I∣=32|\mathcal{I}| = 32, each cell below gives L(P)L(\mathcal{P}) and Disentangle, with its pp-value against random groupings:

LayerContiguousRandom 1Random 2Random 3
00.093, 0.972 (0.005)0.027, 0.987 (0.005)0.565, 0.593 (0.119)0.002, 0.498 (0.328), D=0D = 0
50.057, 0.937 (0.005)0.103, 0.891 (0.005)0.293, 0.709 (0.055)0.159, 0.827 (0.005)
100.396, 0.597 (0.005)0.280, 0.720 (0.005)0.316, 0.675 (0.005)0.310, 0.676 (0.005)

The layer pattern holds across partitions: the best partitions reach leakage 0.027 and 0.057 at layers 0 and 5, whereas no partition goes below 0.280 at layer 10. A single run can depend on the partition: two random partitions at layer 0 and one at layer 5 give weaker abstractions, and these are exactly the runs whose Disentangle is not significantly above random groupings, as the analysis above predicts: mixing makes an abstraction visibly worse, not spuriously better. Since Φλ\Phi_\lambda needs no ground truth, Algorithm 1 can be run from several initial partitions, keeping the lowest objective (here 0.027, 0.057 and 0.280 at layers 0, 5 and 10). On the gender task (layer 5), random partitions give leakage 0.203–0.284 vs. 0.245 for contiguous blocks, all with D(P)=1D(\mathcal{P}) = 1. Leakage at layer 10 also stays at 0.32–0.40 for ∣I∣=16|\mathcal{I}| = 16, 24 and 32 (Table 3).

Why the remaining limitation does not affect our contributions. The partition limits resolution, not validity: whatever the partition, an abstraction found by Algorithm 1 is evaluated with its own high-level game and can be tested by interventions, and the framework (definitions, criteria and Algorithm 1) applies unchanged to any low-level mediators, including finer blocks or learned features such as sparse autoencoder features (Section 7).

Thus, the initial partition can affect a single run, but in a way our criteria detect, and the layer-wise conclusions hold across random partitions and block sizes: they reflect the separability of the model's mechanisms, not the contiguous partition.

Reviewer_6i4yOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a way to generate causal abstractions for mechanistic interpretability by defining two metrics: completeness (where intervention of low-level components can be wholly explained through their high-level variables) and distinctness (where the behaviors of the high-level components are different). The authors use Shapley values to quantify both of these.

Critical strengths

Understanding how to construct causal abstractions provide a good first step towards both formalizing mechanistic interpretability and also providing a way to systematically create interpretations.

Critical weaknesses

The paper is so incredibly difficult to follow (even for causal abstraction paper standards and to the extent that I feel like it's entirely AI-generated) that I doubt this will be useful to the community. I highly recommend the authors clean up the notation in the paper, or potentially use more standard notion like in (GIZC+25, RSSC25, etc.). I also do not understand what are the falsifiable claims being tested in the the experiments.

Questions for authors
  • It is also unclear to me what the motivation of the work is. Do the authors envision this to be a better way to perform circuit discovery (i.e. group nodes in the circuit to make it more granular and easier to analyze)?
  • How is the greedy algorithm applied in Algorithm 1 different from just applying something like ACDC, but just grouping neurons / attention heads in the same layer that yield similar effects? For larger models, how is this tractable?
4 comments

We thank the reviewer for their time and valuable feedback. Below, we reproduce your comments and respond to each in turn.

The paper is so incredibly difficult to follow (even for causal abstraction paper standards and to the extent that I feel like it's entirely AI-generated) that I doubt this will be useful to the community. I highly recommend the authors clean up the notation in the paper, or potentially use more standard notion like in (GIZC+25, RSSC25, etc.). I also do not understand what are the falsifiable claims being tested in the the experiments.

Our response:

Notation and clarity. Our notation follows the structural causal model framework (Pearl, 2009), the standard language of causal mediation analysis, on which our framework builds. [RSSC25] uses similar notation but does not address mediation, so our indirect-effect quantities have no direct counterparts there. Some causal abstraction papers, including [GIZC+25], use notation built around mappings between low- and high-level models; Section 3.4 of [GIZC+25] also discusses mediation within causal abstraction. To make the connection explicit, the revised paper adds a table translating our notation into that of [GIZC+25] (Appendix C.1, reproduced below).

For attribute kk, the base input is b:=ψ(Xx′k,U)b := \psi(\boldsymbol{X}_{x'^k}, \boldsymbol{U}) and the source input is s:=ψ(Xxk,U)s := \psi(\boldsymbol{X}_{x^k}, \boldsymbol{U}); Def. 52 of [GIZC+25] writes the change as x→x′x \to x' with base xx, so their (x,x′)(x, x') is our (x′k,xk)(x'^k, x^k).

Ours[GIZC+25]Ref.
ML\mathcal{M}^L, MH\mathcal{M}^Hlow-level model L\mathcal{L}, high-level model H\mathcal{H}Def. 33
MiM^ilow-level variable (a coordinate block at one layer)Def. 36
φ\varphiinput alignment map πXIn\pi_{\boldsymbol{X}^{\mathrm{In}}}Def. 31
Y(M(S)(Xxk,Xx′k,U))Y(\boldsymbol{M}^{(S)}(\boldsymbol{X}_{x^k}, \boldsymbol{X}_{x'^k}, \boldsymbol{U}))ProjY(Solve(Lb∪IntInv(L,⟨s⟩,⟨MS⟩)))\mathrm{Proj}_Y(\mathrm{Solve}(\mathcal{L}_{b \cup \mathrm{IntInv}(\mathcal{L}, \langle s \rangle, \langle \boldsymbol{M}_S \rangle)}))Def. 44
TE(k;xk,x′k)TE(k; x^k, x'^k)E[TotalEffect(L,b,s,Y)]\mathbb{E}[\mathrm{TotalEffect}(\mathcal{L}, b, s, Y)]Def. 52
IEsingleML(k,i;xk,x′k)IE^{\mathcal{M}^L}_{\mathrm{single}}(k, i; x^k, x'^k)E[IndirectEffect(L,b,s,Y,Mi)]\mathbb{E}[\mathrm{IndirectEffect}(\mathcal{L}, b, s, Y, M^i)]Def. 52
P\mathcal{P}, HjH^jpartition Π\Pi; HjH^j is a variable merge of {Mi:i∈Ij}\lbrace M^i : i \in \mathcal{I}_j \rbraceDefs. 31, 37
IESHAPIE_{\mathrm{SHAP}}, LL, DD, Φλ\Phi_\lambdano counterpart (our contribution)–

Usefulness to the community. Given specified input attributes, our framework automatically discovers high-level mediators that separate their causal contributions to model outputs, and its criteria show when such separation fails. Existing causal abstraction analyses typically require the high-level mediator structure itself to be specified in advance and then test it; our framework needs only the attributes of interest, so it supports exploratory analysis when the relevant mechanisms are not known. Thus, it complements hypothesis-testing analyses by proposing the high-level structure that such analyses can then test.

Regarding AI-generated writing: as stated in the paper's AI use statement, we used generative AI to assist with writing and code, which we reviewed and tested; the scientific formulation, methodology, and findings are our own.

What the experiments test. The experiments do not merely demonstrate the algorithm; they test two empirical questions about given input attributes, and each could have been answered negatively.

  1. Can the causal effects of the attributes be isolated in distinct high-level mediators? The answer would be no if the attributes shared a dominant group (D(P)=0D(\mathcal{P}) = 0) or if leakage were no lower than for random groupings of the same blocks, against which we compare every abstraction (200 random partitions). This does happen: with ∣I∣=4|\mathcal{I}| = 4 or 8, attributes share a dominant group in five layer–task combinations (Tables 3 and 8); with four attributes, leakage is no lower than for random groupings (Appendix C.5); and two random initial partitions at layer 0 give weak abstractions, one with D(P)=0D(\mathcal{P}) = 0, which our criteria and interventional scores flag (Appendix C.6). With ∣I∣=32|\mathcal{I}| = 32 contiguous blocks, both tasks give D(P)=1D(\mathcal{P}) = 1 at every layer, and interchange interventions that use neither Shapley values nor our criteria confirm the agreement abstractions (Disentangle above all 200 random partitions, p=0.005p = 0.005; Appendix C.3).

  2. Does this separability vary across layers? With ∣I∣=32|\mathcal{I}| = 32, leakage is low at layers 0 and 5 (L(P)=0.093L(\mathcal{P}) = 0.093 and 0.0570.057) but high at layer 10 (0.3960.396; Table 3), and the independent interventions show the same pattern (Cause 0.960 and 0.895 vs. 0.258). Comparable quality at layer 10 would have contradicted this. We report it as an empirical observation in the studied settings, not as a general property of later layers. We tested the most direct alternative explanation, that contiguous blocks mix attributes more at later layers, by rerunning Algorithm 1 with random initial partitions: the best partitions reach leakage 0.027 and 0.057 at layers 0 and 5, whereas no partition goes below 0.280 at layer 10 (Appendix C.6); Qwen3.5-9B-base shows the same pattern (Appendix B).

Thus, the experiments test falsifiable hypotheses about specified input attributes and report where separation fails as well as where it succeeds.

Questions For Authors: It is also unclear to me what the motivation of the work is. Do the authors envision this to be a better way to perform circuit discovery (i.e. group nodes in the circuit to make it more granular and easier to analyze)?

Our response:

Motivation and relation to circuit discovery. Given specified input attributes, our motivation is to automatically discover the high-level mediators that separate their causal effects, rather than relying on a human-specified high-level model. While related to circuit discovery, our approach addresses a different, complementary question. Circuit discovery typically identifies a subnetwork sufficient to preserve a target behavior, revealing which components are responsible for that behavior. In contrast, our method groups all mediators in a given layer according to their causal contributions to different input attributes, revealing how attribute-specific causal effects are organized and separated within the representation. For example, rather than identifying the components necessary for grammatical agreement, our framework investigates how the effects of grammatical number and person are distributed across high-level mediator groups. Thus, our goal is not a better circuit discovery method, but to provide a complementary approach to understanding the causal organization of internal representations.

How is the greedy algorithm applied in Algorithm 1 different from just applying something like ACDC, but just grouping neurons / attention heads in the same layer that yield similar effects? For larger models, how is this tractable?

Our response: Algorithm 1 and ACDC solve different problems. Both are greedy, but over different objects: ACDC greedily removes edges while preserving a task metric, whereas Algorithm 1 greedily merges groups of mediators to minimize leakage under the distinctness constraint.

Algorithm 1 does not simply group neurons or attention heads in the same layer that have similar effects. It searches for the grouping that best achieves two properties: completeness (each attribute's effect concentrates in one abstract mediator) and distinctness (different attributes have different dominant mediators). Grouping mediators with similar effects does not by itself yield an interpretable high-level abstraction.

First, effects measured one component at a time can miss the effect entirely: at layer 0 (∣I∣=32|\mathcal{I}| = 32), each block's conventional IE is below 1.2×10−51.2 \times 10^{-5} (0.03% of the total effect in sum), because the effect appears only when many blocks are intervened on jointly, and k-means clustering of these effects is no better than random partitions (p=0.26p = 0.26; Appendix C.4). Second, even clustering effects that capture interactions fails: we apply k-means clustering to the low-level Shapley IEs IESHAPMLIE^{\mathcal{M}^L}_{\mathrm{SHAP}} in Table 4 (layer 0, ∣I∣=32|\mathcal{I}| = 32) and calculate the high-level Shapley IE IESHAPMHIE^{\mathcal{M}^H}_{\mathrm{SHAP}} of each resulting group. We then obtain the following (total effects: TENumber=0.358TE_{\mathrm{Number}} = 0.358, TEPerson=0.202TE_{\mathrm{Person}} = 0.202):

MethodGroup sizesNumber IEs (H1, H2, H3)Person IEs (H1, H2, H3)LD
k-means on Shapley IEs20 / 10 / 2(0.364, −0.009, 0.002)(0.180, 0.013, 0.009)0.0700
Algorithm 115 / 16 / 1(0.017, 0.309, 0.032)(0.192, 0.007, 0.003)0.0931

Both attributes are concentrated in the same group (D(P)=0D(\mathcal{P}) = 0), so clustering by similar effects fails to separate them, whereas Algorithm 1 attributes number and person to distinct groups. The same holds at layer 10, and at layer 5 the clustering leaks substantially more (L(P)=0.268L(\mathcal{P}) = 0.268 vs. 0.0570.057).

We acknowledge that tractability remains a challenge. Algorithm 1 needs O(N∣I∣4)O(N|\mathcal{I}|^4) coalition evaluations and O(KBN∣I∣4)O(KBN|\mathcal{I}|^4) forward passes for ∣I∣|\mathcal{I}| low-level mediators, NN permutations, KK attributes and BB reference pairs (now in Section 5). With caching, the four-attribute gender task (∣I∣=16|\mathcal{I}| = 16) needed 3.1M forward passes and about 2 hours with GPT-2 small on one GPU, and our batched implementation, which returns identical partitions, runs ∣I∣=32|\mathcal{I}| = 32 in 8–45 minutes per layer on one A100 GPU. Our method is less scalable than circuit discovery because it solves a harder problem: Shapley indirect effects account for interactions among mediators, which requires evaluating coalitions of mediators rather than one component at a time. Improving tractability is left for future work.

Thank you again for your valuable comments. As the discussion period is coming to a close, we would be grateful to know whether our responses and the revised manuscript address your concerns. We would be happy to clarify any remaining points.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript

We thank all reviewers for their constructive comments. We have uploaded a revised manuscript in which all changes are marked in red, and we respond to each reviewer's comments individually below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.