- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a methodology for analyzing how changes in input features affect model outputs using Shapley value-based indirect effects. The method satisfies an additive decomposition property, whereby the contributions of individual inputs sum to the total effect on the output. The authors evaluate the methodology using GPT-2 and Qwen3.5-9B-base on two tasks: third-person singular -s agreement and gender bias.
Critical strengths
- The authors evaluate their methodology at two levels of abstraction using two LLM models.
- The authors provide implementation details that support the reproducibility of the experiments.
- The paper is well written and includes examples that help clarify the proposed methodology.
Critical weaknesses
-
Contribution: The authors should clarify the main contributions of the paper. One concern is the similarity of the proposed method to the original SHAP (SHapley Additive exPlanations) method, which is well established in the XAI literature, along with its many variants. I understand that the main distinction may lie in the use of indirect effects, but it is completly clear how the proposed method differs from the original SHAP approach. What happens if the original SHAP strategy is applied to the proposed low-level and high-level mediator interventions? Would it yield the same results? I recommend that the authors work more in the list of contributions at the beginning of the paper. In my view, applying the proposed method to the two selected tasks demonstrates its evaluation and applicability, rather than constituting a contribution in itself.
-
Experiments and Complexity: The authors mention that contiguous blocks are used as low-level mediators due to computational constraints. However, it is unclear how they ensure that these blocks contribute to the same behavior. Could the authors provide evidence or further justification for this assumption? Additionally, I would recommend to specify the computational complexity (runtime or number of operations), especially for the representation abstraction algorithm, which the authors acknowledge is computationally expensive.
Questions for authors
I have included questions above.
3 comments
Experiments and Complexity: The authors mention that contiguous blocks are used as low-level mediators due to computational constraints. However, it is unclear how they ensure that these blocks contribute to the same behavior. Could the authors provide evidence or further justification for this assumption? Additionally, I would recommend to specify the computational complexity (runtime or number of operations), especially for the representation abstraction algorithm, which the authors acknowledge is computationally expensive.
Contiguous blocks. We do not assume that the coordinates in a block contribute to the same behavior, and our method does not need this. Blocks are units of intervention, not assumed mechanisms: Algorithm 1 groups them only by their measured causal contributions, and interventions on whole groups confirm the agreement abstractions (Appendix C.3). We use contiguous blocks because they encode no prior knowledge: the order of residual-stream coordinates carries no meaning, so a contiguous partition is a fixed partition chosen without looking at the data. A block that mixed attributes would contribute to leakage, so the low leakage at layers 0 and 5 ( and with ) suggests that the blocks there are not substantially mixed. As direct evidence, we reran Algorithm 1 with random partitions of the coordinates into blocks (Appendix C.6). With , every random partition yields distinct dominant groups, Disentangle above random groupings (), and the same layer pattern: separation at layers 0 and 5 and weaker completeness at layer 10. At the paper's resolution, , the layer pattern also holds across partitions (best leakage 0.027 and 0.057 at layers 0 and 5 vs. at least 0.280 at layer 10); a single run can depend on the partition, but the weaker runs are exactly those our criteria and interventional scores flag (Appendix C.6). Thus, our results do not rely on the coordinates of a block sharing a role, and the layer-wise conclusions do not rely on the contiguous partition.
Complexity. At the step with groups, Algorithm 1 scores candidate merges with permutations of coalitions each, giving coalition evaluations and forward passes for attributes and reference pairs. With caching, the four-attribute gender task () needed 3.1M forward passes and about 2 hours with GPT-2 small on one GPU. We also implemented a batched version of Algorithm 1 that returns identical partitions; on one A100 GPU it runs in 8–45 minutes per layer. The revised Section 5 and Appendix C.6 report this analysis and the runtimes. Thus, the cost is polynomial in and depends on the model only through the cost of one forward pass.
Thank you for your clarifications. They helped me better understand the methodology.
We thank the reviewer for their time and valuable feedback. Below, we reproduce your comments and respond to each in turn.
Our response:
Relation to SHAP. Shapley value-based attribution methods share the same aggregation formula; they differ in the cooperative game they define, that is, the players, the value function and the baseline, and hence in the quantity they explain. SHAP is one such method: its players are input features, and it explains a prediction. In our game, the players are internal mediators, the value of a coalition is the outcome when its mediators take their treatment-induced states and the others their reference-induced states (averaged over contexts), and the quantity explained is the total effect of changing one input attribute.
Would SHAP give the same result? Yes, by design. SHAP applied to this game, explaining Mi(Xxk,U) with baseline Mi(Xx′k,U) and averaged over P−k,U, is Baseline Shapley and yields IESHAPML(k,i;xk,x′k); over groups it yields IESHAPMH(k,j;xk,x′k). We now state this in the Introduction. Our contribution is therefore not a new aggregation formula but (i) the mediation game, which decomposes the total effect over causally non-ordered, interacting mediators, where individual natural indirect effects do not add up to the total effect, and (ii) its use to define, construct and validate abstractions. Because mediators interact, the effect of a group differs from the sum of its members' effects, so abstractions must be evaluated with their own high-level games rather than by summing low-level scores; this is why simpler groupings of low-level effects fail (Appendix C.4).
Contributions. We agree that applying a method is evaluation, and we revised the list so that the third item states the empirical finding rather than the application:
We formulate a Shapley value-based indirect effect that decomposes the total effect of changing an input attribute over multiple causally non-ordered mediators.
We propose a framework that, unlike prior causal abstraction objectives, derives a high-level causal model by grouping mediators to separate the causal contributions of different input attributes: completeness and distinctness criteria and a greedy algorithm that constructs abstractions.
We show empirically, in two tasks on GPT-2 small and Qwen3.5-9B-base, that attribute-specific abstractions can be discovered without specifying a high-level model and validated by independent interventions, and that separation is clear at early and intermediate layers but weaker at later layers.
Thus, the novelty lies in the mediation game and in using it to derive and validate abstractions, not in the Shapley formula; the experiments contribute an empirical finding about how attribute-specific effects are organized across layers.