- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses an important problem: how to accurately assess the causal impact of a model component on a given task, given the presence of interactions between components that occur when performing the task. Naive approaches to this problem are prohibitively expensive.
The paper thus works in the same setting as classical Shapley-style analysis (see below). Where the method goes beyond Shapley-style analysis in its introduction of the role of witnesses and also in defining the learning problem that allows it to do subset selection rather than individual-component attribution.
The paper introduces the use of witnesses (components with frozen outputs in the causal reach of the mediators) as a way to separate interacting sets of components. It then defines a learning problem to identify sets of components that appear to have strong effect together (either in necessity or sufficiency or both). The learning problem addresses subset selection for a problem evaluating model effect size on counterfactual inputs.
Critical strengths
The problem that the paper attacks is an important one, one that has bedeviled the circuit-discovering community for some time. The paper adds good conceptual ideas to the problem (witnesses, mainly, and the idea of witness-integrated effects) and it shows that one can define a learning problem that can be addressed via gradient descent to attack the subset-selection problem that is implied by the witness-integrated effects concept.
Critical weaknesses
I think the general problem the authors attack is important, and the paper represents a solid effort to do so. However it falls short for me for three reasons.
- The evaluation is problematic. The paper focuses its conclusions on the "tuned" values of the evaluation metrics using the CRS version of each score. The paper's conclusion that the method outperforms attribution baselines on average is mainly based on the "tuned" results. There are a number of problems:
-
First, the tuned configurations were chosen using the same scores the paper reports. Tuning is performed over five categories of configuration settings using a process that optimizes for the evaluation metric. When not tuning configuration for each given tesk/model setting, ie, using the "default" configuration, Table 1 shows that the results are essentially the same as Per-head NIE. Thus the value of the subset-selection aspect of the work seems be called in question.
-
Second, the settings used for "tuned" in each row of Figure 4 are different, ie, the setting used to choose Net, Faithfulness, and Kill are all differerent for the same model and task. The paper should pick one configuration per cell for all three goals.
-
Third, introducing the CRS version of each score is favorable to the paper's methods. CRS favors small circuits, which the paper's methods also favor.
-
Fourth, some baselines are tunable (eg EAP-IG) but they were not tuned as far as I can tell, unlike the proposed method.
-
The main conceptual contribution of the paper is witness-pinning. However, pinning only helps on one out of nine model-metric combinations, and the default method does not seem to use witness pinning.
-
The portion of the paper that estimates equation (5) is addressing the problem of how to assign credit to a set of contributors when there are interactions between contributors. This general problem has been studied using Shapley value as its tool.
I believe that the sampling estimator in Appendix A produces the Shapley ranking. In fact it appears that the method used by Algorithm 2 is the stratified Shapley representation used by SVARM (Kolpaczki et al., AAAI 2024).
There is literature following this direction already. It should be cited and compared against:
- Neuron Shapley (Ghorbani & Zou, 2020): seems to me to have almost the same goal, finding a few high-impact components cheaply by sampling.
- Shapley Head Pruning (Held & Yang, EACL 2023)
- Asymmetric Shapley values (Frye et al., 2020) - this addresses to some extent the same problem that addition of witnesses in this paper does
- Interaction indices: Grabisch & Roubens 1999; Shapley-Taylor (Sundararajan et al., 2020); Faith-Shap (Tsai et al., 2023). The paper needs to compare its methods against a sampled Shapley ranking.
Questions for authors
See above.
4 comments
Response to Reviewer ayjz (2/2)
(Continued from the previous comment, part 1/2.)
Third, introducing the CRS version of each score is favorable to the paper's methods. CRS favors small circuits, which the paper's methods also favor.
Yes, this is true (Point #8). This is by design: a circuit is useful when a few components carry the behavior, and removing more components risks damaging other behaviors. Under CPR and 1−CMD, which weight all circuit sizes equally, our net lead is larger (0.69 against 0.56 for per-head NIE; under CRS, 0.54 against 0.48). New: Table 7 (App. F).
Fourth, some baselines are tunable (eg EAP-IG), but they were not tuned as far as I can tell, unlike the proposed method.
Thanks for pointing this out (Point #5). In our revised experiments, we run a grid search over the EAP-IG hyperparameter, and additionally consider signed and unsigned variants. The best score out of these becomes the "task-optimized EAP-IG". We run MAttr (which we note is concurrent work to ours) at its default values only due to compute and time constraints; however, our main comparison to MAttr (in Figure 1) uses JuntaLearner's defaults, so we believe this comparison is fair. We also further discuss this in Appendix C.
This general problem has been studied using Shapley value as its tool.
Thank you for the pointer to this literature! We now include sampled Shapley as a baseline (Point #4). In particular, with all set sizes weighted equally, our Algorithm 2 gives the Shapley ranking and is closely related to Stratified SVARM (App. A). Notably, JuntaLearner is the method primarily used in our experiments, and does not estimate (5) directly.
Sampled Shapley is now a baseline on the ten GPT-2 and Qwen3 cells. On GPT-2, the six rankings of Table 8 are within 0.04 of each other on every mean. On Qwen3, Shapley has the highest mean kill and faithfulness (0.78 and 0.57, against 0.77 and 0.54 for Task-optimized) but a much lower net (0.19, against 0.49 for Task-optimized and 0.35 for our default), because its circuits also remove the control tasks. Sampled Shapley is >100x more expensive than per-head NIE, and would become infeasible on Llama. Cf. Appendix G and Table 8.
We hope the answers and the revised paper resolve your concerns.
Response
I've read and considered the authors' responses. However the results suggest that key parts of the method are equivalent to Shapley, and so the responses don't demonstrate in my view that there is a significant enough advance over prior work to merit raising my score.
Factual correction: JuntaLearner is not equivalent to Shapley
We thank the reviewer for reading our responses. We respectfully disagree that key parts of the method are equivalent to Shapley, and we apologize if our presentation gave that impression.
-
The equivalence holds only for the Monte Carlo estimator, in one special case, not for what JuntaLearner learns.
-
The two methods differ in results and in cost. If they were equivalent, they would score alike. The figure shows both on Qwen3-1.7B (App. G, Fig. 32 shows both models; see this more clearly in this figure:
) Qwen3-1.7B. Left: net CRS per task, sampled Shapley against Task-optimized JuntaLearner. Right: minutes per task on one GPU
- Results. On GPT-2 small all rankings are close. On the larger Qwen3-1.7B, sampled Shapley has lower net than Default JuntaLearner in every task (by 0.07 to 0.27; mean 0.19 against 0.35, and 0.49 for Task-optimized). It ranks first heads that every computation needs, many in layer 0, so its circuits also remove the control tasks.
- Cost. At our budgets, sampled Shapley takes 124 to 252 times as long as per-head NIE on both models, and 25 to 190 times as long as Default JuntaLearner.
-
The revised paper states the differences in App. A (end of the SVARM paragraph) and in App. G (paragraph "Differences from JuntaLearner"). App. A ends with: "JuntaLearner trains on a ladder of a few sizes and, in its held regimes, with witnesses, so its target is not the Shapley value." We summarize them here.
a. The estimator, not the learning objective. Without witnesses, and with all set sizes weighted equally, the expected score of the sampler in Algorithm 2 is an increasing affine function of the Shapley value (App. A). JuntaLearner instead learns a distribution over sets by gradient descent on the expected set score , and that objective is not a Shapley value.
b. Sets are drawn from learned logits, not uniformly. Shapley averages a head's marginal contribution over all coalitions. The link to WISE holds at initialization, when the logits are uniform (Prop. 1), and even there it covers only the first set size, one term of the Shapley average.
c. Only a few set sizes. Training runs on a short ladder (e.g. 72-32-16-4-1). With only some sizes weighted, the score is neither Shapley nor Banzhaf.
d. A different readout. The ranking comes from recorded gradients and, on ladders ending at , from single-head scores (Eq. 9), not from estimated Shapley values.
e. Witnesses change the game. In the held regimes we pin downstream heads at their clean values, so is no longer the NIE game, and no Shapley identity applies to any estimator.
Since these points are in the appendices, they are easy to miss. We will move a short version into the main text (Sec. 4, in the paragraph on sampled Shapley values) and make App. A and App. G more explicit.
Response to Reviewer ayjz (1/2)
We thank the reviewer for their time and feedback on our manuscript. We address the reviewer's concerns below and in the revised PDF. The PDF on this page is now the revised version; the submitted version remains available under Revisions. In the revised PDF, we annotated all changes with specific numbers, which we refer to as (Point #); points raised only by this reviewer are marked with "A" in magenta, and concerns shared with other reviewers are marked in blue.
This is correct, thank you for pointing it out! Selection used the same held-out pairs we report scores on. We have therefore rescored the selected configurations and the baselines on fresh prompt pairs that no run had drawn. Mean scores drop by at most 0.03, and per-head NIE, which involves no selection, drops as much as our configurations, so selecting on the held-out pairs did not inflate the reported scores. The net margin over per-head NIE holds (0.08 on fresh pairs, against 0.07 on the held-out pairs). Cf. Appendix C. (Point #1).
It's true that in aggregate over models and datasets, we tie per-head NIE; however, sets and witnesses help in various places (Point #2 and Point #3, respectively):
We've introduced more experiments and discussion on witnesses in Appendix D (cf. Figure 19).
(Continued in the next comment.)