acceptodds
Under review as a conference paper at ICLR 2027

Matryoshka attribution: Learning to attribute language model outputs to representations and weights

Abstract

Attributing language model outputs to their internal computations is an open problem in interpretability. Existing methods, which use causal interventions, gradients, or learnable masks, either are infeasibly expensive or struggle to identify actual causally-important internal computations. We propose framing attribution as the problem of identifying nested subsets of internal components which minimise a downstream loss. To learn this task, we introduce **Matryoshka Attribution** (MAttr), a mask learning method that parametrises the mask with a simple differentiable sigmoid top- operator. We supervise training over all sparsities simultaneously by randomising over training, resulting in a learned ordering of components by attribution score. MAttr achieves number 1 on the official leaderboard of the Mechanistic Interpretability Benchmark (Mueller et al., 2025); our method identifies sparse and task-transferrable circuits across varying circuit bases. As a practical application, we show that MAttr can be trained with reinforcement learning to identify weight changes responsible for downstream behaviours in LLM finetuning. We train MAttr on refusal judge scores and find that restoring of Llama 3.1 8B Instruct's weights to their base model state is sufficient to remove refusals while maintaining capabilities. We view MAttr as a successful formulation of interpretability into a learnable objective that we can tackle with gradient descent, and encourage future work along these lines.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, attribution, language model internals

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_wmDtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a new attribution and circuit extraction method (MAttr). The method incorporates several improvements, most notably a Matryoshka method that allows them to learn circuits of multiple sizes simultaneously (by setting a random top-k budget in each learning step), and sigmoid-based top-k masks. The top-k masking avoids requiring an explicit sparsity loss, and the sigmoid-based mask allows for differentiation. The paper compares the performance of MAttr to state of the art methods such as EAP and Integrated Gradients, and finds that MAttr performs significantly better. The paper applies the method to Llama-3.1-8B-Instruct, showing which weights are responsible for refusal, and removing refusal by replacing those weights with base model weights.

Critical strengths

The method outperforms baselines on the highly relevant task of circuit extraction.

The algorithm is a simple iterative method learning continuous mask scores.

A single run of MAttr provides circuits at all sparsity levels which eliminates a major hyperparameter that had to be tuned for previous circuit extraction methods.

Critical weaknesses

The method's success relies on the Faithfulness (CPR) metric, and scores substantially closer to baselines on the Compactness metric. The CPR metric’s faithfulness is unbounded: It can be larger than 1 if the circuit achieves a better / higher logit difference than the full model. This is a known issue in circuit analysis. Additionally CPR calculates the area under the curve on a linear x-axis which gives little weight to small circuits. The paper acknowledges the issues with CPR and shows the Compactness metric to account for them. However, MAttr performs similarly to Integrated Gradients (IG) on the Compactness metric. This is particularly surprising given that MAttr’s first step is equivalent to centred IG (under SGD, in expectation, as the paper states), so I would expect MAttr to significantly improve over IG.

Questions for authors

Why doesn’t MAttr beat IG on Compactness, especially considering MAttr’s first step similar to IG?

Is MAttr’s very good performance on the CPR metric due to the unbounded logit difference, or due to the linear x-axis scaling? Can you quote metrics with faithfulness capped to 1, and with logarithmic x-axis scaling (separately)?

2 comments

Thank you for your thoughtful review! On your questions:

  1. Overall, this is because Compactness is much easier to saturate than CPR. In fig 9b in the appendix, we find that Compactness saturates very quickly in training while CPR keeps smoothly improving. Furthermore, if you look at the IIA curves in fig 14 (which Compactness is an AUC of), all methods essentially have a score of 0 at small circuits which jumps to 1 at some threshold. The only way to improve the score further would be to prune components from the smallest circuit getting an IIA of 1: this may simply not be possible if all the components involved are truly helpful for the behaviour.
  2. On the other hand, CPR rewards sorting units in order from helpful to harmful for the behaviour: this gives more signal for learning the attribution ranking since the logit difference can smoothly change across changes in rankings. We did find that a linear-AUC for CPR still shows MAttr succeeding; we will post scores shortly.

Here is the log-AUC of normalised logit difference, as a log counterpart to CPR. The results are node-level (attention heads + MLPs), on Llama for SVA, Arithmetic and ARC-E, and Qwen for IOI. SVA and Arithmetic are averaged over 4 subtasks each.

MethodSVAArithmeticARC-EIOI
MAttr0.670.731.600.96
IG0.600.570.600.77
Expected Gradients0.600.560.610.78
Attribution patching (I×G)0.550.030.600.26
AttnLRP0.560.470.510.61
Node Pruning0.460.470.600.67
DBM0.550.470.500.63
Random0.040.030.050.07

Additionally, here is the linear-scale version of Compactness, reported on the same setting:

MethodSVAArithmeticARC-EIOI
MAttr0.9710.9630.9820.953
IG0.9670.9240.9780.937
Expected Gradients0.9680.9010.9780.934
Attribution patching (I×G)0.9560.1320.9730.655
AttnLRP0.9600.8820.9700.886
Node Pruning0.9280.9200.8710.905
DBM0.9600.8160.8990.879
Random0.1900.1410.2490.251

MAttr is the best-performing method in both cases.

Reviewer_SAsMOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new technique for attributing behaviours to a subset of model features, Matryoshka Attribution (MAttr). The technique beats a wide array of other comparators by a substantial margin on a well known open source benchmark, and on a behavioural evaluation the authors perform.

In general, their technique falls within the realm of mask-learning when it comes to input-output attributions. The advantages to those methods are that they are architecture agnostic, however they do require solving a training objective per architecture for each problem, as opposed to gradient-based methods, which require no such optimization. This is not necessarily a weakness of the method, but it is worth noting.

They specifically design a method that learns one global attribution vector across a dataset and learns a ranking across all sparsity levels simultaneously via. random sampling. This makes it substantially more well-suited for application than other mask-based methods with occasionally will require a fitting per example and/or per sparsity.

They evaluate their method on MIB and MIB+, which is an open-source benchmark for circuit detection, where they convincingly set the state of the art score, and additionally demonstrate AI safety relevant capabilities by showing that MAttr can identify a small subset of behaviour specific weights for refusal.

Finally, they show the method's applicability to non-standard settings for attribution by adapting to reinforcement learning.

Critical strengths

The authors benchmark on Mechanistic Interpretability Benchmark (MIB), a large open source benchmark which is open to submissions from the community. The authors show that their technique outperforms a wide array of comparators by a significant margin. This is an excellent result.

Their method is theoretically well-justified with clear links to causal abstraction and previous attribution methods.

The authors provide a behavioural study which directly ties their method to control of a well known interpretability phenomenon. This helps to establish that the technique is significant for researchers attempting to steer model behaviour.

Critical weaknesses

The authors only consider a single behavioural study. In light of their other evidence, this is not as significant a limitation, but it would be interesting to see if their headline finding can be replicated for say, reasoning behaviours [Venhoff 25, Ward 25] or other safety relevant features [Turner 25, Soligo 25, Betley 25].

The authors leave out comparisons of the other models on the MIB benchmark using the CMD scores, reporting instead only CPR. From looking at the MIB leaderboard, I can see that MAttr performs substantially worse on CMD. I do not think this is necessarily a massive weakness of the paper, but it would

Questions for authors

You report only CPR scores for the MIB leaderboard comparison -- did you omit CMD for any specific reason?

2 comments

Thank you for your review and pointing us to other behaviours to try our RL attribution approach on! We are very interested in safety-relevant applications of our method; the main reason to focus on refusal is because prior work reports a variety of interesting baselines to compare against (e.g. pure RL finetuning, and abliteration).

You raise a relevant point on CMD: overall, we found CMD is not obviously compatible with our method since it requires signed attribution scores but our method doesn't produce signed scores. We will quote our response to reviewer bEcH for elaboration on this point:

A key detail about CMD in appendix D.1 of the MIB paper is that it requires ranking components by absolute attribution score. As you rightly point out in your question 1, MAttr is insensitive to constant shifts, meaning 0 is an arbitrary reference point (only relative MAttr scores matter), and therefore taking the absolute value of the scores is a totally arbitrary operation. Since the conceptual goal of CMD is to measure "how closely the circuits reproduce the full model’s behavior", that's mainly why we introduce the Compactness metric as a complementary measure for how small a circuit is needed to do the task, which doesn't require absolute ranking. Table 8 shows very similar Compactness scores across losses.

The official MIB leaderboard reports that edge-level MAttr @ 5k train steps has a CMD score of 0.19, while EAP-IG-inputs has 0.06. However, this sorts MAttr scores by absolute value, which as we point out above is conceptually arbitrary.

More conceptually, we also point to this part of sec. 3: "MAttr is actually an instance of what Bilodeau et al. (2024) call for in response to their result: “a method that directly optimizes the task” by allowing the practitioner to define a loss function, rather than defining an axiomatically determined outcome." If a practitioner were to prioritise CMD as a metric, then with MAttr one could define a CMD-relevant differentiable loss function and train on that. Enabling this flexibility is the key contribution of our work.

We should have discussed this in the paper and will include this in the final version along with the leaderboard scores.

I appreciate the clarification!

Yes, the lack of signed scores makes sense. In hindsight, I should have realized this, but very much appreciate the authors clarifying this for me.

In light of this, I believe my concerns are fully resolved. One can always produce more results in more settings, but it sounds like the authors already intend to pursue more safety-related approaches with their work.

This work definitely warrants publication and I am happy to increase my score.

Reviewer_bEcHOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Matryoshka Attribution (MAttr), a method for attributing NNs behavior to internal components or weight changes. The authors show that MAttr outperforms the evaluated baselines on the Mechanistic Interpretability Benchmark’s CPR metric. They also validate this across component granularities and different tasks. Additionally, the work extends the method to weight attribution using RL, showing that reverting 1% of Llama 3.1 8B Instruct’s parameters to their base-model values largely removes refusal while broadly preserving performance on other tasks.

Critical strengths
  • The experiments provide strong evidence that MAttr improves over the baselines on MIB's CPR, often by a wide margin. Results across several models, tasks, and component types make this more convincing than an improvement on a single circuit-finding task.
  • The authors introduce MIB+, extending the MIB benchmark with subject–verb agreement and additional arithmetic tasks, and evaluations over individual neurons and sparse autoencoder features. This tests whether the method works beyond whole attention heads and MLP blocks.
  • The authors tune and re-evaluate competing methods rather than relying on their default settings. They also examine where the rankings disagree and find that MLP blocks account for much of the disagreement.
  • The work also proposes an extension to weight attribution, which identifies which changes between two checkpoints matter for a particular behavior. The results on Llama 3.1 8B Instruct show that this extension makes the technique useful for concrete applications beyond circuit localization.
  • I appreciate the clear writing and presentation. The algorithms and ablations make the method and its main design choices relatively easy to follow.
Critical weaknesses
  • MAttr performs very well on CPR, but the paper does not report its performance on MIB’s integrated circuit-model distance (CMD) metric, which measures how closely the circuits reproduce the full model’s behavior rather than strengthen the target answer. The leaderboard results for MIB show MAttr behind several baselines on CMD. This limits the broad claim of better circuit discovery and deserves a discussion in the paper. Table 8 does include a CMD training objective, but still only reports CPR and compactness.
  • It would be great to see a more qualitative analysis of what MAttr finds that other methods miss, or vice versa. The ranking comparisons and known-component analyses are great, but do not explain much about the roles of the differing components. Even one small example with targeted interventions would show whether the higher scores reflect a better insight into the model's circuit.
Questions for authors
  • Why is CMD not reported alongside CPR? Could the authors provide CMD results for both the main MAttr configuration and the CMD-trained variant in Table 8, compared with the strongest baselines? Please also clarify how MAttr scores would be converted into circuits for CMD, given that adding the same constant to all MAttr scores leaves its masks unchanged, but can change their ranking by absolute value.
  • Could you give one concrete example where MAttr and a strong baseline select different components, and explain what targeted interventions show about their roles?
2 comments

Thank you for your thoughtful engagement with our work! On weaknesses/questions:

  1. CMD
    1. A key detail about CMD in appendix D.1 of the MIB paper is that it requires ranking components by absolute attribution score. As you rightly point out in your question 1, MAttr is insensitive to constant shifts, meaning 0 is an arbitrary reference point (only relative MAttr scores matter), and therefore taking the absolute value of the scores is a totally arbitrary operation. Since the conceptual goal of CMD is to measure "how closely the circuits reproduce the full model’s behavior", that's mainly why we introduce the Compactness metric as a complementary measure for how small a circuit is needed to do the task, which doesn't require absolute ranking. Table 8 shows very similar Compactness scores across losses.
    2. The official MIB leaderboard reports that edge-level MAttr @ 5k train steps has a CMD score of 0.19, while EAP-IG-inputs has 0.06. However, this sorts MAttr scores by absolute value, which as we point out above is conceptually arbitrary.
    3. More conceptually, we also point to this part of sec. 3: "MAttr is actually an instance of what Bilodeau et al. (2024) call for in response to their result: “a method that directly optimizes the task” by allowing the practitioner to define a loss function, rather than defining an axiomatically determined outcome." If a practitioner were to prioritise CMD as a metric, then with MAttr one could define a CMD-relevant differentiable loss function and train on that. Enabling this flexibility is the key contribution of our work.
  2. On qualitative analysis, this is a great point! Due to space reasons we decided to prioritise the quantitative aspects of our work.
    1. You might find appendix H interesting as we try to analyse and compare attribution ranking across methods there. One interesting observation is that the attention head a9.h6 in GPT-2 (a known name mover head) is ranked harmful by IG/EG/IntInv, but not MAttr. We think this could be related to the "hydra effect" (where components become helpful only if other ones are inactive) but didn't investigate further.
    2. A deeper qualitative investigation would involve trying to understand the roles individual components that are ranked highly by attribution play. One could use all sorts of more computationally expensive interp tools to do this once the important components have been found, e.g. logit/J-lens, max activating exemplars over a corpus, and targetted causal interventions. We'd love to explore this in future work.

On the CMD point, we will explain our reasoning on our focus on CPR more clearly in the final version of the paper and include more official leaderboard results.

I appreciate the authors response, and willingness to really engage with the rebuttal. They have addressed my points, and so I have increased my score accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.