- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a new attribution and circuit extraction method (MAttr). The method incorporates several improvements, most notably a Matryoshka method that allows them to learn circuits of multiple sizes simultaneously (by setting a random top-k budget in each learning step), and sigmoid-based top-k masks. The top-k masking avoids requiring an explicit sparsity loss, and the sigmoid-based mask allows for differentiation. The paper compares the performance of MAttr to state of the art methods such as EAP and Integrated Gradients, and finds that MAttr performs significantly better. The paper applies the method to Llama-3.1-8B-Instruct, showing which weights are responsible for refusal, and removing refusal by replacing those weights with base model weights.
Critical strengths
The method outperforms baselines on the highly relevant task of circuit extraction.
The algorithm is a simple iterative method learning continuous mask scores.
A single run of MAttr provides circuits at all sparsity levels which eliminates a major hyperparameter that had to be tuned for previous circuit extraction methods.
Critical weaknesses
The method's success relies on the Faithfulness (CPR) metric, and scores substantially closer to baselines on the Compactness metric. The CPR metric’s faithfulness is unbounded: It can be larger than 1 if the circuit achieves a better / higher logit difference than the full model. This is a known issue in circuit analysis. Additionally CPR calculates the area under the curve on a linear x-axis which gives little weight to small circuits. The paper acknowledges the issues with CPR and shows the Compactness metric to account for them. However, MAttr performs similarly to Integrated Gradients (IG) on the Compactness metric. This is particularly surprising given that MAttr’s first step is equivalent to centred IG (under SGD, in expectation, as the paper states), so I would expect MAttr to significantly improve over IG.
Questions for authors
Why doesn’t MAttr beat IG on Compactness, especially considering MAttr’s first step similar to IG?
Is MAttr’s very good performance on the CPR metric due to the unbounded logit difference, or due to the linear x-axis scaling? Can you quote metrics with faithfulness capped to 1, and with logarithmic x-axis scaling (separately)?
2 comments
Here is the log-AUC of normalised logit difference, as a log counterpart to CPR. The results are node-level (attention heads + MLPs), on Llama for SVA, Arithmetic and ARC-E, and Qwen for IOI. SVA and Arithmetic are averaged over 4 subtasks each.
| Method | SVA | Arithmetic | ARC-E | IOI |
|---|---|---|---|---|
| MAttr | 0.67 | 0.73 | 1.60 | 0.96 |
| IG | 0.60 | 0.57 | 0.60 | 0.77 |
| Expected Gradients | 0.60 | 0.56 | 0.61 | 0.78 |
| Attribution patching (I×G) | 0.55 | 0.03 | 0.60 | 0.26 |
| AttnLRP | 0.56 | 0.47 | 0.51 | 0.61 |
| Node Pruning | 0.46 | 0.47 | 0.60 | 0.67 |
| DBM | 0.55 | 0.47 | 0.50 | 0.63 |
| Random | 0.04 | 0.03 | 0.05 | 0.07 |
Additionally, here is the linear-scale version of Compactness, reported on the same setting:
| Method | SVA | Arithmetic | ARC-E | IOI |
|---|---|---|---|---|
| MAttr | 0.971 | 0.963 | 0.982 | 0.953 |
| IG | 0.967 | 0.924 | 0.978 | 0.937 |
| Expected Gradients | 0.968 | 0.901 | 0.978 | 0.934 |
| Attribution patching (I×G) | 0.956 | 0.132 | 0.973 | 0.655 |
| AttnLRP | 0.960 | 0.882 | 0.970 | 0.886 |
| Node Pruning | 0.928 | 0.920 | 0.871 | 0.905 |
| DBM | 0.960 | 0.816 | 0.899 | 0.879 |
| Random | 0.190 | 0.141 | 0.249 | 0.251 |
MAttr is the best-performing method in both cases.
Thank you for your thoughtful review! On your questions: