- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a benchmark for evaluating attribution methods on transformers. The authors train small decoder-only models on four algorithmic tasks where it is known by construction which input positions can affect the output, and measure how much attribution mass falls on positions that cannot. They compare fourteen scoring rules across seeds and report findings on training dynamics during grokking, sparse parity, the choice of IG baseline, and the statistical significance of method rankings.
Critical strengths
-
Using algorithmic tasks to get exact input-level ground truth is a good idea. Benchmarks built on natural data rely on approximate relevance labels, whereas here relevance is known by construction. The proposed false-positive mass (FPM) does not need a threshold and is easy to interpret.
-
The paper is open about its limitations: it reports how many seeds failed and notes the resulting selection effect. The whole pipeline runs in about one GPU-day, and the code and models are released, so the benchmark should be easy to reuse.
Critical weaknesses
- Convergence of the training-dynamics results (Section 7).
I agree that FPM changes across checkpoints. But the claim that the parity gap "does not close" with more training is based on a short window after the transition. Parity is trained for 30k steps, and in Figure 2b the curves still seem to be going down at the last checkpoint. The raw-attention curve for modular addition in Figure 2a also seems to be still changing at the end, so the final values in Table 2 may not be stable.
- Limited model scale and architecture.
All models have only 2-3 layers, and inputs are 10-40 tokens long. Depth may affect layer-aggregating methods and gradient-based methods differently, so the ranking could change in deeper models. FPM also depends on sequence length. Scale could affect training dynamics as well; for example, parity takes longer to learn as the input gets longer, so the transitions in Section 7 may be specific to this setting. In addition, all models are decoder-only, while attribution is often applied to encoder-based classifiers, and the paper does not test whether the conclusions carry over. As the authors note, the full pipeline runs in about one GPU-day on a laptop GPU (Section 5), so these ablations seem feasible.
- Novelty of the IG baseline finding.
I am not sure this should be a main finding. IG is defined with respect to a baseline, so it is expected that changing the baseline changes the attributions a lot. Measuring how large the effect is with exact ground truth is still useful, but I think it should be framed as a quantification rather than a new result.
Questions for authors
-
Section 1 says only checkpoints that pass the accuracy threshold are evaluated, but Section 7 also evaluates intermediate checkpoints where held-out accuracy is at chance. How should FPM be interpreted at those checkpoints?
-
Do the parity and modular-addition FPM curves stabilize with longer training?
-
FPM drops mass on structural tokens before scoring. The paper reports that BOS mass is small, but I could not find the READOUT mass. Could you report it for all methods?
-
Do the method rankings hold for deeper models, longer inputs, or encoder-only architectures?