acceptodds
Under review as a conference paper at ICLR 2027

How Much Attribution Mass Lands on Tokens That Cannot Matter? A Faithfulness Benchmark for Transformers

Abstract

Attribution methods for transformers are usually judged by whether their heatmaps look plausible to a human reader. We take a more direct route. We train small decoder-only transformers on four algorithmic tasks (sparse parity, indexed retrieval, associative recall, and modular addition with distractors) in which every input position is, by construction, either causally relevant or provably unable to affect the output. Any attribution mass landing on a provably irrelevant position is an error, with no threshold, reference model, or human judgment involved in that verdict. Under one pre-registered protocol we evaluate fourteen scoring rules, and every number carries a -based confidence interval over 8 to 11 independently trained seeds. Our most surprising finding concerns training dynamics: through the grokking transition on modular addition, the faithfulness of attention-based methods degrades sharply and becomes far more seed-dependent, while test accuracy rises to 100%. Explanations of the better model are worse. Second, cheap attribution and causal measurement diverge exactly where feature interactions matter: on sparse parity a resample-ablation oracle reaches 0.031 false-positive mass while every attribution method stays above 0.49, a gap that rank-preserving sharpening cannot close, so the failure is misidentification rather than diffuse scoring. Third, protocol choices can outweigh the choice of method: on the three tasks with first-order structure, moving integrated gradients from a marginal-mean baseline to a zero baseline raises false-positive mass by factors of 2 to 22. Fourth, paired tests show the top two methods are statistically indistinguishable on two of four tasks, so single-seed rankings among mid-ranked methods carry little information. We release the task generators, model zoo, attribution harness, and evaluation suite.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, explainability, transformers, attribution, faithfulness, mechanistic interpretability, causal inference, attention, feature attribution, evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XpurOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a benchmark for evaluating attribution methods on transformers. The authors train small decoder-only models on four algorithmic tasks where it is known by construction which input positions can affect the output, and measure how much attribution mass falls on positions that cannot. They compare fourteen scoring rules across seeds and report findings on training dynamics during grokking, sparse parity, the choice of IG baseline, and the statistical significance of method rankings.

Critical strengths
  1. Using algorithmic tasks to get exact input-level ground truth is a good idea. Benchmarks built on natural data rely on approximate relevance labels, whereas here relevance is known by construction. The proposed false-positive mass (FPM) does not need a threshold and is easy to interpret.

  2. The paper is open about its limitations: it reports how many seeds failed and notes the resulting selection effect. The whole pipeline runs in about one GPU-day, and the code and models are released, so the benchmark should be easy to reuse.

Critical weaknesses
  1. Convergence of the training-dynamics results (Section 7).

I agree that FPM changes across checkpoints. But the claim that the parity gap "does not close" with more training is based on a short window after the transition. Parity is trained for 30k steps, and in Figure 2b the curves still seem to be going down at the last checkpoint. The raw-attention curve for modular addition in Figure 2a also seems to be still changing at the end, so the final values in Table 2 may not be stable.

  1. Limited model scale and architecture.

All models have only 2-3 layers, and inputs are 10-40 tokens long. Depth may affect layer-aggregating methods and gradient-based methods differently, so the ranking could change in deeper models. FPM also depends on sequence length. Scale could affect training dynamics as well; for example, parity takes longer to learn as the input gets longer, so the transitions in Section 7 may be specific to this setting. In addition, all models are decoder-only, while attribution is often applied to encoder-based classifiers, and the paper does not test whether the conclusions carry over. As the authors note, the full pipeline runs in about one GPU-day on a laptop GPU (Section 5), so these ablations seem feasible.

  1. Novelty of the IG baseline finding.

I am not sure this should be a main finding. IG is defined with respect to a baseline, so it is expected that changing the baseline changes the attributions a lot. Measuring how large the effect is with exact ground truth is still useful, but I think it should be framed as a quantification rather than a new result.

Questions for authors
  1. Section 1 says only checkpoints that pass the accuracy threshold are evaluated, but Section 7 also evaluates intermediate checkpoints where held-out accuracy is at chance. How should FPM be interpreted at those checkpoints?

  2. Do the parity and modular-addition FPM curves stabilize with longer training?

  3. FPM drops mass on structural tokens before scoring. The paper reports that BOS mass is small, but I could not find the READOUT mass. Could you report it for all methods?

  4. Do the method rankings hold for deeper models, longer inputs, or encoder-only architectures?

Reviewer_6VrYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study addresses the lack of confound-free, input-level ground truth to evaluate transformer attribution methods. Deletion-based metrics push inputs off-distribution, natural-data benchmarks rely on approximate labels, and mechanistic benchmarks (Tracr, InterpBench) give ground truth for circuit components rather than input tokens. To address this concern, the paper trains small transformers on four algorithmic tasks in which every input position is by construction either relevant or provably irrelevant, and proposes false-positive mass (FPM), the fraction of attribution mass on irrelevant positions. 14 scoring rules are evaluated under one fixed protocol on 8–11 seeds per task that meet a held-out accuracy threshold, using t-based confidence intervals and paired tests. The paper reports that (i) on modular addition, attention-based FPM rises from about 0.01 to 0.2–0.5 through grokking, (ii) on sparse parity, a resample-ablation oracle reaches 0.031 FPM while every attribution method stays above 0.49, (iii) switching the IG baseline from the marginal mean to zero changes FPM by 2–22× on three tasks, and (iv) the top two methods are statistically indistinguishable on two of four tasks.

Critical strengths
  1. The paper addresses a well-motivated problem and provides exact ground truth at the input token level. Extant faithfulness evaluation avoids circularity in two ways. However, both introduce a confound: deletion pushes inputs off-distribution, and natural-data benchmarks approximate their labels. Mechanistic benchmarks (Tracr and InterpBench) provide ground truth at the level of heads and circuit components, but not at the input token level. Here, irrelevance is held fixed by the task definition, and a unit test in the released code confirms that changing any irrelevant position never changes the target. Consequently, FPM is an error rate in itself and needs no threshold, reference model, or human judgment.

  2. The paper's design is careful to ensure that task-level ground truth also holds at the model level. Causal importance is measured by resample ablation rather than zero or mean ablation, which avoids pushing inputs off-distribution. Only models that pass a held-out accuracy filter are scored, thereby excluding models that may rely on shortcuts unrelated to the task. The oracle (parity FPM 0.031) and Figure 1 show that the models depend on the relevant positions; thus, failures can be attributed to the explanation methods rather than to the models. The paper also separates tasks into those with a fixed, relevant set and those where it changes per example, which limits how much can be read from fixed-set tasks, in which a positional prior alone achieves FPM 0.

  3. The paper presents controlled experiments to defend its key claims against alternative explanations. The sharpening analysis applies a rank-preserving transform that concentrates mass on top-ranked tokens. This analysis distinguishes whether the parity failure stems from diffuse mass or from wrong token selection. Even at γ = 8, the best method remains 76× above the oracle, and some methods worsen, confirming that the failure is attributed to misidentification. In addition, the proposed failure mechanism is tested through a prediction. Since each bit’s effect flips sign in XOR, saliency, discarding sign, should beat gradient×input, which multiplies by it, and it does (0.496 vs. 0.729). Moreover, the attention sink (BOS) is measured separately rather than counted as an error, and both IG baselines are reported, thereby avoiding a comparison that favors a particular setting.

  4. The statistical design is much more rigorous than usual in this discipline. The results are also reproducible. Every number carries a t-based interval over 8–11 seeds, and between-seed and between-example variance are reported separately. Methods are compared using paired tests on identical examples and checkpoints. Strong trivial baselines, such as a positional prior, are included. This design directly bolsters practical conclusions: the top two methods are statistically indistinguishable on parity and recall, and on modular addition, between-seed standard deviations (up to 0.35) exceed most gaps between methods, so single-seed rankings are unreliable. The authors also report the number of failed seeds as an outcome and release those models, making the selection effect transparent. I recomputed Tables 2, 3, 4, and 6 and the paired t-tests from the released per-seed result files, and they match the paper.

Critical weaknesses
  1. The headline training-dynamics result is concentrated in a few seeds and the final checkpoint interval. Its alignment with grokking is not shown. In the released per-seed results, the last-layer attention becomes uniform (FPM 0.800) in only 4 of 9 seeds, whereas the other five range from 0.03 to 0.61. Hence, the reported mean of 0.474 ± 0.272 is a mixture of two groups rather than a gradual degradation. In these four seeds, FPM stays at 0.05–0.24 through step 15,686 and jumps only at the final 50k checkpoint. Gradient×input rises over the same interval. As Figure 2 doesn't show held-out accuracy, it is not clear as of whether this jump coincides with grokking. Also, the proposed Fourier-based explanation has not been tested. Figure 2 with held-out accuracy overlaid, per-seed trajectories, and I think that per-layer FPM should be provided in the rebuttal.

  2. FPM does not penalize missed relevant tokens. Putting all its mass on one relevant token scores FPM = 0. On retrieval, the two best methods by FPM (attention×grad 0.003, raw attention 0.007) have precision@2 of 0.54 and 0.56 in the released per-seed results; thus, they miss one of the two relevant tokens. In this task, the FPM ranking runs in the opposite direction to the AUROC ranking (ρ = −0.44) and the causal-Spearman ranking (ρ = −0.47), both computed from Tables 2, 3, and 6. Therefore, the statements that rank metrics “do not reverse” the picture, that causal Spearman “ranks methods consistently with it,” and that raw attention “can be trusted on retrieval-style circuits” do not hold on the retrieval. It would be great to see a complementary metric for missed tokens reported alongside FPM.

  3. The method comparison brings protocol confounds. First, in the released code, the IG zero baseline sets the positional embedding to zero, whereas the marginal baseline keeps it. By this, the “factors of 2 to 22” may in part reflect the removal of positional information. Secondly, KernelSHAP uses a single baseline per example. On parity, about half of the examples have the same XOR output for the input and the baseline; thus, all Shapley values are zero, and FPM falls to the uniform value (0.925). In a simulation with an idealized parity model under the same protocol, FPM is 0.47 with exact Shapley values and 0.53 with 256-sample KernelSHAP, close to the paper’s 0.593. Without a position-preserving IG baseline and a multi-baseline KernelSHAP, it is hard to generalize the paper's claims that “every attribution method stays above 0.49” and that “the baseline matters more than the method”.

  4. It is more likely to misinterpret the validation statistics. The task–model agreement in Table 4 (0.464–0.696) is equal to the maximum Spearman correlation attainable against binary labels with ties, given the number of relevant positions in each task. Hence, agreement is essentially perfect, yet the text reads it as “low enough to justify measuring both.” Furthermore, the stated reason for skipping multiple-comparison correction (“every conclusion either clears p < 10⁻³ or fails to reach 0.05”) is not consistent with the paper’s own reported p = 2.2×10⁻³, although the conclusions still hold after correction.

Questions for authors

See my comments in the Critical Weaknesses.

Reviewer_BdX5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines attribution mass on transformers. The authors contribute a novel benchmark, a set of experiments on four tasks (sparse parity, indexed retrieval, associative recall, and modular addition with distractors) showing shifts in faithfulness during training, and metrics with statistical results. Their findings suggest that attention-based methods degrade during training and become seed-dependent. Their benchmark, code, and evaluations are released in the supplementary material.

Critical strengths
  1. A clear strength of the paper is the contribution of the benchmark task generator. In the way they have constructed it, the input positions are known so that they can directly measure whether attribution methods assign importance to irrelevant tokens.
  2. Another clear strength of the paper is the evaluation protocol across seeds with the relevant statistical tests.
Critical weaknesses
  1. The grokking results and claims are perhaps a bit too strong. Firstly, grokking is a key part of the paper, but it's not defined in the paper (I had to look that up). The paper does not clearly show when grokking occurs for each seed or how the attribution changes align with that transition.
Questions for authors
  1. How is the grokking transition identified for individual training runs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.