acceptodds
Under review as a conference paper at ICLR 2027

Attribution Evaluation Needs an Ablation, Not an Annotation

Abstract

Feature attributions are evaluated by comparing what they highlight against an annotation. We show that such comparisons do not measure what they are read as measuring. On sequences where ablation demonstrates that a model uses neither the annotated feature nor the region the attribution itself identifies, attributions still locate the annotation at 5–14× the empirically measured chance level. The effect holds across 15 transcription factors, two architectures and three attribution methods, disappears when the annotated interval is compositionally rewritten, and a variance decomposition puts  40% of the score's variance on sequence identity against under 2% on the choice of method. We then measure the contaminating term rather than infer it. Scoring a model against the annotation of a factor it was never trained on sets model usage to zero by construction, and what survives is measurable: 0.22× chance without training, 1.69× after training on the benchmark, 5.45× for a byte-pair architecture. Carrying the decomposition through the argmax the metric performs identifies both terms from measured recovery alone, and the usage term is stable at 3.22±0.26 across factors whose reported scores range from 0.31 to 0.58. The architecture that appears to produce better attributions confers more recovery on annotations it never learned, and shows less usage. The protocol reproduces the effect on protein binding sites, text rationales and natural textures at 1.5–2.2×, 2.25× and 11.2×, an ordering the decomposition predicts — the protein range spanning two negative-set designs, both far below the genomic effect; images are not exempt where a composition-preserving manipulation exists. Ten corrections to the score fail, and the identification says why. A rank-based alternative improves run-to-run reproducibility on every factor in all three sequence domains and costs less than the attribution it evaluates. We derive the runs a comparison needs — four paired, fourteen unpaired — and the second kind is what this literature reports from single runs.

Primary area:
interpretability and explainable AI
Keywords:
feature attribution, interpretability evaluation, ablation, reproducibility, XAI, genomics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WTiWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is concerned with the evaluation of attribution methods. Specifically, it argues that attribution evaluation should not be conducted based on ground-truth annotations but rather based on interventions to assess which parts of the input have been truly important for the model. To this end, the paper focuses on DNA sequence data as it allows for meaningful interventions. In brief, by intervening on windows of the sequence their influence on the model can be computed, which then can be used to evaluate attributions.

Critical strengths
  • I like the idea of using DNA sequence data for attribution evaluation.
  • I agree with the claim that annotations have limitations for attribution evaluation that ablations can solve.
Critical weaknesses
  • W1 The paper is very hard to read and seems to rely heavily on AI writing. Overall, the writing of the paper is very hard to understand for me and gives the strong impression of being AI generated with insufficient human revision. To give a few examples:
    • The abstract is full of empirical results instead of intuitively conveying the intuition or high-level idea behind the paper.
    • In Sec. 3.1, there is no intuition on what W∗W* is.
    • The paper uses a lot contrastive structures usually found in AI generated writing. E.g., L. 055 "The claim is about a class of metric, not about a domain.", L. 147: "What s is not.", and L. 151: "This is a decomposition assumption, not a derivation. It asserts additivity, not independence:".
    • The Table in L. 402 has no caption.
    • Figure 1c has overlapping text.
    • The main paper is lacking a proper discussion or conclusion.
  • W2 The entire premise of the paper is inprecise. The abstract and introduction both start with the claim that feature attributions are evaluated by comparing them to annotations. While it is true that there exist protocols doing so, there are various established evaluation practices that don't do that. For example, deletion-based approaches (see Samek et al., 2016 - Evaluating the visualization of what a deep neural network has learned and Hooker et al., 2019 - A benchmark for interpretability methods in deep neural networks). The paper is too broad in its claims. Further, the motivation (L. 041-048), i.e., that alignment-based evaluations of attribution maps do not consider faithfulness with respect to the model, is well established knowledge in the community (e.g., Nauta et al. 2023 - From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI) which is not really acknowledged.
  • W3 Insufficient discussion and conclusions. The evaluation relies on Assumption 1, which generally does not need to hold. The Limitations in the very end of the Appendix briefly acknowledge this, but in my opinion the main paper should more carefully discuss when this assumption holds and when not. There are various scenarious where the evaluation protocol is not faithful (which generally is okay in XAI evaluation but must be clearly communicated). Further, for an attribution evaluation paper, the actual evaluation of attribution methods is really sparse. I would have expected a table listing different attribution methods and how they perform, alongside established protocols to see if the novel evaluation yields additional value.
Questions for authors

From my current perspective, the writing and conclusions of the paper are in a state that cannot be sufficiently improved in a rebuttal. I therefore have no questions whose answers would materially affect my assessment.

1 comment

W1. Each specific point is addressed in the revised PDF. The three sentences at lines 55, 147 and 151 are rewritten. The abstract is rewritten; ten numbers remain, each attached to one claim. Section 3 now says in words what ss is before using it: the part of the attribution at a position that would be there whether or not the model used that position. A Conclusion section is added. The table at line 402 and four others now carry captions. The overlapping text in Figure 1(c) is corrected.

W2. The reviewer is right that the premise was stated too broadly. Deletion-based protocols (Samek et al. 2017; Hooker et al. 2019) evaluate attributions with no annotation, and the limits of annotation-based evaluation are documented (Nauta et al. 2023). The revised Section 1 opens with "commonly evaluated", Section 2 cites these works and confines the claim to the annotation-overlap class, and Section 2 now positions the paper against the works that construct the ground truth (BAM, FunnyBirds, Zhou et al. 2022). What the paper adds to the established point that overlap is not faithfulness is a measurement: the chance level of the overlap score on an annotation that cannot be constructed, obtained by scoring a model against a factor it never learned, and the decomposition that says what the score is made of.

W3. On Assumption 1: the revised Section 3.2 states it as an assumption, says what it asserts (additivity only; uu and ss may covary) and what would falsify it, and the general response reports the one test we could run (the simplex correction of Majdandzic et al. 2023, which leaves the excess recovery in place on low-dependence sequences and enlarges the floor). The cases where the protocol is not faithful are those where the ablation itself is uninformative, which Section 7.4 describes (SP1 and TBP, where the median drop is near zero), and the Conclusion now says this in the main text. On the evaluation of attribution methods: three methods are compared under both the overlap score and the proposed one (Table 4, Figure 3), and Appendix I adds a comparison on a constructed ordering (IG with 2, 8 and 32 integration steps) where the correct order is known. We agree that a broader table of methods would strengthen the paper and we have not added one in the revision.

Reviewer_QvLsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether annotation-based attribution metrics reflect the features actually used by a model. Through ablation experiments, the authors show that attributions can achieve high recovery scores even when the model does not rely on the annotated features. They further analyze the sources of this bias and propose an ablation-based evaluation metric.

Critical strengths

S1. The ablation-based evaluation provides an interesting perspective on assessing attribution faithfulnes

S2. The experiments cover multiple models, attribution methods, and datasets across different domains.

Critical weaknesses

W1. The cross-factor evaluation assumes that a model does not use motifs it was not trained on. However, different factors may share similar sequence features. Although the authors exclude highly similar motifs, this does not necessarily mean the model cannot use features from other factors. The validity of this assumption needs further justification.

W2. The paper assumes an additive decomposition of attribution scores and a Gumbel noise distribution. However, these assumptions are not directly validated. It is unclear how much the estimated model usage depends on them.

W3. The proposed Ablation-Prediction AUC does not work well across all tasks. It can only be computed for 11 of the 17 factors, and achieves an AUC of just 0.54 on the image texture task. Its applicability to other tasks remains unclear.

W4. In the image experiments, only 4 out of 10 training runs successfully learned the task, and the main results are based on these 4 runs. The experiments also use a constructed texture classification task rather than a standard object recognition task. It is unclear whether the findings would generalize to more realistic image classification settings.

W5. The abstract states that the attribution method accounts for less than 2% of the variance, but Table 4 reports 3.7% for GATA1.

W6. The abstract claims that the rank-based metric improves reproducibility across all factors, but Section 7.2 shows that it can only be computed for 11 of the 17 genomic factors. This limitation should be mentioned.

Questions for authors

Q1. In Appendix G, the chance level for the image task is set to 0.0625. However, the evaluation uses overlapping windows. Should the expected random recovery be higher when window overlap is considered?

Q2. Some labels overlap in Figure 1(c), making the figure difficult to read. The layout could be improved.

Q3. Please refer to weakness

1 comment

W1 (cross-factor assumption; shared features). The PWM screen excludes pairs whose motifs are similar; it does not address factors whose motifs differ but co-occur in the same windows, which is the reviewer's point. A model trained on A could learn B's motif if B's sites co-occur in A's positives. We have not measured this. The revision states it as a limitation of the cross-factor construction (Section 5.5), and we will report the frequency with which each benchmark factor's motif occurs in the positive windows of each other factor, with the floor recomputed on the pairs where that frequency is low. On the purpose-built dataset, where positives for different factors are drawn independently, the floor is 0.44×0.44\times and the low-dependence excess persists, so co-occurrence cannot be the whole account.

W2 (additivity and Gumbel not validated). Both are assumptions, and the revised Section 3.2 says so: additivity is stated as an assumption, with what it asserts (additivity only; uu and ss may covary) and what makes it testable; the Gumbel form fixes the link function. What depends on them is the identified usage term of Section 5.5 (the 3.22±0.263.22\pm0.26), and the diagnostic for that term is its stability across factors whose raw scores differ by a factor of two. The headline results, recovery on low-dependence sequences at 55--14×14\times chance and the floor, are measured and do not pass through either assumption. A direct validation on a synthetic task with planted, graded usage is the open test; we have not run it and the revision says so.

W3 (AUC: 11 of 17; 0.54 on textures). Both are stated in Section 7.2 and the abstract now agrees with them. On textures the AUC is at chance because usage is bimodal (a run uses the patch throughout or not at all), so the median threshold separates nothing; Appendix G explains this. On a constructed ordering (integration steps 2, 8, 32) the AUC orders the variants correctly on CTCF, GATA1 and MAX, and raw recovery orders them the same way, so the test shows the score does not fail on a known ordering and no more; Appendix I reports it with that reading.

W4 (image experiments). We agree. Four of ten runs learned the task, the appendix reports all ten and why the other six are not read (median ablation drop 0.000, so every example is nominally low-dependence). The revision removes the phrasing in Sections 1 and 5.6 that presented the texture result as more than a demonstration that the protocol runs on images where a composition-preserving manipulation exists; an object-recognition setting needs such a manipulation, which is the gap the revised contribution 4 names.

W5, W6 (abstract). Both correct; traced in the general response. The abstract transposed two bounds (method share under 4%, run share under 2%) and omitted the 11-of-17 restriction. It is rewritten.

Q1 (chance level for the image task). The reviewer is right, and we thank them for it. The score is the overlap of the annotation with the highest-summing 32×3232\times32 window, so the chance level is that of a uniformly placed window, π0=(32/97)2=0.109\pi_0 = (32/97)^2 = 0.109, the two-dimensional form of the L/(N−L+1)L/(N-L+1) we use for sequences; 322/1282=0.062532^2/128^2 = 0.0625 is the chance level for a random pixel, which the submitted version used by mistake. Every multiple in Appendix G and Section 5.6 is divided by 1.74: the diagonal is 6.70±0.186.70\pm0.18 (was 11.67), the floor 3.73±0.073.73\pm0.07 (6.49), recovery on low-dependence examples 6.44±0.496.44\pm0.49 (11.21), the shuffle 1.34±0.261.34\pm0.26 (2.34), and the untrained model 0.950.95 (1.66), now at chance. The ordering and the reading are unchanged: the floor remains 56% of the diagonal, low-dependence recovery remains indistinguishable from the diagonal, and the shuffle brings recovery to near chance. The revised appendix states the correction.

Q2 (Figure 1(c)). The figure is regenerated and the revised.

Reviewer_mct1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that annotation-based attribution metrics mix together three things: how easy a feature is to find, whether the model uses it, and whether the attribution is accurate. In TF-binding prediction with HyenaDNA and DNABERT-2, attributions recover motifs at 5 to 14 times chance even when ablation indicates that the model does not use them. The authors measure this contamination with a cross-factor "floor": 0.22× for the untrained model, 1.69× for the trained model, and 5.45× for DNABERT-2. They use a Gumbel-based decomposition to separate usage from salience and propose a rank-based AUC and a power analysis for the number of runs.

Critical strengths

The controls are careful. They include matched and symmetric ablations, a verified chance level, and a reliability estimate. I found the cross-factor floor particularly informative.

Critical weaknesses
  1. The paper needs to place its contribution more clearly in the existing literature. Its core argument is the plausibility/faithfulness distinction discussed by Jacovi & Goldberg (2020). Several uncited studies already evaluate attributions against measured or controlled model reliance, including BAM (Yang & Kim, 2019), Zhou et al. (AAAI 2022), ROAR (Hooker et al., 2019), and FunnyBirds (ICCV 2023). FunnyBirds also contradicts the claim that object images cannot be manipulated. The paper omits Majdandzic et al. (2022, 2023) on spurious genomic attributions as well. I see the floor and Proposition 1 as the novel contributions.

  2. The identification in Proposition 1 depends on a fragile assumption: α∆ must be equal at the diagonal and the floor, even though s changes with training. The DNABERT-2 usage term is 1.50 ± 1.40, which leaves considerable uncertainty.

  3. The interpretation of s is unclear. Uniform placement lowers the floor from 1.69× to 0.44×, suggesting that a positional prior accounts for much of the effect.

  4. The evidence does not support the breadth of the generality claims. The image experiments use one texture task, which only 4/10 seeds learned. The protein result is not significant under one design, and the effect holds for 10/15 factors.

  5. The proposed rank AUC still needs validation. It cannot be computed for 6/17 factors, fails on textures, and has not been shown to rank attribution methods correctly.

  6. Some reported numbers disagree. Table 1 and Fig. 2 give 15.05 and 10.5, respectively, while the protein ablation drop is reported as both 0.558 and 0.538.

Questions for authors
  1. What does the floor reveal that BAM, ROAR, or FunnyBirds cannot?
  2. Can you validate Proposition 1 in a setting where usage is known by construction?
  3. How much of the floor comes from a positional prior?
  4. Can the rank AUC recover a known ordering of attribution methods?
1 comment
AuthorsOpenReview ↗

Response to Reviewer mct1

Q1. What does the floor reveal that BAM, ROAR or FunnyBirds cannot? Those three construct the ground truth, so the model's use of the feature is known by construction. The floor is for the case where the ground truth is measured and cannot be constructed, such as a motif determined by ChIP-seq: it gives the chance level of the overlap score itself, how often an attribution lands on an annotation the model never learned. BAM and FunnyBirds supply no such number for a measured annotation; ROAR evaluates faithfulness by retraining and does not evaluate an overlap score. The revised Section 2 adds a paragraph on these works (and Zhou et al. 2022, Jacovi & Goldberg 2020, Majdandzic et al. 2023), corrects our statement about object images (FunnyBirds supplies both the chance level and the manipulation, by construction; Section 1 now says "natural object photographs"), and states this positioning.

Q2. Can Proposition 1 be validated where usage is known by construction? Partly, with what we have. The texture task of Appendix G is such a setting: a 32×3232\times32 patch is placed into a background, and its usage is known from the ablation to be either near zero (six of ten runs) or near 0.5 (four runs). The identification returns a usage term near zero in the first group and positive in the second, the direction the construction fixes; the bimodality means the task cannot test the magnitude. A full test needs a synthetic sequence task with graded, planted usage, which we have not run; we state this as the open validation of Proposition 1 in the revised Section 3.2.

Q3. How much of the floor comes from a positional prior? About seventy percent on the benchmark. Uniform placement lowers the floor from 1.5×1.5\times to 0.44×0.44\times, so the benchmark's centre bias supplies the difference; the revised Section 5.5 says so in these words (the submitted "three quarters" was computed from a floor of 1.69 that the recomputation described in the general response replaces with 1.52±0.281.52\pm0.28). The claim does not rest on the magnitude. With the positional prior removed, recovery on low-dependence sequences is still 1.51.5--9.7×9.7\times chance in ten of twelve factors, and the diagonal-to-floor ratio is 29 against 11 on the benchmark. What the prior adds is a higher floor, which is the reason Section 5.5 prescribes measuring the floor on each dataset.

Q4. Can the rank AUC recover a known ordering? We constructed one by varying the number of integration steps of IG (2, 8, 32), which degrades the Riemann approximation monotonically. The AUC at the median threshold is 0.55, 0.67 and 0.75 on CTCF, 0.50, 0.63 and 0.63 on GATA1, and 0.50, 0.62 and 0.71 on MAX: it separates two steps from the rest on all three factors and separates eight from thirty-two on CTCF and MAX but not on GATA1. Raw recovery orders the three variants the same way (CTCF 0.05, 0.49, 0.59; GATA1 0.01, 0.41, 0.50; MAX 0.00, 0.22, 0.34), so this shows that the score does not fail on a known ordering, and no more. The revised Appendix I reports it with that reading.

Proposition 1. Three things are assumed: additivity (Assumption 1), Gumbel window noise, and that αΔ/σ\alpha\Delta/\sigma is the same at the diagonal and the floor. The third is the one the reviewer identifies, and it is the assumption under test: the identified usage term is 3.22±0.263.22\pm0.26 on HyenaDNA across factors whose reported scores span 0.31--0.58, and 1.50±1.401.50\pm1.40 on DNABERT-2. We report the second because it is informative: where the term is unstable, the model does not use the motif, and the composition screen reaches the same verdict independently. We do not claim the identification holds everywhere; we claim it holds where its own diagnostic says it does, and the revised Section 3.2 states additivity as an assumption, what it asserts (additivity only; uu and ss may covary) and what makes it testable.

Generality. We agree the claims were wider than the evidence. The revision confines the texture result to a demonstration that the protocol runs where a composition-preserving manipulation exists (four of ten runs learned the task; all ten are reported), keeps the "n.s." on the protein design where the ratio of 1.5 is not significant, and reports the five factors where the effect is absent: three between chance and 5×5\times, and two below chance (TFAP2C 0.7, ZNF143 0.5), which have the highest attribution-collapse rates (Section 5.4).

Rank AUC. Computable on eleven of seventeen factors, at chance on textures for the reason Appendix G gives (usage is bimodal there, so the median threshold separates nothing), and tested on a constructed ordering in Q4. The abstract now says eleven of seventeen.

Numbers. Table 1 vs. Figure 2(b) and 0.558 vs. 0.538 are traced and corrected in the general response.

Comments on the paper

3 comments

We thank the reviewers for a careful reading. This response covers three points that several reviews raise; each review is then answered separately. A revised paper accompanies it, and every change is listed in part 2 of this comment.

Numerical inconsistencies. We traced each of the four to its source.

Abstract "under 2% on the choice of method" vs. Table 4 (GATA1 3.7%). The abstract transposed two bounds that Section 5.2 states correctly: the method share of variance is under 4% and the run share under 2%. The abstract is rewritten. Abstract "on every factor" vs. 11/17. The rank-based score is computable on eleven of seventeen factors; Section 7.2 states which six are excluded and why. The abstract now says eleven of seventeen. Protein ablation drop 0.558 (Section 5.6) vs. 0.538 (Appendix F). The three seeds give 0.369, 0.511 and 0.538; the Section 5.6 value was a transcription error, now 0.369–0.538. Table 1 diagonal 15.05 vs. Figure 2(b) 10.5. These were two aggregations of the same cross-factor runs. The figure showed the five-factor mean from one run; the table pooled every available seed-by-factor diagonal entry, which weights factors by how many runs were available for them (SP1 and TBP have one run, the other three have five). Neither is the quantity the text discusses. The revised Table 1 reports the diagonal as the mean over the three factors on which the identification of Section 3 is defined (CTCF, GATA1, MAX: 17.1×) and states SP1 and TBP in the caption (0.31× and 0.30×), as Figure 2(a) already showed them. The floor had the same coverage problem; it is now computed on the eight model–data pairs present in all five runs, 1.52 ± 0.28 over runs (previously 1.69 ± 0.38 over a changing set of pairs). The dependent sentences change accordingly: the diagonal-to-floor ratio on the benchmark is 11 (was 9), the positional regularity of the dataset accounts for about seventy percent of the contamination (was "three quarters"), and the DTD floor is 9% of its diagonal (was 11%). The same recomputation corrected three entries in the floor column of the appendix table of per-factor identification (MAX, SP1, TBP). No conclusion changes.

Prior work. The reviewers are right that BAM (Yang & Kim, 2019), FunnyBirds (Hesse et al., 2023), Zhou et al. (2022), ROAR (Hooker et al., 2019), Majdandzic et al. (2023) and Jacovi & Goldberg (2020) belong in Section 2, and that our statement that object images "supply neither" a chance level nor a manipulation is wrong for synthetic datasets: FunnyBirds supplies both by construction. The revised Section 2 adds a paragraph that positions this work against them. The distinction we rely on, between annotations that can be constructed and annotations that must be measured, is theirs. What these works do not supply is a chance level for the overlap score on an annotation that cannot be constructed, such as a motif determined by experiment, and that is what the cross-factor floor measures. Section 1 now says "natural object photographs"; Section 5.6 is qualified the same way.

Simplex correction (Majdandzic et al., 2023). We ran it. HyenaDNA takes token embeddings, so the one-hot projection does not apply directly; the analogous correction averages integrated gradients over the four single-nucleotide baselines, which places the baseline at the centroid of the four embeddings and removes the component common to them. With the original [PAD] baseline the modified pipeline reproduces every reported value exactly. Under the correction, recovery rises on every factor, and it rises more on the floor than on the diagonal: the diagonal on CTCF goes from 15.2× to 18.2× chance, the mean floor from 1.31× to 3.04× (17 of 20 model–data pairs increase), and the diagonal-to-floor ratio over CTCF, GATA1 and MAX falls from 14 to 7.5. On held-out sequences the model does not use (ablation-measured usage below 0.05), recovery is unchanged on CTCF and GATA1 (paired change +0.018 [−0.025, +0.062], n = 288; −0.008 [−0.098, +0.086], n = 91) and rises on MAX, from 8.3× to 14.1× chance (+0.115 [+0.080, +0.153], n = 591). The excess recovery the paper reports is therefore not a property of the off-simplex gradient component: removing that component leaves it in place where it does not raise it, and raises it on annotations the model never learned. The new Appendix I reports all five factors and the ablation-prediction AUC under the correction (0.72 on CTCF, 0.70 on GATA1 and 0.75 on MAX at the median threshold, against 0.75, 0.63 and 0.71 with the [PAD] baseline).

Changes in the revised PDF. Abstract (rewritten; the two corrections above). Section 1 (natural object photographs; contribution 4 reworded). Section 2 (new paragraph, six references). Section 3.2 (the dependence of s on training stated as a paragraph; additivity named as an assumption and what it does and does not assert). Section 5.1 (raw recovery stated as such). Section 5.5 (Table 1 and the dependent sentences). Section 5.6 (0.369–0.538; the DTD comparison table; qualification of the image result). Appendix C (floor values; three entries of the identification table). Captions added to five previously uncaptioned tables. A Conclusion section. Appendix I (simplex correction; constructed ordering for the rank-based score). Figure 1(c) relaid out. Three sentences rewritten at the lines the reviewer cites. No result or claim is changed except as stated under item 4.

I would like to remind the authors that they can post a rebuttal version of their paper using "Edit > Rebuttal version" on this page. I encourage them to incorporate all the changes they have made during the rebuttal period into the paper, and to highlight any pertinent changes they would like reviewers to note using coloured text, margin notes, or the changebar package. This will make it easier for reviewers to see how the paper has ended up as a result of their changes, and for the AC to assess both the final version of the paper and how it was changed during the rebuttal period.

Rebuttal versions can be posted multiple times (you do not need to only post one version) and you can not post a rebuttal version after the author-reviewer discussion period closes at the end of 18 Nov.

When submitting the rebuttal version, please also fix the minor style changes changes from the template (missing ICLR header).

Thanks, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.