- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper is concerned with the evaluation of attribution methods. Specifically, it argues that attribution evaluation should not be conducted based on ground-truth annotations but rather based on interventions to assess which parts of the input have been truly important for the model. To this end, the paper focuses on DNA sequence data as it allows for meaningful interventions. In brief, by intervening on windows of the sequence their influence on the model can be computed, which then can be used to evaluate attributions.
Critical strengths
- I like the idea of using DNA sequence data for attribution evaluation.
- I agree with the claim that annotations have limitations for attribution evaluation that ablations can solve.
Critical weaknesses
- W1 The paper is very hard to read and seems to rely heavily on AI writing. Overall, the writing of the paper is very hard to understand for me and gives the strong impression of being AI generated with insufficient human revision. To give a few examples:
- The abstract is full of empirical results instead of intuitively conveying the intuition or high-level idea behind the paper.
- In Sec. 3.1, there is no intuition on what is.
- The paper uses a lot contrastive structures usually found in AI generated writing. E.g., L. 055 "The claim is about a class of metric, not about a domain.", L. 147: "What s is not.", and L. 151: "This is a decomposition assumption, not a derivation. It asserts additivity, not independence:".
- The Table in L. 402 has no caption.
- Figure 1c has overlapping text.
- The main paper is lacking a proper discussion or conclusion.
- W2 The entire premise of the paper is inprecise. The abstract and introduction both start with the claim that feature attributions are evaluated by comparing them to annotations. While it is true that there exist protocols doing so, there are various established evaluation practices that don't do that. For example, deletion-based approaches (see Samek et al., 2016 - Evaluating the visualization of what a deep neural network has learned and Hooker et al., 2019 - A benchmark for interpretability methods in deep neural networks). The paper is too broad in its claims. Further, the motivation (L. 041-048), i.e., that alignment-based evaluations of attribution maps do not consider faithfulness with respect to the model, is well established knowledge in the community (e.g., Nauta et al. 2023 - From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI) which is not really acknowledged.
- W3 Insufficient discussion and conclusions. The evaluation relies on Assumption 1, which generally does not need to hold. The Limitations in the very end of the Appendix briefly acknowledge this, but in my opinion the main paper should more carefully discuss when this assumption holds and when not. There are various scenarious where the evaluation protocol is not faithful (which generally is okay in XAI evaluation but must be clearly communicated). Further, for an attribution evaluation paper, the actual evaluation of attribution methods is really sparse. I would have expected a table listing different attribution methods and how they perform, alongside established protocols to see if the novel evaluation yields additional value.
Questions for authors
From my current perspective, the writing and conclusions of the paper are in a state that cannot be sufficiently improved in a rebuttal. I therefore have no questions whose answers would materially affect my assessment.
W1. Each specific point is addressed in the revised PDF. The three sentences at lines 55, 147 and 151 are rewritten. The abstract is rewritten; ten numbers remain, each attached to one claim. Section 3 now says in words what s is before using it: the part of the attribution at a position that would be there whether or not the model used that position. A Conclusion section is added. The table at line 402 and four others now carry captions. The overlapping text in Figure 1(c) is corrected.
W2. The reviewer is right that the premise was stated too broadly. Deletion-based protocols (Samek et al. 2017; Hooker et al. 2019) evaluate attributions with no annotation, and the limits of annotation-based evaluation are documented (Nauta et al. 2023). The revised Section 1 opens with "commonly evaluated", Section 2 cites these works and confines the claim to the annotation-overlap class, and Section 2 now positions the paper against the works that construct the ground truth (BAM, FunnyBirds, Zhou et al. 2022). What the paper adds to the established point that overlap is not faithfulness is a measurement: the chance level of the overlap score on an annotation that cannot be constructed, obtained by scoring a model against a factor it never learned, and the decomposition that says what the score is made of.
W3. On Assumption 1: the revised Section 3.2 states it as an assumption, says what it asserts (additivity only; u and s may covary) and what would falsify it, and the general response reports the one test we could run (the simplex correction of Majdandzic et al. 2023, which leaves the excess recovery in place on low-dependence sequences and enlarges the floor). The cases where the protocol is not faithful are those where the ablation itself is uninformative, which Section 7.4 describes (SP1 and TBP, where the median drop is near zero), and the Conclusion now says this in the main text. On the evaluation of attribution methods: three methods are compared under both the overlap score and the proposed one (Table 4, Figure 3), and Appendix I adds a comparison on a constructed ordering (IG with 2, 8 and 32 integration steps) where the correct order is known. We agree that a broader table of methods would strengthen the paper and we have not added one in the revision.