acceptodds
Under review as a conference paper at ICLR 2027

Accuracy, Architecture, or Method: What Determines XAI Quality in Vision Models?

Abstract

Explainable Artificial Intelligence (XAI) is an emerging topic which aims at making opaque AI models’ decisions transparent and understandable. Various methods have been proposed to explain deep learning models across different domains. What still remains an ongoing debate is the question on how the explanation quality of XAI methods should be evaluated. Therefore, numerous quantitative metrics have been suggested which are supposed to measure certain explanation properties such as localization, faithfulness, complexity and randomization. Previous studies, mostly using off-the-shelf pretrained backbones, have indicated that explanations might be influenced by three factors: the model architecture, the model performance, and the XAI method. In addition, the interplay between these three factors has not been investigated yet. Our study contributes to this research gap by determining how much these three factors actually impact the nominal explanation quality of plain image classifiers. We therefore train 26 different Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) from scratch, spanning model sizes from 3.4M to 210M parameters. In our study we use the same training parameters on a 50-class ImageNet subset with pixel-level object masks, and score five XAI methods with twelve evaluation metrics at eight checkpoints per model, covering accuracies from 16% to 91%. From our image classification scenario three results emerge: 1. Accuracy has the largest impact on localization. On the other hand, complexity, faithfulness and randomization metrics are mainly influenced by the XAI method and do not significantly improve as the model learns. 2. With ongoing training, perturbation-based XAI methods tend towards outperforming gradient-based methods on transformers regarding most metrics we measure. The opposite behaviour can be observed for CNNs. 3. Within a model architecture it are the normalization layers that have the highest impact on localization metrics. Simply increasing the number of model parameters, however, does not yield measurable improvements regarding any XAI metric.

Primary area:
interpretability and explainable AI
Keywords:
Explainable Artificial Intelligence, Evaluation, Localization, Faithfulness, Robustness, Randomization, Complexity, Computer Vision

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qba2OpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether explanation quality in image classification depends mainly on model accuracy, architecture or XAI method. The authors train 26 CNNs, vision transformers and hybrid architectures from scratch on ImageNet-S50 under one training setup and evaluate five attribution methods with twelve performance metrics at eight checkpoints per model. They report that accuracy drives localization, while faithfulness, complexity and randomization depend mostly on the XAI method. They also find that perturbation methods overtake gradient methods on transformers during training but not on CNNs, and that normalization layers affect explanations more than model size.

The paper addresses a recent and important topic. I have, however, strong doubts on the validity of the paper with respect to AI generation artifacts, see below, for which reason I recommend a rejection.

Critical strengths

The study design is the paper’s main strength: training all 26 backbones from scratch under one regime and scoring explanations along the full training trajectory separates the effect of accuracy from that of architecture, which comparisons of converged off-the-shelf models cannot do. The choice of a multi-category metric suite, together with an explicit analysis of metric redundancy, is very helpful.

Critical weaknesses

The paper seems to contain hallucinated and incorrect references, which raises the question do which degree this paper actually the paper was quality checked after the draft had been generated by AI (see AI usage statement at the end of the paper).

In particular: "Kilian Göller, Haadia Amjad, Steffen Seitz, Carsten Knoll, and Ronald Tetzlaff. How a model’s architecture and performance influences explainability: A study of Grad-CAM and CRAFT in CNN-based image classification. In Explainable Artificial Intelligence (xAI), 4th World Conference. Springer, 2026." This paper seems not to exist. However, in the paper mentions Göller et al. (2026) as one of only two cited works that evaluate explanations across training checkpoints. The paper is thus core related work this submission builds upon and delineates from. The authors describe it as narrow in scope (two XAI methods, three CNNs, no transformers). Since this reference appears to be hallucinated, the paper’s account of its most closely related work cannot be trusted.

Further potential hallucination artifacts can be found here: (b) "Deevyankar Agarwal, Mario Alberto Berbís, Antonio Luna, Vivian Lipari, Julien Brito Ballester, and Isabel de la Torre-Díez. Automated medical diagnosis of Alzheimer’s disease using an EfficientNet convolutional neural network. Journal of Medical Systems, 47(1):57, 2023." This paper exists, but the second author is Manuel Álvaro Berbís, not Mario Alberto Berbís. (c) "Anna Hedström, Leander Weber, Dilyara Krakowczyk, Dina Bareeva, Franz Motzkus, Wojciech Samek, Sebastian Lapuschkin, and Marina M-C Höhne. Quantus: An explainable AI toolkit for responsible evaluation of neural network explanations and beyond. Journal of Machine Learning Research, 24(34):1-11, 2023a.". This paper exists, but the third and fourth authors are Daniel Krakowczyk and Dilyara Bareeva.

These examples indicate that the AI generated draft was not properly validated and raises the question to which degree the content itself was checked by a human. I thus have strong doubts on the validity of the work since principal related work it builds upon seems not to exist. My conclusion is that the paper is thus not yet ready for publication in a top-tier conerence.

Questions for authors

I recommend the authors to closely review (at least) the related work and to fix AI-generation artifacts. Or, in case I am wrong, please provide the Göller et al. paper or a publisher link to proove its existance.

Reviewer_QAWhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the quality of XAI explanations across four main properties, using 26 model architectures trained from scratch to test how architecture, model performance, and the XAI method itself interact in shaping the explanation. The findings show that higher accuracy generally leads to better localization, that different XAI method families have different advantages for different model backbones, and that normalization improves localization.

Critical strengths
  • The study design is really positive: training 26 backbones from scratch and choosing different checkpoints varying in accuracy is a nice evaluation. The extensive evaluation on the 208 resulting model states therefore offers a good ground for the evaluation, with a small weakness of using only ImageNet with 50 classes and being limited to one dataset.

  • The narrative of the paper is clear and easy to follow. Usually, having the three RQs you are investigating in one section is something I would prefer. However, you wrote and designed your paper with your experiments so that splitting them into clear sections delivered a clear story.

Critical weaknesses
  • The authors focused a lot on localization; however, localization in this manner means the segment of an object that needs to be classified. However, is this truly a good explanation? Consider faithfulness and complexity: there are well-known quantitative studies for deletion and insertion checks, but only removal is tested, while replacement of pixel/superpixel values has also been shown to have immense impacts. I would therefore not agree that localization in this manner should be the preferred headline metric.

  • While you acknowledge limitations regarding your design choices, I see the largest and most significant weakness in the XAI methods setup, since the choice of hyperparameters of XAI methods has a large influence on the model’s performance, such as the segmentation technique in models such as LIME or SHAP [1,2]. For example, the referenced ones here would probably return the exact area of the original image. Thus, what the paper completely ignores is that the issues the authors discuss here are not new in the XAI community at all. On the one hand, it is totally fair that not all improvements of LIME, Grad-CAM, SHAP, etc., can be tested in a paper, and the contributions and insights are still relevant, but it should be acknowledged and at least discussed that improvements already exist for specific methods that improve the explanations. Otherwise, the last 7 years of XAI research would be ignored.

  • In general, your results showed some really interesting patterns, such as the deviation in behavior between model families. However, what I was mainly missing was a discussion about WHY these patterns exist, or at least some hypotheses.

All of these weaknesses heavily influenced my rating, as addressing them is, in my opinion, needed to support the acceptance of the paper.

[1] Beyond Pixels: Enhancing LIME with Hierarchical Features and Segmentation Foundation Models, P. Knab et al., ECML 2025

[2] Explain Any Concept: Segment Anything Meets Concept-Based Explanation, A. Sun et al., NeurIPS 2023

Questions for authors
  • Why is IG in Figure 3(a) performing the worst, and why is Grad-CAM exhibiting the largest variance? Can you also discuss why transformers are easier to localize with KernelSHAP than with Grad-CAM in this plot? Also, in Figure 5(a), please discuss why the two families diverge so much.

  • What is your opinion of using ground-truth masks as indicators of areas that “explain” how the model thinks? Is it not more like a human preference to check if the model classifies an image according to our semantic intuition? Suppose the class can already be identified by the background alone, which would have a stronger signal for the model than the semantic unit itself. In this scenario, we would say the XAI method or the model is bad, but it has done its job correctly by finding the features/pixels that most easily distinguish classes.

  • Future work: I really acknowledge the work the authors have put into this paper, especially the design of the study with the training of the models from scratch. However, I believe that your paper needs to address the limitations of different XAI techniques that have seen advancements in the last couple of years, as well as include a discussion of the patterns that you found during your experimentation. I would not say that you must test many more XAI techniques on all models, but selectively testing several ones on selected checkpoints and model architectures could be an additional experiment. However, I would like to hear your thoughts on why you designed the study like this in the first place.

5 comments
AuthorsOpenReview ↗

Response to Reviewer QAWh (Part 1/3)

We thank the reviewer once again for the constructive review and for the clarifying reply to our questions. Below we answer each point. Section, figure and table numbers refer to the revised paper we uploaded.

CW1: Localization as headline metric; sensitivity of faithfulness to the perturbation strategy

Localization. We agree that agreement with an object mask is not by itself a good explanation. It measures plausibility (does the map point where a human expects the evidence?), not faithfulness. The revision says so in a new Discussion paragraph, "Limitations due to metrics" (Section 7); see also our answer to Q2. We keep localization AUC as the headline localization metric because its chance level (0.5) is independent of the image. We no longer present localization as the headline of explanation quality as a whole:

  • faithfulness now has four metrics, all reported in the main text;
  • Finding 1 now states explicitly that training yields better-localized, but not overall better explanations, and the Discussion states that training alone does not necessarily improve faithfulness.

Faithfulness. We added Pixel Flipping (Bach et al., 2015) next to IROF, Faithfulness Estimate and Faithfulness Correlation. It sharpens one conclusion:

  • the deletion-based scores (IROF, Pixel Flipping) rise with training, Pixel Flipping on 96% of trajectories;
  • but both divide the deletion curve by the model's unperturbed class probability, which itself grows with accuracy, while the correlation-based metrics do not improve (Section 4, footnote 2).

So deletion scores compared across checkpoints or models largely reflect the model's confidence.

Perturbation strategy. Following the reviewer's clarification, the new Discussion paragraph addresses this topic directly.

  • All four of our deletion metrics fill removed pixels or superpixels with black.
  • Insertion into a neutral image (Petsiuk et al., 2018), replacement with blurred or other image content, or imputation as in ROAD (Rong et al., 2022) can change the scores and even the method ranking. The perturbed inputs leave the training distribution (Hooker et al., 2019), and CNNs and transformers react to this differently (Jain et al., 2022).

We did not run insertion or alternative fill strategies in the limited time available.

CW2: Hyper-parameters of the XAI methods; improved method variants; positioning

Hyper-parameters. We agree that the explanations depend on them. Every non-default setting is listed in Table 3 and described in Appendix B, and a single fixed setting is used for all models and checkpoints. Each deviation from the Quantus and Captum defaults was based on a measurement:

  • 7×7 coalition grid for LIME and KernelSHAP: at 14×14, KernelSHAP's split-half reliability is +0.005 and its AUC is at chance; at 7×7 it is +0.60 (LIME +0.90).
  • LIME's Lasso penalty is scaled to the logit range, which removes an artificial sparseness trend.
  • Faithfulness Correlation uses 100 runs instead of 10.
  • Effective Complexity: ε = 10⁻² instead of 10⁻⁵.
  • IROF is summarized by its median rather than its mean.

Improved variants. A new related-work paragraph acknowledges that improved successors exist, for example LIME and SHAP with coalitions from hierarchical or foundation-model segmentations (Knab et al., 2025; Sun et al., 2023). It also explains why we keep the general-purpose methods in their standard form: so that differences can be attributed to accuracy, architecture and method family rather than to method-specific tuning. As the reviewer suggested in Q3, we also tested more recent, architecture-adapted methods on selected architectures:

  • LeGrad and LibraGrad on all 72 transformer states;
  • LRP with AttnLRP rules on the 13 LayerNorm and attention backbones.

This refines Finding 2. Generic gradient methods (Grad-CAM, IG) fall behind the perturbation methods on transformers, but the better of LeGrad and LibraGrad beats both perturbation methods on all nine transformers, and LRP with AttnLRP rules draws level with them (Section 5).

Positioning. We fully agree with the reviewer's reply: explanation faithfulness, sensitivity to methodological choices, and the link between model characteristics and explanation quality are well-established concerns. The revision now says explicitly that we do not claim this sensitivity as new (Section 2). Our contribution is the joint, controlled measurement of how much accuracy, architecture and method each contribute, and how they interact.

CW3: Why do the patterns exist?

Each finding (Sections 4 to 6) is now structured as Observation, Interpretation and Implication; the Interpretation paragraph explains why we think the pattern arises. We present these as hypotheses consistent with our data and with prior work.

(continued in Part 2)

AuthorsOpenReview ↗

Response to Reviewer QAWh (Part 2/3)

Q1: IG worst, Grad-CAM most variable, KernelSHAP vs Grad-CAM on transformers, divergence in Figure 4

  • Why IG localizes worst. IG accumulates raw input gradients and inherits their high-frequency artefacts, which pooling and striding create in CNNs (Muzellec et al., 2024); Section 4 now states this. A pixel-wise AUC against a smooth mask penalizes this noise, whereas Grad-CAM, LIME and KernelSHAP score 7×7 regions, which acts as a low-pass filter.
  • Why Grad-CAM varies most. The variance in Figure 3(a) is mainly the spread across backbones: Grad-CAM has a standard deviation of 0.089 localization AUC at convergence, compared to 0.018 to 0.044 for the other methods. Grad-CAM is applied to the last convolutional or attention layers, which have a resolution of 7×7 (14×14 for VGG). Upsampling this coarse map mislocates relevance, especially for smaller objects. This is visible in Figure 11 (Appendix D), where Grad-CAM struggles to fully cover smaller objects such as the street sign or the television screen, and shows a localization offset that becomes clearly visible especially at higher accuracies.
  • Why KernelSHAP beats Grad-CAM on transformers. KernelSHAP needs no spatial layer and averages 0.66 AUC on the nine transformers, against 0.62 for Grad-CAM. Grad-CAM leads only on the three Swin models; on the CNNs it outperforms LIME and KernelSHAP on 16 of 17 models.
  • Why the two families diverge in Figure 4 (Section 5, Interpretation). Attention and LayerNorm break the conservation of gradient-based relevance (Ali et al., 2022), and gradients are far noisier in ViTs (Achtibat et al., 2024), which hurts the generic Grad-CAM and IG. Swin's local window attention keeps the spatial layout Grad-CAM relies on, whereas PVTv2 and MaxViT mix tokens globally. The perturbation methods only need a well-behaved response to masking, which ViTs provide (Naseer et al., 2021) and CNNs less so (Jain et al., 2022). This occlusion robustness is learned during training, which is consistent with the families diverging only later in training. The new, transformer-adapted methods test this account: every method that repairs propagation through attention and LayerNorm (AttnLRP rules, LibraGrad, LeGrad) closes or even reverses the gap.

Q2: Ground-truth masks as a target for explanations

This is an important open question in XAI. We agree that object masks encode a human expectation (plausibility, or "coherence" in the taxonomy of Nauta et al., 2023) and not the model's decision rule. Overlap with segmentation masks is nevertheless a widely used evaluation; Nauta et al. list 13 papers that use it.

Requiring a model to focus on the whole object can be excessive (Elisha et al., 2026). Our Pointing Game and Top-K Intersection metrics partly account for this, because they score only the most relevant part of the map.

Models do use background cues. However, Xiao et al. (2021) find that more accurate models, although they still exploit background cues, are less misled by changes in the background, which fits our accuracy-localization trend. Background reliance is often a shortcut that fails out of distribution (Geirhos et al., 2020). Humans may also use context (e.g. we associate lions with the savannah), but a lion in the snow is still a lion. Requiring a classifier to rely on the object is therefore a reasonable desideratum, and some work enforces it during training (Chefer et al., 2022).

In the reviewer's scenario, a class identifiable from the background alone, a faithful explanation should point at the background, and a mask would wrongly penalize it. Such classes exist in ImageNet (Singla & Feizi, 2022). This is why we read localization together with faithfulness metrics, and why the revision now states this limitation explicitly (Section 7).

We also note that our setting is object classification: every image in ImageNet-S50 is labelled by the object it depicts, so the object is the intended evidence for the class. In scene classification (e.g. Places365), by contrast, the background itself carries the class, and an object mask would not be a meaningful reference. Accordingly, human and CNN (Grad-CAM) attention agree far less on landscapes without distinct objects than on images of single objects (Müller et al., 2024).

(continued in Part 3)

AuthorsOpenReview ↗

Response to Reviewer QAWh (Part 3/3)

Q3: Why this study design?

Our motivation is a practical one: in safety verification, a human operator has to decide, partly on the basis of explanations, whether a system is safe to deploy. This decision is only reliable if the operator knows whether a good or poor explanation reflects the model itself, or merely the chosen XAI method, the architecture or the training state of the model. To our knowledge, no previous study has measured these three factors jointly under controlled conditions.

The design follows from the research gap we target. Prior comparisons evaluate XAI methods on converged, off-the-shelf backbones, where architecture and attained accuracy are confounded. Training every backbone from scratch under one recipe, keeping checkpoints across the accuracy range, and applying one evaluation protocol is, in our view, the smallest design in which accuracy, architecture and method can be separated and deliver meaningful insights.

It costs 26 training runs and 1040 explanation runs scored with 13 metrics. For that reason we limited the core method set to standard methods that apply to every backbone in the same form, and the data to one dataset with pixel-level masks.

We agree with the reviewer's suggestion to test newer techniques selectively, and did so:

  • LeGrad and LibraGrad on all transformer checkpoints (144 additional runs);
  • LRP with transformer-specific rules on the 13 LayerNorm and attention backbones.

As explained above (CW2), these do not fundamentally change our study's outcome, but add new insights to it. The released pipeline can be used to score further method variants, such as segmentation-based LIME and SHAP, on the same checkpoints.

We hope these changes address the reviewer's concerns, and we thank the reviewer again for the open discussion. In case we missed any open points, we would be glad if the reviewer could point us towards them.

References:

  • Achtibat et al. AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers. ICML 2024.
  • Ali et al. XAI for Transformers: Better Explanations through Conservative Propagation. ICML 2022.
  • Bach et al. On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation. PLoS ONE 2015.
  • Chefer, Schwartz, Wolf. Optimizing Relevance Maps of Vision Transformers Improves Robustness. NeurIPS 2022.
  • Elisha, Cohen, Barkan, Koenigstein. Rethinking Saliency Maps: A Cognitive Human Aligned Taxonomy and Evaluation Framework for Explanations. AAAI 2026.
  • Geirhos et al. Shortcut Learning in Deep Neural Networks. Nature Machine Intelligence 2020.
  • Hooker et al. A Benchmark for Interpretability Methods in Deep Neural Networks. NeurIPS 2019.
  • Jain et al. Missingness Bias in Model Debugging. ICLR 2022.
  • Knab, Marton, Bartelt. Beyond Pixels: Enhancing LIME with Hierarchical Features and Segmentation Foundation Models. ECAI 2025.
  • Müller et al. Do Humans and Convolutional Neural Networks Attend to Similar Areas during Scene Classification: Effects of Task and Image Type. Applied Sciences 2024.
  • Muzellec et al. Saliency Strikes Back: How Filtering out High Frequencies Improves White-Box Explanations. ICML 2024.
  • Naseer et al. Intriguing Properties of Vision Transformers. NeurIPS 2021.
  • Nauta et al. From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI. ACM Computing Surveys 2023.
  • Petsiuk, Das, Saenko. RISE: Randomized Input Sampling for Explanation of Black-box Models. BMVC 2018.
  • Rong et al. A Consistent and Efficient Evaluation Strategy for Attribution Methods. ICML 2022.
  • Singla, Feizi. Salient ImageNet: How to Discover Spurious Features in Deep Learning? ICLR 2022.
  • Sun et al. Explain Any Concept: Segment Anything Meets Concept-Based Explanation. NeurIPS 2023.
  • Xiao, Engstrom, Ilyas, Madry. Noise or Signal: The Role of Image Backgrounds in Object Recognition. ICLR 2021.
Reviewer_nRhCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper is concerned with studying the effects of accuracy, model architecture, and XAI method on attribution quality. To this end, the paper evaluates different attribution methods on various ImageNet-50 models (across different training points (to control for accuracy) and architectures). The attribution quality is assessed via a subset of the Quantus (Hedström et al., 2023) benchmark to measure four attribution quality dimensions, i.e., localization, faithfulness, complexity, and randomization. The paper highlights three main findings: (1) higher accuracy leads to better attribution localization, (2) perturbation-based attribution methods outperform gradient-based approaches on vision transformers and vice versa for CNNs, and (3) normalization plays an important role.

Critical strengths
  • The paper identifies an important gap in the literature.
  • The writing style and figures were (mostly) clear and easy to follow.
  • The control for accuracy via different training steps is interesting.
Critical weaknesses
  • W1 The choice of the XAI methods is insufficient. The paper only considers five XAI methods which, in my opinion, is not sufficient for drawing meaningful conclusions. In particular, modern ViT-specific backpropagation-based attribution methods such as LeGrad (Bousselham et al., 2025) and LibraGrad (Mehri et al., 2025) are missing and could change the conclusion in finding two, i.e., that perturbation-based attribution methods outperform gradient-based approaches on vision transformers.
  • W2 The findings are rather descriptive than explanatory. Findings 1-3 lack insights into why the observed properties emerge. For example, in finding 1 (higher accuracy=higher localization), a potential reason could be that higher accuracy models need to focus more on the object of interest, and hence, localization improves. This lack of explanations leads to W3.
  • W3 Lack of take-home messages. After reading the paper, I do not necessarily know what to do with the gained insights. Finding 2 is interesting, but in line with related work and hence not really novel; for example, LibraGrad (Mehri et al., 2025) also establishes that standard gradient-based attribution methods do not perform well on ViTs. Findings 1 and 3 do not seem too actionable.
  • W4 Overreliance on localization. I understand that localization is an established metric in attribution evaluation, however, it also comes with the strong limitation that it does not say anything about the faithfulness of an explanation. As all three findings rely mostly on this localization metric (e.g., finding 1 only shows strong effects for localization while findings 2 and 3 only focus on localization in figures 5 and 6), I am not sure if the overarching research question ("what determines XAI quality") is really addressed by the experiments.
  • W5 Correlation of metrics within one quality dimension. The paper argues that metrics for the same quality are highly redundant and hence only reports one representative metric per quality dimension in the main paper (L. 234-238). However, looking at Figure 10 in Appendix C the metrics for faithfulness and randomization do not seem to be very redundant.
Questions for authors
  • Why did you choose the specific XAI methods and how would the conclusions change if more recent attribution methods are included?
  • Can you provide explanations for why the different findings occur?
  • What actionable insights do findings 1 and 3 provide?
6 comments
AuthorsOpenReview ↗

Response to Reviewer nRhC (Part 1/4)

We thank the reviewer for the detailed and constructive review. Several of the reviewer's points led to substantive additions in the revision: LeGrad and LibraGrad on all transformer checkpoints, LRP on all 26 backbones, Pixel Flipping as a fourth faithfulness metric, explanations of why each pattern occurs, and explicit statements and recommendations in the Discussion section. We first answer the weaknesses (W1 to W5) and then the questions (Q1 to Q3). Section, figure and table numbers refer to the revised paper.

Weaknesses

W1: Choice of XAI methods

Our study is motivated by settings such as safety verification, where a human operator has to decide, partly on the basis of explanations, whether a system is safe to deploy. Such a decision is only reliable if the operator knows whether a good or poor explanation reflects the model itself, or merely the chosen XAI method, the architecture or the training state of the model. To separate these factors, we evaluate not only converged models but checkpoints along the whole training trajectory, which the reviewer kindly acknowledged as a strength. This design requires XAI methods that apply in the same form to every backbone and every checkpoint: a method that exists only for some architectures cannot separate the effect of architecture from the effect of method. We therefore chose Grad-CAM, IG, LRP, LIME and KernelSHAP, which are widely used representatives of both the backpropagation-based and the perturbation-based family and work on CNNs and transformers alike.

We agree, however, that this limits Finding 2 to generic methods, and we have tested this limit directly. We extended LRP to all 26 backbones (previously 13). In addition, we scored LeGrad (Bousselham et al., 2025) and LibraGrad (Mehri et al., 2025) on all 72 transformer states (9 transformers × 8 checkpoints) with all metrics. Both methods were defined for isotropic ViTs with a class token and have no published form for hierarchical backbones, so we adapted them (Appendix B): LeGrad on the last-stage attention layers, and LibraGrad as FullGrad+ under the Libra gradient rules.

The reviewer was right that this changes Finding 2. At convergence, the better of LeGrad and LibraGrad beats both perturbation methods on all nine transformers, by 0.048 to 0.141 localization AUC (Section 5), and LibraGrad is the most faithful method on the transformers under all four faithfulness metrics (Table 4). LRP with transformer rules draws level with the perturbation methods. Both adapted methods also follow Finding 1: their localization rises with accuracy on every transformer (new right-hand panels in Figure 3(a)). What remains of the original finding concerns the generic gradient methods (Grad-CAM, IG): they fall behind the perturbation methods on transformers but not on CNNs. At convergence, a perturbation method is ahead on 6 of 9 transformers but on only 1 of 17 CNNs, and this gap widens during training. We have rewritten Finding 2 accordingly.

(continued in Part 2)

AuthorsOpenReview ↗

Response to Reviewer nRhC (Part 2/4)

W2: Findings are descriptive rather than explanatory

Each finding (Sections 4 to 6) is now structured as Observation, Interpretation and Implication. The Interpretation paragraph gives our explanation of the pattern; we present these explanations as hypotheses consistent with our data and with prior work, not as proven mechanisms.

Finding 1 (localization follows accuracy). We adopt the reviewer's hypothesis. Classification training produces channels that respond selectively to particular objects (Zhou et al., 2015), and their number grows during training (Bau et al., 2017); more accurate models are also less misled by changes in the background (Xiao et al., 2021). Any method that reads out the model's evidence therefore moves onto the object as that evidence forms during training.

Finding 2 (method families diverge on transformers). Attention and LayerNorm break the conservation of gradient-based relevance (Ali et al., 2022), and gradients are far noisier in ViTs (Achtibat et al., 2024), which hurts the generic Grad-CAM and IG. Perturbation methods depend only on how the model responds to masking; ViTs degrade gracefully under occlusion (Naseer et al., 2021), CNNs less so (Jain et al., 2022). This robustness is learned, which fits the observation that the method families separate only during training. The new experiments test this account directly: every method that repairs the propagation through attention and LayerNorm (AttnLRP rules, LibraGrad, LeGrad) closes or even surpasses the gap. Swin, whose attention is local, keeps the spatial layout Grad-CAM needs and is the one transformer family where Grad-CAM stays ahead.

Finding 3 (normalization layers matter, not model size). At inference, BatchNorm reduces to a fixed per-channel scaling and shift that can be merged into the preceding convolution, so it does not alter how relevance is propagated. Its benefit more likely stems from the learned weights, which is consistent with BatchNorm also improving LIME and KernelSHAP; one possible explanation is the smoother optimization landscape it induces (Santurkar et al., 2018). LayerNorm, by contrast, remains on the propagation path and violates relevance conservation, which would explain why only the methods that propagate through the network are penalized. Model width and depth leave the propagation path unchanged.

W3: Lack of take-home messages; novelty

As described in W1, our goal is to tell practitioners who rely on explanations, for example in safety verification, and also developers of XAI methods what an explanation score actually reflects. Today, XAI methods are usually proposed and compared on converged, off-the-shelf backbones, often different ones in different papers. Our results show that such comparisons are not comparable: if paper A and paper B present their XAI methods on different architectures and at different accuracies, their scores reflect the backbone and its training state as much as the method. The Discussion (Section 7) now opens with five explicit statements derived from the results:

  1. A localization score mostly measures how well the model is trained. Accuracy has the biggest impact on localization (Figure 3(b)), so a fair comparison between XAI methods or backbones regarding localization is only possible at matched accuracy. This is currently not standard practice when new XAI methods are published or compared.
  2. Depending on the evaluation metric, training alone does not necessarily improve faithfulness. Deletion-based scores reflect a model's confidence rather than the faithfulness of its explanations.
  3. On transformers, generic gradient-based methods fall behind perturbation-based ones, and architecture-adapted methods take the lead (see W1).
  4. The "best" XAI method changes during training. A method that works best for a converged model is not necessarily the best in an early training state; the method families start tied and separate during training.
  5. Normalization layers have a higher impact on explanations than model size. Adding BatchNorm at equal parameter count is worth 17 to 37 accuracy points of localization; increasing the number of parameters within a family shows no clear effect.

(continued in Part 3)

AuthorsOpenReview ↗

Response to Reviewer nRhC (Part 3/4)

(W3, continued)

Following these statements, a new "Recommendations" paragraph addresses the two groups directly. For practitioners, there is no universally best XAI method: which method is suitable depends on the model architecture and its training state, so the choice should be validated on the model at hand rather than taken from benchmarks of other backbones. For developers of XAI methods, a new method should be compared against existing ones on the same architectures and at the same training stages; otherwise a reported gain may reflect the backbone or its accuracy rather than the method.

Actionable insights.

  1. Insight 1 (Finding 1): Report XAI scores together with model accuracy, and compare methods only at matched accuracy. Otherwise explanation quality is not comparable.
  2. Insight 2 (Findings 1 and 2): Choose the XAI method for the model at hand and its training state: generic gradient methods on CNNs, architecture-adapted or perturbation methods on transformers.
  3. Insight 3 (Finding 3): If explanations matter, changing the normalization layers has a stronger impact than just increasing the model’s width and depth.

Novelty relative to LibraGrad. We agree that LibraGrad has already shown that standard gradient methods fail on converged isotropic ViTs, and the revision acknowledges this (Section 7). Our study extends this result in two ways. Beyond isotropic ViTs, the failure also occurs on hierarchical transformers (Swin, PVTv2, MaxViT), and our adaptations of LeGrad and LibraGrad resolve it there as well, surpassing both the perturbation-based and generic gradient XAI methods. However, how individual XAI methods perform is not the key point of our study. Our main contribution is to show that method rankings are not stable across architectures and training stages, so that conclusions drawn on one model cannot simply be transferred to another.

W4: Over-reliance on localization

We agree with the reviewer that the submitted paper version was weighted towards localization, and that this did not fully support the overarching research question. The revision rebalances the paper towards faithfulness. We added Pixel Flipping (Bach et al., 2015), so the paper now reports four faithfulness metrics (Section 3.4; Figures 3(b), 4, 7 and 12; Tables 4 to 6).

Finding 1 now addresses faithfulness explicitly. The correlation-based metrics (Faithfulness Estimate, Faithfulness Correlation) do not improve with training. The deletion-based metrics (IROF, Pixel Flipping) do rise with accuracy, Pixel Flipping on 96% of trajectories, but both divide the deletion curve by the model's unperturbed class probability, which itself grows with accuracy (footnote 2). Training therefore yields better-localized, but not more faithful explanations.

Findings 2 and 3 now go beyond localization as well. The figures the reviewer refers to have been replaced: Figure 4 now shows every gradient and perturbation method along training, for CNNs and transformers, in localization, faithfulness and complexity. For faithfulness it shows one deletion-based metric (IROF) and one correlation-based metric (Faithfulness Correlation); the other metric of each pair behaves alike. The deletion-based metrics rise for every method with the model's confidence, while under the correlation-based ones IG loses faithfulness with training. In complexity, LIME stays the most and KernelSHAP the least concise method, and in randomization Grad-CAM is the most class-discriminative method on all 26 models. The clear architecture split of Finding 2 is therefore specific to localization, and Finding 2 now says so. Table 5 (Appendix E) compares the two method families for every metric, including significance tests between CNNs and transformers, and Table 4 compares all seven methods on the transformers across all categories. For Finding 3, Table 6 gives the BatchNorm ablation for every metric, and the new Figure 12 shows all four faithfulness metrics for the VGG pairs: at matched accuracy, BatchNorm improves the correlation-based faithfulness metrics for every method except IG, and Pixel Flipping for Grad-CAM and LRP.

Finally, a new Discussion paragraph, "Limitations due to metrics" (Section 7), states that localization measures plausibility, not faithfulness. We keep localization as the headline category because it is where accuracy has a strong effect, but we now report the results for the other metric categories as findings in their own right.

(continued in Part 4)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.