acceptodds
Withdrawn from ICLR 2027

The Explainability Illusion: Must ViTs Exhibit Patch-Decoding Interpretability?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Transformers are widely assumed to exhibit token-decoding explainability. In particular, in vision transformer (ViT) classifiers, decoding individual patch tokens has become a common diagnostic tool due to its tendency to yield human‑interpretable outputs. In light of the increasing popularity of this approach, an important question is whether token‑decoding explainability an inherent property of transformers that should be expected to hold across architectures, scales, and training objectives, or merely an artifact of some implicit bias? In this paper we systematically study token‑decoding explainability across diverse ViT settings, spanning architectures, scales, and training objectives, using both controlled experiments and real-world models. We find that while some models indeed exhibit aligned and interpretable token decodings, others exhibit either inverted explainability maps or a complete lack of correlation with the image semantics. As we show, these tendencies toward different explainability behaviors are not intrinsic to the architecture and can vary arbitrarily with changes in the optimization landscape. We provide theoretical support for these observations by demonstrating a setting where a single fixed toy transformer architecture can provably implement the optimal classification rule in multiple ways, which differ in their patch-decoding explainability patterns. Our results show that token‑decoding explainability is not an inherent property of trained transformers, but rather an outcome contingent on the problem setting and thus should not be a-priori assumed to hold.

Primary area:
interpretability and explainable AI
Keywords:
explainable ai, vision transformer

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VfkhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper challenges the common assumption that zero-shot patch-token decoding provides reliable explanations for Vision Transformers. Through experiments on a wide range of ViTs and vision-language models, combined with controlled synthetic studies and a toy theoretical model, the authors argue that patch-decoding explanations can be aligned, inverted, or uninformative, while preserving the same predictive performance.

Critical strengths

Extensive empirical evaluation: Analysis spans many classifier and contrastive VLM architectures, scales, and training regimes.

Theoretical support: Section 5 provides a constructive example showing that the same optimal classifier can admit different explainability regimes.

Critical weaknesses

Overall, I see value in the scale of the empirical study, rather than in the claimed conceptual discovery. Inverted CLIP visualisation was previously documented, and the theoretical message of Section 5 closely follows established non-identifiability and implementation-invariance arguments. The paper’s clear added value is the systematic characterisation of patch-decoding regimes across model families and controlled training settings. I would find the contribution more convincing if the claims were narrowed accordingly, the closest theoretical literature were incorporated, and the distinction between explaining an input-output function and inspecting implementation-specific representations were addressed.

The authors miss to put into context relevant past work.

On Identifiability in Transformers [A] proves that, under conditions involving sequence length and attention-head dimension, different attention distributions can produce the same contextual embeddings. The paper explicitly frames this as a structural identifiability problem and concludes that self-attention distributions are not directly interpretable.

The submission’s construction is not identical. Brunner et al. study attention-weight identifiability, whereas this paper studies head-decoded patch-token representations and gives aligned versus inverted spatial regimes. But the theoretical story is sufficiently close that Section 5 should be positioned as a specialisation or extension of transformer non-identifiability, not as introducing the general insight.

  1. Attention is not Explanation [B] shows empirically that very different attention distributions can yield equivalent predictions, while learned attention may be poorly correlated with gradient-based importance measures. Again, the object being interpreted differs: Jain and Wallace analyse attention distributions; this paper analyses direct decoding of patch representations. Nevertheless, the epistemic point is almost the same: an intuitive token-level visualisation is not uniquely constrained by predictive behaviour. The subsequent Attention is not not Explanation is also relevant because it cautions that such negative conclusions depend heavily on the definition of “explanation” and on the experimental test used. It proposes comparisons across seeds and adversarially trained alternatives, which are directly pertinent to this submission’s claims about training-seed dependence.

[A] Brunner et al., On Identifiability in Transformers, ICLR 2020 https://iclr.cc/virtual_2020/poster_BJg1f6EFDB.html [B] Jain and Wallace, Attention is not Explanation, NAACL 2019 https://arxiv.org/abs/1902.10186 https://aclanthology.org/N19-1357/

Additionally: The toy construction uses a specialised embedding and non-standard classification head, making the connection to real-world ViTs less convincing.

Theoretical result is illustrative rather than explanatory: It shows that multiple regimes are possible but does not explain why real models converge to a specific regime.

Overall: Interesting empirical study with useful evidence, but the novelty of the central conceptual claim and the theoretical contribution appears overstated.

Questions for authors

none

2 comments
AuthorsOpenReview ↗

Response to Reviewer Vfkh (1/2)

I would find the contribution more convincing if the claims were narrowed accordingly, the closest theoretical literature were incorporated, and the distinction between explaining an input-output function and inspecting implementation-specific representations were addressed.

Overall: Interesting empirical study with useful evidence, but the novelty of the central conceptual claim and the theoretical contribution appears overstated.

The authors miss to put into context relevant past work.

We thank the reviewer for pointing out these important relevant works. This line of work was a miss on our part, and we modified our paper accordingly. We added clarifications regarding our scope throughout the text, including narrowing our claims, emphasizing our focus on the explainability of a specific zero-shot approach, and incorporating the relevant theoretical literature (see Abstract, and Secs 1,2, 3.1, 5 and 6, and Apps. D.1 and E).

As the reviewer mentions, all three works focus on attention distributions, while our analysis specifically targets patch-token decoding.

It should be noted that the term "identifiability" used in ref. [A] refers to input-dependent identifiability for a single sample. In contrast, our theoretical construction in Sec. 5 addresses token-decoding weight identifiability given the classification head over the entire training set (or the global input-output mapping across all inputs). Namely, we show that the token-decoding maps weights of a model are not uniquely determined by the input-output function it implements, and that different implementations of that function lead to different explainability behaviors. Additionally, the results of [A] hold only for cases where the sequence length is larger than the head dimensions, whereas our results don’t have such a constraint (the sequence length NN is not dependent on the attention head dimension K+1K+1). Nevertheless, it is certainly an important relevant work, which we missed.

Regarding “Attention is not Explanation” [B1] and “Attention is not not Explanation” [B2], we agree that the underlying story is similar on a high level, except again, that they analyze attention matrices and we analyze token-decoding maps. We updated the manuscript to reflect the similarity of the themes. We note that a main concern in [B2] regarding the adversarial setup in [B1] is resolved in our case. Specifically, whereas in [B1] the links between the attention distribution and the trained model are removed, our construction here shows the adversarial example can be a direct result of a fully trained model. Furthermore, we do not claim a hard negative result as in [B1], but rather a softer cautionary tale on the use of zero-shot patch decoding explainability maps, similarly to [B2].

Please see the revised Secs 1,2, 5 and 6, and Apps. D.1 and E.

AuthorsOpenReview ↗

Response to Reviewer Vfkh (2/2)

Additionally: The toy construction uses a specialised embedding and non-standard classification head, making the connection to real-world ViTs less convincing.

It is correct that the construction in Sec.5 uses a non-standard head and embeddings. However, the mere purpose of this construction is to exemplify that a model can implement the same classification rule in infinitely many ways, which differ in their explainability patterns.

The logarithmic head is not the cause for the existence of multiple explainability regimes. Specifically, in App. E.6 we remove it, leaving a linear head as in real-world ViTs. This linear head predicts the same class and leads to the exact same three regimes for the same α\alpha values. We used the logarithm in Sec. 5 only to force the optimality of the model in terms of the logits, as it ensures the class-token output equals the exact posterior of the optimal classifier.

As for the nonlinear embedding used in the proof, it is relevant to real-world ViTs. Specifically, since it acts on each patch separately, it can be approximated to arbitrary accuracy by a per-token MLP block, which is common in ViTs. Therefore, a ViT with two layers can approximate our toy model on any bounded range of inputs, and regenerate these explainability behaviors. We can set the MLP of the first layer to approximate the embedding ϕ\phi, ignoring its attention part. Then, the second layer can implement the rest of our toy model with the provided attention weights. Apart from LayerNorm, which our toy model omits, every component of the construction is thus a standard ViT component. We added this point to App. E.3.

Theoretical result is illustrative rather than explanatory: It shows that multiple regimes are possible but does not explain why real models converge to a specific regime.

Indeed, we do not yet have a clear understanding on why real models converge to specific regimes. The main contribution of our work is the observation that the common assumptions of patch-decoding interpretability does not hold in the wild, and cannot be assumed to hold a-prioi. We hope a more fine-grained understanding of different mechanisms can be achieved in future works.

Reviewer_NskoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper explores the validity of using lens based methods like LogitLens to decode what patch tokens encode in vision transformers. Specifically, it evaluates, given an image containing an object of a particular class, whether the patch tokens within the object have a higher logit value for the class as compared to background tokens. A positive difference between average values inside and outside is termed as aligned, negative is termed as inverted, and a similar value is termed as unexplainable. The paper then performs a study across models, training types, and data, to show that the explainability can significantly differ, even when controlled for each of the design choices.

Critical strengths
  1. The problem addressed by the paper is important and interesting. Lens based methods are popularly used as a tool to decode what a token encodes. Given the complexity of transformer models, there is no reason to expect locality of patch tokens in images to hold, but this is still a common assumption. Evaluating whether a patch token indeed stores semantic information corresponding to the same region in an image is therefore an important sanity check and evaluation metric to better understand information flow in models.
Critical weaknesses
  1. There seems to be a critical issue in how the metric is formulated. The model was never trained to have the classification head applied to patch tokens and at arbitrary layers. While this has been a common method to "interpret" models, the finding this paper makes that some models have low "aligned explainability" could simply point to the process being flawed. It is not clear, e.g., that an "inverted explainability" result necessarily correlates with the patch tokens of the background encoding semantic information of the class, which would be the actual interesting question. So I feel the observations from this paper should first question the procedure that has been used, instead of just using it to make claims about the characteristics of various models and layers. The metric is also currently very coarse, and does not account for, e.g., some parts of the object being well aligned and others being poorly aligned. A finer-grained analysis would be necessary.
  2. The novelty and insights from this paper seem highly limited. A wide-scale evaluation is performed, but at a high level the only conclusion seems to be that different models may exhibit different "explainability", which is a limited insight that is also not new, as also pointed out in the paper (L117-127). The experiments with the synthetic setup could have been interesting, but in its current form, the analysis is done based on the noise levels used for the data generation process, and therefore it is very unclear how any of it could transfer to realistic setups. There does not seem to be any link between any property of the model, data (in a realistic setup), or training strategy, to the "explainability" score. This in itself could be insightful, but even this claim is under question given the concerns regarding the interpretation from Weakness 1.
Questions for authors

Please refer to the weaknesses. Substantial clarifications or answers that point to a serious misunderstanding on my part would be helpful.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Nsko (1/2)

There seems to be a critical issue in how the metric is formulated. The model was never trained to have the classification head applied to patch tokens and at arbitrary layers. While this has been a common method to "interpret" models, the finding this paper makes that some models have low "aligned explainability" could simply point to the process being flawed. It is not clear, e.g., that an "inverted explainability" result necessarily correlates with the patch tokens of the background encoding semantic information of the class, which would be the actual interesting question. So I feel the observations from this paper should first question the procedure that has been used, instead of just using it to make claims about the characteristics of various models and layers.

The reviewer is absolutely correct that aligned/inverted token-decoding explainability maps do not imply anything about the model, but are rather only a property of this (flawed) explainability procedure. This is the core claim of the paper. In particular, we claim that when the explainability map shows some patches as hotter, it does not necessarily imply that these image patches are most responsible for the classification outcome.

The reviewer is correct that this claim was not directly demonstrated in the original manuscript. Following the reviewer’s comment, we added in App. G a perturbation test [1,2,3] that directly compares between the explainability power of the maps in aligned and inverted models. The perturbation test works by gradually masking image patches according to their explainability map, either from hottest to coldest or vice versa, and measuring how the classification accuracy of the model changes. If the maps actually relate to semantic information, then the classification accuracy should degrade rapidly as the hottest patches are removed and it should degrade slowly as the coldest patches are removed.

We plot the accuracy against the percentage of removed patches for all supervised ViT-based classifiers in Fig. S20. As can be seen, in the inverted models (ViT-B/32+B/16 (SAM) and DeiT-Ti/16+S/16) the curves of the coldest-patches-removed-first, which correlate with object-patches, degrade very quickly, whereas the hottest patches degrade slowly - meaning the semantic information needed for classification is encoded in the object patches, even though the maps show them as cold. This proves directly that the interpretability obtained via these zero-shot patch-decoding approaches does not correlate with what the model “sees”.

We added clarifications in the text regarding the paper’s claim (specifically in the Abstract, and Secs. 1, 3.1, and 6). We thank the reviewer for this important comment.

The metric is also currently very coarse, and does not account for, e.g., some parts of the object being well aligned and others being poorly aligned. A finer-grained analysis would be necessary.

Our metric indeed does not capture object sub-parts. However, we note that better granularity is implicitly captured by the perturbation tests mentioned above, which mask individual patches rather than whole objects. These tests demonstrate that the explainability maps do not necessarily indicate which patches affect the model’s classification the most. This is the main point that the paper highlights - token-decoding explainability is inherently flawed.

AuthorsOpenReview ↗

Response to Reviewer Nsko (2/2)

The novelty and insights from this paper seem highly limited. A wide-scale evaluation is performed, but at a high level the only conclusion seems to be that different models may exhibit different "explainability", which is a limited insight that is also not new, as also pointed out in the paper (L117-127). The experiments with the synthetic setup could have been interesting, but in its current form, the analysis is done based on the noise levels used for the data generation process, and therefore it is very unclear how any of it could transfer to realistic setups. There does not seem to be any link between any property of the model, data (in a realistic setup), or training strategy, to the "explainability" score. This in itself could be insightful, but even this claim is under question given the concerns regarding the interpretation from Weakness 1.

Please note that while the observation that CLIP exhibits inverted maps is not new, we are the first to reveal that many ViT classifiers exhibit the same phenomenon. Furthermore, our theory also constitutes a novel contribution, as it demonstrates how the same transformer model can implement the same function in infinitely different ways, some of which exhibit aligned explainability and some exhibit inverted explainability. This also relates to the reviewer’s first comment - we demonstrate that inverted maps do not necessarily indicate that the model relies on the background to classify the object. A model can implement the precise same classification rule in either aligned or inverted ways.

We agree with the reviewer that our analysis does not clearly pinpoint a single cause for the emergence of inverted explainability. However, this is because there seems to be no single factor that deterministically determines the explainability regime. One of our key observations is that the data distribution can affect the explainability of a model (Sec. 4). Another factor that affects the explainability is the model architecture (e.g. number of layers). Yet, we show that even with a fixed distribution and model architecture, different training runs can converge to different explainability patterns. This demonstrates that the data distribution and model architecture do not deterministically determine whether a model will be aligned or inverted, they only affect the probability of these two outcomes. We believe that the mathematical mechanism underlying this implicit bias is rather complex and is an intriguing topic for future work.

Our work can be taken as a cautionary observation against unwarranted use of zero-shot patch-decoding interpretability of models. If one needs a zero-shot explainable model, then a better approach is to incorporate an explainability-loss during training to promote alignment [3].

[1] Samek et al. Evaluating the Visualization of What a Deep Neural Network Has Learned. IEEE Transactions on Neural Networks and Learning Systems 2017.

[2] Hooker et al. A Benchmark for Interpretability Methods in Deep Neural Networks. NeurIPS 2019.

[3] Chefer et al. Optimizing relevance maps of vision transformers improves robustness. NeurIPS 2022.

Reviewer_dzgMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors compare the image patch-level decoding of ViT's across several architectures and observe that they do not necessarily produce semantically aligned explanations. They introduce a metric to split architectures into aligned, inverted or unexplained regimes. Controlled experiments on synthetic datasets demonstrate that the resulting explainability regime depends on the data distribution and can vary across training seeds.

Critical strengths
  • The paper evaluates many ViT classifiers and contrastive VLMs across architectures, scales, and training settings. The consistent inversion in CLIP-like models is a strong finding.
  • The synthetic experiments systematically vary data distributions and training seeds, showing that the same architecture can produce different explainability patterns. The theoretical analysis further supports the claim that accurate classification does not require aligned patch decoding.
Critical weaknesses
  • The paper builds upon a single metric that defines explainability by the percentage of images where object patches score higher than background patches. This metric ignores both the magnitude of score differences and whether a model consistently produces aligned or inverted maps across images.
  • The paper focuses exclusively on zero-shot patch-token decoding, a method with well-studied limitations. In particular, applying a classification head trained on CLS tokens to individual patch tokens may produce misleading results due to representation mismatch. Alternative methods, such as Tuned Lens, were developed to address related limitations. Therefore, inverted explanations might reflect limitations of the decoding method rather than the absence of interpretable information. The broad framing of an "explainability illusion" seems too general without evaluating alternative approaches.
  • The authors attribute the different explainability regimes to optimization, training objectives, and data distribution. However, the model comparison in Table 1 mixes many of these effects, while the controlled experiments in Sec. 4 also change the task difficulty and potentially the optimization dynamics. Although the experiments show that these factors influence explainability, I do not see a clear analysis of the underlying mechanisms that drive these differences.
Questions for authors
  • Why is the penultimate layer used for some models?
  • Would a trained linear probe recover aligned explanations in models that show inverted patch decoding?
  • How does your metric separate "unexplainable" models from models that produce a mixture of strongly aligned and inverted maps?
9 comments
AuthorsOpenReview ↗

Response to Reviewer dzgM (1/4)

The paper builds upon a single metric that defines explainability by the percentage of images where object patches score higher than background patches. This metric ignores both the magnitude of score differences and whether a model consistently produces aligned or inverted maps across images.

How does your metric separate "unexplainable" models from models that produce a mixture of strongly aligned and inverted maps?

Regarding the consistency with which a model produces aligned or inverted maps across images, kindly note that our metric measures precisely that. For example, when our metric says 90%, it indicates that the model produces aligned maps for 90% of the images. In other words, it means that the model can be considered aligned with 90% consistency. Likewise, when our metric says 10%, it indicates that the model produces aligned maps for 10% of the images, indicating that the model is inverted with 90% consistency. The reviewer is correct that such a consistency metric may not tell the whole story in the vicinity of 50% (the unexplainable regime). Theoretically, the metric doesn’t differentiate between the case where all images are nearly unexplainable and the case where half of the images are strongly aligned and the other half are strongly inverted. This is the reason we also report 2D histograms for all models in Figs. S1 and S2. These reveal how each model clusters in each regime. For example, in all aligned or inverted explainable models a single cluster is neatly located below or above the red diagonal line, respectively.

Regarding the magnitude of the score difference, the reviewer is correct that our current metric A\mathcal{A} ignores it. To address this concern, we report here Cohen’s dzd_z effect size [1], defined as dz=E[Z]/σzd_z=\mathbb{E}[Z]/\sigma_z, where Z=Sobj−SbgZ=S_\textnormal{obj}-S_\textnormal{bg} for a single image, and the expectation is over images. As common with this measure, we can consider Cohen’s medium effect size, ∣dz∣<0.5|d_z|<0.5, as the unexplainable band. Below we report this metric for the ViT-classifiers and contrastive VLMs of Tab. 1:

Classifiers

ModelA\mathcal{A} (%)dzd_zRegime (A\mathcal{A})Regime (dzd_z)
ViT-B/16 (in1k)98.7+2.37AlignedAligned
ViT-B/16 (MIIL)98.6+2.32AlignedAligned
ViT-S/1698.4+2.25AlignedAligned
ViT-B/16 (AugReg2)98.4+2.32AlignedAligned
ViT-B/1698.4+2.19AlignedAligned
ViT-S/16 (in1k)98.4+2.21AlignedAligned
ViT-B/16 (orig)98.2+2.15AlignedAligned
ViT-L/16 (orig)98.1+2.21AlignedAligned
DINOv2-B/14+reg97.9+1.94AlignedAligned
ViT-L/1697.8+2.09AlignedAligned
DINOv2-s/14+reg97.7+1.89AlignedAligned
ViT-B/32 (in1k)97.7+2.03AlignedAligned
ViT-B/3297.5+2.03AlignedAligned
DINOv2-g/14+reg96.8+1.71AlignedAligned
DINOv2-s/1496.7+1.72AlignedAligned
DINOv2-B/1494.4+1.49AlignedAligned
DeiT-B/1692.2+1.41AlignedAligned
DINOv2-g/1475.6+0.66AlignedAligned
ViT-Ti/1670.5+0.51AlignedAligned
DINOv2-L/1463.5+0.43UnexplainableUnexplainable
DINOv2-L/14+reg41.1-0.22UnexplainableUnexplainable
ViT-B/32 (SAM)14.4-0.98InvertedInverted
ViT-B/16 (SAM)6.4-1.29InvertedInverted
DeiT-Ti/164.1-1.52InvertedInverted
DeiT-S/161.4-1.98InvertedInverted
AuthorsOpenReview ↗

Response to Reviewer dzgM (2/4)

Contrastive VLMs

ModelA (%)dzd_zRegime (A)Regime (dzd_z)
SigLIP-B/3222.5-0.74InvertedInverted
OpenCLIP-B/16 (LAION80M-3B)13.8-1.01InvertedInverted
OpenCLIP-L/14 (LAION80M-3B)12.1-1.10InvertedInverted
OpenCLIP-B/16 (LAION400M-3B)11.5-1.12InvertedInverted
OpenCLIP-B/16 (LAION80M-13B)8.5-1.25InvertedInverted
OpenCLIP-L/14 (LAION80M-34B)7.0-1.44InvertedInverted
OpenCLIP-B/16 (LAION80M-34B)7.0-1.37InvertedInverted
OpenCLIP-B/32 (LAION2B-34B)6.3-1.44InvertedInverted
MetaCLIP-H/144.6-1.33InvertedInverted
CLIP-L/144.3-1.58InvertedInverted
OpenCLIP-B/16 (LAION400M-34B)3.5-1.46InvertedInverted
OpenCLIP-B/16 (LAION2B-3B)3.5-1.60InvertedInverted
OpenCLIP-B/16 (LAION2B-13B)3.5-1.61InvertedInverted
OpenCLIP-L/14 (LAION2B-3B)3.4-1.62InvertedInverted
OpenCLIP-B/16 (LAION400M-13B)3.3-1.66InvertedInverted
MetaCLIP-L/143.2-1.76InvertedInverted
MetaCLIP-G/143.0-1.81InvertedInverted
OpenCLIP-H/14 (LAION2B-34B)2.9-1.87InvertedInverted
OpenCLIP-L/14 (LAION2B-34B)2.8-1.91InvertedInverted
MetaCLIP-B/322.6-1.73InvertedInverted
CLIP-B/322.3-1.91InvertedInverted
OpenCLIP-g/14 (LAION2B-34B)2.2-2.00InvertedInverted
OpenCLIP-B/16 (LAION2B-34B)2.2-1.92InvertedInverted
OpenCLIP-g/14 (LAION2B-13B)2.0-1.97InvertedInverted
CLIP-B/162.0-1.80InvertedInverted
MetaCLIP-B/161.5-1.95InvertedInverted

As can be seen, in terms of explainability regimes, this metric agrees with the consistency metric A\mathcal{A} reported in the paper. Yet, this metric does show how some models are deeper in their regime than others.

We thank the reviewer for this suggestion, we added this analysis under App. F.

[1] Jacob Cohen, “Statistical power analysis for the behavioral sciences” (1988)

AuthorsOpenReview ↗

Response to Reviewer dzgM (3/4)

The paper focuses exclusively on zero-shot patch-token decoding, a method with well-studied limitations. In particular, applying a classification head trained on CLS tokens to individual patch tokens may produce misleading results due to representation mismatch. Alternative methods, such as Tuned Lens, were developed to address related limitations. Therefore, inverted explanations might reflect limitations of the decoding method rather than the absence of interpretable information. The broad framing of an "explainability illusion" seems too general without evaluating alternative approaches.

Would a trained linear probe recover aligned explanations in models that show inverted patch decoding?

Indeed, our work focuses on zero-shot patch-token decoding, and we only investigate the explainability illusion for that particular explainability method. As noted also by Reviewer Nsko, these approaches are still commonly used, assuming that the head trained on CLS tokens will generate viable results when applied to individual patch tokens . Therefore, our aim in this paper is to shed light on exactly how these assumptions do not hold, and reveal how such explainability methods are flawed. We do not claim the models themselves do not contain interpretable information, but rather that the explainability provided by zero-shot patch-token decoding methods, may be flawed. We added clarifications throughout the text (specifically in the Abstract, and Secs 1, 3.1, and 6) that we consider the explainability with regards to zero-shot patch-token decoding methods, and how other approaches, including trainable methods, may succeed to produce aligned-explainable results.

Indeed, adding trainable elements to patch decoding has been shown to increase alignment of explainability maps, e.g., [2] which included losses aimed to directly affect the alignment of explainability maps during training. This is similar to what was shown in Tuned Lens. Specifically, adding a learnable linear probe masks how the zero-shot use of a classification head not trained to be applied on patch tokens is flawed. We trained such a linear probe on the inverted-explainable models from Tab. 1 and tested the explainability with this probe:

ModelOriginal A\mathcal{A}A\mathcal{A} w/ linear probeOriginal dzd_zTrained dzd_zOriginal regimeTrained regime
ViT-B/32-SAM14.4%79.7%−0.980+0.788InvertedAligned
ViT-B/16-SAM6.4%67.2%−1.291+0.412InvertedMarginally aligned by A\mathcal{A}; unexplainable by dzd_z
DeiT-Tiny/164.1%94.4%−1.521+1.537InvertedAligned
DeiT-Small/161.4%92.8%−1.984+1.446InvertedAligned

Indeed, learned representation alignment helps produce aligned-explainable maps in most cases, reinforcing our claim of zero-shot patch decoding being a flawed process. Interestingly, this does not mitigate inverted-explainability completely in all models, as ViT-B/16-SAM is only marginally aligned according to A\mathcal{A}, and unexplainable under dzd_z. However, we emphasize that we focus on zero-shot approaches as they are commonly regarded as legitimate interpretability approaches under assumptions that do not hold.

[2] Chefer et al. Optimizing relevance maps of vision transformers improves robustness. (NeurIPS 2022)

Comments on the paper

1 comment

As the discussion period is about to end, we would like to kindly ask if the reviewers have remaining concerns we should address. We believe we have addressed all comments and questions raised by the reviewers and would greatly appreciate the opportunity to address any additional feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.