- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper challenges the common assumption that zero-shot patch-token decoding provides reliable explanations for Vision Transformers. Through experiments on a wide range of ViTs and vision-language models, combined with controlled synthetic studies and a toy theoretical model, the authors argue that patch-decoding explanations can be aligned, inverted, or uninformative, while preserving the same predictive performance.
Critical strengths
Extensive empirical evaluation: Analysis spans many classifier and contrastive VLM architectures, scales, and training regimes.
Theoretical support: Section 5 provides a constructive example showing that the same optimal classifier can admit different explainability regimes.
Critical weaknesses
Overall, I see value in the scale of the empirical study, rather than in the claimed conceptual discovery. Inverted CLIP visualisation was previously documented, and the theoretical message of Section 5 closely follows established non-identifiability and implementation-invariance arguments. The paper’s clear added value is the systematic characterisation of patch-decoding regimes across model families and controlled training settings. I would find the contribution more convincing if the claims were narrowed accordingly, the closest theoretical literature were incorporated, and the distinction between explaining an input-output function and inspecting implementation-specific representations were addressed.
The authors miss to put into context relevant past work.
On Identifiability in Transformers [A] proves that, under conditions involving sequence length and attention-head dimension, different attention distributions can produce the same contextual embeddings. The paper explicitly frames this as a structural identifiability problem and concludes that self-attention distributions are not directly interpretable.
The submission’s construction is not identical. Brunner et al. study attention-weight identifiability, whereas this paper studies head-decoded patch-token representations and gives aligned versus inverted spatial regimes. But the theoretical story is sufficiently close that Section 5 should be positioned as a specialisation or extension of transformer non-identifiability, not as introducing the general insight.
- Attention is not Explanation [B] shows empirically that very different attention distributions can yield equivalent predictions, while learned attention may be poorly correlated with gradient-based importance measures. Again, the object being interpreted differs: Jain and Wallace analyse attention distributions; this paper analyses direct decoding of patch representations. Nevertheless, the epistemic point is almost the same: an intuitive token-level visualisation is not uniquely constrained by predictive behaviour. The subsequent Attention is not not Explanation is also relevant because it cautions that such negative conclusions depend heavily on the definition of “explanation” and on the experimental test used. It proposes comparisons across seeds and adversarially trained alternatives, which are directly pertinent to this submission’s claims about training-seed dependence.
[A] Brunner et al., On Identifiability in Transformers, ICLR 2020 https://iclr.cc/virtual_2020/poster_BJg1f6EFDB.html [B] Jain and Wallace, Attention is not Explanation, NAACL 2019 https://arxiv.org/abs/1902.10186 https://aclanthology.org/N19-1357/
Additionally: The toy construction uses a specialised embedding and non-standard classification head, making the connection to real-world ViTs less convincing.
Theoretical result is illustrative rather than explanatory: It shows that multiple regimes are possible but does not explain why real models converge to a specific regime.
Overall: Interesting empirical study with useful evidence, but the novelty of the central conceptual claim and the theoretical contribution appears overstated.
Questions for authors
none
2 comments
Response to Reviewer Vfkh (2/2)
Additionally: The toy construction uses a specialised embedding and non-standard classification head, making the connection to real-world ViTs less convincing.
It is correct that the construction in Sec.5 uses a non-standard head and embeddings. However, the mere purpose of this construction is to exemplify that a model can implement the same classification rule in infinitely many ways, which differ in their explainability patterns.
The logarithmic head is not the cause for the existence of multiple explainability regimes. Specifically, in App. E.6 we remove it, leaving a linear head as in real-world ViTs. This linear head predicts the same class and leads to the exact same three regimes for the same values. We used the logarithm in Sec. 5 only to force the optimality of the model in terms of the logits, as it ensures the class-token output equals the exact posterior of the optimal classifier.
As for the nonlinear embedding used in the proof, it is relevant to real-world ViTs. Specifically, since it acts on each patch separately, it can be approximated to arbitrary accuracy by a per-token MLP block, which is common in ViTs. Therefore, a ViT with two layers can approximate our toy model on any bounded range of inputs, and regenerate these explainability behaviors. We can set the MLP of the first layer to approximate the embedding , ignoring its attention part. Then, the second layer can implement the rest of our toy model with the provided attention weights. Apart from LayerNorm, which our toy model omits, every component of the construction is thus a standard ViT component. We added this point to App. E.3.
Theoretical result is illustrative rather than explanatory: It shows that multiple regimes are possible but does not explain why real models converge to a specific regime.
Indeed, we do not yet have a clear understanding on why real models converge to specific regimes. The main contribution of our work is the observation that the common assumptions of patch-decoding interpretability does not hold in the wild, and cannot be assumed to hold a-prioi. We hope a more fine-grained understanding of different mechanisms can be achieved in future works.
Response to Reviewer Vfkh (1/2)
We thank the reviewer for pointing out these important relevant works. This line of work was a miss on our part, and we modified our paper accordingly. We added clarifications regarding our scope throughout the text, including narrowing our claims, emphasizing our focus on the explainability of a specific zero-shot approach, and incorporating the relevant theoretical literature (see Abstract, and Secs 1,2, 3.1, 5 and 6, and Apps. D.1 and E).
As the reviewer mentions, all three works focus on attention distributions, while our analysis specifically targets patch-token decoding.
It should be noted that the term "identifiability" used in ref. [A] refers to input-dependent identifiability for a single sample. In contrast, our theoretical construction in Sec. 5 addresses token-decoding weight identifiability given the classification head over the entire training set (or the global input-output mapping across all inputs). Namely, we show that the token-decoding maps weights of a model are not uniquely determined by the input-output function it implements, and that different implementations of that function lead to different explainability behaviors. Additionally, the results of [A] hold only for cases where the sequence length is larger than the head dimensions, whereas our results don’t have such a constraint (the sequence length N is not dependent on the attention head dimension K+1). Nevertheless, it is certainly an important relevant work, which we missed.
Regarding “Attention is not Explanation” [B1] and “Attention is not not Explanation” [B2], we agree that the underlying story is similar on a high level, except again, that they analyze attention matrices and we analyze token-decoding maps. We updated the manuscript to reflect the similarity of the themes. We note that a main concern in [B2] regarding the adversarial setup in [B1] is resolved in our case. Specifically, whereas in [B1] the links between the attention distribution and the trained model are removed, our construction here shows the adversarial example can be a direct result of a fully trained model. Furthermore, we do not claim a hard negative result as in [B1], but rather a softer cautionary tale on the use of zero-shot patch decoding explainability maps, similarly to [B2].
Please see the revised Secs 1,2, 5 and 6, and Apps. D.1 and E.