- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper is concerned with computing attributions for Vision Transformers (ViTs). Specifically, it identifies a critical failure mode of layer-wise relevance propagation (LRP) in ViTs: signals in the residual stream and the attention/MLP updates that cancel each other, can lead to attribution explosion (Sec. 3). Based on this insight, the paper proposes an improved LRP version (ResLRP) that accounts for this issue while provably satisfying desirable guarantees such as conservation (Sec. 4). Finally, the paper conducts a variety of different experiments, demonstrating consistent gains with the proposed ResLRP.
Critical strengths
- The paper is well-written and easy to follow.
- The paper has a clear, logical structure: problem identification, fix, empirical support.
- The identified problem (exploding attribution scores when residual cancellation happens) and the fix are convincing.
- The empirical analysis is extensive and nicely combines a variety of established benchmark evaluations (5.1) with an ablation to establish that the residual cancellation is indeed an important factor of the proposed method (Sec. 5.2). Further, the paper extends the experiments to sparse autoencoders and vision-language models (Sec. 5.3-5.4).
Critical weaknesses
I currently do not see critical weaknesses that would hinder me from accepting the paper. However, I have a few minor comments:
- seems to be not defined which could help readers less familiar with LRP.
- L. 216 "Unless stated otherwise, Rup is obtained from Rin by swapping the subscripts in and up:" - it took me a while to understand what this sentence is trying to say.
- Do you have an intuition on why this residual cancellation would be less pronounced in language models?
- Do you think the method could also improve LRP for language models even if the effect is less pronounced?
Questions for authors
Nothing critical. For minor points please see weaknesses.
3 comments
I thank the authors for the further clarification. After reading the reply, alongside the other reviews and their replies, I maintain my initial score and still vote for accepting the paper.
We would like to express our heartfelt gratitude for your exceptionally encouraging feedback. Your highly positive assessment of our work means a great deal to us and is truly motivating. We sincerely appreciate the time and consideration you have devoted to reviewing our manuscript, as well as your continued support.
We would like to express our sincere gratitude for your careful review of our manuscript. We are glad that the underlying diagnosis and proposed solution resonated with you, and truly appreciate your positive assessment. Your constructive feedback has also helped us further strengthen the clarity and presentation of our work.
W1 — Undefined Rout. Thank you, we agree and will add an explanation. LRP explains a prediction with a single backward pass, similar to backpropagation. It starts at the target logit, whose relevance is its value. Each layer then splits the relevance of its output among its inputs in proportion to their forward contributions, so that the total is conserved. In practice, this is a backward pass with a modified gradient g~: the relevance of any tensor x is Rx=x⊙g~x. Rout is the relevance this pass has assigned to the residual output zout after going through all layers above it. The rule in Eq. (1) splits Rout into Rin and Rup, separately for every token and channel. Rin continues backward along the residual stream, and Rup goes into the attention or MLP block.
W2 — L. 216 wording. Thank you for pointing out this confusing wording. We will clarify it in the final manuscript.
W3 — Why less cancellation in LLMs? We have a hypothesis, but testing it rigorously was beyond the scope of this paper unfortunately, and we plan to address it in future work. Two observations from the paper motivate it. First, cancellation is strongest in the early ViT layers (Fig. 2a). Second, within a layer it concentrates on patches that contain edges or have high saliency (Fig. 2c). Early layers of vision networks are known to learn low-level, edge-like filters (Zeiler & Fergus, 2014; Yosinski et al., 2014), and ViTs learn similar filters in their patch embedding (Dosovitskiy et al., 2021). Such filters must respond strongly to small changes in local contrast. A cancellation is one way to achieve this sensitivity: when zout=zin+zup is the sum of two large branches of opposite sign, a small relative change in either branch changes zout a lot. Language models, on the other hand, operate on symbolic text manifolds, so there is no continuous low-level signal that demands such sensitivity. Since this remains a hypothesis without conclusive evidence, we did not state it in the paper.
W4 — ResLRP for LLMs. We thank the reviewer for this valuable suggestion, which prompted us to strengthen the paper further. We applied ResLRP to language models and observed no improvement over AttnLRP. This is consistent with our analysis: residual amplification is clearly weaker in LLMs than in ViTs and AttnLRP heatmaps of LLMs are already clean. With little amplification, there is little for ResLRP to correct. We will add this negative result to the paper, as it supports cancellation as the cause of the noise in ViT heatmaps. Interestingly, this does not apply to LLM backbones in VLMs. Applying ResLRP on LLMs in VLMs improves the faithfulness. We believe this is due to the fact that image domain features of the vision backbone leak into the langauge backbone.
Thank you again for taking your time reviewing our paper, we are truly encouraged by the positive review.