- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to efficiently generate fine-grained vision-language interaction explanations for dual encoders such as CLIP. The authors propose a gradient-difference framework that masks a target phrase or image region, computes the difference between the matching-score gradients before and after masking, and applies this difference to Grad-CAM, GAME, and Grad-ECLIP, yielding Grad-CAM+, GAME+, and Grad-ECLIP+. The framework is evaluated on ImageNet, MS COCO, and Flickr30k using phrase-targeted deletion/insertion, the Pointing Game, computational cost, and user studies. The results show that Grad-ECLIP+ generally achieves performance comparable to existing second-order methods while substantially reducing computational and memory costs.
Critical strengths
-
The problem is important and practically relevant. As discussed in the Introduction, ordinary first-order attribution can identify which image regions or text tokens are important to the overall image-text similarity, but it cannot further determine which image region is associated with a particular phrase. Understanding such fine-grained relationships is valuable for assessing whether the model relies on meaningful vision-language correspondences.
-
The efficiency results are convincing. Table 1 shows that, under the evaluated configurations, Grad-CAM+, GAME+, and Grad-ECLIP+ require substantially less processing time and memory than exCLIP and FIxLIP. Tables 2 and 5 further indicate that this efficiency improvement does not substantially compromise phrase-targeted deletion/insertion or localization performance.
-
The core idea is clear and easy to understand. The intuition is straightforward: if a textual concept affects the contribution of certain image regions to the matching score, masking that concept should change the gradients of those regions. The proposed framework directly reuses existing gradient-based explanation methods and provides a substantive and memorable insight rather than a complicated recipe composed of many modules.
Critical weaknesses
-
Gradient difference may reflect attribution redistribution rather than semantic correspondence. For example, after masking "car," the model may reallocate more matching contribution to "dog," producing a large negative gradient difference in the dog region. This indicates a competitive dependency between the two concepts, but it does not imply that the dog region semantically corresponds to "car." The paper also does not explain how handles negative values: ReLU would discard competitive interactions, whereas taking the absolute value could incorrectly highlight non-target regions. This issue concerns Sections 3.2-3.3 and Equations (5) and (7).
-
The current experiments do not rule out this alternative explanation. Tables 2 and 5 mainly test whether high-scoring regions localize the target object, but they do not report signed gradient changes in target and non-target regions. Consequently, these results cannot distinguish semantic correspondence from global attribution redistribution. The authors should report signed interaction maps and separately analyze positive and negative gradient differences in target and non-target regions.
Questions for authors
-
How does handle negative gradient differences in Grad-CAM+, GAME+, and Grad-ECLIP+? Does the implementation use ReLU, absolute values, clipping, or min-max normalization? Please clarify the exact processing used for Figures 1 and 3 and Tables 2 and 5.
-
What are the results in Tables 2 and 5 when explanations are generated separately using $ \Delta G_P^+=\max(\Delta G_P,0)$ and $ \Delta G_P^-=\max(-\Delta G_P,0)?$ Do the positive and negative components have different interpretations?
The text and token labels in Figure 3 are too small to read comfortably at the normal page size. I suggest increasing the font and overall visualization size.
2 comments
Rebuttal to reviewer F83V (Part 2)
What are the results in Tables 2 and 5 when explanations are generated separately using and ? Do the positive and negative components have different interpretations?
We thank the reviewer for raising the distinction between the effects that positive and negative gradient differences may produce on the attribution map.
We will add a new Appendix section to the revised paper to discuss the effects of the positive and negative components in the gradient differences. To investigate the effects of the positive and negative gradient differences, we apply the and to the Grad-ECLIP+ and evaluate the deletion, insertion, and pointing game as suggestions. In the deletion and insertion metrics, both the target phrase and the caption without the phrase are used to calculate the score changes.
Table A. Deletion and insertion AUC for Grad-ECLIP+ on ViT-B/32 using the positive gradient component ().
| Scoring text | Flickr30k deletion ↓ | Flickr30k insertion ↑ | MS COCO deletion ↓ | MS COCO insertion ↑ | ImageNet deletion ↓ | ImageNet insertion ↑ |
|---|---|---|---|---|---|---|
| Phrase | 0.3837 | 0.8917 | 0.3359 | 0.8958 | 0.0668 | 1.0819 |
| Caption without phrase | 0.5421 | 0.6293 | 0.7739 | 0.6220 | 0.5597 | 0.4570 |
Table B. Deletion and insertion AUC for Grad-ECLIP+ on ViT-B/32 using the negative gradient component ().
| Scoring text | Flickr30k deletion ↓ | Flickr30k insertion ↑ | MS COCO deletion ↓ | MS COCO insertion ↑ | ImageNet deletion ↓ | ImageNet insertion ↑ |
|---|---|---|---|---|---|---|
| Phrase | 0.8876 | 0.3865 | 0.8951 | 0.3419 | 1.0946 | 0.0660 |
| Caption without phrase | 0.6407 | 0.5533 | 0.6195 | 0.7835 | 0.5072 | 0.6831 |
Table C. Interaction recognition for Grad-ECLIP+ on ViT-B/32 using each gradient component separately.
| Gradient component | Flickr30k PGA ↑ | Flickr30k PGE ↑ | MS COCO PGA ↑ | MS COCO PGE ↑ | ImageNet PGA ↑ | ImageNet PGE ↑ |
|---|---|---|---|---|---|---|
| Positive () | 77.57% | 59.28% | 86.01% | 63.91% | 92.59% | 69.35% |
| Negative () | 27.42% | 24.97% | 22.63% | 19.09% | 3.30% | 4.30% |
Tables A, B, and C show that the positive and negative gradient differences prioritize different regions. Across all three ViT-B/32 datasets, maps generated from achieve lower deletion AUC and higher insertion AUC for the target phrase, as well as higher pointing-game accuracy and energy within the annotated target region. When scored against the caption without the phrase, maps generated from achieve better deletion and insertion AUCs. These results indicate that the positive gradient component is more closely aligned with visual evidence for the target phrase. The negative component can highlight regions that support the remaining caption, consistent with attribution redistribution after masking.
The text and label size in Figure 3 are too small.
Thanks for pointing that out. We will enlarge the font size and visualization when more space is available if preparing the final version.
Rebuttal to reviewer F83V (Part 1)
How does handle negative gradient differences in Grad-CAM+, GAME+, and Grad-ECLIP+?
(1) The sign handling in our paper:
We will clarify the sign handling in Eq. (7). After calculating the gradient difference, the ΔGP substitutes the gradient used in the standard Grad-ECLIP, GradCAM, and GAME, where Grad-CAM+ and Grad-ECLIP+ apply ReLU after channel aggregation, and GAME+ clips negative attention-weighted responses before propagation. Therefore, the final attribution maps are all based on positive gradients, which highlight regions/words that contribute positively to the matching score. In Figures 1 and 3, the heat maps are min-max normalized to 0 to 1 before visualization. In Table 2, the deletion and insertion ranks the patch values in descending order. In Table 5, the peak location and energy inside the target mask are calculated on the positive heatmaps.
The relationship between positive gradient differences and the use of ReLU:
For positive and negative gradient differences, we should clarify that using ReLU does not mean discarding negative gradient differences or competitive interactions, since both positive and negative gradient differences are multiplied by the feature representations, and ReLU is adopted at the outermost calculation, after the channel aggregation, as in the standard Grad-ECLIP and Grad-CAM. The usage of ReLU aims to keep only the matched patch-token interactions that contribute positively to the image-text matching score. Therefore, our method takes both the positive and negative gradient differences into the patch-token interaction calculation. Then, the results in the experiments below show the evidence that, in the gradient differences, the positive component is more closely aligned with visual evidence for the target phrase, while the negative component can suppress the responses of regions that support the remaining caption.