- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the vulnerability of VLM explanations to adversarial perturbations. It proposes X-Shift, a white-box attack that redirects the explanation heatmap of a predicted class toward a target region while preserving the original prediction. The method combines normalization-invariant explanation objectives with prediction-preservation losses. Experiments on three datasets and three CLIP backbones show explanation redirection on 45 selected image-backbone pairs.
Critical strengths
-
The paper is generally well written, organized, and easy to follow. The problem formulation is clear, and the proposed attack objectives are consistent with the goal of manipulating explanations while preserving predictions.
-
The results demonstrate explanation redirection while maintaining the original predictions on the evaluated samples.
Critical weaknesses
-
Several existing studies have already demonstrated that model explanations can be manipulated while preserving predictions, although most do not focus on VLM. The paper discusses these studies in Section 2 but does not include direct experimental comparisons with them.
-
The current baselines do not adequately establish the advantage of X-Shift over existing explanation-targeted attacks. Given the white-box setting, demonstrating successful explanation manipulation is less surprising without stronger competing methods. This also makes it difficult to assess the methodological novelty beyond adapting existing ideas to CLIP-based explanations.
-
The motivation for Section 3.4 and Proposition 1 is not entirely convincing. The proposition characterizes gradient directions in explanation-map space, but it remains unclear how this property translates into more effective optimization of input perturbations. In particular, the theoretical result does not directly establish the practical advantage of the proposed attack.
-
In Table 3, removing the contrast loss improves IoU→t from 0.86 to 0.92, whereas removing the shape loss substantially reduces it to 0.46. This suggests that the shape loss accounts for most of the redirection effect, while the necessity of the contrast loss remains unclear.
Questions for authors
-
Can the authors compare X-Shift with existing explanation-targeted attacks under the same white-box setting and perturbation budget?
-
What is the practical role of Proposition 1 in the attack design? Can the authors further explain how the normalization-invariance property in explanation-map space improves optimization in input space? If available, an ablation that changes only the normalization strategy would be useful.
-
Given that removing the contrast loss improves the reported IoU→t, what motivates retaining it in the final objective? Could the authors provide further analysis of the interactions between the contrast, shape, and preservation losses?
-
Why select the 45 selected image-backbone pairs rather than the entire datasets? Also, given the relatively limited transferability reported in Appendix C, could the authors clarify the scope of their generalization claims?