- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces X-shaped Sparse Pixel Attack (XSPA), which investigates whether highly sparse perturbations with fixed geometry can disrupt the shared visual-textual representations of vision-language models. XSPA restricts perturbations to two intersecting diagonal lines, modifying approximately 1.04% of image pixels. It combines classification-oriented optimization, semantic attraction and suppression, and regularization on perturbation magnitude and linewise smoothness. Experiments on COCO report substantial degradation in zero-shot classification, image captioning, and visual question answering across multiple models, suggesting that VLMs remain vulnerable even under strongly constrained perturbation support.
Critical strengths
-
The experimental evaluation is relatively comprehensive, covering zero-shot classification, image captioning, and VQA across four CLIP-style encoders and six downstream VLMs. The paper also examines optimization budgets, perturbation geometry, and loss components, while the appendix provides matched-budget comparisons, perceptual measurements, and seed-stability analyses.
-
The paper has a clear motivation: investigating whether VLMs remain vulnerable when perturbations are restricted in both spatial coverage and geometric freedom. The fixed X-shaped support provides a concrete setting for examining how constrained visual perturbations affect semantic behavior across downstream tasks.
Critical weaknesses
-
The novelty appears limited: line-shaped adversarial scratches (Jere et al., 2019) and diagonal perturbations against multimodal models (Botocan et al., 2024) already establish geometrically constrained sparse attacks, albeit under different threat models. XSPA mainly pairs existing optimization components with a fixed two-diagonal mask, and a matched-budget saliency Top-k baseline achieves higher targeted ASR (94.44% vs. 83.44%), leaving the value of this geometry unclear.
-
Claims of generalization and targeted cross-task transfer need qualification. Section 4.4’s cross-dataset results are missing (unresolved “Table ??”). Table 7 reports strong targeted classification but zero targeted-caption and joint three-task successes on 214 jointly eligible images, with limited targeted VQA success, suggesting degradation rather than transfer to a shared target. GPT-based judges lack human validation, and caption-score calibration shows low agreement (ICC = 0.301).
Questions for authors
-
Could the authors provide the cross-dataset results referenced in Section 4.4 and clarify whether these experiments evaluate classification alone or also captioning and VQA?
-
How should the claimed target-oriented cross-task transfer be interpreted given the zero targeted caption and joint success rates in Table 7? Could independent human evaluation further validate the reported semantic degradation?