acceptodds
Under review as a conference paper at ICLR 2027

XSPA: Crafting Imperceptible X-Shaped Sparse Adversarial Perturbations for Transferable Attacks on VLMs

Abstract

Vision-language models (VLMs) rely on a shared visualtextual representation space to support tasks such as zeroshot classification, image captioning, and visual question answering (VQA). Although this shared space enables strong cross-task generalization, it also creates a critical vulnerability: subtle visual perturbations may propagate through the common embedding space and induce correlated semantic failures across tasks. This issue is especially concerning in interactive and decision-support scenarios, yet it remains unclear whether VLMs are still fragile under highly constrained, sparse, and geometrically fixed perturbations. We propose Xshaped Sparse Pixel Attack (XSPA), an imperceptible structured attack that restricts perturbations to two intersecting diagonal lines. Compared with dense perturbations and flexible localized patches, XSPA operates under a much stricter attack budget, offering a stringent test of VLM robustness. Within this sparse support, XSPA jointly optimizes a classificationoriented objective, cross-task semantic guidance, and regularization on perturbation magnitude and linewise smoothness, inducing both transferable misclassification and semantic drift in captioning and VQA while preserving visual subtlety. Under the default setting, XSPA modifies only about 1.04% of image pixels. Experiments on the COCO dataset show that XSPA consistently degrades performance across zero-shot classification, image captioning, and VQA. Zero-shot accuracy drops by 52.33 points on OpenAI CLIP ViT-L/14 and 67.00 points on OpenCLIP ViT-B/16, while GPT-4-evaluated caption consistency decreases by up to 58.60 points and VQA correctness by up to 44.25 points. These results show that even highly sparse and visually subtle perturbations with fixed geometric priors can substantially disrupt cross-task semantics in VLMs, revealing an important robustness gap in current multimodal systems.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Adversarial Attack, Sparse Adversarial Perturbation, Imperceptible Attack, Cross-task Semantic Disruption, Multimodal Robustness, CLIP, X-shaped Sparse Attack

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NsKMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces X-shaped Sparse Pixel Attack (XSPA), which investigates whether highly sparse perturbations with fixed geometry can disrupt the shared visual-textual representations of vision-language models. XSPA restricts perturbations to two intersecting diagonal lines, modifying approximately 1.04% of image pixels. It combines classification-oriented optimization, semantic attraction and suppression, and regularization on perturbation magnitude and linewise smoothness. Experiments on COCO report substantial degradation in zero-shot classification, image captioning, and visual question answering across multiple models, suggesting that VLMs remain vulnerable even under strongly constrained perturbation support.

Critical strengths
  1. The experimental evaluation is relatively comprehensive, covering zero-shot classification, image captioning, and VQA across four CLIP-style encoders and six downstream VLMs. The paper also examines optimization budgets, perturbation geometry, and loss components, while the appendix provides matched-budget comparisons, perceptual measurements, and seed-stability analyses.

  2. The paper has a clear motivation: investigating whether VLMs remain vulnerable when perturbations are restricted in both spatial coverage and geometric freedom. The fixed X-shaped support provides a concrete setting for examining how constrained visual perturbations affect semantic behavior across downstream tasks.

Critical weaknesses
  1. The novelty appears limited: line-shaped adversarial scratches (Jere et al., 2019) and diagonal perturbations against multimodal models (Botocan et al., 2024) already establish geometrically constrained sparse attacks, albeit under different threat models. XSPA mainly pairs existing optimization components with a fixed two-diagonal mask, and a matched-budget saliency Top-k baseline achieves higher targeted ASR (94.44% vs. 83.44%), leaving the value of this geometry unclear.

  2. Claims of generalization and targeted cross-task transfer need qualification. Section 4.4’s cross-dataset results are missing (unresolved “Table ??”). Table 7 reports strong targeted classification but zero targeted-caption and joint three-task successes on 214 jointly eligible images, with limited targeted VQA success, suggesting degradation rather than transfer to a shared target. GPT-based judges lack human validation, and caption-score calibration shows low agreement (ICC = 0.301).

Questions for authors
  1. Could the authors provide the cross-dataset results referenced in Section 4.4 and clarify whether these experiments evaluate classification alone or also captioning and VQA?

  2. How should the claimed target-oriented cross-task transfer be interpreted given the zero targeted caption and joint success rates in Table 7? Could independent human evaluation further validate the reported semantic degradation?

Reviewer_hBjgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

VLMs share a visual-textual representation space, and the authors argue that attacking this shared space can cause failures across many tasks at once; they therefore study how vulnerable VLMs are when the support, geometry, and optimization freedom of the perturbation are all restricted. To this end, they propose XSPA, which restricts perturbations to an X-shaped mask formed by two intersecting line segments. On a CLIP surrogate, XSPA jointly optimizes a classification loss, losses that pull the image representation toward a target text pool and push it away from a source text pool, and regularizers on perturbation magnitude and along-line smoothness, using MI-FGSM-style momentum, input diversity, and L∞ projection; the resulting images are transferred to downstream models without adaptation. The authors claim that XSPA is the first X-shaped sparse pixel attack and that it is imperceptible and transferable. Experiments use 300 COCO images to evaluate zero-shot classification on four CLIP-style encoders and image captioning and VQA on six VLMs, with GPT-based judges for the open-ended tasks; XSPA is compared with natural-light, shadow, and ITA attacks, and the ablations also compare different mask shapes.

Critical strengths
  1. The setting is relatively new. The paper attacks VLMs with a perturbation restricted to a fixed, thin X-shaped region, optimized on a CLIP surrogate and transferred to captioning and VQA models. I am not aware of prior work that combines these elements, although the X shape itself has been used before (see W1). Whether VLMs can still be attacked when the attacker has little control over where the perturbation goes is a question worth studying.

  2. The evaluation is broad and the appendix is careful. The paper tests four CLIP encoders and six VLMs on three tasks, and XSPA lowers performance on most of them. The appendix uses multiple seeds and proper paired statistics. The authors also report results that do not favor their method, for example that Top-k pixel selection works better than the X shape, and that no image succeeds on all three targeted tasks. This makes it easier to see what the method can and cannot do.

Critical weaknesses
  1. The main claims go beyond what the paper’s own results show. The paper says XSPA is imperceptible, transfers across tasks, and that the X shape is the key to its success. The appendix does not really support this. The X lines can be seen in the examples, and the authors themselves say in the appendix that attack success should not be taken as evidence of imperceptibility. For cross-task transfer, none of the 214 eligible images fools all three targeted tasks at once, and the caption results depend on a GPT judge with low absolute agreement and no human check. For the X shape, choosing the top-k pixels by gradient works better at the same budget, and moving or rotating the X matters much less than the number of pixels. Figure 5(k) does put the X ahead of other fixed shapes, but its lead over the circle is small and untested (see Q3). The X shape is also not new. CSPA (Ran et al.) already restricts perturbations to a fixed X shape with a similar pixel budget, and the paper cites it but does not mention this overlap. The real differences are the fixed location, the bounded perturbation and the transfer to VLMs. The contribution should be stated in these terms, rather than as the first X-shaped attack. This is the main reason for my score. After reading the appendix, what is clearly supported is that a sparse attack on a fixed X can fool the CLIP surrogate. That is a much weaker result than the title, abstract and conclusion suggest.

  2. The comparison with baselines is not reliable. All Clean and baseline numbers in Tables 1–3 are exactly the same as in the ITA paper, so they seem to have been taken from that paper rather than re-run, and this is not stated. It is also not clear that XSPA was evaluated in the same way. In Table 3, the Clean and baseline scores are all whole numbers but the XSPA scores are not, which suggests XSPA was tested on a different set of questions. If so, even the drops relative to Clean compare results from different evaluation sets. In addition, OpenFlamingo and BLIP-2 (ViT-L) are listed under EVA-CLIP, although both use a CLIP ViT-L/14 encoder, and the paper does not say which surrogate was used for which model. So it is hard to tell how much of the drop comes from real transfer. Since these tables are the main evidence that XSPA beats prior attacks, I cannot accept that conclusion as it stands. Clean and the baselines should be re-run under the same setup.

Questions for authors
  1. The Clean and baseline numbers in Tables 1–3 are identical to those reported in the ITA paper. Section 4.1 says the judge prompt and rubric are shared across all methods, but it does not say whether the Clean and baseline results were re-computed in this work or taken from ITA, or whether the same images and questions were used. The numbers also raise questions about the sample sizes. In Table 3, the Clean and baseline scores are whole numbers while the XSPA scores are not. In Table 1, EVA-CLIP accuracy is 63.71%, which is not possible on 300 images, although Appendix B says classification uses all 300 image IDs. Could the authors report the number of samples behind each column, and re-run Clean and the baselines under the same setup as XSPA?

  2. The paper claims XSPA is transferable, but it is not clear which results reflect actual transfer. The surrogate is stated only for the ablations and the appendix comparisons (OpenCLIP ViT-B/16, Table 4), not for the main experiments in Tables 1–3, which also use a different number of steps. If the surrogate is one of the four encoders in Table 1, that column is white-box. If the examples for each VLM were crafted on its own vision encoder, as in the ITA protocol, then most results in Tables 2–3 are close to white-box on the visual side. The only setting that appears to be cross-encoder is Appendix E, where targeted transfer is very weak, but its surrogate is not stated either. Could the authors state the surrogate for every experiment, and give a table of surrogate vs. target model that separates white-box, same-encoder and cross-encoder results?

  3. Some questions in Figure 5. (a) The target and source text losses are meant to help transfer to captioning and VQA, but Figure 5(l) only reports CLIP-side metrics on the surrogate, and there the full loss differs from the other two variants by about 2 points or less. Is there any result showing that these losses help on the downstream tasks?

(b) In Figure 5(k), the X beats the circle by only about 2 points, while ch anging the angle of the X alone moves U-ASR by about 4–5 points in Figure 5(g), and the angle, center and length of the X were all selected from these sweeps. Were the other shapes, especially the circle, tuned in the same way? What is the “Held-out 240” split? Also, Figure 5 shows single runs without error bars. Since the authors already use paired bootstrap and McNemar tests in Appendix C, could they do the same for Figure 5(k), so it is clear whether the advantage of the X is significant?

(c) Given that Top-k selection is stronger at the same budget (Appendix C), and that the center and angle of the X matter much less than the number of pixels, what do the authors conclude about how much geometric freedom is needed?

  1. The title and abstract describe XSPA as imperceptible. Is this meant with respect to humans? The paper does not include any human study. Apart from PSNR in the ablations, the only quantitative evidence in the main text is the GPT-4o naturalness score, and XSPA still scores clearly lower than clean images (about 2.7–2.9 vs. 3.65). Appendix D also says that attack success should not be taken as evidence of imperceptibility. Could the authors explain what they mean by “imperceptible” and what this claim is based on? If there is no human study, should the word be removed from the title and abstract?
Reviewer_i9mCOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces XSPA, a structured sparse adversarial attack constraining perturbations within fixed X shaped diagonal masks to evaluate cross task vulnerability of vision language models.

Critical strengths

This paper introduces XSPA, a structured sparse adversarial attack constraining perturbations within fixed X shaped diagonal masks to evaluate cross task vulnerability of vision language models.

Critical weaknesses
  1. The core empirical claim of strong cross task targeted transfer does not hold up under supplementary analysis. Even though XSPA yields large zero shot classification degradation on surrogate CLIP encoders, targeted failures rarely propagate simultaneously across classification, captioning and VQA on the same adversarial sample. As shown in Appendix E, zero joint three task targeted success is observed. The significant caption performance drop measured by GPT 4 judge only has moderate inter GPT correlation and low ICC scores (Appendix F), casting doubt on the reliability of LLM as judge metrics as primary evidence for semantic drift.
  2. The structured and cross modal adversarial attacks have been well studied. Fixing perturbation geometry to an X shape is a straightforward engineering instantiation rather than fundamental methodological innovation. Under identical pixel budget, saliency driven Top k attack achieves substantially higher attack success rate than XSPA (Appendix C). The paper acknowledges X mask is not optimal, yet the main text over emphasizes attack effectiveness without sufficiently tempering conclusions.
Questions for authors

The experimental setup contains confounding factors. Transfer evaluation mixes diverse encoders and VLMs without systematic control. Geometric ablations mostly tune mask density, so it remains unclear whether performance gains stem from X shape itself or simply larger effective perturbation budget.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.