acceptodds
Withdrawn from ICLR 2027

Exposing Vulnerabilities in Visible-Infrared VLMs: A Unified Adversarial Framework with Cross-Task Transferability

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision–language models (VLMs) have made substantial progress in multimodal understanding and have achieved strong performance across diverse vision–language tasks. However, existing VLM security studies remain largely confined to single-modality settings, predominantly in the visible spectrum, leaving adversarial robustness in visible–infrared (VIS–IR) scenarios underexplored. This omission is consequential for real-world perception systems, where VIS–IR sensing is widely deployed to sustain reliable semantic understanding under challenging imaging conditions. Motivated by this underexplored cross-modal threat setting, we propose a curved-edge fractal adversarial patch (CFPatch), the first unified adversarial patch framework for cross-modal attacks against VIS–IR VLMs. Specifically, CFPatch takes triangular fractal geometry as its base and transforms rigid straight-edged primitives into Bezier-curved elements, preserving fractal self-similarity for multi-scale adversarial effects while enhancing structural expressiveness through smoother contours, richer directional variation, and flexible shape deformation. Complementing this global geometric disruption, we introduce a modality-specific Fraser-spiral rendering mechanism that injects fine-grained texture distortions and misleading perceptual cues tailored to visible and infrared imagery. Through this geometry–appearance coupling, CFPatch forms a dual-level perturbation: the curved fractal geometry disrupts global shape perception, while Fraser-spiral rendering corrupts local appearance interpretation via modality-specific texture interference. We further use expectation over transformation (EOT) to improve patch robustness against common image-level transformations. Extensive experiments show that CFPatch effectively fools VIS–IR VLMs and outperforms standard patch baselines in both attack effectiveness and robustness. Moreover, adversarial samples optimized for the zero-shot classification task transfer well to image captioning and visual question answering (VQA), demonstrating the strong cross-task transferability and generalizability of CFPatch across downstream tasks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision-Language Models, Adversarial Patch, Visible-Infrared Perception, Cross-Task Transferability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_atVQOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CFPatch, a VIS–IR adversarial patch combining shared curved-fractal geometry with spectrum-specific textures. It uses PSO with EOT to jointly attack zero-shot classification and evaluates the resulting adversarial samples on image captioning and VQA without further optimization.

Critical strengths

The joint success criterion requires both visible and infrared predictions to fail on the same sample, preventing strong results on one branch from masking failure on the other. Reusing the adversarial samples for captioning and VQA also provides useful evidence that the attack effects extend beyond classification.

Critical weaknesses
  1. The main experiments use infrared images synthesized from visible images and general-purpose VLMs adapted through LoRA. Although the appendix includes real LLVIP data, the results do not establish effectiveness against dedicated infrared VLMs trained on real thermal imagery. This limits the evidence supporting the claimed contribution to VIS–IR VLM security.

  2. The shape and texture design relies heavily on hand-crafted structures and numerous hyperparameters. The motivation for curved fractals and Fraser patterns is not sufficiently convincing, and their generality remains unclear. Other shape or color parameterizations could provide similar flexibility, but the paper does not establish why these particular choices are preferable.

Questions for authors
  1. The main benchmark consists of COCO images and synthetic infrared counterparts. The real-data evaluation is limited to pedestrian classification on LLVIP, and the victim models are LoRA-adapted versions of general-purpose models. Dedicated infrared models such as IRGPT and Thermo-VL are not evaluated, leaving the applicability to real infrared perception unclear.

  2. Thermal infrared intensity depends on temperature, emissivity, and surrounding radiation rather than visible color alone. CFPatch generates visible colors and infrared grayscale textures from a shared coordinate field, but does not establish a material-based relationship between these appearances. An insulating patch can produce a relatively uniform cool region, whereas the proposed grayscale texture contains spatial intensity variations whose physical realization is unexplained.

  3. Table 1 reports substantially higher VIS-only ASR than TOUAP, but the comparison provides insufficient detail to explain this gap. TOUAP explicitly optimizes a grid of patch colors, offering considerable appearance flexibility. The limited information on baseline adaptation and optimization leaves me unconvinced that TOUAP was sufficiently optimized for this setting.

  4. The comparison omits recent methods such as CDUPatch, despite citing it as related work.

  5. The paper does not explain how the VQA questions are constructed or selected. Reusing the corresponding annotations does not specify the question set or selection procedure, reducing the reproducibility of the quantitative results.

  6. Extending the attack to VLMs does not by itself establish substantial methodological novelty. The particular shape and color models still lack a convincing motivation. The term “cross-modal” is unclear here, since VIS and IR are both visual inputs, while VLMs combine vision and language.

Reviewer_yBPROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

To address the lack of research on adversarial robustness in visible-infrared (VIS–IR) bimodal scenarios, the paper proposes a unified adversarial patch for Vision–language models (VLMs): CFPatch. To achieve both global geometric interference and local fine-grained interference, CFPatch introduces Bézier curves into a three-layer recursive equilateral triangle and a Fraser spiral rendering mechanism to inject modality-specific textures into both modalities. Experimental results show that the current method ranks best among comparative methods. However, due to the limited number of comparative methods, the current experimental results are not sufficiently objective and comprehensive.

Critical strengths

The problem formulation is novel, extending from a single modality to a joint attack on both visible and infrared modalities. The adversarial objective has also been changed from traditional detection and tracking to the currently popular semantic reasoning of vision-language models. Moreover, the experimental design is comprehensive, including zero-shot comparisons with other adversarial methods, cross-task validation of image captioning and visual question answering, and ablation studies of individual components. These experiments collectively enhance the credibility of the results.

Critical weaknesses
  1. Overall innovation of the paper is insufficient. It merely represents a simple combination of existing methods rather than an essential innovation to the underlying architecture;
  2. Three contributions in Chapter 1 should be reorganized. Currently, one is about innovation, and two are about experiments, which makes the paper appear to lack innovation;
  3. Section 3.6 provides insufficient justification for the choice of the particle swarm optimization (PSO) method. Besides PSO, there are other intelligent optimization methods that also satisfy the problem conditions;
  4. The zero-shot experiment has too few comparative methods, making it difficult to obtain convincing, objective, and comprehensive experimental results;
  5. The captioning and VQA conclusions rely on GPT-5 as the sole judge. The paper does not report judge calibration against human ratings, inter-rater agreement, and sensitivity to prompt or decoding choices.
Questions for authors
  1. How are the positions and the area of the patches determined? Is there any discussion of the impact of different positions or different areas on adversarial effectiveness?
  2. Is CFPatch one fixed parameter vector shared across all evaluation images, optimized on a disjoint training set, or independently optimized for each pair?
  3. Has the adversarial effectiveness of this method been tested on commercial vision-language models such as GPT and Gemini?
Reviewer_bGgdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes CFPatch for adversarial attacks on visible and infrared vision–language models. It combines shared curved-fractal geometry with modality-specific textures, optimized using PSO and EOT. Evaluation covers zero-shot classification and transfer to captioning and VQA, primarily on RGB images with synthesized infrared counterparts, with additional classification experiments on real LLVIP pairs.

Critical strengths

The topic of Vulnerabilities in Visible-Infrared VLMs is intetesting. The evaluation of transfer from classification to captioning and VQA is useful.

Critical weaknesses
  1. The distinction from existing RGB–thermal adversarial patch frameworks is not sufficiently clear. The paper should discuss and compare with more recent related methods, rather than mainly comparing against Unified Patch and TOUAP.

  2. It is unclear how much of the contribution comes from a new cross-modal attack framework versus applying a different patch design to VLMs. The claim of the “first unified adversarial patch framework” for VIS–IR VLMs is therefore too strong without a more thorough comparison.

  3. The experiments attack visible and infrared models separately and define joint success as both models failing. Whether this represents vulnerabilities of an integrated VIS–IR VLM needs clarification.

  4. The experimental validation of the VIS–IR setting is limited. The main experiments use infrared images converted from RGB images, which may not adequately represent real thermal observations. Although Appendix G includes real LLVIP pairs, this evaluation is limited to pedestrian classification with CLIP backbones, without corresponding baseline comparisons or captioning and VQA results.

Questions for authors
  1. What are the main differences from existing RGB–thermal cross-modal adversarial patch frameworks? Please provide more discussion and comparisons with recent related work to support the novelty claim.
  2. Does “VIS–IR VLM” refer to a model jointly processing visible and infrared inputs, or separate visible and infrared-adapted VLMs? Please clarify the scope of the claimed cross-modal vulnerabilities.
  3. Why are the main experiments conducted on synthesized infrared images? Can the real-data evaluation include baseline comparisons and the captioning and VQA tasks?
  4. Have the authors considered physical attacks? Can a single physical patch realize the separately optimized visible and infrared appearances?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.