acceptodds
Withdrawn from ICLR 2027

Revealing Rainy-Day Vulnerabilities in Vision-Language Models: A Two-Stage Semantic Decoupling Approach

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision–Language Models (VLMs) are trained on image–text pairs collected under canonical visual conditions and achieve strong performance on multimodal tasks. However, their robustness to real-world weather conditions, and the stability of cross-modal semantic alignment under such structured perturbations, remain insufficiently studied. In this paper, we focus on rainy scenarios and introduce the first adversarial framework that exploits realistic weather to attack VLMs, using a two-stage, parameterized perturbation model based on semantic decoupling to analyze rain-induced shifts in decision-making. In Stage 1, we model the global effects of rainfall by applying a low-dimensional global modulation to condition the embedding space and gradually weaken the original semantic decision boundaries. In Stage 2, we introduce structured rain variations by explicitly modeling multi-scale raindrop appearance and rainfall-induced illumination changes, and optimize the resulting non-differentiable weather space to induce stable semantic shifts. Operating in a non-pixel parameter space, our framework generates perturbations that are both physically grounded and interpretable. Experiments across multiple tasks show that even physically plausible, highly constrained weather perturbations can induce substantial semantic misalignment in mainstream VLMs, posing potential safety and reliability risks in real-world deployment. Ablations further confirm that illumination modeling and multi-scale raindrop structures are key drivers of these semantic shifts.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision-Language Models, Adversarial Robustness, Weather Perturbations

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4fy6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates rain-based attacks targeting vision-language models. Specifically, it proposes a two-stage approach: optimizing global rain layer blending coefficients to reduce CLIP-style semantic margins, and using CMA-ES to optimize multiscale raindrops and spatial illumination parameters under perceptual and structural constraints. The evaluation covers zero-shot classification tasks based on multiple CLIP backbone networks, and the resulting images are transferred to image description and visual question-answering (VQA) models.

Critical strengths
  1. The paper introduces a structured and physically interpretable adversarial framework that jointly models multi-scale raindrops and illumination variations.
  2. Extensive experiments demonstrate strong attack effectiveness across multiple CLIP-based models and transferability to image captioning and VQA tasks.
Critical weaknesses
  1. The objective of this paper is not clear. While the paper claims to reveal the vulnerability of VLMs in rainy-day scenarios, its methodology is based on adversarial attacks using simulated raindrops rather than real-world rainy-day scenarios.
  2. What components does the proposed semantic decoupling mechanism actually decouple? The paper describes semantic decoupling as separating semantic manipulation from physical appearance synthesis. However, both stages modify image-space weather perturbations and optimize semantic similarity objectives.
Questions for authors
  1. Figure 2 does not clearly illustrate the optimization objectives and parameter interactions between the two stages, making the overall workflow less intuitive.
  2. Both arXiv:2009.09205 and arXiv:2302.14267 explore adversarial attacks that utilize simulated raindrop effects; the authors should discuss these studies.
  3. How to calculate the CMA-ES fitness values for candidates that did not make it into the Top-K subset.
  4. The number of model queries and computational costs of the two stages are not clearly compared with baseline methods.
Reviewer_Kk6yOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the vulnerability of vision-language models to rainy-weather perturbations. The method first optimizes the blending weights between a fixed rain layer and the original image, then jointly searches over multiscale raindrop and illumination parameters. Attacks developed for classification are subsequently transferred to image captioning and visual question answering. The experiments use digitally synthesized weather on COCO images, supplemented by component ablations, extended evaluations, and human perceptual assessments.

Critical strengths

The paper demonstrates the effects of structured weather perturbations on multiple vision-language models and directly transfers classification attacks to image captioning and visual question answering, making the study relevant across tasks. The raindrop and illumination parameterization is clearly structured, while component ablations, extended evaluations, and human perceptual assessments provide supporting evidence.

Critical weaknesses

The independent contribution of two-stage semantic decoupling has not been sufficiently established. The first stage pre-optimizes the input, but it remains unclear how this differs from a general reduction in the decision margin. The current experiments lack direct joint-search and random-weather controls under matched budgets and visibility conditions, making it difficult to distinguish the effects of the stage design from those of the search procedure. In addition, the paper emphasizes physical consistency and risks in real environments, yet all experiments use digital synthesis on COCO images, and the generated images have limited realism. Extended evaluations and human visibility assessments cannot substitute for validation under real rainfall. The overall darkening in the examples also needs to be distinguished from the effects of raindrop structure, and the content should be confirmed to remain recognizable to humans after the attack.

Questions for authors
  1. The two-stage method should be compared with direct joint search starting from the original image and with random weather search, while matching the parameter space, total query budget, and visibility conditions, to clarify the contribution of the stage separation itself.
  2. Can the generated raindrops and illumination changes represent actual rainfall and more fully support the paper's claims about physical consistency and real environments? The authors could provide additional experiments using real rainy-weather images or controlled imaging setups.
  3. Could the authors separately compare the effects of raindrop structure and ordinary darkening to distinguish semantic attacks from a general reduction in visibility?
  4. The criteria for selecting valid samples also need to be clarified.
Reviewer_DftTOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses the problem of developing physically grounded adversarial perturbations in images to attack vision-language models. The type if physically grounded perturbation that is introduced in this paper is the introduction of realistic rain into an image. The paper presents a two-stage process for generation of such perturbations. The first stage adjusts the additive mixture of raindrops to the image. The second stage further optimises the perturbations to the image. An empirical evaluation is provided on several CLIP variants against several benchmark attacks.

Critical strengths

The paper presents an interesting idea for generating naturalistic adversarial perturbations to images.

Critical weaknesses

The novelty of the proposed approach is limited, as there are existing approaches for creating naturalistic adversarial perturbations to images. While the idea of using weather-related perturbations is interesting, their usefulness is questionable for many types of image settings, such as indoor settings, face recognition, scenes with a clear sky, etc.

Based on the images presented in Figure 3 of the paper, the quality of the perturbed images seems to be significantly degraded, in terms of being much darker than the original images. The rain drops have a rather uniform appearance, which may make them easier to detect and filter than other forms of perturbations.

Questions for authors

Have you tested any methods for detecting or filtering the adversarial perturbations of your approach and the existing approaches, to see which are more resilient to such defences?

Can you provide examples of how the perturbed images from the other benchmark methods as a visual comparison of the naturalness of the images from each method?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.