acceptodds
Withdrawn from ICLR 2027

When Weather Misleads Retrieval: Atmospheric Attacks on Remote Sensing Vision-Language RAG

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal RAG systems rely on vision-language retrievers to ground visual queries in external evidence, yet the retrieval stage itself remains underexplored as an adversarial target in remote sensing. We introduce an atmospheric retrieval attack that modifies only the input image, leaving the retriever, generator, and knowledge base unchanged. The attack overlays parameterized cloud- and haze-like patterns and optimizes them to attract image representations toward target atmospheric evidence while suppressing source-scene evidence, with additional constraints on rank separation and visual naturalness. We evaluate the attack on a seven-dataset remote sensing RAG benchmark with five CLIP-style retrievers and downstream vision-language generators. Across retrievers, optimized atmospheric perturbations consistently outperform clean inputs and multiple handcrafted, random, and fixed baselines in promoting weather-related evidence into top-ranked retrievals. On GeoRSCLIP ViT-B/32, Weather@5 rises from 0.71% to 43.29%. The resulting retrieval corruption further induces measurable weather hallucinations and semantic shifts in downstream generation. These results reveal retrieval-stage evidence grounding as a distinct vulnerability of remote sensing vision-language RAG.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Remote Sensing Multimodal Retrieval-Augmented Generation, Remote Sensing, Vision-Language Retrieval, Adversarial Attacks, Retrieval Hijacking, Atmospheric Perturbations, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_V1qbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents an adversarial attack for RAG-based systems in the Remote Sensing domain. They introduce a method of generating plausible clouds and haze in remote-sensing imagery that trips the retriever component to shift towards more weather-related context, thus shifting also the answers in the same direction. The method consists of an optimization over several parameters that control clouds and haze appearance, and is validated via comparison with un-optimized version or more simplistic noise.

Critical strengths

None

Critical weaknesses
  1. The plausibility of the generated weather artifacts is not supported. On the contrary, in the appendix E2 they show 21 out of 30 humans correctly judged a modified image as being digitally altered.
  2. The introduced attack does not produce an unintended effect. The attack introduces weather patterns, and the retriever shifts towards weather documents. Same for the generator. The attack would be consequential if we could use the method to elicit information that contradicts conditions on the ground.
  3. The RAG setup is forced. Usually RAG is used to inject external knowledge that could help answer the user query. Here, in all 7 datasets used for evaluation, all the data needed for the answer is supplied by the user: the image. It is not clear how adding other items from the same dataset can improve the downstream generation, because queries always target the associated image. Furthermore, the retrieved evidence (Eq 3) does not depend on the query, only on the image (i.e. the same context is retrieved regardless of the user asking about land cover or disaster mapping). This further limits the formulation as RAG.
Questions for authors
  1. Can you please clarify the W@5 metric and what role it has? Is it a coincidence that on Sample100 top W@5 corresponds to 303 / 700? Is that how Sample303 dataset was constructed?
  2. Are there cases where the response or the retrieved context is actually adversarial, not just weather-related ?
Reviewer_iumzOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The authors define an "adversarial" RAG attack as one where the input image is modified to have synthetic clouds added to it, fully visible to the human observer, and the system then focusses its response on the clouds that were added to the input image. Yes, this reviewer does not "get" the paper: I don't understand why that result is noteworthy. Thanks to please clarify this in your rebuttal, and in future revisions to the manuscript.

Critical strengths

The authors add synthetic clouds, and the systems all say "clouds". It relates to my confusion about the purpose of this paper (please see "weaknesses", and "author questions") that this seems very unremarkable to me. Thanks to please clarify things for me in your rebuttal.

Critical weaknesses

It took me until line 192 of the paper (on p. 4, in the section Method, Problem formulation and threat model) to understand what the task was and why it might be interesting. I think the authors can do a better job of explaining things in the introduction in a way that someone outside their subfield will understand their paper sooner.

Pardon my lack of imagination, but: why would I care about the adversarial scenario that the paper describes? Given that the adversary is modifying the input query image in a way that is completely visible to a human observer, why is it an adversarial deception at all that an image with clouds painted over it comes back described as clouds? Suppose an adversary substituted a photo of a cat. Might the label come back, "cat"? In you scenario's point of view, would that be good or bad?

An "adversarial" attack usually means one that are not apparent to a human. What difference between human and model perception does this attack exploit? Please show human judgments of the original scene on attacked images. If people still identify the land use correctly while the RAG system reports weather, that would be a real vulnerability. If people also mainly see clouds, it would seem that the system is behaving correctly. Please clarify for me: what am I missing in my analysis above?

Questions for authors

The paper’s claim seems to depend on optimized clouds doing more than equally heavy random clouds. Can you compare optimized vs. random/fixed CloudWeb at matched coverage and opacity? And since the generator sees the clouded image, in what sense is mentioning fog a hallucination rather than an accurate description of the input?

Reviewer_ZjAmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies a novel input-space adversarial vulnerability for remote-sensing multimodal RAG. Unlike prior corpus- or output-targeted attacks, the proposed CloudWeb injects realistic cloud/haze perturbations into query images to hijack retrieval. It optimizes a multi-objective loss to pull embeddings toward weather-related evidence while keeping the retriever, generator and knowledge corpus fixed. Evaluations on seven remote-sensing datasets with five retrievers and six VLMs show that CloudWeb effectively triggers retrieval hijacking, leading to downstream weather hallucinations and semantic shifts. The work also presents visualizations, ablations, transfer and robustness tests, human evaluation and defensive insights.

Critical strengths
  1. Novel realistic threat model for multimodal RAG security. Most prior RAG adversarial attacks focus on corpus poisoning; vision adversarial attacks for remote sensing mostly target end‑task prediction. This paper identifies a new threat: attackers inject plausible cloud/haze atmospheric patterns only on input images to hijack retrieval stage, without modifying deployed models or vector databases. This threat is practically relevant for real‑world remote‑sensing multimodal RAG services accepting user‑uploaded satellite / aerial images.
  2. Comprehensive multi‑faceted evaluation. The work conducts rich evaluations: seven remote‑sensing datasets, five CLIP‑style retrievers, six downstream VLMs, ablation for loss & perturbation components, post‑processing robustness, cross‑model transfer, CAM mechanism visualization, human‑model perceptual audit and failure‑case analysis. It not only measures retrieval‑stage metrics but quantitatively measures downstream hallucination propagation, which solidifies the end‑to‑end risk analysis.
Critical weaknesses
  1. Baseline comparisons are partially incomplete for retrieval‑oriented adversarial attacks. The paper compares against Gaussian blur, brightness haze, random noise cloud and fixed non‑optimized CloudWeb. However, it lacks comparison against existing input‑space targeted embedding attacks. While authors mention them as future work in the main text, adding at least one such adversarial baseline would better demonstrate whether cloud‑haze physical prior brings unique advantages compared with generic pixel‑level embedding attacks.
  2. Limited discussion of practical real‑world attack constraints. The attack optimizes parameters via differential evolution with offline access to retriever encoders. The black‑box transfer results show performance drop when transferring across retrievers. It remains unclear how well CloudWeb would perform under strict real‑world black‑box settings, where the adversary cannot access target retriever weights or embeddings at all. The practical operational risk magnitude is not sufficiently quantified.
Questions for authors
  1. You leave unconstrained pixel-level targeted retrieval attacks to future work. What is the expected performance gap between CloudWeb and generic pixel-level embedding-steering attacks and, if feasible, could you provide a small-scale empirical comparison on a subset to quantify what the atmospheric plausibility constraint costs in attack power? Beyond human-perceived realism, does the parameterized cloud haze generator bring intrinsic advantages for hijacking vision–language retrieval?
  2. In your transfer experiment, the perturbation is optimized on GeoRSCLIP as a surrogate and applied to the other retrievers. In practical deployment the adversary may not know which retriever backs the target system: (a) how sensitive is transfer success to the choice of surrogate? (b) Beyond retriever weights, the optimization requires a (proxy) corpus to construct the target and source evidence sets Tq ​ and Sq ​ — how realistic is this knowledge for an external attacker, and how much does performance degrade when Tq ​ / Sq ​ are built from a proxy corpus rather than the deployed one?
  3. In your human perceptual audit, 21 of 30 evaluators flagged the perturbation as artificial, while the multi-model ensemble was near-balanced — yet both audits evaluated a single optimized image. Would these conclusions hold over a random sample of successful attacks? For practical deployment, what fraction of such adversarial samples can be detected by simple heuristic image-level detectors (e.g., cloud-mask statistics, frequency or contrast cues) that do not rely on internal model weights or embeddings?
  4. Appendix F.2 proposes multiple defensive strategies. Have you performed any preliminary quantitative experiments to validate any of them (e.g., the instability-score-based consistency-aware retrieval of Eq. 25)? If not, what are the main practical challenges of deploying these defenses on real-world remote-sensing RAG pipelines?
  5. In Table 10, cloud-targeted hijacking performs far better than fog-haze and smoke-mist targets even though Top5-Changed remains high for all three groups. Is this gap primarily caused by semantic alignment within the vision–language embedding space, or by the visual morphology limitation of your perturbation generator? Note that the three target groups may also differ in evidence-pool size within the corpus; a controlled comparison would help disentangle these factors.
  6. Several nominally identical quantities disagree across the paper: on Sample100 with GeoRSCLIP, W@5 = 43.29% in Table 1, 41.14% in Table 6, and 40.86% for the full model in Fig. 7, with Fig. 5 implying yet another baseline (37.43–43.14%). Are these different runs, subsets, or re-optimizations? Relatedly, please specify the corpus construction (text sources, "Full" size) and the exact criterion for labeling evidence as "weather-related", since the W@k metric depends on it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.