acceptodds
Under review as a conference paper at ICLR 2027

From Regions to Decisions: Multimodal Adversarial Attacks on Remote Sensing Vision-Language Models

Abstract

Remote sensing vision–language models (RS-VLMs) translate wide-area Earth observations into disaster-related decisions, yet their predictions may hinge on sparse local evidence and subtle variations in prompt formulation. Existing attacks typically perturb images under fixed linguistic conditions or optimize surrogate representations, overlooking the evolving interaction between visual evidence and language. We propose RS-RDA, a region-to-decision white-box multimodal attack. RS-RDA localizes decision-relevant evidence by combining proposal reliability, vision–language relevance, and semantic complementarity, and then optimizes bounded perturbations within them. After each visual round, RS-RDA compares the clean and adversarial answer distributions together with regional gradients to identify the induced score shift, the dominant incorrect competitor, and the visual regions driving the decision change. These signals guide the generation and selection of semantics-preserving adversarial prompts, enabling the prompt attack to adapt to the evolving decision boundary. Across five heterogeneous RS-VLMs, RS-RDA achieves average attack success rates of 93.89%, 90.64%, and 97.29% on building-damage counting, disaster-type recognition, and object-relation reasoning, respectively. These results exceed the strongest task-level baseline averages by 52.09%, 35.50%, and 24.00%, respectively. These results establish RS-RDA as an effective method for evaluating the adversarial robustness of RS-VLMs in disaster-scene perception and reasoning tasks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Remote Sensing Vision--Language Models, Multimodal Adversarial Attacks, Adversarial Robustness, White-Box Attacks, Disaster-Scene Understanding.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_m8xKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RS-RDA, a region-to-decision white-box multimodal adversarial attack method for remote sensing vision-language models (RS-VLMs). It first combines region reliability, vision-language relevance, and semantic complementarity to localize decision-relevant regions and applies bounded perturbations within them. Then, by comparing clean and adversarial answer distributions along with regional gradients, it identifies score shifts, dominant incorrect competitors, and decision-changing regions, thereby generating and selecting semantics-preserving adversarial prompts to adapt the attack to the evolving decision boundary.

Critical strengths

see summary

Critical weaknesses

The following concerns need to be addressed by the authors one by one.

1)The paper employs SAM to segment remote sensing images. Is the segmentation quality consistently satisfactory for every image in the attack dataset?

2)The text font in the framework figure is too small and does not conform to standard formatting conventions.

3)In the equations, some variables are typeset in bold. What is the purpose of this emphasis? Are these variables intended to be highlighted as particularly important?

4)The authors should explicitly clarify whether the baselines are also permitted an equivalent strength of prompt search.

5)Why is the ASR of the baseline methods in Table 1 substantially lower than that of the proposed method? Is there a reasonable explanation for this discrepancy?

Questions for authors

see weakness

Reviewer_iAh4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies task-preserving adversarial robustness of remote-sensing vision–language models on disaster imagery. RS-RDA constructs a fixed regional mask using SAM proposal reliability, RemoteCLIP task relevance, and DINOv2 diversity, then alternates target-model answer-ranking optimization with decision-state-guided prompt reformulation. Evaluation covers five models and three multiple-choice tasks: building-damage counting, disaster-type recognition, and object-relation reasoning. Averaging the model-wise results in Table 1 yields attack success rates of 93.89%, 90.64%, and 97.29%, respectively, as reported in Section 5.2. The contribution is a coherent combination of localized evidence selection and adaptive multimodal optimization.

Critical strengths
  1. The localized, task-preserving threat model is explicit and technically concrete. Target-independent regional masks and deterministic prompt mutations preserve the ordered answer set and question intent (Sections 2–4; Appendix C.4). Figures 2–4 illustrate legal wrong-option predictions with unchanged answer contents, supporting the existence of genuine decision errors rather than only formatting failures.
  2. Evidence extends beyond headline ASR. Table 2 and Appendices F–G provide component and region-selection experiments across tasks; Table 4 reports paired full-image distortion reductions averaging 31.55% for RMSD and 37.06% for LPIPS. The text acknowledges cases in which full-image optimization performs better, appropriately limiting any universal claim about regional attacks.
Critical weaknesses
  1. The main comparison does not isolate algorithmic superiority from different attack conditions. Appendix E.3 explicitly gives RS-RDA target-model white-box gradients while retaining surrogate-only transfer settings for the baselines. Tables 7–8 also use different budgets and update counts: RS-RDA uses 8/255 and 30 updates, several baselines use 2/255 and 10 steps, and VQAttack uses 16/255 and 40 steps.
  2. The success metric mixes semantically wrong decisions with unavailable outputs. Appendix E.4 counts malformed responses, inference exceptions, and missing checkpoint outputs as successes, but their proportions and whether they are attack-induced are not reported. Availability failure can be a legitimate attack objective; infrastructure failure is not interchangeable with a legal wrong answer. MAE/MRE exclude invalid outputs without reporting valid counts. Consequently, the headline ASRs do not establish what fraction of samples actually exhibit the claimed erroneous decisions.
Questions for authors
  1. Can you provide an equal-access, equal-8/255-budget comparison with a straightforward target-native answer-ranking attack, matching optimization and prompt-selection effort? Which gains remain under these controls?
  2. For each model–task pair, please report the clean-correct cohort size, legal correct/wrong answers, malformed outputs, inference exceptions, and missing checkpoints. Separate semantic-error ASR from availability failure and provide the valid BDC counts used for MAE/MRE.
  3. What cohorts, seeds, or settings explain the full-method DTR average of 90.64% in Table 1 versus 96.46% at step 30 in Table 9? Please also define precisely which prompts and computational effort remain in the ‘without adaptive prompt’ and ‘without alternating optimization’ interventions.
Reviewer_HiguOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes RS-RDA, a multimodal attack on remote sensing vision-language models. It selects task-relevant image regions and alternates regional perturbation optimization with prompt evolution guided by the model's decision state. Experiments report high attack success rates across five models and three disaster-related tasks.

Critical strengths
  1. The paper studies a useful robustness problem for remote sensing VLMs. The evaluation covers five models and three tasks.

  2. The ablations examine regional support, prompt adaptation, and alternating optimization.

Critical weaknesses
  1. The baseline comparison is not convincing. RS-RDA access to target gradients while the transfer baselines use surrogate models. Several baselines also use smaller perturbation budgets and fewer updates. These differences make it difficult to attribute the large gains to the proposed method.

  2. This method incorporates a lot of modules and hyperparameters. How were the hyperparameters such as region thresholds, ensemble coefficients, and prompt utility weights selected?

Questions for authors
  1. Could you compare against PGD and AutoAttack using the target model's option scores at ϵ=8/255\epsilon=8/255.

  2. What is the computational cost of RS-RDA, compared with other methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.