acceptodds
Withdrawn from ICLR 2027

EvoNAT: Natural Language-Guided Evolution of Adversarial Images for Large Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Natural adversarial images induce semantic errors in large vision-language models (LVLMs) while remaining visually natural. Existing source-image attacks typically search for local pixel perturbations around benign images, tying the search to the source and limiting the visual content that can be explored. We introduce EvoNAT, an output-only black-box framework that evolves structured visual descriptions in semantic space using image-captioning feedback. EvoNAT generates complete adversarial scenes without requiring a source image. We further extend this approach to source-conditioned editing with EvoNAT-Edit, which evolves source-grounded edit specifications and renders each candidate from the original image. Both branches use image-captioning feedback to optimize semantic deviation, while visual question answering (VQA) is held out to evaluate cross-task transfer. Across seven open-weight and proprietary LVLMs, EvoNAT achieves relative gains of 53.4% and 52.8% in the seven-model mean cosine-based IC-Attack and VQA-Attack scores, respectively, over the strongest baseline on each task. Its normalized text-only LLM Judge scores also show relative gains of 2.0% on image captioning and 4.2% on VQA. The generated images achieve a 1.2% higher MUSIQ score and a 14.3% higher human naturalness rating than the strongest source-free baselines on the respective metrics. In source-conditioned editing, EvoNAT-Edit achieves relative gains of 36.2% and 22.7% in the mean cosine-based IC-Attack and VQA-Attack scores, and 29.6% and 28.0% in the corresponding LLM Judge scores. Its edited images achieve 5.8% higher TOPIQ-NR and HyperIQA scores than the strongest editing baselines. These results demonstrate that semantic-space evolution supports effective natural adversarial image generation both from scratch and through source-conditioned editing.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Natural Adversarial Sample, Large Vision-Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_3Ry7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates natural adversarial example generation for LVLMs. The proposed EvoNAT framework performs source-free attacks via natural language-guided evolution, enabling adversarial image generation without relying on a specific source image. The authors further extend the framework with a source-conditioned editing variant. Experimental results across multiple models and tasks demonstrate the effectiveness of the proposed attack.

Critical strengths

Natural adversarial example generation is an interesting research topic. EvoNAT achieves good performance on multiple evaluation metrics and datasets.

Critical weaknesses
  • The writing issues: This paper is not easy to follow.

In my view, a good abstract and introduction should tell the readers three aspects: “What”, “Why” and “How”.

“What is EvoNAT?” => a source-free adversarial image generation method that leverages natural language-guided evolution. (Well, it looks interesting)

“How does it work?” => it introduces evolutionary search over structured visual descriptions on objects, attributes and scene context. A text-to-image diffusion model is used to render the adversarial image. (It sounds reasonable)

“Why do we do this research?” => Previous work limits the range of visual content that can be explored AND THEN? (This paper largely misses the discussion on this part)

  • The baselines are not sufficient.

As an attacker, two core metrics should be considered: attack success rate and stealthiness. Regardless of whether the adversarial image is generated from a source image or not. Therefore, the authors should compare with multiple black-box attack methods [1][2] to sufficiently verify the superiority of source-free attacks.

[1] Li et al. A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1. NeurIPS’25.

[2] Jia et al. Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment. NeurIPS’25.

Questions for authors

About the “Why” in writing issues:

  • Why is source-free better than source-based methods? Do they achieve better ASR? Do they have better stealthiness? Are they computationally friendly?

  • Why introduce EvoNAT-Edit? Especially since it is still source-conditioned and seems inconsistent with the motivation of this paper.

  • Why use evolutionary search instead of gradient-based optimization? Faster? More interpretable?

Reviewer_pFRMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EvoNAT proposes an output-only black-box framework for generating natural adversarial images for LVLMs. Instead of perturbing pixels around a fixed source image, it evolves structured natural-language descriptions of objects, attributes, actions, and scene context, renders them into images, and uses the semantic difference between the intended description and the LVLM’s caption as feedback. The paper also introduces EvoNAT-Edit, which applies the same idea to source-conditioned image editing.

Critical strengths
  • Searching directly in semantic space allows the method to explore much broader changes in objects, attributes, actions, and scene context than conventional source-image attacks.

  • EvoNAT only needs textual feedback from a proxy LVLM and does not require access to gradients, hidden states, or model parameters, which makes the setting relevant to black-box and proprietary models.

  • The authors test both source-free generation and source-conditioned editing across seven LVLMs, including both open-weight and proprietary models, and also report visual-quality and human naturalness results in addition to attack scores.

Critical weaknesses
  • The motivation for EvoNAT is not fully convincing. The paper emphasizes that source-image attacks are restricted to a local neighborhood, but prior generative and language-guided methods already move beyond this constraint by synthesizing or searching for adversarial visual content without relying on local pixel perturbations. The remaining claimed gap is that prior work does not provide a single output-only framework covering both full-scene generation and source-conditioned editing. This seems more like a framework-level unification than a new attack capability, and its practical significance is unclear. As a result, the main novelty appears to lie in combining structured semantic search, existing evolutionary operators, and LVLM output feedback, rather than introducing a fundamentally new attack setting or capability.

  • The method treats the generated description as ground truth, but some images may already be visually ambiguous or fail to render the description faithfully. In that case, different LVLMs making the same “mistake” may reflect shared visual ambiguity rather than transferable adversarial behavior. The paper does not clearly separate these two effects. For example, in Figure 5, the generated image for the “folded-paper carousels” does not clearly depict carousels at all and visually resembles umbrellas or pavilion-like structures, making the models’ responses quite reasonable rather than necessarily adversarial failures.

  • The held-out “VQA” task is still very close to image captioning. As shown in the Appendix, the VQA prompt asks the model to describe, in a single continuous sentence, the image type, object color and identity, appearance, style, action, background, location, weather, scene dynamics, and overall atmosphere. This is essentially another holistic description task with a different prompt format, rather than a clearly distinct VQA task with focused questions and ground-truth answers. Therefore, the reported transfer may mainly reflect robustness across two similar caption-style prompts, rather than genuine cross-task transfer.

  • EvoNAT evaluates many candidates over multiple generations and then selects strong examples from the search history. It is unclear whether the baselines receive a comparable number of image generations and model queries. A same-budget random search or best-of-NN baseline would be important here.

  • The attack has very limited controllability. The objective only encourages the model output to differ from the intended description. The attacker cannot reliably control how the model is wrong or force a specific target prediction. This makes the practical security impact weaker than that of a targeted attack.

  • Using only one proxy LVLM makes the generality of the search unclear. All adversarial concepts are evolved using Qwen3-VL-8B as the proxy. It would be useful to repeat the search with LLaVA, InternVL, or another architecture and see whether EvoNAT still works. More importantly, do different proxies discover different vulnerabilities? If they all converge to similar ambiguous concepts such as mushroom-like jellyfish or umbrella-like carousels, that would actually suggest that the method is mainly finding naturally ambiguous images rather than proxy-specific adversarial weaknesses.

  • The same all-mpnet-base-v2 embedding space is used to guide evolution and to compute the main cosine-based attack metric. This means EvoNAT may preferentially discover description–response pairs that are far apart specifically in this embedding space, rather than examples that are consistently judged as severe semantic errors. The paper therefore needs to show that the reported gains remain consistent across multiple independent text encoders or semantic similarity metrics; otherwise, it is difficult to tell whether the method is finding genuine LVLM failures or simply exploiting one embedding space.

Questions for authors

See Weaknesses

Reviewer_JfB7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes EvoNAT, an output-only black-box framework for generating or editing “natural adversarial images” against large vision-language models (LVLMs). It searches in semantic space rather than pixel space. Each candidate is a structured visual description with a template and the description is converted into a “generated description” and rendered with a text-to-image model.

Critical strengths

Some benefit from proposed method: The method searches over interpretable semantic attributes without requiring gradients, logits, or model internals.

The authors propose both EvoNAT and EvoNAT-Edit which share a consistent optimization framework supporting both source-free generation and source-conditioned editing.

Experiments across seven LVLMs demonstrate consistent improvements over baselines in both image captioning and held-out VQA metrics.

Critical weaknesses

The paper argues that semantic-space evolution overcomes the limitations of local pixel-space attacks by enabling global visual exploration. However, searching over a predefined textual schema introduces a different, potentially severe restriction on the reachable image space, and the paper provides no evidence that this restricted search space offers broader or more effective adversarial exploration. More importantly, the effective adversarial search space is determined not only by the textual candidates but also by what the image generator can faithfully render. This dependency restricts exploration to the generator's learned visual distribution and introduces uncertainty about whether the discovered failure cases reflect vulnerabilities of the attacked LVLM or limitations of the renderer.

Critical evaluation issue: The attack metrics compare LVLM responses against intended descriptions without verifying whether the generated images actually match those descriptions, potentially misclassifying rendering failures as successful adversarial attacks. In another word, the “ground truth” against which LVLM responses are scored is the intended description​, but not the actual content of the rendered image.

Key implementation details are missing, and the source-free final-selection description is inconsistent about whether visual quality influences selection.

Questions for authors

What exact criteria govern final image selection, and does visual quality influence this process?

Can the authors provide complete evolutionary configurations, generation prompts, editing pipeline details, and LLM Judge specifications?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.