acceptodds
Under review as a conference paper at ICLR 2027

Evolutionary Adversarial Attacks Meet LLMs: Semantic-aware Adaptive Optimization Against Vision-Language Pre-trained Models

Abstract

Targeted attacks against black-box vision-language models remain challenging because transfer-based optimization must simultaneously achieve precise semantic steering, cross-model transferability, and bounded image perturbations without access to victim gradients or internal representations. Although evolutionary algorithms provide a natural gradient-free solution, their search is typically semantically blind and relies on fixed optimization strategies. We introduce LEA, an LLM-driven evolutionary adversarial framework that turns conventional evolutionary search into a semantic-aware adaptive optimization process. LEA first constructs original and target semantic anchors to explicitly characterize the desired semantic transition. More importantly, rather than directly generating adversarial perturbations, LEA employs the LLM as a high-level controller that interprets the evolving search state and adaptively coordinates semantic region priors, operator selection, and search parameters. A heterogeneous evolutionary operator pool then performs the low-level gradient-free perturbation search, enabling complementary global exploration and local refinement. Experiments on SMALLCAP, ViECap, and LLaVA-13B demonstrate consistent improvements over representative black-box and transfer-based baselines. On SMALLCAP, LEA improves Target ASR from 39.7% to 53.2% over the strongest targeted baseline while increasing the average CLIP score from 63.78 to 68.76. These results show that high-level LLM reasoning can effectively control evolutionary adversarial search and expose the vulnerability of vision-language systems to targeted semantic manipulation under transfer-based black-box settings.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
VLP, LLM, Adversarial Attack, Multi-objective

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Hr3kOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

LEA combines LLM-guided semantic control with evolutionary search to generate targeted transferable adversarial images against vision-language models.

Critical strengths

This paper uses an LLM to adapt search regions, operators, and parameters based on optimization progress. Consistent improvements across three victim models.

Critical weaknesses

i) Limited scope. The experiments show that the attack performs well on VLP models but has limited effectiveness on LLaVA, with target ASR% 10.7 and Fool Rate 20.4%. The results corroborate my previous observation that attacks on VLP models differ substantially from those on multimodal large language models (MLLMs). Consequently, the work’s narrow focus on VLP models limits its relevance to future research, e.g., attacks on frontier MLLMs (GPT, Opus, etc.).

ii) Overly complex design without design intuition. The method combines an LLM with multiple evolutionary algorithms and search strategies, raising questions about its generalizability. With so many components and hyperparameters, it is unclear which design choices actually drive the performance gains and why.

Questions for authors

Please see the weaknesses above. I also wonder why Stable Diffusion is needed to generate a reference image when CLIP already aligns text and image representations? The rationale for this design, along with its connection to attack transferability, needs to be clarified.

Reviewer_cUU3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LEA, an LLM-guided evolutionary adversarial attack framework for targeted black-box attacks against vision-language models. The method constructs original and target semantic anchors using vision-language representations and diffusion-generated images. Experiment demonstrate the proposed method can achive better performance than several ccomapred method baselines.

Critical strengths
  1. Integrating LLM-based semantic reasoning with evolutionary optimization offers an interesting perspective on targeted adversarial attacks against vision-language models.
  2. LEA achieves higher targeted attack success rates than the compared methods across three victim models.
Critical weaknesses
  1. The Introduction provides only a brief discussion of related work, with limited references to existing studies. A more comprehensive discussion is needed to better contextualize the proposed method.
  2. It is unclear why evolutionary optimization is preferred over commonly used gradient-based attacks, particularly since the surrogate model is differentiable. In particular, given the substantial computational overhead of evolutionary methods, the authors should clarify their advantages and provide comparisons with gradient-based approaches. The proposed method constructs semantic anchors to guide adversarial optimization. However, it remains unclear why aligning adversarial representations with these anchors on the surrogate model should lead to successful targeted attacks on other models. Further analysis is needed to explain the underlying transferability mechanism.
  3. Experiments are only conducted on three models and a single dataset, and seem to focus mainly on embeddings rather than different tasks and datasets, making it difficult to verify the effectiveness and transferability of the proposed method.
  4. The paper lacks a thorough investigation of the shared vulnerabilities across different victim models, the underlying reasons for successful adversarial transfer, and the sensitivity to different perturbation budgets. Such analyses would provide deeper insights into the effectiveness and limitations of the proposed method.
Questions for authors

Please refer to Critical Weaknesses.

Reviewer_zmQQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose a novel adversarial black-box attack method for modern vision-language models.
In a nutshell, the proposed method assumes black-box access to the victim model and tries to build an adversarial image through evolutionary search guided by an LLM controller.
Starting with a desired textual target, the method attempts to generate potential attacks, evaluate them, and guide the exploration procedure with a language model.
The proposed method showed promising results on a number of benchmarks, outperforming other approaches.

Critical strengths
  • The paper considers an important and timely problem of VLM adversarial attacks, which has critical use cases in real-world applications.
  • The method is logical, and the authors properly address the motivation behind introducing a new type of black-box attack leveraging an LLM as an attack search controller.
  • The empirical validation features extensive evaluation, demonstrating an improvement over existing approaches.
Critical weaknesses
  • The motivation for black-box adversarial attacks is clear, but the paper does not benchmark its method on frontier models that already include a number of safeguards and safety measures. As these are a primary target for such a methods, benchmarking on a (near) frontier model would make the impact much stronger.
  • One of the core components of the pipeline is a language model that steers the attack through evolutionary optimization. The paper does not demonstrate a comprehensive evaluation of how different LLMs perform in this particular context. It would also be interesting to see how attack success changes when using various thinking / query budgets.
  • The motivation for using an LLM controller is clear, though it would be good to demonstrate reasoning traces from the controller model for a better understanding of its reasoning / how the model decides on the optimization of the attack.
Questions for authors
  • Could you evaluate the method on a frontier or near-frontier VLM to assess its effectiveness in a more realistic setting?
  • How sensitive is attack success to the choice of controller LLM?
  • How does attack success change with the controller’s reasoning budget and the number of victim-model queries?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.