acceptodds
Withdrawn from ICLR 2027

Discrete Text-Conditioned Universal Multimodal Attacks on Vision–Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language pre-trained (VLP) models support diverse multimodal tasks but remain vulnerable to universal adversarial perturbations. Continuous textual surrogates can create a mismatch between optimization and the discrete edits applied at inference, while a shared word has context-dependent effects. We propose an optimization-based framework that combines online vocabulary-constrained text learning with edited-caption conditioning: the current shared word is inserted into complete captions, which are re-encoded to condition universal image-perturbation learning. Representation-level disruption and bidirectional cross-modal competition target image–text correspondence, with auxiliary fusion guidance when supported by the source architecture. Experiments span six VLPs configurations and three vision-language tasks. Extensive experiments demonstrate competitive white-box attack performance, and effective transfer to downstream vision-language tasks without further optimization.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Adversarial Perturbation, Visual-Language Models, Discrete Text

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rGtoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a universal multimodal attack that combines online vocabulary-constrained text optimization with image-perturbation learning conditioned on re-encoded edited captions. The framework also incorporates bidirectional competition and architecture-dependent fusion guidance. Experiments across six model configurations demonstrate strong white-box attack performance and varying degrees of transfer to other models and downstream tasks.

Critical strengths

The central idea is well motivated. The distinction between optimizing a continuous text representation and applying a discrete word replacement at inference is meaningful. Conditioning image updates on complete, re-encoded edited captions provides a sensible way to account for the replacement word’s contextual effects.

Critical weaknesses
  1. Important comparisons are missing. Although the paper discusses DO-UAP and HRA, neither is evaluated experimentally. Both are closely related to the proposed approach. Downstream experiments compare only against C-PGC, despite ARG-Attack’s competitive retrieval performance, leaving the empirical advance incompletely established.

  2. The full framework’s transfer advantage is also limited. Mean black-box I2T/T2I ASR is 40.00%/60.39%, compared with 40.55%/57.53% for ARG-Attack, so the average improvement is concentrated in T2I retrieval. In the ALBEF-source ablation, the full framework (A4) performs slightly below component D alone (A1): 50.58%/71.25% versus 50.95%/71.32%. Although bidirectional competition improves white-box performance, its additional complexity does not yield a consistent transfer benefit in the reported ablation.

Questions for authors
  1. What are the values of all loss hyperparameters, how were they selected, and were they shared across source models?

  2. How does the method compare with DO-UAP and HRA under matched attack constraints, checkpoints, and computational budgets? Can ARG-Attack be included in downstream comparisons?

Reviewer_2Qo3OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on universal multimodal adversarial attacks against vision-language pre-trained (VLP) models. It proposes a framework that jointly optimizes a universal image perturbation and a shared textual substitution through online text learning and edited-caption-conditioned image optimization.

Critical strengths
  1. The paper focuses on a traditional and classical research question.
  2. The paper conducts experiments on extensive vision-language-pretrained models,
Critical weaknesses
  1. Lack of Semantic Preservation Constraints. The proposed method only limits textual perturbations to a single-word substitution without explicitly enforcing semantic preservation. However, even replacing a single word can significantly change the meaning of a caption and break its original correspondence with the image. This raises concerns about the stealthiness of the attack. Moreover, I believe semantic preservation is particularly important for text-based and multimodal adversarial attacks, as mentioned in prior works such as BERT-Attack[1] and VLATTACK[2].

[1] BERT-ATTACK: Adversarial Attack Against BERT Using BERT. EMNLP 2020

[2] VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Models. NeurIPS 2023

  1. Limited Technical Novelty. The proposed method mainly combines existing techniques, including universal image perturbations, constrained word substitution, and cross-modal attack objectives. Although the authors introduce edited-caption-conditioned image optimization, I am not fully convinced that this provides sufficient technical novelty compared with existing universal multimodal attack methods.

  2. Limited Evaluation on Modern Multimodal Large Language Models. Recent multimodal research has increasingly shifted from traditional vision-language pre-trained (VLP) models to multimodal large language models (MLLMs). However, the experiments are limited to relatively traditional VLP architectures, such as CLIP, ALBEF, TCL, X-VLM, and BLIP. The paper does not evaluate more recent MLLMs, such as Qwen-VL or LLaVA. I believe this limits the practical relevance and generalizability of the proposed attack, particularly given the growing importance of MLLMs in real-world applications.

Questions for authors

Please refer to the weakness section.

Reviewer_AmxBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a universal multimodal adversarial attack which alternates between discrete vocabulary constrained text learning and edited caption conditioned image perturbation optimization, augmented with bidirectional competition and fusion guided losses.

Critical strengths

This paper proposes a universal multimodal adversarial attack which alternates between discrete vocabulary constrained text learning and edited caption conditioned image perturbation optimization, augmented with bidirectional competition and fusion guided losses.

Critical weaknesses
  1. The core edited caption conditioning is based on the prior multimodal UAP work including C PGC and HRA. Straight through estimator vocabulary projection, contrastive representation disruption, and rank based pairwise cross modal losses are well established building blocks in existing adversarial literature. The incremental combination of these modules does not deliver substantial conceptual insight beyond existing universal multimodal attack pipelines.
  2. The black box transfer performance is inconsistent and lacks convincing overall improvements. While the method achieves strong white box ASR on the source model, black box transfer results are mixed. In several cross model settings, ARG Attack obtains competitive or even better attack transfer performance. The ablation further confirms that full model components improve white box effectiveness but do not consistently boost average black box transfer. The paper fails to adequately diagnose why performance gains do not generalize reliably across different source target model pairs.
  3. The evaluation has important missing controls. The paper only compares against selected multimodal universal baselines. It lacks rigorous ablations isolating whether performance gains stem mainly from the edited caption conditioning or from extra capacity introduced by bidirectional competition and fusion guidance. Moreover, there is no human evaluation to verify semantic plausibility of the single word textual trigger; the discrete word substitution budget offers no guarantee of natural looking text modifications.
  4. The paper mixes results across six heterogeneous VLP architectures, multiple downstream tasks, and different evaluation datasets. Some key metrics are scattered across tables without unified summary. It is difficult for readers to systematically understand under which model configurations the proposed method brings tangible advantages.
  5. The attack requires per source model retraining of universal perturbations. This heavy computational overhead weakens its practical appeal as a universal probing tool for real world multimodal robustness. The ethical statement only briefly mentions dual use risks without analyzing real world applicability of the learned universal image text triggers.
Questions for authors

The paper provides no analysis explaining why edited caption conditioning should improve cross model generalization, relying purely on empirical observations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.