acceptodds
Withdrawn from ICLR 2027

Diversify and Agree: Reliable Surrogate Expansion for Transferable Adversarial Attacks on MLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) remain vulnerable to transfer-based adversarial attacks, where imperceptible perturbations optimized on accessible surrogate models induce unseen MLLMs to produce attacker-specified responses. While enlarging the surrogate ensemble with diverse representation paradigms improves transferability, the set of accessible surrogates is limited in practice, and collecting or training additional models is costly. In this work, we ask whether richer transferable guidance can be extracted from an existing surrogate set without introducing new models. To this end, we propose DA-Attack, a diversify-and-agree framework comprising two components. Randomized Surrogate Expansion (RSE) diversifies each surrogate by randomly pruning a small fraction of its parameters at every iteration, turning a single model into a stream of online-sampled variants. Since these variants may yield noisy or conflicting gradients, Intra-Surrogate Neighborhood Consensus (ISNC) measures the directional agreement among gradients from the original surrogate and its variants, and assigns greater weight to directions consistently supported across the neighborhood, producing a reliable consensus update. Extensive experiments on eight open-source and four closed-source MLLMs demonstrate that DA-Attack achieves state-of-the-art attack performance. For example, it raises the average targeted attack success rate across the 12 victim models from 70.4% to 82.1% on NIPS 2017, and from 28.5% to 45.6% on Flickr30K, compared with the strongest baseline. The anonymized source code is provided in the supplementary materials.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multimodal Large Language Models, Transferable Adversarial Attacks, Surrogate Expansion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aoE9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DA-Attack, a targeted transfer attack that leverages a fixed ensemble of pretrained vision encoders and does not query the target MLLM during optimization. Its contrastive objective pulls the adversarial features toward those of the target image while pushing them away from those of the source image. RSE retains each original encoder and augments the ensemble with variants generated via random parameter pruning. ISNC assigns weights based on the cosine similarity between gradient directions; the normalized gradients are first aggregated within each model and then averaged across models, followed by a momentum update under an ℓ∞ constraint. The authors evaluate the method on twelve target MLLMs on NIPS 2017 and Flickr30K, using the semantic similarity of generated image captions as the metric, and report better targeted transfer performance than existing baselines.

Critical strengths
  1. Broad experimental coverage. The evaluation covers two datasets and twelve victim models, including both open-source and closed-source models. Component ablations and sensitivity analyses of the number of randomized variants and pruning probability provide evidence of the method’s effectiveness and transferability.
  2. The comparison of consensus strategies provides a useful empirical insight. Table 5 shows that consensus weighting among variants of the same surrogate performs well, whereas additionally applying consensus weighting across different surrogates reduces performance. This suggests that stabilizing update directions within a surrogate and preserving directional diversity across surrogates may serve distinct roles.
Critical weaknesses
  1. The methodological novelty is limited. RSE follows RaPA’s random parameter pruning strategy, while ISNC primarily introduces agreement-based gradient weighting over sampled variants. Previous work has already explored gradient conflict projection for vision-language model attacks, as in TGGR, and adaptive ensemble weighting with disparity-reduced filtering, as in AdaEA. Although the specific mechanisms differ, the paper lacks sufficient comparisons or deeper analysis to establish a substantive contribution beyond combining existing parameter randomization and gradient processing ideas.
  2. The source and computational cost of the improvements remain insufficiently established. Runtime, peak memory, and matched-compute comparisons are missing, making it difficult to assess the gains relative to the additional computational overhead. Furthermore, Table 5 does not adequately isolate the effects of computing losses independently for each surrogate, gradient normalization, and the proposed components, limiting the attribution of improvements.
Questions for authors
  1. Previous studies such as TGGR and AdaEA have explored gradient conflict resolution or adaptive aggregation to improve attack transferability. Could the authors provide a deeper analysis of why directional agreement among variants of the same surrogate improves transfer to unseen models? Is there evidence that higher within-surrogate gradient agreement corresponds to higher attack success on victim models?
  2. Were the baselines matched in total forward and backward evaluations? What are the per-image attack generation time and peak GPU memory usage? Which comparisons in Table 5 keep the loss formulation and gradient normalization unchanged?
Reviewer_qLxsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DA-Attack for transferable adversarial attacks on MLLMs. It introduces Randomized Surrogate Expansion (RSE) to generate diverse surrogate variants through random parameter pruning and Intra-Surrogate Neighborhood Consensus (ISNC) to aggregate gradients based on directional agreement. Experiments on NIPS 2017 and Flickr30K demonstrate improved attack performance across 12 MLLMs.

Critical strengths
  1. The method achieves substantial improvements over competitive baselines across multiple open-source and closed-source MLLMs.

  2. The framework is relatively simple and does not require training additional surrogate models. The authors also provide several ablation studies to examine its effectiveness.

Critical weaknesses

(1) Limited novelty. RSE closely follows the random parameter pruning strategy of RaPA, while ISNC primarily combines gradient normalization and cosine similarity-based weighting. The paper does not sufficiently establish its methodological distinction from existing approaches, particularly through direct comparisons with RaPA-style pruning under the same attack setting.

(2) Insufficient justification for ISNC. ISNC assumes that stronger directional agreement among gradients from randomized variants provides more reliable transferable guidance. However, variants of the same surrogate may share model-specific biases. The paper lacks direct evidence connecting intra-surrogate gradient agreement with cross-model transferability.

(3) In Table 5, the transition from "+RSE*" to "+RSE" also changes how the attack objective is computed, increasing the average ASR from 73.75% to 79.00%. Without evaluating this change independently of RSE, the contributions of the two modifications cannot be fully disentangled.

(4) RSE introduces additional forward and backward computations through multiple randomized paths. However, the paper lacks runtime, memory, and computationally matched comparisons, making it difficult to assess whether the performance improvements justify the additional computational cost.

(5) The reported ASR relies on GPT-4o-mini with a fixed GPTScore threshold of 0.5. The sensitivity of the results to this threshold and the choice of evaluator is not investigated, leaving the reliability of the targeted attack success measurements insufficiently established.

Questions for authors
  1. Section 4.1 states that RSE is applied only to three CLIP encoders, whereas Algorithm 1 appears to generate randomized variants for all surrogate models. Could the authors clarify the actual implementation?

  2. Several hyperparameters in Algorithm 1, including the consensus smoothing parameter, are not explicitly specified in the experimental settings. Could the authors provide their values to facilitate reproducibility?

Reviewer_oZd4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper aims to improve targeted transfer attacks against unseen MLLMs using a set of randomized surrogate model parameters. The proposed RSE creates temporary parameter-pruned variants of the surrogates. The proposed ISNC then weights attack gradients according to their directional agreement within each surrogate's neighborhood. The resulting directions are combined across surrogates to optimize the adversarial image.

Critical strengths

The reported attack performance is good, with improvements across multiple victim MLLMs. The method reuses pretrained surrogates without additional model training, and the experiments include both open-source and closed-source victims.

Critical weaknesses
  1. RSE looks very close to RaPA (CVPR 2026), which already uses random parameter pruning during attack optimization to generate multiple surrogate variants. Even the default pruning probability of 0.03 for CLIP has been used there. The paper cites RaPA and follows its pruning scope, but the additional insight of RSE is still limited. Fig. 4 shows that this probability works well in the tested setting; it does not establish a new explanation of MLLM transferability.

  2. The source of useful surrogate diversity needs a better comparison. The paper already uses five surrogates, but with the same online computation, would additional independently pretrained checkpoints or models with different pretraining data be more effective than randomized parameter variants?

  3. Table 5 has an attribution issue. The baseline obtains 58.50 ASR, +RSE* obtains 73.75, +RSE obtains 79.00, and +RSE+ISNC obtains 83.50. However, the starred version concatenates surrogate features, while the non-starred version computes an attack objective separately for each surrogate. This objective change gives a 5.25-point difference between the two RSE variants. I do not see a per-surrogate-objective-only baseline without RSE or ISNC.

Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.