acceptodds
Under review as a conference paper at ICLR 2027

From Fooling CNNs to Fooling MLLMs: Generating Unrestricted Adversarial Examples via Diffusion-Based Style Injection

Abstract

White-box adversarial attacks that directly backpropagate on multimodal large language models (MLLMs) or API query-based black-box attacks against MLLMs are both resource-intensive. Motivated by the observation that MLLMs suffer from difficulties in recognizing images of non-common styles, we propose a transferable StyleUAE to explore the vulnerability of MLLMs to stylized adversarial examples. StyleUAE is a low-resource attack that generates unrestricted adversarial examples (UAEs) on small models. To improve the transferability of UAEs, we first propose a diffusion-based style injection method to weaken the recognition ability of MLLMs, and then jointly attack a CNN and the cross-attention maps of the diffusion model in latent space to further disrupt discriminative cues. Considering that excessively stylizing and disrupting latent space may generate deformed UAEs, we leverage the harmonized image prior to reduce noticeable stylization, and further design structure-preserving components to preserve the original structure. Extensive experiments show that, compared with state-of-the-arts, StyleUAE achieves average ASRs (gains) of 77.5% (+45.8%), 80.9% (+56.7%), 77.2% (+21.4%), and 81.3% (+9.8%) when attacking open-source MLLMs, commercial MLLMs, zero-shot vision-language models, and visual unimodal classifiers, respectively. In addition, our UAEs exhibit satisfactory appearances and achieve competitive image quality scores. The anonymous code repository is available at https://anonymous.4open.science/r/StyleUAE.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Unrestricted adversarial examples, adversarial transferability, diffusion models, multimodal large language models, style transfer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TLMVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StyleUAE, a method that uses CNN, diffusion inversion and style transfer techniques to generate AEs that transfers to MLLMs. Experiments show its effectiveness on imagenet.

Critical strengths
  1. Pratical attack setting. This attacks is black-box and targets frontier MLLMs.
  2. Strong Transferability. The reported ASR on eight MLLMs is 79.7% on average, compared with 62.0% of MABA
Critical weaknesses
  1. Missing baselines. The paper’s evaluation remains limited to the conventional ImageNet classification setting and omits several recent baselines, including M-Attack (NeurIPS 2025), FOA-Attack (NeurIPS 2025), and MPCAttack (CVPR 2026). These baselines are more related to your threat model (MLLMs). Some your selected baselines, e.g., APA, were not originally designed to target MLLMs !!

  2. A major gap in image quality. The paper does not report reference-based metrics such as SSIM or LPIPS. No-reference metrics can assign high scores to natural-looking images even when their content has changed substantially from the original !!! These scores therefore provide insufficient evidence that the adversarial examples preserve the original image content.

Questions for authors

See weaknesses. Weakness 2 is the most serious issue. More: Can u explain why the proposed design produces more transferable adversarial examples.

2 comments

Thank you for your comment. All concerns have been answered as follows.

Q1. Weakness 1.1: Evaluation remains limited to the conventional ImageNet classification setting.

A1. The proposed evaluation is not limited to the conventional setting. Besides ASR on CNNs and ViTs, we also define ASR on zero-shot VLMs and ASR on MLLMs following Chain of Attack (CVPR 2025), which proposed an LLM-based ASR. Please see Appendix I.1.

Q2. Weakness 1.2: Missing baselines against MLLMs.

A2. In the experiments of the original manuscript, we have compared our method with recent baselines against MLLMs, including MABA (USENIX Security 2026) and VEAttack (ICLR 2026). We did not compare with M-Attack, FOA-Attack and MPCAttack for the following reasons: (1) Our method is an untargeted attack, whereas these attacks are designed for targeted attacks. Improperly modifying them to suit the untargeted attack may lead to unfair comparisons. (2) Our method is an unrestricted attack, whereas these methods are perturbation-restricted attacks.

In Tables 13–15 of Appendix K.6, we add the comparisons with them. M-Attack and FOA-Attack are modified for untargeted attacks following MPCAttack, and the untargeted version of MPCAttack is used. Tables 13 and 14 show that our method achieves the highest average ASRs. Table 15 shows that our method achieves the best scores on no-reference image quality metrics. The results of ours and MPCAttack (the second-best) are shown below. For more results, please see Appendix K.6.

Table 13. ASR (%) of black-box attacks on CNNs and ViTs.

AttacksVGG-19Mob-v2ConvNeXtViT-BSwin-BDeiT-BBEiT-BPVTv2AVG ↑
MPCAttack94.486.378.748.072.956.885.685.376.0
StyleUAE93.592.879.174.983.077.767.481.981.3

Table 14. ASR (%) of black-box attacks on open-source and commercial MLLMs.

AttacksBLIP-2InBLIPLLaVAQwen3.7Kimi2.6GPT5.6AVG ↑
MPCAttack81.960.087.963.171.372.672.8
StyleUAE74.881.276.578.281.980.378.8

Table 15. No-reference image quality assessment comparison.

AttacksHyperIQA ↑DBCNN ↑ARNIQA ↑MUSIQ-AVA ↑NIMA-AVA ↑MUSIQ-KonIQ ↑TReS ↑
MPCAttack0.31980.51720.55573.62193.849641.462448.5595
StyleUAE0.56830.65870.61864.11544.594451.525177.4289

Q3. Weakness 1.3: Why include baselines that were not originally designed to target MLLMs?

A3. Besides tests on MLLMs, we also test the transferable performance when attacking CNNs and ViTs, so the baselines (e.g., APA) targeting small models are selected. These baselines and ours rely on small CNN-based surrogate models to generate AEs. We want to check whether these similar low-resource methods can fool MLLMs, so we add them to comparisons of attacks on MLLMs.

Q4. Weakness 2: Image quality assessment.

A4. Considering that clean images are not available as references in some applications, we select no-reference metrics to evaluate image quality of UAEs. The metrics used, such as MUSIQ-AVA and NIMA-AVA, are usually employed for aesthetic evaluation. It means that UAEs with high scores of no-reference metrics won't arouse people's suspicion, and will probably evade human supervision to launch attacks. According to the definition, UAEs are allowed to change substantially from the original image, as long as UAEs retain visible cues of the original class for human inspection. Based on the definition, our method's preservation of the original image content does not mean preserving fine details, but preserving image structure, shape, etc., that are related to human recognition clues. We have reported the reference-based metrics SSIM and LPIPS in Table 18 of Appendix K.7. Although our attack does not obtain better results, it generates satisfactory UAEs which can be recognized by humans. Please refer to the randomly selected 80 pairs of clean images and UAEs in Figure 15 of Appendix N.6.

Q5. Explain why the proposed design produces more transferable adversarial examples.

A5. We enhance the transferability via jointly (1) disturbing the semantic features of CNN surrogate models in latent space, (2) destroying discriminative cues in cross-attention maps, and (3) injecting style to induce semantic disruptions. The benefits of each enhancement are supported by the ablation study in Table 4 of the manuscript, with the results reproduced below.

Classifiers AVG (%) ↑Zero-shot VLMs AVG (%) ↑Open-source MLLMs AVG (%) ↑
Only style injection36.233.545.8
Only attack CNN66.638.419.2
Only attack diffusion model32.135.729.9
StyleUAE81.377.277.5

Thank you again for your reviews.

Reviewer_Bk5SOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes StyleUAE, an untargeted transferable attack that generates Unrestricted Adversarial Examples (UAE) on small models. To induce deeper semantic disruptions, StyleUAE jointly attacks the CNN and diffusion model in latent space while injecting style information into UAEs. Extensive experiments show that StyleUAE achieves excellent black-box transferability when attacking visual unimodal classifiers, zero-shot vision-language models, open-source MLLMs, and commercial MLLMs.

Critical strengths
  1. Combining style injection with classifier-based and diffusion-attention objectives is an interesting approach to generating UAEs.
  2. The reported transferability gains over existing methods are promising.
  3. The authors explicitly examine the trade-off between attack effectiveness and visual quality. StyleUAE achieves a human content-recognizability score of 97.9% with RN-50 as the surrogate model, and the appendix includes examples of both poor visual quality and failed attacks.
Critical weaknesses
  1. The method combines style injection with classifier and diffusion-attention objectives in a compelling way. However, it remains unclear whether joint optimization provides a benefit beyond combining existing methods sequentially. Since the style-injection mechanism is closely related to StyleID and the attack objectives draw on DiffAttack, I suggest comparing StyleUAE with a matched-compute baseline that first applies StyleID and then performs a DiffAttack-style optimization.
  2. The evaluation does not clearly distinguish model misrecognition from a change in the image’s visible class. Attack success is measured against the original ImageNet label, while the content recognizability evaluation explicitly allows the adversarial image to depict a different category (Appendix H.2–H.3). For example, if an image originally labeled “husky” comes to look like a wolf, an MLLM captioning it as a wolf may be responding accurately to the adversarial image, even though the current metric counts this as a successful attack. An independent, blinded human assessment of whether the original fine-grained class remains visible, followed by ASR reported on the class-preserving subset, would clarify how much of the measured success reflects misrecognition.
  3. Qwen3.7 is the sole judge of MLLM captions, including captions produced when Qwen3.7 is the attacked target. Moreover, the reported ASR includes partial credit for broad or ambiguous category descriptions. Although Table 10 gives a strict ASR of 68.1% for StyleUAE when only fully successful cases are counted, it does not provide corresponding strict results for every baseline. Please report the counts of scores 0, 0.5, and 1, together with strict ASR for all methods, and validate a stratified sample using independent human judges or a separate evaluator.
  4. Section 5.4 states that StyleUAE achieves the best or second-best image-quality results. Table 3 supports this claim for four of the seven reported metrics, but StyleUAE ranks third or lower on the other three, while HAM ranks first on six metrics. Please qualify the statement to reflect the variation across metrics.
Questions for authors
  1. Table 2 reports 29.5 seconds per image for StyleUAE but does not make clear whether retrieval, MLLM selection, model loading, and diffusion optimization are all included. Does the 29.5-second timing include the entire per-image style-selection stage?
3 comments

Thank you for your comment. All concerns have been answered as follows.

Q1. Weakness 1: Provide the benefit of joint optimization beyond combining existing methods sequentially.

A1. We think the sequential attack, which first applies StyleID and then executes DiffAttack, may benefit ASR, but has a risk of deteriorating image quality. Because StyleID first destroys the original latent, it is hard to preserve the original image structure and shapes in the next DiffAttack. We compare our StyleUAE with the sequential attack under the same settings as in Table 1 of the main manuscript. The results show that the sequential attack obtains better ASRs when attacking CNNs and ViTs, while it performs worse than StyleUAE when attacking open-source MLLMs. StyleUAE achieves much better image quality scores, verifying the gains of the proposed joint optimization (see Table 25 of Appendix L.6).

Table 25: Comparison with the sequential attack. All UAEs are generated using RN-50 as the surrogate model.

MethodClassifiers AVG (%) ↑Zero-shot VLMs AVG (%) ↑Open-source MLLMs AVG (%) ↑NIMA-AVA ↑TReS ↑
StyleID + DiffAttack93.887.361.24.181260.1529
StyleUAE81.377.277.54.594477.4289

Q2. Weakness 2: Distinguish model misrecognition from a change in the image’s visible class.

A2. We report ASR on the class-preserving subset in Table 19 of Appendix K.8. With our image quality-preserving schemes, the vast majority of UAEs retain visible cues of the original class for human inspection. The experimental results further confirm that this ASR (i.e., misrecognition rate) is only a little lower than the original reported ASR.

Q3. Weakness 3: Show score counts, strict ASR for all methods, and validation using a separate evaluator.

A3. We add detailed counts for calculating ASR and strict ASR for all methods in Table 11 of Appendix K.5. The results from a separate evaluator using Qwen3.8 Flash are also reported in Table 12 of Appendix K.5. Being consistent with the original results, our method achieves the highest average ASR. Comparisons with MABA are shown below. Please see more results in Table 11 of Appendix K.5.

Table 11: Score counts and ASR (%) of black-box attacks on open-source and commercial MLLMs, which are evaluated by Qwen3.7. Each cell reports the counts of scores 0, 0.5, and 1, respectively, totaling 1,000. The last two columns report the average ASR and strict ASR.

SurrogateAttacksBLIP-2InBLIPLLaVAQwen3.7Kimi2.6GPT5.6Claude4.6Gemini3.1ASR (%) ↑Strict ASR (%) ↑
CLIP-L/14MABA202/329/469127/310/563166/372/462283/297/420270/268/462279/273/448270/265/465251/281/46862.047.0
RN-50StyleUAE113/279/60864/248/68873/324/603104/227/66989/184/72796/201/70381/187/73285/195/72079.768.1

Table 12: Score counts and ASR (%) of black-box attacks on open-source and commercial MLLMs, which are evaluated by Qwen3.8. Each cell reports the counts of scores 0, 0.5, and 1, respectively, totaling 1,000. The last two columns report the average ASR and strict ASR.

SurrogateAttacksBLIP-2InBLIPLLaVAQwen3.7Kimi2.6GPT5.6ASR (%) ↑Strict ASR (%) ↑
CLIP-L/14MABA202/329/469126/311/563165/372/463283/295/422270/266/464279/271/45062.647.2
RN-50StyleUAE113/279/60864/248/68873/323/604104/226/67089/182/72996/197/70778.966.8

Q4. Weakness 4: Qualify the image-quality statement.

A4. We have modified the statement in Section 5.4.

Q5. Does the 29.5-second timing include the entire per-image style-selection stage?

A5. The reported timing does not include the style-selection stage. This stage takes an extra 0.015 s per image on an NVIDIA RTX 3090 GPU.

Thank you for the clarification. After considering the authors’ responses and the other reviewers’ comments, I maintain my original rating.

Thank you for your consideration of our response.

Reviewer_5LB1OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a type of adversarial attack using unrestricted adversarial examples using small surrogate models and diffusion based stylisation with constraints

Critical strengths

-attack success rate appears high against baselines in tables 1 and 2

Critical weaknesses

-no explicit section outlining what the threat model is -unclear what the deeper underlying mechanism is that is responsible for attack transferability -not clear enough why we need yet another adversarial attack -not clear what the core technical innovation is compared to prior work -proposed attack is claimed to be low-resource, but it is not clear what separates low from high resource attack

Questions for authors

-provide detailed description of threat model -what is the deeper underlying mechanism responsible for attack transferability? -why do we need yet another type of adversarial attack? -what is the core technical innovation compared to prior work? -what is precise difference between a low and high resource attack

3 comments

Thank you for your comment. All concerns have been answered as follows.

Q1. Describe the threat model.

A1. White-box and transferable black-box attacks are considered. In the white-box setting, the attacker is assumed to have full access to the attacked model, and generates an adversarial example (AE) from a clean image via backpropagating on the model. In the transferable black-box setting, the attacker has no access to the target model and generates an AE on a fully accessible surrogate model, which is then transferred to attack the target model. Following the definition of an unrestricted adversarial example (UAE), the attack strength is not restricted, as long as the UAE remains human-recognizable. In our work, our method performs white-box attacks on small CNN or ViT models (e.g., RN-50, ViT-B), and transfers the UAEs generated on these small models to attack inaccessible models (e.g., commercial GPT5.6). Details in Appendix C.

Q2. What is the deeper underlying mechanism responsible for attack transferability?

A2. The mechanism of transferable adversarial attacks is an open problem. As indicated by FIA (ICCV 2021) and NAA (CVPR 2022), different deep models depend on semantic object-aware features for classification. It can be assumed that the AE, which is generated through disrupting the shared object-aware features of the surrogate model, has transferability to fool the target model. Following such a routine, we enhance the transferability via (1) disturbing the semantic features of CNN-based surrogate models in latent space, (2) destroying discriminative cues in cross-attention maps, and (3) style injection to induce semantic disruptions of multimodal large language models (MLLMs).

Q3. Why do we need yet another type of adversarial attack?

A3. We are unsure what you mean by “another type of adversarial attack.” Could you please clarify this point?

Q4. What is the core technical innovation?

A4. The core innovations of our work are twofold: (1) we first propose a stylized diffusion-based unrestricted attack to enhance transferability; (2) we design style image selection, structure preservation, and a latent distribution constraint to generate human-recognizable UAEs, which solves the challenge of diffusion-based attacks that tend to generate deformed UAEs. Our method shows a low-resource black-box pathway to reveal the vulnerability of commercial MLLMs using small CNN-based surrogate models.

Q5. What is the precise difference between a low and high resource attack?

A5. We distinguish between a low and high resource attack based on the parameter scale of the surrogate model. An attack that uses a CNN or ViT with small-scale parameters as the surrogate model is taken as a low resource attack, which can run on GPUs with small memory capacity to generate AEs quickly, such as our attack generating AEs on a CNN model RN-50. In contrast, high resource attacks usually use the surrogate models with large-scale parameters (e.g., LLaVA-1.5-7B) for attack optimization. These attacks require GPUs with large memory capacity or need much time to generate AEs, such as Fit the Distribution (NeurIPS 2025).

Thankyou, acknowledging your response.

Thank you for your acknowledgement of our response. It has given us great encouragement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.