acceptodds
Under review as a conference paper at ICLR 2027

SimAttack: A Simple and Effective Transferable Untargeted Adversarial Attack on Large Vision-Language Models

Abstract

Large Vision-Language Models (LVLMs) have demonstrated strong multimodal understanding and reasoning capabilities, yet remain vulnerable to transferable adversarial attacks, underscoring the need for rigorous robustness evaluation under black-box settings. Existing transfer-based untargeted attacks often rely on ground-truth textual annotations to obtain transferable optimization signals. However, requiring such annotations for every attacked input imposes a strong practical assumption, since they are generally unavailable in real-world scenarios. To address this limitation, we propose SimAttack, a simple and effective transfer-based black-box untargeted attack that derives transferable optimization signals solely from surrogate vision encoder representations without ground-truth textual supervision. By leveraging visual representations from different local regions and encoder layers, SimAttack reduces reliance on a limited subset of surrogate encoder representations and improves adversarial transferability. Extensive experiments demonstrate that our method consistently outperforms existing state-of-the-art attack methods across heterogeneous LVLMs and diverse downstream tasks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI Safety, LVLMs, Transferable Adversarial Attacks, Untargeted Attacks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yX9uOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SimAttack generates transferable black-box attacks on vision-language models by disrupting spatially aligned patch features across all layers of a surrogate vision encoder, without requiring ground-truth text annotations.

Critical strengths

The paper provides a pratical attack setting.

Critical weaknesses
  1. Limited novelty. The crop design, title, and overview figure closely resemble M-Attack. The layer patch objectives resembles VLAttack. The main change appears to be adapting targeted attacks to an untargeted setting, which offers insufficient methodological novelty for ICLR acceptance in my view.
  2. Narrow scope. The paper focuses solely on untargeted attacks, without demonstrating broader implications or applications beyond degrading task performance.
Questions for authors

Why transferable? The paper reports empirical improvements but does not clearly explain why the proposed design improves cross-model transferability or provide sufficient analysis of the underlying mechanism. Could the auther please give any explanation?

4 comments
Reviewer_yX9uOpenReview ↗

Followup comment

I agree with Reviewer ZAGx and am leaning towards score 1 (Clear rejection).

AuthorsOpenReview ↗

Response to Reviewer yX9u

Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).


W1: Thank you for your comment. SimAttack is motivated by a key challenge in transferable black-box untargeted attacks without ground-truth textual annotations: deriving transferable optimization signals from surrogate vision encoders. To this end, SimAttack incorporates visual representations from diverse local regions and encoder layers to reduce reliance on a limited subset of surrogate representations. Specifically, unlike M-Attack's target feature alignment, CPLRS addresses the spatial correspondence required for clean–adversarial feature disruption. By applying identical crop-and-resize transformations to both images, CPLRS avoids feature differences caused by mismatched image content. ALPFD incorporates corresponding patch features across all encoder layers within these spatially aligned local views. The ablation results support these designs. CPLRS improves AvgSRR from 19.00% with independent cropping to 26.99% (Table 3), while combining CPLRS and ALPFD achieves 49.10% AvgSRR, outperforming either component alone (Table 2).


W2: Thank you for your valuable comment. Untargeted adversarial attacks are an important research direction for evaluating LVLM robustness, as demonstrated by recent studies [1][2][3][4]. While targeted attacks evaluate whether adversaries can steer LVLMs toward predefined target responses, untargeted attacks assess whether LVLMs can maintain semantically valid understanding of the original visual content under adversarial perturbations. These two settings therefore examine complementary aspects of adversarial robustness.

This setting also present two distinct challenges. First, without predefined target semantics, insufficient optimization guidance can cause perturbations to overfit surrogate-specific features or limited visual attributes, compromising transferability. Second, LVLMs have an open-ended output space in which multiple semantically valid responses may correspond to the same image and prompt, making it difficult to identify transferable optimization signals. Existing methods such as MABA and SAF address these challenges using ground-truth textual annotations. SimAttack further considers their absence and derives optimization signals solely from a pretrained surrogate vision encoder.

We evaluate SimAttack across five heterogeneous LVLMs, five datasets, and three downstream tasks: image captioning, visual question answering, and hallucination evaluation. The consistent performance degradation across these settings (Table 1) demonstrates its effectiveness in evaluating black-box robustness across diverse models and tasks.


Q1: Thank you for your valuable comment. SimAttack improves cross-model transferability by incorporating diverse visual representations across spatial local regions and encoder layers, reducing reliance on a limited subset of surrogate representations. Specifically, CPLRS resamples local regions across iterations to capture diverse visual information while preserving spatial correspondence between clean and adversarial images, ensuring that feature discrepancies primarily reflect adversarial perturbations. ALPFD uniformly incorporates patch features from all encoder layers, avoiding dependence on specific layers whose effectiveness may vary across victim models and downstream tasks.

The main experiments demonstrate that SimAttack consistently outperforms all baselines across all 25 dataset–LVLM combinations, achieving the highest Average SRR on each of the five victim LVLMs (Table 1). The ablation results further support this explanation. CPLRS improves AvgSRR from 19.00% with independent cropping to 26.99% (Table 3), while all-layer optimization achieves 22.34% AvgSRR compared with 12.17% using only the final layer (Table 4). Combining CPLRS and ALPFD further improves AvgSRR to 49.10% (Table 2). Together, these results support the effectiveness of exploiting spatially diverse and hierarchical visual representations for improving cross-model transferability.

Reviewer_yX9uOpenReview ↗

Followup comment

Specifically, unlike M-Attack's target feature alignment, CPLRS addresses the spatial correspondence required for clean–adversarial feature disruption. By applying identical crop-and-resize transformations to both images, ...

The method is presented as a combination of A and B, yet the comparison contrasts your component A (M-Attack) with M-Attack’s component B, and vice versa. This does not clearly establish the novelty of either component. What needs to be clarified is how your crop-and-resize strategy differs from the corresponding strategy in M-Attack.

transferability by incorporating diverse visual representations across spatial local regions and encoder layers, reducing reliance on a limited subset of surrogate representations..

You just repeated your method design. But why the design work? The author still didn't justify the transferability beyond reporting the table numbers.

Again, i'm leaning towards score 1 (Clear rejection).

AuthorsOpenReview ↗

Response to Reviewer yX9u

Thank you for your thoughtful comments.

The method is presented as a combination of A and B, yet the comparison contrasts your component A (M-Attack) with M-Attack’s component B, and vice versa. This does not clearly establish the novelty of either component. What needs to be clarified is how your crop-and-resize strategy differs from the corresponding strategy in M-Attack.

While CPLRS in SimAttack employs crop-and-resize transformations, its novelty lies in the local-view pairing and optimization strategy. Unlike M-Attack, which aligns the features extracted from cropped adversarial images with those of either the global or independently cropped target image. CPLRS applies identical crop-and-resize transformations to clean and adversarial images, ensuring that corresponding features originate from the same spatial region. This avoids content-induced discrepancies unrelated to adversarial perturbations and enables local sampling to integrate with ALPFD's all-layer patch feature disruption.

You just repeated your method design. But why the design work? The author still didn't justify the transferability beyond reporting the table numbers.

Optimizing adversarial perturbations on a fixed full-image view or a selected subset of encoder layers may lead to excessive dependence on specific surrogate representations, limiting cross-model transferability and task generalization. To address this issue, SimAttack seeks to disrupt a broader range of visual information and representations. Specifically, CPLRS Specifically, CPLRS diversifies the visual content emphasized during optimization by sampling different local regions across iterations, while preserving spatial correspondence to ensure that feature discrepancies primarily reflect adversarial perturbations. ALPFD broadens the range of visual representations targeted for disruption by incorporating patch features from all encoder layers, reducing dependence on any particular layer or subset of layers. Together, CPLRS and ALPFD enable perturbation optimization across diverse local views and encoder layers, reducing reliance on specific surrogate representations and improving transferability across different LVLMs and downstream tasks.

Reviewer_ZAGxOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a black-box transferable, non-targeted adversarial attack method, SimAttack for LVLM, aiming to enhance the transferability of adversarial samples between different LVLMS and downstream tasks without relying on real text annotations. Specifically, this method utilizes a pre-trained visual encoder as an alternative model and combines two core modules.

Critical strengths
  1. The authors conducted a systematic set of experiments across five distinct LVLM architectures and five datasets, covering tasks such as image captioning, VQA, and hallucination evaluation. SimAttack consistently outperformed baseline methods across all evaluated model-dataset combinations, demonstrating strong cross-model transferability.

  2. The overall methodology is relatively straightforward and does not require the training of additional attack models. Furthermore, the authors performed ablation studies, examining CPLRS, ALPFD, layer selection, layer weighting, and the use of various alternative visual encoders, which provide insight into how specific design choices influence the final attack performance.

Critical weaknesses
  1. The motivation requires a more compelling justification, as obtaining captions for images is neither particularly difficult nor dependent on ground-truth data; for instance, one could easily generate them using a simple model like GPT-4o or BLIP at a minimal cost.
  2. According to Algorithm 1 in Appendix A, the initial perturbation is set to zero (δ(0)=0\delta^{(0)}=0). Meanwhile, ALPFD operates by minimizing the cosine similarity between the features of the original image and those of the adversarial image. Since CPLRS applies identical cropping operations to both images, their features are initially identical. Mathematically, the gradient of the cosine similarity between two identical, non-zero feature vectors with respect to either vector is zero.
  3. The two core modules of this paper are primarily built upon existing techniques. CPLRS essentially employs random cropping during the optimization process, maintaining consistent spatial transformations across both the original and adversarial images, a concept highly similar to established "input transformation-based" transferable attacks. ALPFD utilizes cross-layer patch-feature cosine similarity for perturbation, which overlaps significantly with existing multi-layer feature-based attack methods. Consequently, the proposed approach appears to be a combination of existing random input transformation and multi-layer feature perturbation techniques, lacking a convincing new mechanism or theoretical analysis to substantiate its unique contribution.
  4. The authors select only PA-Attack, MABA, and MPCAttack as primary baselines, failing to include comparisons with more directly relevant methods such as VEAttack, the original BSA, or existing attack methods augmented with random cropping strategies. In particular, VEAttack is highly aligned with the optimization objective of this paper. Furthermore, the authors modify PA-Attack (originally designed for a gray-box setting) by substituting its encoder with CLIP-L/14, while other baselines retain their original surrogate configurations. This experimental setup lacks strict control over the surrogate models and optimization conditions, which may compromise the fairness of the comparison. The authors should re-evaluate their method under a unified surrogate configuration and computational budget to verify whether "SimAttack" still significantly outperforms a simple combination of existing attack methods and standard input augmentation techniques.
  5. This paper defines an "untargeted attack" as one that induces an LVLM to generate semantically incorrect responses, yet it primarily measures attack effectiveness through declines in task-specific metrics such as CIDEr, SPICE, CLIPScore, VQA Accuracy, and POPE F1. In the context of open-ended image captioning, a drop in evaluation scores does not necessarily imply that the model has generated semantically erroneous content. Therefore, incorporating a "GPT-as-a-judge" evaluation is recommended. Furthermore, the paper should report the values ​​for all individual metrics rather than presenting only their averages.
  6. ALPFD requires the extraction and optimization of patch features from all layers of the visual encoder during each iteration. Compared to methods that utilize only a subset of layers or the final layer alone, this approach likely incurs additional computational and VRAM overhead. However, the paper fails to provide data regarding attack generation time, VRAM consumption, or performance comparisons under a fixed computational budget.
  7. The primary experiments are limited to a perturbation budget of ϵ=16/255\epsilon=16/255 and a fixed 30 iterations. The study lacks stability analysis across varying perturbation budgets, iteration counts, and random seeds.
  8. The paper does not validate the transferability of the attacks on actual closed-source LVLMs. Notably, the visual encoder in LLaVA-v1.6 shares the same architecture as the CLIP-L/14 surrogate model used in the study; the authors should clarify whether the specific checkpoints overlap. This is crucial to avoid misinterpreting results obtained under the condition of a shared encoder as evidence of true cross-architecture transferability. Moreover, for a "gray-box" approach, the performance is significantly worse than I would have expected. I believe that at least one existing method, VEAattack, could achieve results comparable to those presented here, which raises concerns regarding the authenticity and completeness of the reported reproduction.
Questions for authors
  1. Could the authors provide comparisons with VEAttack, BSA, and existing feature-based attacks augmented with the same random crop transformation?
  2. Does SimAttack still demonstrate a substantial improvement when the surrogate model, optimization steps, and computational budget are controlled consistently across all methods?
  3. Can the authors check all baseline method was reproduced correctly in the experiment, and the results are also correct.
4 comments
AuthorsOpenReview ↗

Response to Reviewer ZAGx (Part 1 of 2)

Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).


W1: Thank you for your valuable comment. We agree that models such as BLIP or GPT-4o can generate image captions at relatively low cost. However, captions generated by auxiliary models may omit relevant visual details or contain inaccurate information, making the resulting semantic guidance dependent on the captioning model. Furthermore, caption generation introduces additional computational overhead and model dependencies. The motivation of our study is to investigate whether effective transferable untargeted attacks can be achieved without auxiliary textual supervision, which avoiding these limitations of text-based guidance.


W2: Thank you for your question. We agree that with zero initialization and identical cropping, the clean and adversarial images produce identical features. Consequently, the gradient of the cosine similarity with respect to either feature is zero in exact arithmetic, preventing the first perturbation update. To understand why the attack still proceeds in practice, we inspected the gradients at the first iteration and confirmed small nonzero residuals caused by finite-precision computation. These residual gradients produce nonzero perturbation updates through the sign-based optimization, allowing subsequent iterations to proceed. Therefore, we adopt uniform random initialization to avoid this issue, which provides a non-degenerate starting point and aligns with prior work, including VEAttack and PA-Attack. We have updated the algorithm and implementation, rerun all experiments, and revised the reported results accordingly.


W3: Thank you for the comment. We agree that input transformations and multi-layer feature attacks are established techniques, but our core contribution is to derive transferable optimization signals solely from a public available vision encoder, without textual supervision. To this end, CPLRS resamples local regions across iterations to provide diverse visual information, while applying the same crop-and-resize transformation to clean and adversarial images within each iteration so that feature comparisons focus on perturbation-induced changes. ALPFD minimizes the average cosine similarity between corresponding clean and adversarial patch features across all encoder layers, avoiding predefined layer preferences that may not generalize across unknown victim models and downstream tasks.


W4 & Q1 & Q2: Thank you for your comment. To enable comparison under our transfer-based black-box, image-only setting, we adapt VEAttack to optimize perturbations on a surrogate vision encoder and BSA touse only its image-encoder feature disruption objective. All methods are evaluated with the same attack iterations, step size, and perturbation budget. Except for MPCAttack, which retains its original multi-paradigm surrogate configuration, all methods use CLIP-L/14 as the surrogate model. As shown in Table 1, SimAttack consistently outperforms these baselines under the same evaluation protocol.

Table 10 shows that MABA and MPCAttack already take substantially longer than SimAttack while achieving weaker attacks. We therefore allocate more iterations to the faster VEAttack and BSA for comparison. As shown in the computational-budget comparison under W6 in Part 2 of our response, SimAttack at 30 iterations outperforms both baselines at 50/70 iterations across all captioning metrics on both victim models.


W5: Thank you for your valuable suggestion. We acknowledge that standard captioning metrics does not fully capture semantic errors, so we further evaluate the semantic consistency of generated captions using an LLM-as-a-judge evaluation. Specifically, we adopt CLAIR [1], which uses an LLM (GPT-6 Luna in our evaluation) to assess whether generated and reference captions describe the same image and has demonstrated strong correlation with human judgments of caption quality. Results are reported in Qwen2.5-VL-7B/InternVL3-8B order; lower CLAIR and higher SRR indicate stronger attacks. † denotes methods adapted to our transfer-based black-box setting. As shown in the table below, SimAttack achieves the lowest CLAIR scores on both models, providing further evidence of semantic degradation in generated captions. The individual captioning metrics of are reported in Appendix C.3 (Table 12).

MethodCLAIR↓SRR(%)↑
Clean0.8330/0.85980.00/0.00
PA-Attack†0.6266/0.677524.78/21.21
VEAttack†0.6231/0.629325.20/26.82
MABA0.5846/0.583329.82/32.17
BSA†0.4382/0.488447.40/43.20
MPCAttack0.2060/0.105775.27/87.70
SimAttack0.1102/0.029586.77/96.57
AuthorsOpenReview ↗

Response to Reviewer ZAGx (Part 2 of 2)

W6: Thank you for your comment. SimAttack obtained all-layer patch features through a single forward pass per image, without requiring separate passes for each layer. The additional overhead may mainly comes from intermediate feature storage, layer-wise similarity computation, and gradient backpropagation. We have added analyses of computational cost and attack effectiveness under comparable budgets in Appendix C.1 and C.2, respectively.

For computational cost, we compare generation time, GPU memory usage, and attack effectiveness of all methods on 1,000 COCO 2014 images. Adversarial examples are generated on a single NVIDIA RTX 4090 GPU and evaluated against Qwen2.5-VL-7B. Captioning results are reported as raw scores, with lower values indicating stronger attacks. As shown in the table below, SimAttack requires 50.49 minutes and 2.29 GB of GPU memory, consuming less time and memory than MABA and MPCAttack. It outperforms VEAttack, PA-Attack, and BSA with similar memory usage at 1.8–1.9 times their runtime.

MethodTime (min)GPU Memory (GB)CIDErSPICECLIPScoreRefCLIPScore
VEAttack†26.522.2220.5612.7576.33107.80
PA-Attack†27.992.2420.4912.7376.19107.61
MABA215.6014.4820.8812.4675.47106.77
BSA†27.032.3914.969.5567.5496.22
MPCAttack121.9811.347.885.8757.4882.65
SimAttack50.492.293.413.4445.7167.22

For computational budget, we compare SimAttack at 30 iterations with VEAttack and BSA at 50 and 70 iterations, using Qwen2.5-VL-7B and Phi-4-Multimodal as victim LVLMs. Captioning results are reported as raw scores in Qwen/Phi order, with lower values indicating stronger attacks. As shown in the table below, SimAttack achieves better results across all captioning metrics on both models, even when the baselines use more computation time (62.38-63.46 minutes). These results indicate that SimAttack's performance advantage primarily stem from its method design rather than a larger computational budget.

MethodStepsTime (min)CIDErSPICECLIPScoreRefCLIPScore
VEAttack†5042.3720.52/43.4712.82/14.8375.67/72.04106.94/104.21
VEAttack†7063.4620.04/43.2312.46/14.7875.52/72.63106.73/104.89
BSA†5044.3915.55/33.399.75/11.8168.55/66.8497.60/97.13
BSA†7062.3816.56/36.099.91/12.3969.03/68.2098.25/99.07
SimAttack3049.013.42/4.293.44/2.9445.72/44.7967.25/68.07

W7: Thank you for your valuable suggestion. We have conducted experiments on MPCAttack, BSA, and SimAttack under varying attack iterations and perturbation budgets, with results averaged over three random seeds, as reported in Appendix B.3-B.4 (Tables 8-9). For attack iterations, SimAttack consistently outperforms both methods across 10-100 iterations, with diminishing gains beyond 30 iterations. For perturbation budgets, SimAttack consistently outperforms BSA across all budgets. Compared with MPCAttack, it performs worse at 4/255, achieves comparable performance at 8/255, and demonstrates a clear advantage at 16/255.


W8 & Q3: Thank you for your comment. We evaluate SimAttack on five open-source LVLMs with diverse architectures to provide comprehensive coverage, using VLMEvalKit to ensure consistent comparison. To further validate its transferability to commercial closed-source LVLMs, we evaluate SimAttack on three LVLMs using the same 200 COCO2014 images. As shown in the table below, SimAttack achieves AvgSRRs of 63.60%, 46.21%, and 63.43% on GPT-6 Luna, Claude Haiku 5.5, and Gemini 3.8 Flash, respectively, with higher CLAIR SRRs of 82.84%, 69.34%, and 77.86%. AvgSRR averages the SRRs of CIDEr, SPICE, CLIPScore, and RefCLIPScore, excluding CLAIR.

Victim LVLMAvgSRR (%)CLAIR SRR (%)
GPT-6 Luna63.6082.84
Claude Haiku 5.546.2169.34
Gemini 3.8 Flash63.4377.86

We acknowledge that LLaVA-v1.6 and the surrogate vision encoder share the CLIP ViT-L/14 architecture but uses a different input resolution (336×336 vs. 224×224). The other four LVLMs used in our evaluation use different vision encoder architectures. As shown in Table 1, even excluding LLaVA-v1.6, SimAttack still consistently outperforms the strongest baseline MPCAttack across these four LVLMs, yielding an average SRR of 52.25% versus 41.73%.

For baseline reproducibility, we have carefully rechecked all baseline implementations and verified the reported results. For VEAttack and PA-Attack, we preserve their core procedures while moving perturbation optimization from the victim encoder to the surrogate for transfer-based black-box evaluation. Their effectiveness therefore depends on cross-encoder transferability, which differences in feature representations and gradients can weaken relative to the original gray-box setting.

References:

[1] Chan, D. M., et al. CLAIR: Evaluating Image Captions with Large Language Models. EMNLP, 2023.

Thanks for your rebuttal. The concern related to the reliability of results has been solved due to the difference in input resolution of CLIP (needs to emphsize in the paper as these two versions are huge different). Therefore, I increase the score to weak reject. However, the motivation is still not convincing without evidence, the novelty of the method is limited, and there is a lack of a revised paper. At present, I will keep a negative score for this paper.

AuthorsOpenReview ↗

Response to Reviewer ZAGx

Thank you for your careful reassessment and for acknowledging our clarification of the experimental results.

Regarding the motivation, while auxiliary models can generate image captions, the quality of such guidance depends on the captioning model and introduces additional model dependencies and computational overhead, as discussed in our previous response. Therefore, our study investigates whether effective transferable untargeted attacks can be achieved using visual representations alone, without auxiliary textual supervision.

Regarding novelty, we would like to emphasize that CPLRS differs from M-Attack in how local views are paired for optimization. By applying identical crop-and-resize transformations to clean and adversarial images, CPLRS preserves spatial correspondence, avoiding content-induced feature differences and enables local region sampling to be combined with ALPFD's all-layer patch feature disruption.

We have uploaded a revised manuscript with updated results from both the original and supplementary experiments, along with an explicit clarification of the CLIP input-resolution difference in Appendix D.

We appreciate your constructive feedback and hope these revisions address your remaining concerns.

Reviewer_rWxHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies black-box untargeted transferable attacks on LVLMs. The authors point out that some existing methods need ground-truth text annotations to generate adversarial examples, but these annotations are usually not available in real-world attack scenarios. To address this problem, the authors propose SimAttack, which only uses a publicly available vision encoder as the surrogate model to generate transferable adversarial images without ground-truth text annotations.

The method is relatively simple and has two main parts. First, in each iteration, it crops the same region from the clean image and the adversarial image, and randomly selects a new region in the next iteration. Second, it uses patch features from all layers of the vision encoder and calculates the cosine similarity between the corresponding clean and adversarial features. The perturbation is updated by reducing this similarity. The experiments include five LVLMs including Qwen2.5-VL-7B, InternVL3-8B, MiniCPM-o 2.6, Phi-4-Multimodal, and LLaVA-v1.6-7B and five datasets. SimAttack performs better than the compared baselines in all tested settings. The ablation studies also show that removing local region sampling or only using the final layer features leads to lower attack performance.

From my understanding, the main contribution of this paper is not a new optimization algorithm. Instead, it combines visual features from different image regions and different encoder layers to improve the transferability of adversarial examples generated on a surrogate model. The experiments show that this approach can achieve good attack performance without using ground-truth text annotations during attack generation.

Critical strengths
  1. The attack does not require ground-truth text annotations. I think one strength of this paper is that it does not need ground-truth text annotations when generating adversarial examples. It also does not require access to the internal information of the victim LVLMs, and only uses a publicly available vision encoder as the surrogate model. Compared with methods that rely on ground-truth annotations, this setting is more practical for black-box attacks. Also, the method does not need to design different loss functions for different downstream tasks, which makes it easier to apply across different models and tasks.

  2. The experimental results are strong, and the ablation studies are relatively complete. Although the method is simple, the experimental results are quite convincing. The authors evaluate SimAttack on five LVLMs and five datasets, and it performs better than the compared baselines in all tested settings. Some improvements are also quite large. The authors further compare different cropping strategies, layer selections, and layer weighting methods. The ablation results show that removing either CPLRS or ALPFD leads to a clear drop in attack performance. I think these results provide good evidence that both components contribute to the final performance.

Critical weaknesses
  1. The explanation of transferability is not clear. The authors claim that using different regions and all encoder layers can reduce the dependence on the surrogate model. However, I think the experiments mainly show that these two parts improve attack performance. It is still not clear why using more features from the same CLIP model can improve transferability to other models.

  2. Lack of comparison on computational cost. Although all methods use the same number of attack steps, they use different surrogate models and optimization methods. SimAttack needs to calculate patch features from all encoder layers, which may increase the computational cost. I think a comparison of runtime or attack cost would help better understand the improvement.

Questions for authors
  1. About the initialization in Algorithm 1. In Algorithm 1, the authors initialize the perturbation to zero, so the clean and adversarial images are identical in the first iteration. Does the choice of initialization change the results for attacking?

  2. About the improvement in transferability. The authors claim that using different image regions and features from all layers can reduce the dependence on the surrogate model. However, these features still come from the same vision encoder. Could the authors further explain why this can improve transferability to other models?

  3. About the computational cost. SimAttack needs to calculate patch features from all encoder layers in each iteration. Could the authors provide a comparison of runtime or computational cost with the baselines? Does SimAttack still perform better when using a similar computational budget?

  4. Typo. Page 1, Line 39: "esstential" should be corrected to “essential".

1 comment
AuthorsOpenReview ↗

Response to Reviewer rWxH

Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).


Q1: Thank you for your question. We acknowledge that zero initialization makes the clean and adversarial features identical at the first iteration, resulting in a zero gradient in exact arithmetic and preventing the first perturbation update. However, small nonzero gradients arising from finite-precision computation can trigger the first sign-based perturbation update.

To assess the impact of perturbation initialization, we compared zero, gaussian, and uniform random initialization under the same settings as ablation experiments. Results are reported as SRR (%) in Qwen/Phi order. As shown in the table below, no initialization consistently performs best across all settings, indicating that SimAttack is insensitive to the choice of perturbation initialization.

Init.COCOVQAv2POPEAvgSRR
Zero63.48/65.0336.14/40.4643.13/45.5448.97
Gaussian63.09/65.1735.17/41.1744.05/46.8849.26
Uniform61.52/64.9234.81/40.8444.38/48.0649.09

We adopt uniform random initialization to avoid relying on reliance residuals, consistent with VEAttack and PA-Attack. We updated the algorithm and implementation, rerun all experiments, and revised the results accordingly.


W1 & Q2: Thank you for your comment. Although SimAttack uses a single CLIP vision encoder, different local views and encoder layers provide complementary visual information across diverse spatial regions and representation levels. Incorporating these features into adversarial optimization reduces reliance on specific surrogate representations, potentially improving transferability to unseen LVLMs.

Specifically, CPLRS resamples local regions across iterations to diversify the visual information used for optimization, while preserving spatial correspondence between clean and adversarial images within each iteration to focus on perturbation-induced feature changes. ALPFD uniformly incorporates patch features from all encoder layers, avoiding reliance on particular representation depths whose effectiveness may vary across victim models and downstream tasks. The ablation results in Tables 3 and 4 demonstrate the effectiveness of CPLRS and ALPFD, both individually and in combination.


W2 & Q3: Thank you for your valuable suggestion. In SimAttack, patch features from all encoder layers are obtained through a single forward pass per image, without requiring separate passes for each layer. The additional overhead mainly comes from intermediate feature storage, layer-wise similarity computation, and gradient backpropagation. To quantify the computational cost and evaluate attack effectiveness under comparable budgets, we provide two additional analyses below.

For computational cost, we compare generation time, GPU memory, and attack effectiveness on 1,000 COCO 2014 images, generating adversarial examples on a single NVIDIA RTX 4090 GPU and selecting Qwen2.5-VL-7B as victim LVLM. Captioning results are reported as raw scores, with lower values indicating stronger attacks. As shown in the table below, SimAttack requires 50.49 minutes and 2.29 GB of GPU memory, consuming less time and memory than MABA and MPCAttack. It outperforms VEAttack, PA-Attack, and BSA with similar memory usage at 1.8–1.9 times their runtime.

MethodTime (min)GPU Memory (GB)CIDErSPICECLIPScoreRefCLIPScore
VEAttack†26.522.2220.5612.7576.33107.80
PA-Attack†27.992.2420.4912.7376.19107.61
MABA215.6014.4820.8812.4675.47106.77
BSA†27.032.3914.969.5567.5496.22
MPCAttack121.9811.347.885.8757.4882.65
SimAttack50.492.293.413.4445.7167.22

For computational budget, we compare SimAttack (30 iterations) with VEAttack and BSA (50/70 iterations) on 1,000 COCO 2014 images, using Qwen2.5-VL-7B and Phi-4-Multimodal as victim LVLMs. Captioning results are reported as raw scores in Qwen/Phi order, with lower values indicating stronger attacks. As shown in the table below, SimAttack achieves better results across all captioning metrics on both victim models, even when the baselines use more computation time (62.38-63.46 minutes). These results indicate that SimAttack's performance advantage primarily stems from its method design rather than a larger computational budget.

MethodStepsTime (min)CIDErSPICECLIPScoreRefCLIPScore
VEAttack†5042.3720.52/43.4712.82/14.8375.67/72.04106.94/104.21
VEAttack†7063.4620.04/43.2312.46/14.7875.52/72.63106.73/104.89
BSA†5044.3915.55/33.399.75/11.8168.55/66.8497.60/97.13
BSA†7062.3816.56/36.099.91/12.3969.03/68.2098.25/99.07
SimAttack3049.013.42/4.293.44/2.9445.72/44.7967.25/68.07

Answer for Q4: Thank you for pointing out the typo and we have corrected it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.