- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SimAttack generates transferable black-box attacks on vision-language models by disrupting spatially aligned patch features across all layers of a surrogate vision encoder, without requiring ground-truth text annotations.
Critical strengths
The paper provides a pratical attack setting.
Critical weaknesses
- Limited novelty. The crop design, title, and overview figure closely resemble M-Attack. The layer patch objectives resembles VLAttack. The main change appears to be adapting targeted attacks to an untargeted setting, which offers insufficient methodological novelty for ICLR acceptance in my view.
- Narrow scope. The paper focuses solely on untargeted attacks, without demonstrating broader implications or applications beyond degrading task performance.
Questions for authors
Why transferable? The paper reports empirical improvements but does not clearly explain why the proposed design improves cross-model transferability or provide sufficient analysis of the underlying mechanism. Could the auther please give any explanation?
4 comments
Response to Reviewer yX9u
Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).
W1: Thank you for your comment. SimAttack is motivated by a key challenge in transferable black-box untargeted attacks without ground-truth textual annotations: deriving transferable optimization signals from surrogate vision encoders. To this end, SimAttack incorporates visual representations from diverse local regions and encoder layers to reduce reliance on a limited subset of surrogate representations. Specifically, unlike M-Attack's target feature alignment, CPLRS addresses the spatial correspondence required for clean–adversarial feature disruption. By applying identical crop-and-resize transformations to both images, CPLRS avoids feature differences caused by mismatched image content. ALPFD incorporates corresponding patch features across all encoder layers within these spatially aligned local views. The ablation results support these designs. CPLRS improves AvgSRR from 19.00% with independent cropping to 26.99% (Table 3), while combining CPLRS and ALPFD achieves 49.10% AvgSRR, outperforming either component alone (Table 2).
W2: Thank you for your valuable comment. Untargeted adversarial attacks are an important research direction for evaluating LVLM robustness, as demonstrated by recent studies [1][2][3][4]. While targeted attacks evaluate whether adversaries can steer LVLMs toward predefined target responses, untargeted attacks assess whether LVLMs can maintain semantically valid understanding of the original visual content under adversarial perturbations. These two settings therefore examine complementary aspects of adversarial robustness.
This setting also present two distinct challenges. First, without predefined target semantics, insufficient optimization guidance can cause perturbations to overfit surrogate-specific features or limited visual attributes, compromising transferability. Second, LVLMs have an open-ended output space in which multiple semantically valid responses may correspond to the same image and prompt, making it difficult to identify transferable optimization signals. Existing methods such as MABA and SAF address these challenges using ground-truth textual annotations. SimAttack further considers their absence and derives optimization signals solely from a pretrained surrogate vision encoder.
We evaluate SimAttack across five heterogeneous LVLMs, five datasets, and three downstream tasks: image captioning, visual question answering, and hallucination evaluation. The consistent performance degradation across these settings (Table 1) demonstrates its effectiveness in evaluating black-box robustness across diverse models and tasks.
Q1: Thank you for your valuable comment. SimAttack improves cross-model transferability by incorporating diverse visual representations across spatial local regions and encoder layers, reducing reliance on a limited subset of surrogate representations. Specifically, CPLRS resamples local regions across iterations to capture diverse visual information while preserving spatial correspondence between clean and adversarial images, ensuring that feature discrepancies primarily reflect adversarial perturbations. ALPFD uniformly incorporates patch features from all encoder layers, avoiding dependence on specific layers whose effectiveness may vary across victim models and downstream tasks.
The main experiments demonstrate that SimAttack consistently outperforms all baselines across all 25 dataset–LVLM combinations, achieving the highest Average SRR on each of the five victim LVLMs (Table 1). The ablation results further support this explanation. CPLRS improves AvgSRR from 19.00% with independent cropping to 26.99% (Table 3), while all-layer optimization achieves 22.34% AvgSRR compared with 12.17% using only the final layer (Table 4). Combining CPLRS and ALPFD further improves AvgSRR to 49.10% (Table 2). Together, these results support the effectiveness of exploiting spatially diverse and hierarchical visual representations for improving cross-model transferability.
Followup comment
Specifically, unlike M-Attack's target feature alignment, CPLRS addresses the spatial correspondence required for clean–adversarial feature disruption. By applying identical crop-and-resize transformations to both images, ...
The method is presented as a combination of A and B, yet the comparison contrasts your component A (M-Attack) with M-Attack’s component B, and vice versa. This does not clearly establish the novelty of either component. What needs to be clarified is how your crop-and-resize strategy differs from the corresponding strategy in M-Attack.
transferability by incorporating diverse visual representations across spatial local regions and encoder layers, reducing reliance on a limited subset of surrogate representations..
You just repeated your method design. But why the design work? The author still didn't justify the transferability beyond reporting the table numbers.
Again, i'm leaning towards score 1 (Clear rejection).
Response to Reviewer yX9u
Thank you for your thoughtful comments.
The method is presented as a combination of A and B, yet the comparison contrasts your component A (M-Attack) with M-Attack’s component B, and vice versa. This does not clearly establish the novelty of either component. What needs to be clarified is how your crop-and-resize strategy differs from the corresponding strategy in M-Attack.
While CPLRS in SimAttack employs crop-and-resize transformations, its novelty lies in the local-view pairing and optimization strategy. Unlike M-Attack, which aligns the features extracted from cropped adversarial images with those of either the global or independently cropped target image. CPLRS applies identical crop-and-resize transformations to clean and adversarial images, ensuring that corresponding features originate from the same spatial region. This avoids content-induced discrepancies unrelated to adversarial perturbations and enables local sampling to integrate with ALPFD's all-layer patch feature disruption.
You just repeated your method design. But why the design work? The author still didn't justify the transferability beyond reporting the table numbers.
Optimizing adversarial perturbations on a fixed full-image view or a selected subset of encoder layers may lead to excessive dependence on specific surrogate representations, limiting cross-model transferability and task generalization. To address this issue, SimAttack seeks to disrupt a broader range of visual information and representations. Specifically, CPLRS Specifically, CPLRS diversifies the visual content emphasized during optimization by sampling different local regions across iterations, while preserving spatial correspondence to ensure that feature discrepancies primarily reflect adversarial perturbations. ALPFD broadens the range of visual representations targeted for disruption by incorporating patch features from all encoder layers, reducing dependence on any particular layer or subset of layers. Together, CPLRS and ALPFD enable perturbation optimization across diverse local views and encoder layers, reducing reliance on specific surrogate representations and improving transferability across different LVLMs and downstream tasks.
Followup comment
I agree with Reviewer ZAGx and am leaning towards score 1 (Clear rejection).