acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Transferable Targeted Attacks on Multimodal Large Language Models via Causal Intervention

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in vision-language understanding, yet their robustness against targeted adversarial manipulation remains largely unexplored. Existing approaches on transferable targeted attacks mainly optimize target-related feature representations but overlook which visual components truly drive the transfer toward the target semantics. To address this limitation, we introduce a causal-guided framework that leverages target-aware counterfactual intervention to uncover the visual factors contributing to adversarial transfer. To achieve efficient estimation, we develop a gradient-based approximation strategy that evaluates the causal influence of visual components. The identified target-relevant components are then incorporated into a causal-guided feature optimization objective to improve alignment with the desired semantic representation. Extensive experiments across various MLLMs demonstrate that our approach consistently enhances targeted transferability over existing attack methods, validating the effectiveness of causal guidance for generating transferable adversarial examples.

Primary area:
interpretability and explainable AI
Keywords:
Large Vision-Language Models, Transferable Targeted Attacks, Counterfactual Causality

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7YkEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Transferable targeted adversarial attacks aim to generate adversarial examples that can mislead unseen black-box models while consistently driving their outputs toward predefined target responses. Existing targeted transfer attacks on MLLMs mainly improve transferability by aligning the feature representations of adversarial and target images. However, such global or local feature alignment may overlook the different contributions of visual tokens to target transfer, thereby limiting targeted transferability. This paper considers counterfactual interventions that replace the visual token representations of an adversarial image at the l-th layer with the corresponding representations from the target image, and measures their contributions based on the resulting changes in the similarity between the final CLS representation and the target representation. The authors further derive a gradient-based approximation method to efficiently estimate the contributions of visual tokens. An additional optimization objective is then introduced for visual tokens with high contributions to target alignment, thereby improving targeted transferability across unseen black-box MLLMs.

Critical strengths
  1. The paper introduces the idea of counterfactual causal intervention into transferable targeted adversarial attacks on MLLMs and proposes a token importance-based feature optimization method for targeted adversarial attacks.

  2. The paper is well-written and well-organized.

  3. Without controlling for computational overhead, the proposed method achieves the highest ASR and AvgSim compared with the baselines on the target-image semantic transfer task, demonstrating its effectiveness.

Critical weaknesses
  1. Overstated novelty claims. The causal intervention and gradient-based approximation proposed in this paper have been extensively studied in prior work. For example, ''Attribution Patching Outperforms Automated Circuit Discovery'' uses a first-order linear approximation to estimate the effect of replacing internal model activations and identifies model circuits based on the estimated importance. "AtP^{*}: An Efficient and Scalable Method for Localizing LLM Behaviour to Components" employs the same first-order inner-product formulation as the gradient-based approximation in this paper:
\left. \frac{\partial\mathcal{L}(\mathcal{M}(x^{clean}))}{\partial n} \right|_{n=n(x^{clean})}$$ Similar work also exists for ViTs, such as ''Causal Attribution via Activation Patching''. The main contribution of this paper lies in introducing this idea into transferable targeted attacks on MLLMs. However, the claimed contribution of developing a gradient-based approximation to efficiently estimate visual token contributions should not be presented as a major methodological novelty. 2. **Lack of validation of the effectiveness of the gradient-based approximation.** The paper claims that the proposed method identifies visual tokens with genuine causal contributions through counterfactual interventions. However, the accuracy of the first-order gradient approximation depends on whether the displacement toward the target representation is sufficiently small and whether the remaining Transformer layers exhibit sufficiently low nonlinearity along the intervention direction. Neither condition is necessarily satisfied in targeted adversarial attacks. In particular, the source and target images often belong to completely different semantic categories, potentially resulting in large target feature displacements. The authors should at least provide experiments to demonstrate the accuracy of the approximation and the conditions under which it holds, such as comparing the consistency between the first-order approximated contributions and the actual contributions obtained through token replacement. 3. **Lack of computational overhead analysis.** The final attack objective combines the basic target attack loss with an additional token importance-based loss. Beyond the computational cost of the basic attack, the proposed method introduces additional overhead for estimating token contributions. In particular, according to Table 4: $$R=1, ASR = 0.83, AvgSim = 0.645$$ and the main experimental results on GPT-5 in Table 2: $$ASR = 0.83, AvgSim = 0.645.$$ Based on these results, it can be inferred that the main experiments use $R=1$, meaning that token contributions need to be re-estimated at every optimization iteration. The authors should report the wall-clock time overhead and provide performance comparisons under the same computational budget.
Questions for authors

Please see the weaknesses.

9 comments
AuthorsOpenReview ↗

Response to Reviewer 7YkE [1/3]

Thank you for your thoughtful comments. We are glad that you found our manuscript clear and well-organized, and we appreciate your positive feedback.

W1: Overstated novelty claims. The causal intervention and gradient-based approximation proposed in this paper have been extensively studied in prior work. For example, "Attribution Patching Outperforms Automated Circuit Discovery" uses a first-order linear approximation to estimate the effect of replacing internal model activations and identifies model circuits based on the estimated importance. "AtP$^{*}$: An Efficient and Scalable Method for Localizing LLM Behaviour to Components" employs the same first-order inner-product formulation as the gradient-based approximation in this paper:

∂L(M(xclean))∂n∣n=n(xclean)\left. \frac{\partial \mathcal{L}(\mathcal{M}(x^{\mathrm{clean}}))}{\partial n} \right|_{n=n(x^{\mathrm{clean}})}

Similar work also exists for ViTs, such as "Causal Attribution via Activation Patching". The main contribution of this paper lies in introducing this idea into transferable targeted attacks on MLLMs. However, the claimed contribution of developing a gradient-based approximation to efficiently estimate visual token contributions should not be presented as a major methodological novelty.

Response: We thank the reviewer for valuable feedback. We would like to emphasize that our work addresses a different research objective from existing causal attribution studies. While prior works primarily focus on identifying influential model components to interpret model behavior, our work leverages causal attribution to guide transferable targeted attacks on MLLMs. Specifically, we use causal scores to dynamically prioritize visual tokens during adversarial optimization, aiming to improve cross-model transferability. Although we adopt a similar first-order approximation, our key contribution lies in transforming causal attribution from an interpretability tool into an optimization mechanism for transferable adversarial attacks. To further evaluate the effectiveness of our causality-guided optimization, we compare our method against three alternative token selection strategies based on EAP (Attribution Patching Outperforms Automated Circuit Discovery), CAAP (Causal Attribution via Activation Patching), and AtP. As shown in Table 11, our method achieves superior attack transferability, demonstrating the effectiveness of causality-guided visual token selection for adversarial optimization.

Table 11. Comparison of different visual token selection strategies for transferable targeted attacks.

MetricEAPAtPCAAPOurs
ASR ↑\uparrow0.770.770.780.83
AvgSim ↑\uparrow0.6160.6110.6090.645
AuthorsOpenReview ↗

Response to Reviewer 7YkE [2/3]

W2: Lack of validation of the effectiveness of the gradient-based approximation. The paper claims that the proposed method identifies visual tokens with genuine causal contributions through counterfactual interventions. However, the accuracy of the first-order gradient approximation depends on whether the displacement toward the target representation is sufficiently small and whether the remaining Transformer layers exhibit sufficiently low nonlinearity along the intervention direction. Neither condition is necessarily satisfied in targeted adversarial attacks. In particular, the source and target images often belong to completely different semantic categories, potentially resulting in large target feature displacements. The authors should at least provide experiments to demonstrate the accuracy of the approximation and the conditions under which it holds, such as comparing the consistency between the first-order approximated contributions and the actual contributions obtained through token replacement.

Response: We thank the reviewer for this valuable comment. To validate the effectiveness of the first-order approximation, we conducted additional experiments comparing it with exact counterfactual intervention. As shown in Table 12, we evaluate both attack performance (ASR and AvgSim) and the consistency of token contributions (Top-KK overlap). The results demonstrate substantial agreement between the approximated and exact contributions, with a Top-KK overlap of 62.77%, while achieving comparable attack performance (0.83 vs. 0.85 ASR and 0.645 vs. 0.671 AvgSim). Although the approximation does not perfectly reproduce the exact token rankings, it preserves a considerable proportion of the influential tokens identified by exact intervention. More importantly, the small differences in attack performance indicate that these discrepancies do not substantially affect the effectiveness of the resulting adversarial examples. These results support the validity of the first-order approximation.

Table 12. Performance of exact counterfactual intervention and its first-order Taylor approximation.

MethodASR ↑\uparrowAvgSim ↑\uparrowTop-KK Overlap (%) ↑\uparrow
Exact Intervention0.850.671—
First-order Approximation0.830.64562.77
AuthorsOpenReview ↗

Response to Reviewer 7YkE [3/3]

W3: Lack of computational overhead analysis. The final attack objective combines the basic target attack loss with an additional token importance-based loss. Beyond the computational cost of the basic attack, the proposed method introduces additional overhead for estimating token contributions. In particular, according to Table 4:

and the main experimental results on GPT-5 in Table 2:

Based on these results, it can be inferred that the main experiments use R=1R = 1, meaning that token contributions need to be re-estimated at every optimization iteration. The authors should report the wall-clock time overhead and provide performance comparisons under the same computational budget.

Response: Thank you for your valuable feedback. Following the reviewer's suggestion, we conduct an additional computational efficiency analysis, reporting average wall-clock attack time per image, and memory consumption. The results are presented in Table 13. Under the same experimental settings and optimization budget, our method reduces the average attack time from 177.26 to 173.91 s/image and memory consumption from 6.71 to 6.15 GB compared with FOA-Attack. This efficiency improvement is primarily attributed to our causal token identification strategy, which directly identifies influential visual tokens without requiring the additional Patch Token clustering used in FOA-Attack. Meanwhile, our method achieves higher ASR (0.83 vs. 0.72) and AvgSim (0.645 vs. 0.583). These results demonstrate that our method achieves better attack performance with lower overall computational and memory costs, despite re-estimating token contributions at every iteration.

Table 13. Computational efficiency comparison between FOA-Attack and our method.

MethodTime (s/img) ↓\downarrowMemory (GB) ↓\downarrowASR ↑\uparrowAvgSim ↑\uparrow
FOA-Attack177.266.710.720.583
Ours173.916.150.830.645
Reviewer_Cd7MOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a causal-guided framework for obtaining transferable targeted attacks against MLLMs. This framework includes a gradient-based efficient causal approximation to identify visual tokens most likely to contribute to transferability, and a causal-guided optimisation objective that guides high-contribution visual tokens towards the target representation. Results across both closed and open source models show that adversarial images obtained from the proposed framework achieve better attack performance compared to prior methods. The ablation studies further provide insights into the contribution of both the token selection method and the proposed loss function.

Critical strengths

I think the proposed method is intuitive with a clear motivation, with the ablation studies explaining why each component is necessary. The method is also shown to be highly effective, outperforming recent baselines across 8 target models and has an overall high attack success rate.

Critical weaknesses
  1. Tables 3-5 in the ablation studies are obtained from only one target model. Since some of the ablated results seem relatively close to the full method, including results from more target models would make it more convincing that each proposed component is necessary to improve the transferability of adversarial attacks.

  2. Table 4 shows that the optimal update interval for token selection is at every single iteration (R=1), which suggests that the visual tokens with the highest causal scores can vary at each iteration. This weakens one of the central claims that the method uncovers visual factors contributing to adversarial transfer, and indicates that the causal tokens identified are likely dependent on the current stage of the optimisation process rather than purely on visual features of the clean and target images. It would be helpful to report how and how much the top visual tokens change over the entire optimisation process.

Questions for authors
  1. How are the causal tokens selected in Figure 2a, since they could change at every iteration (R=1)?

  2. Is my understanding correct that the token selection only affects the causal loss, which is simply added to a base loss after scaling by lambda? If that is the case, can the proposed method in theory be combined with any other attack method as an additional loss term, to improve transferability?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Cd7M [1/4]

We thank the reviewer for the positive assessment and appreciate the recognition of our method's motivation, component-wise contributions, and effectiveness across eight target models.

W1: Tables 3-5 in the ablation studies are obtained from only one target model. Since some of the ablated results seem relatively close to the full method, including results from more target models would make it more convincing that each proposed component is necessary to improve the transferability of adversarial attacks.

Response: We thank the reviewer for this constructive suggestion. Following your suggestions, we have extended the ablation experiments to additional target models, including GPT-5.6-Terra, Claude Sonnet 5, and Gemini-2.5-Flash, with results shown in Tables 6, 7, and 8. These results further confirm the consistent contributions of individual components across different black-box MLLMs, supporting the robustness of the proposed design. In particular, the additional evaluations provide further evidence of the effectiveness of token intervention, the importance of an appropriate update interval, and the stability of the method under different loss-weight configurations. Overall, the consistent trends across these models suggest that the improvements are not limited to a specific target architecture, but generalize across diverse black-box MLLMs.

Table 6. The performance of different token selection strategies on closed-source MLLMs. Values are reported as ASR / AvgSim.

StrategyGPT-5GPT-5.6-TerraClaude Sonnet 5Gemini-2.5-Flash
Random0.74 / 0.6180.21 / 0.2610.63 / 0.5220.56 / 0.467
Attention0.68 / 0.5560.15 / 0.1990.58 / 0.4840.51 / 0.436
Activation0.71 / 0.5820.19 / 0.2200.61 / 0.4950.53 / 0.444
Gradient Norm0.73 / 0.6210.21 / 0.2290.64 / 0.4810.57 / 0.450
Ours0.83 / 0.6450.26 / 0.3270.69 / 0.5560.61 / 0.488

Table 7. Ablation study of different causal token update strategies on closed-source MLLMs. Values are reported as ASR / AvgSim.

StrategyGPT-5GPT-5.6-TerraClaude Sonnet 5Gemini-2.5-Flash
Static Selection0.71 / 0.5710.19 / 0.2210.59 / 0.4840.46 / 0.429
R=10R = 100.76 / 0.6100.22 / 0.2620.63 / 0.5210.54 / 0.461
R=5R = 50.79 / 0.6200.24 / 0.2590.67 / 0.5280.54 / 0.452
R=1R = 10.83 / 0.6450.26 / 0.3270.69 / 0.5560.61 / 0.488

Table 8. Ablation study of the causal alignment weight λ\lambda on closed-source MLLMs. Values are reported as ASR / AvgSim.

λ\lambdaGPT-5GPT-5.6-TerraClaude Sonnet 5Gemini-2.5-Flash
00.65 / 0.5250.16 / 0.1960.52 / 0.4330.47 / 0.415
10.76 / 0.6000.22 / 0.2540.63 / 0.5220.53 / 0.457
50.83 / 0.6450.26 / 0.3270.69 / 0.5560.61 / 0.488
100.75 / 0.5910.24 / 0.2280.62 / 0.4880.52 / 0.456
AuthorsOpenReview ↗

Response to Reviewer Cd7M [2/4]

W2: Table 4 shows that the optimal update interval for token selection is at every single iteration (R=1R = 1), which suggests that the visual tokens with the highest causal scores can vary at each iteration. This weakens one of the central claims that the method uncovers visual factors contributing to adversarial transfer, and indicates that the causal tokens identified are likely dependent on the current stage of the optimisation process rather than purely on visual features of the clean and target images. It would be helpful to report how and how much the top visual tokens change over the entire optimisation process.

Response: We thank the reviewer for this insightful comment. Our method aims to identify target-oriented causal tokens conditioned on the current adversarial example, rather than static visual factors determined solely by the clean and target images. In addition, we present the results of selected causal tokens using Top-KK overlap across different iteration intervals, as reported in Table 9. The overlap reaches 77.82% between consecutive iterations and gradually decreases to 58.95% when the interval increases to 50 iterations. These results reveal that causal token selection exhibits both temporal consistency and adaptation to the optimization process. Specifically, a substantial proportion of selected tokens remains consistent between adjacent iterations, while the token composition changes over longer optimization intervals. This supports our use of per-iteration token updates, which allows the attack to adapt to evolving causal contributions.

Table 9. Temporal consistency of selected causal tokens across different optimization intervals.

Iteration Gap (Δ\Delta)15102050
Top-KK Token Overlap (%)77.8273.9470.9167.5458.95
AuthorsOpenReview ↗

Response to Reviewer Cd7M [3/4]

Q1: How are the causal tokens selected in Figure 2a, since they could change at every iteration (R=1R = 1)?

Response: The causal tokens visualized in Figure 2a are selected at the initial iteration (t=0t=0). The primary purpose of this visualization is to demonstrate the target-oriented nature of our causal token selection. Specifically, even for the same clean image, different target images lead to distinct selections of causal tokens, indicating that the identified tokens are guided by the target semantics rather than solely determined by the visual content of the clean image. While the selected tokens may evolve during subsequent optimization iterations, Figure 2a focuses on illustrating their target-dependent characteristics.

Reviewer_FSvGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Transferable Targeted Adversarial Attacks in Multimodal Large Language Models, MLLMs. The authors quantify token contributions to target similarity by conducting intermediate-layer replacement interventions, approximately effect first-order gradients, and dynamically select high-contribution tokens for local alignment and global alignment among multiple surrogate models. Although the main method is quite inspiring and shows valuable experimental improvement, its causal interpretation, the effectiveness of the first-order approximation, and the test-target-independent tuning have not been sufficiently validated. The main method is quite inspiring and shows very attractive experimental improvement, but its causal interpretation, the effectiveness of the first-order approximation, and the test-target-independent tuning have not been sufficiently validated.

Critical strengths
  1. Targeted black-box transfer attacks are more challenging than ordinary untargeted attacks; thus, this paper investigates the practical impact of visual information on the semantics output by the target.
  2. The cross-model performance gains observed in the experiments were substantial. This paper reports attack success rates higher than the baseline for several open-source and proprietary MLLMs, with further comparisons based on 1,000 samples provided in the appendix.
Critical weaknesses
  1. The paper adopts the first-order Taylor approximation to estimate the effects of substituting visual word elements. Since there may be feature differences between the source and target images, the accuracy of such an approximation approach is unknown. The authors verify the approximation method's computational efficiency but do not analyze the consistency between the approximation results and the actual intervention effect.
  2. The main experiments in the paper were performed on only 100 samples, and the paper did not report confidence intervals, statistical significance tests, or experimental results for different pairs of random images. Although the appendix includes experiments on 1,000 samples, it is compared with only one baseline method, which is insufficient to demonstrate the method's consistent superiority over other baselines fully.
  3. The paper does not explicitly specify some key experimental settings, such as the attack optimizer, learning rate, number of optimization iterations, perturbation update and constraint methods, random seed, dynamic weight temperature parameter, and generation settings for closed-source models, which will affect the reproducibility of the experimental results.
Questions for authors
  1. We propose to measure the effect of token substitutions by looking at the gradient estimates; however, this approximation is never checked against ground truth substitution. We thus recommend verifying how the actual substitution of tokens compares to what gradient-based estimates predict.
  2. First-order approximations can deviate significantly because of significant image feature differences between the source and target images. This section should discuss the rationale for the validity of first-order approximation given the impact of such feature differences and the effect of the position in the network layer on the approximation error.
  3. Some experimental details are missing; therefore, we strongly recommend that the authors elaborate on those details when discussing the experiments so the results reported herein can be replicated without ambiguity.
5 comments
AuthorsOpenReview ↗

Response to Reviewer FSVG [1/4]

Thank you for your valuable feedback. We sincerely thank the reviewer for the encouraging feedback and for recognizing the potential of our proposed method and the promising results demonstrated in our experiments.

Q1: The paper adopts the first-order Taylor approximation to estimate the effects of substituting visual word elements. Since there may be feature differences between the source and target images, the accuracy of such an approximation approach is unknown. The authors verify the approximation method's computational efficiency but do not analyze the consistency between the approximation results and the actual intervention effect.

W1: We propose to measure the effect of token substitutions by looking at the gradient estimates; however, this approximation is never checked against ground truth substitution. We thus recommend verifying how the actual substitution of tokens compares to what gradient-based estimates predict.

Response: Thanks for your insightful comments. We further evaluate the consistency between the first-order Taylor approximation and exact counterfactual interventions, where each visual token is individually replaced by its corresponding target token. We measure token-level consistency using Top-KK overlap, with KK set to 25% of all visual tokens following our default setting, and also compare the resulting attack performance. As shown in Table 1, the first-order approximation achieves a Top-KK overlap of 62.77%, indicating reasonable consistency with exact intervention in identifying influential tokens, while maintaining comparable attack performance. Combined with the computational efficiency results reported in the manuscript (see Appendix A.3), the first-order approximation avoids costly per-token forward interventions and provides a practical alternative to exact counterfactual intervention.

Table 1. Performance of exact counterfactual intervention and its first-order Taylor approximation.

MethodASR ↑\uparrowAvgSim ↑\uparrowTop-KK Overlap (%) ↑\uparrow
Exact Intervention0.850.671—
First-order Approximation0.830.64562.77
AuthorsOpenReview ↗

Response to Reviewer FSVG [2/4]

W2: The main experiments in the paper were performed on only 100 samples, and the paper did not report confidence intervals, statistical significance tests, or experimental results for different pairs of random images. Although the appendix includes experiments on 1,000 samples, it is compared with only one baseline method, which is insufficient to demonstrate the method's consistent superiority over other baselines fully.

Response: We thank the reviewer for this valuable comment. We have conducted additional experiments and statistical analyses as follows.

  • Robustness across random image pairs. We evaluate our method on four groups of randomly paired source–target images, each containing 200 pairs (800 pairs in total). We assess attack performance using the attack success rate (ASR) and average semantic similarity (AvgSim), where higher values indicate better performance. To evaluate statistical significance, we employ two-sided exact McNemar tests for ASR and two-sided paired t-tests for AvgSim. As reported in Table 2, our method consistently outperforms FOA-Attack across all four groups in both metrics. Overall, our method improves ASR from 0.680 to 0.765 and AvgSim from 0.5524 to 0.6063, with both improvements being statistically significant (p=1.04×10−6p = 1.04 \times 10^{-6} and p=3.31×10−10p = 3.31 \times 10^{-10}, respectively). These results demonstrate the consistent performance gains of our method across different random source–target pairings.
  • Large-scale baseline comparison. We extend the original 1,000-image experiment to include multiple competing baselines. As shown in Table 3, our method achieves the best performance across all four target MLLMs, further demonstrating its effectiveness on a larger evaluation set.

Table 2. Transfer attack performance across four groups of 200 randomly sampled source-target pairs. Values are listed in ASR / AvgSim order.

GroupnnFOA-AttackOursPaired differences [95% CI]Paired pp-values
Group 12000.690 / 0.55550.760 / 0.6150+0.0700+0.0700 [0.0050, 0.1350] / +0.0595+0.0595 [0.0252, 0.0938]4.88×10−24.88 \times 10^{-2} / 7.51×10−47.51 \times 10^{-4}
Group 22000.650 / 0.53650.755 / 0.5975+0.1050+0.1050 [0.0350, 0.1750] / +0.0610+0.0610 [0.0291, 0.0929]5.49×10−35.49 \times 10^{-3} / 2.12×10−42.12 \times 10^{-4}
Group 32000.690 / 0.55600.775 / 0.6055+0.0850+0.0850 [0.0200, 0.1550] / +0.0495+0.0495 [0.0172, 0.0818]2.13×10−22.13 \times 10^{-2} / 2.84×10−32.84 \times 10^{-3}
Group 42000.690 / 0.56150.770 / 0.6070+0.0800+0.0800 [0.0150, 0.1450] / +0.0455+0.0455 [0.0103, 0.0807]2.59×10−22.59 \times 10^{-2} / 1.15×10−21.15 \times 10^{-2}
Overall8000.680 / 0.55240.765 / 0.6063+0.0850+0.0850 [0.0513, 0.1188] / +0.0539+0.0539 [0.0373, 0.0705]1.04×10−61.04 \times 10^{-6} / 3.31×10−103.31 \times 10^{-10}

Table 3. Transferability of different attack methods on closed-source MLLMs across 1,000 test samples. Values are reported as ASR / AvgSim.

MethodGPT-5GPT-5.6-TerraClaude Sonnet 5Gemini-2.5-Flash
AdvDiffVLM0.092 / 0.13950.023 / 0.06160.092 / 0.19150.048 / 0.0825
SSA-CWA0.096 / 0.18650.036 / 0.14970.148 / 0.2240.043 / 0.1394
AnyAttack0.075 / 0.19460.078 / 0.19140.082 / 0.25660.042 / 0.1982
M-Attack0.651 / 0.52910.168 / 0.19510.508 / 0.45600.425 / 0.3878
FOA-Attack0.682 / 0.55260.179 / 0.21690.574 / 0.50580.488 / 0.4236
Ours0.757 / 0.60360.230 / 0.25090.633 / 0.53260.517 / 0.4545
AuthorsOpenReview ↗

Response to Reviewer FSVG [3/4]

W3: The paper does not explicitly specify some key experimental settings, such as the attack optimizer, learning rate, number of optimization iterations, perturbation update and constraint methods, random seed, dynamic weight temperature parameter, and generation settings for closed-source models, which will affect the reproducibility of the experimental results.

Q3: Some experimental details are missing; therefore, we strongly recommend that the authors elaborate on those details when discussing the experiments so the results reported herein can be replicated without ambiguity.

Response: We thank the reviewer for pointing out the missing implementation details. Following FOA-Attack, we use I-FGSM with a step size of 1/2551/255, 300 iterations, and an ℓ∞\ell_\infty perturbation budget of 16/25516/255. The dynamic ensemble weighting temperature is 1.01.0, and the random seed for adversarial example generation is 2023. For closed-source evaluation, we set a generation budget of 500 tokens per image description. Following FOA-Attack's evaluation protocol, we use GPT-4o to evaluate the semantic similarity between adversarial and target image descriptions, with the temperature set to 00, the API seed fixed at 42, and the maximum output length limited to 100100 tokens.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.