- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Transferable targeted adversarial attacks aim to generate adversarial examples that can mislead unseen black-box models while consistently driving their outputs toward predefined target responses. Existing targeted transfer attacks on MLLMs mainly improve transferability by aligning the feature representations of adversarial and target images. However, such global or local feature alignment may overlook the different contributions of visual tokens to target transfer, thereby limiting targeted transferability. This paper considers counterfactual interventions that replace the visual token representations of an adversarial image at the l-th layer with the corresponding representations from the target image, and measures their contributions based on the resulting changes in the similarity between the final CLS representation and the target representation. The authors further derive a gradient-based approximation method to efficiently estimate the contributions of visual tokens. An additional optimization objective is then introduced for visual tokens with high contributions to target alignment, thereby improving targeted transferability across unseen black-box MLLMs.
Critical strengths
-
The paper introduces the idea of counterfactual causal intervention into transferable targeted adversarial attacks on MLLMs and proposes a token importance-based feature optimization method for targeted adversarial attacks.
-
The paper is well-written and well-organized.
-
Without controlling for computational overhead, the proposed method achieves the highest ASR and AvgSim compared with the baselines on the target-image semantic transfer task, demonstrating its effectiveness.
Critical weaknesses
- Overstated novelty claims. The causal intervention and gradient-based approximation proposed in this paper have been extensively studied in prior work. For example, ''Attribution Patching Outperforms Automated Circuit Discovery'' uses a first-order linear approximation to estimate the effect of replacing internal model activations and identifies model circuits based on the estimated importance. "AtP^{*}: An Efficient and Scalable Method for Localizing LLM Behaviour to Components" employs the same first-order inner-product formulation as the gradient-based approximation in this paper:
Questions for authors
Please see the weaknesses.
9 comments
Response to Reviewer 7YkE [2/3]
W2: Lack of validation of the effectiveness of the gradient-based approximation. The paper claims that the proposed method identifies visual tokens with genuine causal contributions through counterfactual interventions. However, the accuracy of the first-order gradient approximation depends on whether the displacement toward the target representation is sufficiently small and whether the remaining Transformer layers exhibit sufficiently low nonlinearity along the intervention direction. Neither condition is necessarily satisfied in targeted adversarial attacks. In particular, the source and target images often belong to completely different semantic categories, potentially resulting in large target feature displacements. The authors should at least provide experiments to demonstrate the accuracy of the approximation and the conditions under which it holds, such as comparing the consistency between the first-order approximated contributions and the actual contributions obtained through token replacement.
Response: We thank the reviewer for this valuable comment. To validate the effectiveness of the first-order approximation, we conducted additional experiments comparing it with exact counterfactual intervention. As shown in Table 12, we evaluate both attack performance (ASR and AvgSim) and the consistency of token contributions (Top- overlap). The results demonstrate substantial agreement between the approximated and exact contributions, with a Top- overlap of 62.77%, while achieving comparable attack performance (0.83 vs. 0.85 ASR and 0.645 vs. 0.671 AvgSim). Although the approximation does not perfectly reproduce the exact token rankings, it preserves a considerable proportion of the influential tokens identified by exact intervention. More importantly, the small differences in attack performance indicate that these discrepancies do not substantially affect the effectiveness of the resulting adversarial examples. These results support the validity of the first-order approximation.
Table 12. Performance of exact counterfactual intervention and its first-order Taylor approximation.
| Method | ASR | AvgSim | Top- Overlap (%) |
|---|---|---|---|
| Exact Intervention | 0.85 | 0.671 | — |
| First-order Approximation | 0.83 | 0.645 | 62.77 |
Response to Reviewer 7YkE [3/3]
W3: Lack of computational overhead analysis. The final attack objective combines the basic target attack loss with an additional token importance-based loss. Beyond the computational cost of the basic attack, the proposed method introduces additional overhead for estimating token contributions. In particular, according to Table 4:
and the main experimental results on GPT-5 in Table 2:
Based on these results, it can be inferred that the main experiments use , meaning that token contributions need to be re-estimated at every optimization iteration. The authors should report the wall-clock time overhead and provide performance comparisons under the same computational budget.
Response: Thank you for your valuable feedback. Following the reviewer's suggestion, we conduct an additional computational efficiency analysis, reporting average wall-clock attack time per image, and memory consumption. The results are presented in Table 13. Under the same experimental settings and optimization budget, our method reduces the average attack time from 177.26 to 173.91 s/image and memory consumption from 6.71 to 6.15 GB compared with FOA-Attack. This efficiency improvement is primarily attributed to our causal token identification strategy, which directly identifies influential visual tokens without requiring the additional Patch Token clustering used in FOA-Attack. Meanwhile, our method achieves higher ASR (0.83 vs. 0.72) and AvgSim (0.645 vs. 0.583). These results demonstrate that our method achieves better attack performance with lower overall computational and memory costs, despite re-estimating token contributions at every iteration.
Table 13. Computational efficiency comparison between FOA-Attack and our method.
| Method | Time (s/img) | Memory (GB) | ASR | AvgSim |
|---|---|---|---|---|
| FOA-Attack | 177.26 | 6.71 | 0.72 | 0.583 |
| Ours | 173.91 | 6.15 | 0.83 | 0.645 |
Response to Reviewer 7YkE [1/3]
Thank you for your thoughtful comments. We are glad that you found our manuscript clear and well-organized, and we appreciate your positive feedback.
Response: We thank the reviewer for valuable feedback. We would like to emphasize that our work addresses a different research objective from existing causal attribution studies. While prior works primarily focus on identifying influential model components to interpret model behavior, our work leverages causal attribution to guide transferable targeted attacks on MLLMs. Specifically, we use causal scores to dynamically prioritize visual tokens during adversarial optimization, aiming to improve cross-model transferability. Although we adopt a similar first-order approximation, our key contribution lies in transforming causal attribution from an interpretability tool into an optimization mechanism for transferable adversarial attacks. To further evaluate the effectiveness of our causality-guided optimization, we compare our method against three alternative token selection strategies based on EAP (Attribution Patching Outperforms Automated Circuit Discovery), CAAP (Causal Attribution via Activation Patching), and AtP. As shown in Table 11, our method achieves superior attack transferability, demonstrating the effectiveness of causality-guided visual token selection for adversarial optimization.
Table 11. Comparison of different visual token selection strategies for transferable targeted attacks.