- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates targeted semantic substitution in vision-language models (VLMs) under small pixel-level perturbation budgets and proposes Token-Level Representation Alignment (TokRA), a white-box attack that aligns the adversarial input's visual tokens with those of a target image at corresponding spatial positions across all post-merger visual streams, including the main stream and three DeepStack streams in Qwen3-VL. The authors argue that successful semantic substitution requires both injecting target semantics and suppressing source semantics, and introduce a strict attack success criterion and a four-state evaluation framework to distinguish different degrees of replacement. Experiments on Qwen3-VL and Qwen2.5-VL demonstrate that targeted semantic manipulation can succeed under small perturbation budgets. The paper also reports semantic fusion, where VLMs combine source and target semantics into coherent but misleading descriptions.
Critical strengths
- The proposed method is conceptually simple and incorporates architectural characteristics of modern VLMs by aligning visual tokens across multiple streams without relying on a specific text prompt.
- The experiments reveal interesting vulnerabilities under small perturbation budgets, while the video evaluation and semantic fusion observations provide useful insights into the behavior of adversarially manipulated VLMs.
Critical weaknesses
- The methodological novelty is limited, as targeted internal representation matching dates back to Sabour et al. [1], while more recent works such as FOA-Attack [2], IPGA [3], and V-Attack [4] have explored fine-grained feature alignment, attacks on vision-language interfaces, and targeted semantic manipulation, respectively. Moreover, CRAFT [5], which introduces token-level targeted object manipulation and evaluates cross-task consistency on the CrossVLAD benchmark, is particularly relevant but not discussed.
- The alignment-location ablation in Appendix F (Table 10) directly challenges the proposed design, as DeepStack-only alignment (50.11% strict ASR) and pre-merger alignment (47.12%) both outperform the proposed full-stream post-merger TokRA (38.17%), leaving the necessity and effectiveness of aligning all post-merger streams insufficiently justified.
- The paper lacks computational-cost analysis and comparisons under matched computational budgets, despite substantial differences among the methods in optimization algorithms, early stopping strategies, gradient computation costs, and video optimization procedures, making their computational efficiency and the fairness of the comparisons difficult to assess.
- Although the restricted white-box threat model is clearly stated, the attack is evaluated only under tightly controlled preprocessing and inference conditions, without testing robustness to common transformations such as JPEG compression, image resizing, video re-encoding, or FPS changes, which limits the practical applicability of the reported findings. Furthermore, the omission of adversarial images from the qualitative examples prevents direct assessment of their perceptual quality, as a small perturbation budget alone does not necessarily guarantee imperceptibility.
[1] Sara Sabour, Yanshuai Cao, Fartash Faghri, and David J. Fleet. Adversarial Manipulation of Deep Representations.
[2] Xiaojun Jia, Sensen Gao, Simeng Qin, Tianyu Pang, Chao Du, Yihao Huang, Xinfeng Li, Yiming Li, Bo Li, and Yang Liu. Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment.
[3] Yiming Cao, Yanjie Li, Kaisheng Liang, and Bin Xiao. Enhancing Targeted Adversarial Attacks on Large Vision-Language Models via Intermediate Projector.
[4] Sen Nie, Jie Zhang, Jianxin Yan, Shiguang Shan, and Xilin Chen. V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs.
[5] Jiale Zhao, Xinyang Jiang, Junyao Gao, Yuhao Xue, and Cairong Zhao. One Object, Multiple Lies: A Benchmark for Cross-task Adversarial Attack on Unified Vision-Language Models.
Questions for authors
- Can the authors provide computational-cost analysis, including runtime, GPU memory usage, and gradient evaluations, as well as comparisons under matched computational budgets?
- How does TokRA differ fundamentally from existing representation-alignment and targeted semantic attacks [1–5], and what specific methodological contributions distinguish it from these prior works?
- Why do DeepStack-only alignment and pre-merger alignment outperform the proposed full-stream post-merger TokRA in Table 10, and how do the authors justify their architectural choices in light of these results?
- How robust is TokRA to common input transformations such as JPEG compression, resizing, video re-encoding, and FPS changes, and can the authors provide adversarial examples to support their claims of perceptual imperceptibility?
2 comments
Thank you for your response. It addressed my questions, and I will maintain my score.
Thanks for the detailed comments. We address alignment position and post-processing robustness in the general response.
Novelty and related work
In terms of novelty, we agree that representation alignment as an adversarial objective was first introduced by Sabour et al. (2016) [1], and we have credited them as the earliest related work in the revised manuscript. We have also added discussion of FOA-Attack [2], IPGA [3], V-Attack [4], and CRAFT [5]. However, these prior methods operate on DNN classifiers or use global/layer-level feature alignment. TokRA instead targets VLMs with LLM backbones and aligns all visual tokens the backbone receives, across multiple streams. A key distinction is that TokRA uses the visual tokens of the target image directly, with no dependence on any prompt. The replacement effect therefore holds across different instructions, unlike output-space methods whose effect is tied to the optimization prompt (Figure 4). Since the target is a concrete image, every attribute in the output traces back to either the source or the target, which is what enables the semantic fusion analysis (Figure 1).
Here is how TokRA stands apart from the specific methods the reviewer lists. FOA-Attack [2] pairs patches across images by optimal transport over feature similarity, so a source patch may end up matched to a spatially distant target patch. TokRA enforces strict spatial correspondence, since the goal is to replace the scene as a whole. IPGA [3] aligns 32 cross-attention query outputs of a surrogate Q-Former with target text embeddings at ϵ=8/255. Since it operates through a surrogate, the perturbation must bridge a transferability gap on top of the alignment objective. TokRA aligns all visual tokens on the victim directly, so this overhead does not apply. V-Attack [4] manipulates value features in the vision encoder to shift a chosen local concept toward a target concept. In other words, it targets individual objects. TokRA operates on the full post-merger token sequence, so the replacement covers the semantics of the entire input.
Comparison with CRAFT
CRAFT [5] is the most closely related work, and we have added it to the related work in the revised manuscript. The main differences concern the alignment target, the perturbation budget, and the modality. CRAFT aligns against text semantics of the target category, so the manipulation is bounded by what a category label can encode. TokRA aligns against visual tokens of the target image, so the full visual content serves as the alignment objective. CRAFT reports results at ϵ=8/255, while TokRA achieves semantic replacement at ϵ≤4/255, half the perturbation budget. Besides, CRAFT evaluates on images only, whereas TokRA additionally covers video.
Computational cost
Regarding the computational cost, we have added GPU timing measurements on an NVIDIA GeForce RTX 5090 (batch size 1; Appendix L, Table 20). Since TokRA's loss is defined on the merger output, backpropagation only needs to pass through the vision encoder and the merger. In contrast, output-space PGD backpropagates through the entire 8B LLM, which is substantially more expensive per step.
With early stopping, output-space PGD takes about 2.0× the per-sample GPU time of TokRA. After removing early stopping and running all methods to 500 steps, output-space PGD's GPU time reaches 3.4× that of TokRA. Since the gradient never enters the LLM backbone, TokRA is the cheapest of the three.
We hope these clarifications and additions address the reviewer's concerns and would welcome reconsideration of the score.
References
[1] Sara Sabour, Yanshuai Cao, Fartash Faghri, and David J. Fleet. Adversarial Manipulation of Deep Representations.
[2] Xiaojun Jia, Sensen Gao, Simeng Qin, Tianyu Pang, Chao Du, Yihao Huang, Xinfeng Li, Yiming Li, Bo Li, and Yang Liu. Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment.
[3] Yiming Cao, Yanjie Li, Kaisheng Liang, and Bin Xiao. Enhancing Targeted Adversarial Attacks on Large Vision-Language Models via Intermediate Projector.
[4] Sen Nie, Jie Zhang, Jianxin Yan, Shiguang Shan, and Xilin Chen. V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs.
[5] Jiale Zhao, Xinyang Jiang, Junyao Gao, Yuhao Xue, and Cairong Zhao. One Object, Multiple Lies: A Benchmark for Cross-task Adversarial Attack on Unified Vision-Language Models.