acceptodds
Under review as a conference paper at ICLR 2027

How Little It Takes to Rewrite Perception: Intrinsic Vulnerability of Vision-Language Models to Targeted Semantic Manipulation

Abstract

Vision Language Models (VLMs) are widely deployed in safety-critical scenarios, and understanding to which extent they can be controlled by adversarial perturbation is a prerequisite for evaluating their trustworthiness. Existing representation-alignment attacks, which make a VLM perceive a target image, achieve limited success at . Therefore, VLMs seems robust to perturbations in this range. We show that this robustness does not hold, as targeted semantic substitution succeeds within the same range. Specifically, we align each stream of the source image with its counterpart in the target image in the victim VLM's post-merger token space, operating under a white-box threat model. We evaluate under a strict success criterion, requiring the model to simultaneously name the target, confirm its presence, and deny the source. In images, target semantics appear at and complete replacement reaches 38% at . On video, complete replacement reaches 35.9% at . We also observe a phenomenon of semantic fusion, where Large Language Model (LLM) rationalizes contradictory visual signals into a coherent narrative.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Adversarial attacks, Vision-Language Models, Targeted semantic replacement

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ikc2OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates targeted semantic substitution in vision-language models (VLMs) under small pixel-level perturbation budgets and proposes Token-Level Representation Alignment (TokRA), a white-box attack that aligns the adversarial input's visual tokens with those of a target image at corresponding spatial positions across all post-merger visual streams, including the main stream and three DeepStack streams in Qwen3-VL. The authors argue that successful semantic substitution requires both injecting target semantics and suppressing source semantics, and introduce a strict attack success criterion and a four-state evaluation framework to distinguish different degrees of replacement. Experiments on Qwen3-VL and Qwen2.5-VL demonstrate that targeted semantic manipulation can succeed under small perturbation budgets. The paper also reports semantic fusion, where VLMs combine source and target semantics into coherent but misleading descriptions.

Critical strengths
  1. The proposed method is conceptually simple and incorporates architectural characteristics of modern VLMs by aligning visual tokens across multiple streams without relying on a specific text prompt.
  2. The experiments reveal interesting vulnerabilities under small perturbation budgets, while the video evaluation and semantic fusion observations provide useful insights into the behavior of adversarially manipulated VLMs.
Critical weaknesses
  1. The methodological novelty is limited, as targeted internal representation matching dates back to Sabour et al. [1], while more recent works such as FOA-Attack [2], IPGA [3], and V-Attack [4] have explored fine-grained feature alignment, attacks on vision-language interfaces, and targeted semantic manipulation, respectively. Moreover, CRAFT [5], which introduces token-level targeted object manipulation and evaluates cross-task consistency on the CrossVLAD benchmark, is particularly relevant but not discussed.
  2. The alignment-location ablation in Appendix F (Table 10) directly challenges the proposed design, as DeepStack-only alignment (50.11% strict ASR) and pre-merger alignment (47.12%) both outperform the proposed full-stream post-merger TokRA (38.17%), leaving the necessity and effectiveness of aligning all post-merger streams insufficiently justified.
  3. The paper lacks computational-cost analysis and comparisons under matched computational budgets, despite substantial differences among the methods in optimization algorithms, early stopping strategies, gradient computation costs, and video optimization procedures, making their computational efficiency and the fairness of the comparisons difficult to assess.
  4. Although the restricted white-box threat model is clearly stated, the attack is evaluated only under tightly controlled preprocessing and inference conditions, without testing robustness to common transformations such as JPEG compression, image resizing, video re-encoding, or FPS changes, which limits the practical applicability of the reported findings. Furthermore, the omission of adversarial images from the qualitative examples prevents direct assessment of their perceptual quality, as a small perturbation budget alone does not necessarily guarantee imperceptibility.

[1] Sara Sabour, Yanshuai Cao, Fartash Faghri, and David J. Fleet. Adversarial Manipulation of Deep Representations.

[2] Xiaojun Jia, Sensen Gao, Simeng Qin, Tianyu Pang, Chao Du, Yihao Huang, Xinfeng Li, Yiming Li, Bo Li, and Yang Liu. Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment.

[3] Yiming Cao, Yanjie Li, Kaisheng Liang, and Bin Xiao. Enhancing Targeted Adversarial Attacks on Large Vision-Language Models via Intermediate Projector.

[4] Sen Nie, Jie Zhang, Jianxin Yan, Shiguang Shan, and Xilin Chen. V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs.

[5] Jiale Zhao, Xinyang Jiang, Junyao Gao, Yuhao Xue, and Cairong Zhao. One Object, Multiple Lies: A Benchmark for Cross-task Adversarial Attack on Unified Vision-Language Models.

Questions for authors
  1. Can the authors provide computational-cost analysis, including runtime, GPU memory usage, and gradient evaluations, as well as comparisons under matched computational budgets?
  2. How does TokRA differ fundamentally from existing representation-alignment and targeted semantic attacks [1–5], and what specific methodological contributions distinguish it from these prior works?
  3. Why do DeepStack-only alignment and pre-merger alignment outperform the proposed full-stream post-merger TokRA in Table 10, and how do the authors justify their architectural choices in light of these results?
  4. How robust is TokRA to common input transformations such as JPEG compression, resizing, video re-encoding, and FPS changes, and can the authors provide adversarial examples to support their claims of perceptual imperceptibility?
2 comments

Thanks for the detailed comments. We address alignment position and post-processing robustness in the general response.

Novelty and related work

In terms of novelty, we agree that representation alignment as an adversarial objective was first introduced by Sabour et al. (2016) [1], and we have credited them as the earliest related work in the revised manuscript. We have also added discussion of FOA-Attack [2], IPGA [3], V-Attack [4], and CRAFT [5]. However, these prior methods operate on DNN classifiers or use global/layer-level feature alignment. TokRA instead targets VLMs with LLM backbones and aligns all visual tokens the backbone receives, across multiple streams. A key distinction is that TokRA uses the visual tokens of the target image directly, with no dependence on any prompt. The replacement effect therefore holds across different instructions, unlike output-space methods whose effect is tied to the optimization prompt (Figure 4). Since the target is a concrete image, every attribute in the output traces back to either the source or the target, which is what enables the semantic fusion analysis (Figure 1).

Here is how TokRA stands apart from the specific methods the reviewer lists. FOA-Attack [2] pairs patches across images by optimal transport over feature similarity, so a source patch may end up matched to a spatially distant target patch. TokRA enforces strict spatial correspondence, since the goal is to replace the scene as a whole. IPGA [3] aligns 32 cross-attention query outputs of a surrogate Q-Former with target text embeddings at ϵ=8/255\epsilon = 8/255. Since it operates through a surrogate, the perturbation must bridge a transferability gap on top of the alignment objective. TokRA aligns all visual tokens on the victim directly, so this overhead does not apply. V-Attack [4] manipulates value features in the vision encoder to shift a chosen local concept toward a target concept. In other words, it targets individual objects. TokRA operates on the full post-merger token sequence, so the replacement covers the semantics of the entire input.

Comparison with CRAFT

CRAFT [5] is the most closely related work, and we have added it to the related work in the revised manuscript. The main differences concern the alignment target, the perturbation budget, and the modality. CRAFT aligns against text semantics of the target category, so the manipulation is bounded by what a category label can encode. TokRA aligns against visual tokens of the target image, so the full visual content serves as the alignment objective. CRAFT reports results at ϵ=8/255\epsilon = 8/255, while TokRA achieves semantic replacement at ϵ≤4/255\epsilon \le 4/255, half the perturbation budget. Besides, CRAFT evaluates on images only, whereas TokRA additionally covers video.

Computational cost

Regarding the computational cost, we have added GPU timing measurements on an NVIDIA GeForce RTX 5090 (batch size 1; Appendix L, Table 20). Since TokRA's loss is defined on the merger output, backpropagation only needs to pass through the vision encoder and the merger. In contrast, output-space PGD backpropagates through the entire 8B LLM, which is substantially more expensive per step.

Forward / BackwardPer-sample timeSteps
TokRA13.1 / 18.0 ms16.0 s500 (for video, 500 per frame pair)
Output-space PGD (early stopping)53.9 / 53.7 ms32.1 s294 (avg.)
Output-space PGD (no early stopping)53.8 / 53.7 ms54.4 s500
PA-Attack21.8 / 28.5 ms25.9 s500

With early stopping, output-space PGD takes about 2.0×2.0\times the per-sample GPU time of TokRA. After removing early stopping and running all methods to 500 steps, output-space PGD's GPU time reaches 3.4×3.4\times that of TokRA. Since the gradient never enters the LLM backbone, TokRA is the cheapest of the three.

We hope these clarifications and additions address the reviewer's concerns and would welcome reconsideration of the score.

References

[1] Sara Sabour, Yanshuai Cao, Fartash Faghri, and David J. Fleet. Adversarial Manipulation of Deep Representations.

[2] Xiaojun Jia, Sensen Gao, Simeng Qin, Tianyu Pang, Chao Du, Yihao Huang, Xinfeng Li, Yiming Li, Bo Li, and Yang Liu. Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment.

[3] Yiming Cao, Yanjie Li, Kaisheng Liang, and Bin Xiao. Enhancing Targeted Adversarial Attacks on Large Vision-Language Models via Intermediate Projector.

[4] Sen Nie, Jie Zhang, Jianxin Yan, Shiguang Shan, and Xilin Chen. V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs.

[5] Jiale Zhao, Xinyang Jiang, Junyao Gao, Yuhao Xue, and Cairong Zhao. One Object, Multiple Lies: A Benchmark for Cross-task Adversarial Attack on Unified Vision-Language Models.

Thank you for your response. It addressed my questions, and I will maintain my score.

Reviewer_VtKrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TokRA: under a white-box setting, it aligns visual tokens of the input image or video with tokens at corresponding positions in a target sample, attempting to simultaneously inject target semantics while suppressing original semantics. The authors evaluate on Qwen3-VL for both images and videos, with supplementary image experiments on Qwen2.5-VL. According to the paper's strict criteria, the complete replacement rate for images is 38.2% at ε=4/255; for videos it is 35.9% at ε=1/255. The authors also analyze the "semantic fusion" phenomenon where both target and original semantics appear simultaneously in the model's descriptions.

Critical strengths
  1. The paper points out that merely causing the model to output target text does not necessarily eliminate original content; TokRA attempts to address both aspects by aligning visual representations.
  2. The authors separately examine whether the target is recognized, whether the target's presence is confirmed, whether the original content is denied, and demonstrate the gap between target semantic injection and complete replacement.
  3. Comparisons are made with output-space baselines and an adapted version targeting PA-Attack, with partial results validated on another VLM.
  4. Threat model clearly articulated. The paper explicitly limits scope to white-box attacks and states that cross-model transferability and robustness after compression/re-encoding were not tested, avoiding packaging results as deployable attacks.
Critical weaknesses

I have the following concerns:

  1. Main results concentrate on Qwen3-VL; video experiments only test this single model. The attack also relies on model-internal gradients, specific preprocessing, and sampling configurations, leaving unclear whether findings hold for other models or under common post-processing.
  2. Results depend on a single LLM judge with no reported human annotation validation; strict success rates and partial replacement states may be influenced by the judging model's assessments.
  3. The variant aligning DeepStack streams achieves higher strict success rate on images than the paper's main method, yet the authors do not adequately explain why merging all streams degrades performance or justify the primary method choice.
  4. The paper primarily describes perturbations via pixel-level ε constraints without systematic perceptual quality evaluation; thus "low-budget" results do not equate to proven imperceptibility to human observers.
Questions for authors

see weakness

2 comments

Thanks for the positive evaluation and the constructive suggestions on model scope. We address evaluation reliability, alignment position, and post-processing robustness in the general response.

Model scope

Regarding the concern about single-model evaluation, Table 3 already provides image results on Qwen2.5-VL, which uses a different visual encoder and has no DeepStack streams. At ϵ=4/255\epsilon = 4/255, strict ASR is 33.0% and Q3 pass rate is 79.5%, consistent with Qwen3-VL. Following the suggestion, we have additionally run video experiments on Qwen2.5-VL (Appendix I, Table 13). At ϵ=1/255\epsilon = 1/255 and 2/2552/255, TokRA achieves 14.8% and 28.9% strict ASR, with Q3 pass rates of 52.3% and 76.6%, respectively. Since the two models differ substantially in visual encoder and architecture, the consistent trends indicate that TokRA's effectiveness is not tied to Qwen3-VL specifically.

We hope these clarifications address the reviewer's concerns.

The authors have addressed the question I was most concerned about, and I also found the relevant description of the experiment in the paper. Therefore, I plan to maintain my original positive score.

Reviewer_2f2vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TokRA, a white-box attack that aligns source and target visual representations position by position across the main and DeepStack post-merger streams. It evaluates targeted semantic substitution using questions testing target identification, target presence, and source denial. Experiments use 469 image pairs and 128 video pairs, primarily on Qwen3-VL, with additional image results on Qwen2.5-VL. Reported strict attack success reaches 38.2% for images at ε = 4/255 and 35.9% for videos at ε = 1/255. The paper also describes outputs combining source and target semantics.

Critical strengths
  1. The low-budget results provide useful robustness evidence. Evaluating quantized 8-bit images and losslessly stored video frames strengthens the observation that small perturbations can induce substantial semantic changes under the specified white-box conditions.

  2. The evaluation distinguishes target injection from source suppression. Testing several independent questions reveals limitations that a single optimized output string would miss. The breakdown into retention, partial replacement, complete replacement, and degeneration is also informative about attack behavior across perturbation budgets.

Critical weaknesses
  1. The strongest controlled ablation does not support the central design rationale. Table 10 reports 50.11% strict ASR for DeepStack-only alignment and 47.12% for pre-merger alignment, compared with 38.17% for the proposed alignment of all post-merger streams. These results challenge the emphasis on aligning every visual input after merging as the explanation for effectiveness. Moreover, Appendix E explicitly states that the adapted PA-Attack also performs position-wise alignment, whereas Section 4.3 describes it as using a global objective. Consequently, the contribution cannot be attributed simply to positional correspondence. The paper needs to reconcile these findings and establish which design choice actually produces the improvement.

  2. The reliability and interpretation of the success labels remain unresolved. All responses are scored by one LLM judge without human validation, and some examples conflict with the stated rules. In Figure 6, Q4 receives “Yes” and is marked successful, although Table 1 requires “No” for source denial. Figure 8 marks “bowl” as successful target identification in a cat-to-sauce example, which also needs reconciliation with the label-matching rules. These may be presentation errors, but an audit is necessary to determine whether quantitative scoring is affected. Additionally, passing three category-level questions supports a narrower conclusion than complete replacement of the scene’s semantics.

Questions for authors
  1. Why do DeepStack-only and pre-merger alignment outperform full TokRA in Table 10? Do these variants retain their advantage across budgets and on videos, and how should the proposed method’s central contribution be interpreted?

  2. Can you release the judge prompt and provide a human-validated audit of success labels, explicitly resolving the inconsistencies in Figures 6 and 8? Do corrected labels materially change strict ASR?

  3. What are the Q1–Q4 results on unperturbed source inputs? Are pairs filtered to ensure that the source is initially recognized and the target is initially absent, particularly for multi-object images and semantically related categories?

1 comment

Thanks for the detailed comments, and especially for catching the PA-Attack description inconsistency and the caption errors in Figures 6 and 8. We address the alignment position in the general response.

Ablation variants at other budgets and on video

The reviewer has raised a question about the description of PA-Attack, noting that Section 4.3 characterizes it as using a global objective while Appendix E shows it performs position-wise alignment. This was a presentation error, and we have corrected Section 4.3 to accurately reflect that the adapted PA-Attack also performs position-wise alignment (Equation 8). That said, our finding that target injection and source suppression are independent rests on each baseline's own output pattern alone.

  • Output-space PGD. Q1 and Q3 pass rates rise with budget while Q4 stays low (Figure 4). Injecting target semantics alone does not suppress source semantics.
  • PA-Attack. Q4 pass rate rises with budget while Q1 stays near zero (Tables 8–9). Suppressing source semantics alone does not inject target semantics.

The ablation in Table 10 covers ϵ=4/255\epsilon = 4/255 on images, and the finding that this budget suffices for semantic replacement holds across all variants, regardless of which one scores highest.

Each observation follows from one method's output pattern alone, so this conclusion does not depend on how PA-Attack compares with TokRA.

Caption errors and scope of conclusions

The reviewer identified two inconsistencies in the qualitative examples and asked whether they affect the quantitative scores. We checked both cases. In Figure 6, "person" appears in both source and target, making Q4 impossible to answer correctly — such pairs can only fail Q4 and therefore lower strict ASR. In Figure 8, "bowl" and "sauce" do not match and should be judged as a failure. Both errors are confined to the figure captions. The quantitative labels in the dataset are correct, and we have corrected both captions.

The reviewer also notes that passing three category-level questions supports a narrower conclusion than complete replacement of the scene's semantics. We agree, and have changed "complete replacement" to "category-level replacement" throughout the manuscript. After the category is replaced, attributes of the source scene may still be preserved, as shown in Figure 1 (semantic fusion).

Clean-input baselines and sample-pair filtering

The reviewer asks for Q1–Q4 results on unperturbed inputs and whether pairs are filtered to ensure the source is recognized and the target is absent. On unperturbed source inputs (Qwen3-VL, image / video; Appendix K, Table 15), Q1 matches the target at 1.1% / 4.7%, Q3 answers "yes" at 7.9% / 19.5%, Q4 answers "yes" at 99.8% / 99.2%, and strict ASR is 0.0% / 0.0%. These numbers confirm that the model correctly recognizes clean inputs and that no pair trivially passes strict ASR before perturbation.

To address the Q3 false-positive concern, we additionally report results on a filtered subset that excludes pairs where Q3 ≠\neq "no" on the clean source, using each model's own outputs (432 image pairs and 103 video pairs on Qwen3-VL; Appendix K, Tables 16–18). On this subset, TokRA's strict ASR at ϵ=4/255\epsilon = 4/255 is 38.9% (38.2% before filtering), and output-space PGD's is 24.1% (unchanged). Results remain stable after filtering.

We hope these corrections address the reviewer's concerns and would welcome reconsideration of the score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Thanks to all three reviewers for the careful reading and constructive feedback. It is encouraging that the injection–suppression evaluation (2f2v, VtKr) and the semantic fusion observations (VtKr, ikc2) resonated. We address the common questions below.

Scope of the contribution

We recognize that this work may be read as an attack contribution, but our main finding is that the apparent robustness of VLMs to representation alignment attacks at small perturbation budgets reflects a limitation of prior attack formulations, not genuine model robustness.

Alignment position

All three reviewers are concerned about why DeepStack-only and pre-merger alignment outperform TokRA in Table 10. The design principle behind TokRA is to align all visual input that the LLM backbone directly receives, rather than selecting architecture-specific subsets. For Qwen3-VL, this means aligning four post-merger streams, while for Qwen2.5-VL it is simply the single merger output. As Table 10 shows, subset variants can indeed score higher on Qwen3-VL at ϵ=4/255\epsilon = 4/255. DeepStack-only reaches 50.11% and pre-merger reaches 47.12%, both above TokRA's 38.17%. In other words, TokRA trades peak ASR on one specific architecture for a principle that generalizes across models. Since all variants achieve semantic replacement at this budget, the paper's central conclusion holds regardless of which alignment position is chosen.

Reliability of the LLM judge

Reviewers 2f2v and VtKr are concerned about relying on a single LLM judge without human validation. To address this, we ran rule-based scoring as an independent check on all 12,750 samples (Appendix J). The two methods agree on 99.69% of labels, and method rankings remain unchanged, so we retain the original LLM-based scores as the main results.

Perceptibility

Reviewers VtKr and ikc2 point out that a small perturbation budget does not by itself guarantee imperceptibility, and that adversarial examples should be shown for direct assessment. We agree with this concern. We have replaced "imperceptible" with "small-budget" throughout the manuscript to avoid overclaiming. At ϵ=4/255\epsilon = 4/255, each pixel changes by at most 1.6% of the intensity range. Adversarial examples are now included in the appendix (Appendix M) for visual inspection.

Post-processing robustness

Reviewers VtKr and ikc2 ask whether TokRA is robust to common post-processing such as JPEG compression, resizing, and video re-encoding. We deliberately scope the threat model to exclude these factors (Section 4.3), so that the reported ϵ\epsilon isolates the cost of semantic substitution alone.

Corrections and additions (blue for additions, red for corrections)

  • Reviewer 2f2v pointed out that Section 4.3 describes PA-Attack as using a global objective, while Appendix E states it performs position-wise alignment. We have corrected Section 4.3 and the corresponding discussion in Section 5.1.
  • Following Reviewer 2f2v's examples in Figures 6 and 8, we have update figures to match the stated evaluation rules.
  • All three reviewers asked about the alignment position (Table 10). We have stated the design principle in Section 4.1: TokRA aligns all visual input that the LLM backbone directly receives.
  • Reviewer ikc2 raised concerns about novelty relative to prior representation-alignment attacks. We have credited Sabour et al. [1] as the earliest related work in the first contribution, and added discussion of CRAFT [5].
  • As Reviewer 2f2v noted, passing three category-level questions supports a narrower conclusion than complete semantic replacement. We have changed "complete replacement" to "category-level replacement" accordingly.
  • Regarding Reviewer 2f2v's question about unperturbed baselines, we have added results on a filtered subset that excludes clean Q3 false positives (Appendix K).
  • Reviewers 2f2v and VtKr both raised concerns about LLM judge reliability. We have added rule-based scoring validation with 99.69% agreement over 12,750 samples (Appendix J).
  • Reviewers ikc2 and VtKr remarked on perceptual quality. We have added adversarial examples (Appendix M) and replaced "imperceptible" with "small-budget" throughout.
  • Following Reviewer ikc2's request, we have added GPU timing comparisons (Appendix L).
  • Following Reviewer VtKr's suggestion, we have added Qwen2.5-VL video results (Appendix I, Table 13).

References

[1] Sara Sabour, Yanshuai Cao, Fartash Faghri, and David J. Fleet. Adversarial Manipulation of Deep Representations.

[5] Jiale Zhao, Xinyang Jiang, Junyao Gao, Yuhao Xue, and Cairong Zhao. One Object, Multiple Lies: A Benchmark for Cross-task Adversarial Attack on Unified Vision-Language Models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.