acceptodds
Under review as a conference paper at ICLR 2027

VL-ReasonHijack: A Cross-Modal Collaborative Adversarial Attack Method for Vision-Language Model Reasoning

Abstract

Vision-language models (VLMs) have developed rapidly with an emerging capability of multimodal reasoning, which aims to generate reasoning chains that connect visual evidence with commonsense knowledge, scientific concepts, or mathematical relations before producing answers. This capability creates an under-explored robustness risk: adversarial inputs can change not only the final answer, but also the reasoning chain that appears to justify it based on misleading visual evidence. Existing multimodal attack methods mainly manipulate final answers or disrupt reasoning processes, which may cause notable problems in generated texts like answer-rationale mismatch or abrupt changes in the reasoning form. Thus, we study reasoning hijacking, where an adversarial input induces a wrong answer together with a reasoning chain that supports it. To this end, we propose VL-ReasonHijack, a cross-modal collaborative adversarial attack for VLM reasoning. Given a fake reasoning chain as the hijacking objective, VL-ReasonHijack alternates between text-side question rewriting and image-side reasoning-aware optimization. Specifically, the question rewriting component emphasizes visual details used by the fake chain while preserving the original answer for a human reader. The reasoning-aware optimization component applies key reasoning atom weighting (KRAW) with a contrastive loss to reinforce reasoning-critical phrases in the fake chain. Across three mainstream VLMs and four multimodal reasoning benchmarks, VL-ReasonHijack achieves the highest answer-level attack success rates (ASR) on all the benchmark-model pairs, while producing more textually plausible reasoning outputs as reflected by lowest detection rates in most settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
adversarial attack, vision-language models, multimodal reasoning, reasoning hijacking, robustness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_83zoOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper aims to hijack the reasoning chain of LVLMs rather than merely manipulating their final outputs. Previous adversarial attack methods mainly target the final prediction, which may lead to inconsistency between the intermediate reasoning evidence and the final answer. To address this issue, the authors propose VL-REASONHIJACK, a two-stage white-box cross-modal adversarial generation framework. Specifically, VL-REASONHIJACK first rewrites the text-side question and then introduces KRAW to optimize the visual input. Experiments on three LVLMs and four datasets demonstrate that the proposed method achieves state-of-the-art attack performance.

Critical strengths
  • The paper explores cross-modal optimization for attacking the reasoning process of LVLMs, rather than focusing only on the final output. This setting is potentially more challenging than conventional output-targeted attacks.

  • The proposed method achieves strong attack performance and is also shown to bypass existing defense mechanisms, suggesting that reasoning-level attacks may expose limitations that are not fully captured by conventional adversarial evaluation.

Critical weaknesses
  • lack representative works as comparisons:

[1] Li et al. A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1. NeurIPS 2025.

[2] Jia et al. Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment. NeurIPS 2025.

[3] Nie et al. V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs. CVPR 2026.

These works are closely related to adversarial attacks on MLLMs/LVLMs and should either be included as baselines or explicitly discussed. Without such comparisons, it is difficult to say previous works can not affect the reasoning chain.

  • writing issues: There are so many Appendix.XX in the Experiment Section. In particular, in the MAIN RESULTS section, there are 4 experiments and analyses all in the supplementary material, causing the experimental section to look confusing and not sufficiently support the experiments.

I strongly suggest reorganizing the experimental section and moving the most important results and analyses into the main paper.

Questions for authors
  • Why does the paper not discuss or compare against representative recent adversarial attacks on LVLMs/MLLMs, such as the works listed above? Is the reason that the adversarial examples generated by these methods mainly manipulate the final output while their reasoning chains remain benign or semantically consistent? If so, I think this distinction should be explicitly discussed and empirically demonstrated.

  • What is the number of iterations used in the proposed method? 1) As the authors describe the optimization as being conducted “in an iterative adversarial manner,” I believe the effect of the iteration number on attack performance is an important experiment, particularly for supporting the claimed “collaborative” nature of the method. 2) I checked the pseudocode, and T seems to be the number of iterations (t = 1 to T), but in the Input part it is written as “PGD budget T,” which I suppose should be the perturbation budget ϵ? Please clarify the exact meaning of T and provide further discussion or experiments on how the number of iterations affects the attack results.

7 comments

Thank you for pointing out the issues regarding the comparison with recent representative methods, the organization of the experimental section, and the description of the iteration settings. We have added evaluations of the three attack methods you suggested and clarify each point below. Text revisions are highlighted in blue.

W1 / Q1: Comparison and discussion of recent adversarial attacks (Part 1/2)

We did not compare with these three works in the submission mainly because the attack settings differ. Our method and the main baseline Stop Reasoning (Stop Reasoning! When Multimodal LLM with Chain-of-Thought Reasoning Meets Adversarial Image, COLM 2024) both optimize the adversarial image directly with the victim model's gradients. M-Attack (A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1, NeurIPS 2025), FOA-Attack (Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment, NeurIPS 2025), and V-Attack (V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs, CVPR 2026) instead generate adversarial images with CLIP surrogate models and transfer them to closed-source models. In response to your comment, we evaluated them on our task with their official code, and their answer flip rates range from 23.6% to 35.0%. By changing image semantics, these methods influence model outputs and may also change the generated reasoning. Our distinction lies in explicit control over the attack target: we form a target chain from an image-specific false visual premise, reasoning that supports a specified wrong option, and the final answer, and optimize the image-question pair toward this chain.

1. Settings and attack targets.

MethodDataEvaluation task and promptSuccess criterion
M-AttackNIPS 2017 adversarial competition images (ImageNet-compatible), paired with target imagesImage description, e.g., "Describe this image in one concise sentence, no longer than 20 words."Target semantic keyword matching rate (KMR), and the proportion of descriptions with semantic similarity above 0.3 (ASR)
FOA-AttackSame as aboveImage description, e.g., "Describe this image in one concise sentence, no longer than 20 words."LLM-judged semantic similarity between the generated description and the target-image description; a score above 0.5 counts as success
V-AttackCOCO, ImageNetImage description, and identification questions about the main object, e.g., "What is the large creature that occupies most of the frame?"Mean GPT-4o score of 0, 0.5, or 1, based on the presence of the original and target concepts
VL-ReasonHijack (Ours)A-OKVQA, ScienceQA, MathVista, MathVerseMultiple-choice multimodal reasoning QA with rationale-then-answer promptingASR: the final answer differs from the ground truth. THSR: the final answer matches the pre-selected wrong option.

M-Attack and FOA-Attack make the adversarial image close to a target image in the feature space of surrogate vision encoders, and V-Attack replaces one object concept in the image with another through value features. Their attack targets are defined only by image semantics and are independent of the specific question, options, and reasoning. Our work studies multi-step reasoning QA, including the knowledge-intensive A-OKVQA and ScienceQA and the visual mathematical reasoning benchmarks MathVista and MathVerse (Section 4.1). In these tasks:

  • The answer depends on the question itself and the reasoning process, not only on "what is in the image". Many images are charts, geometric figures, or complex scenes without a single salient entity that can be directly replaced.
  • Our attack target is a complete reasoning chain. The model must output the pre-specified wrong option and give a reasoning chain that supports this option and rests on a false visual premise (Section 3.1, Eq. (1)).

W1 / Q1 (continued, Part 2/2)

2. Measurements on our task (Appendix C.9 of the revised version, Table 15). We evaluated the three methods with their official code on our three victim models. The evaluation uses a fixed random sample of 200 A-OKVQA questions, excluding those that each model answers incorrectly on clean inputs. The attacks follow the official settings: perturbation budget ε=16/255, 300 steps, and transfer from ensembles of CLIP surrogate models. The target concept of V-Attack is set to our pre-selected wrong option.

AccessMethodMetricQwen2.5-VL-7BInternVL3.5-8BLLaVA-OV-1.5-8B
Black-box transferM-AttackASR29.4%30.8%25.3%
FOA-AttackASR30.6%29.6%24.7%
V-AttackASR35.0%30.8%23.6%
V-AttackTHSR23.3%24.3%14.9%
White-boxVL-ReasonHijackASR98.1%98.3%99.5%
VL-ReasonHijackTHSR97.7%97.8%98.9%

VL-ReasonHijack results are from Table 1.

The FOA-Attack paper reports that, on its image description task, M-Attack and FOA-Attack achieve success rates of 52.6% and 70.7% on the same Qwen2.5-VL-7B. On our task, their answer flip rates are only 29.4% and 30.6%. Replacing the semantics of an entity in the image does not necessarily change the answer to a question that requires multi-step reasoning, and it is difficult for such attacks to steer the answer reliably to a specified wrong option. Even when V-Attack sets the target concept to the wrong option, only 14.9%–24.3% of the samples output that option.

3. Revision. Section 2 of the revised version adds M-Attack and FOA-Attack and explains how the three works differ from ours in threat model, task, and success criterion. The Baselines paragraph in Section 4.1 also states why the three works are reported separately rather than in the main table. V-Attack is also discussed in Section 2 and Table 6 in Appendix B.1 of the revised version. The measurements above are added to the revised version as Appendix C.9, and the entries of the three works in Table 6 are updated accordingly.

W2: Reorganizing the experimental section and moving key evidence into the main paper

We have reorganized the experimental section. Of the four appendix analyses cited in the original Main Results, THSR has been moved into the main table, the significance tests remain in the appendix, and the two supplementary diagnostics, the image audit and cross-model transfer, are no longer cited in the main results. The five consecutive appendix references in the experimental setup have also been merged into a single reference, reducing the appendix references in the experimental section from 19 to 11.

1. Main Results (Section 4.2). The revised section presents the following results and analyses:

  • THSR is merged into Table 1. Each VL-ReasonHijack cell reports ASR / THSR / Detection Rate, and the separate THSR table in the original appendix has been removed. The text now reports the difference between ASR and THSR directly. On A-OKVQA and ScienceQA, THSR is within 0.9 points of ASR for every victim model, while the gap is larger on MathVista and especially MathVerse, e.g., 63.0% ASR versus 55.0% THSR for Qwen2.5-VL on MathVerse. The text also gives the ASR improvement of VL-ReasonHijack over the strongest baseline (4.9–15.0 points).
  • The image audit and cross-model transfer are supplementary diagnostics and are no longer cited in this section. They remain in Appendices C.4 and C.7.
  • The significance tests remain in Appendix C.2, and the text states the conclusion: VL-ReasonHijack achieves significantly higher ASR in all 36 comparisons with the three Stop Reasoning settings.
  • A one-sentence result on the newer Qwen3.5-4B is added (in response to another reviewer), with details in Appendix C.8.

2. Section order. The ablation study (Section 4.3, Table 2) now precedes the human evaluation of question rewriting (Section 4.4, Table 3), so the text proceeds as "overall effectiveness → component contributions → rewrite validity → visual dependence analysis". The ablation section also directly reports the sensitivity results. On 431 A-OKVQA samples, disabling question rewriting lowers ASR from 95.1% to 83.8%, and the default rewrite interval of 50 and suppression weight of 0.3 achieve the highest ASR among the tested settings.

3. Visual dependence analysis (Section 4.5, Table 4). The original version pointed to the likelihood analysis in the appendix with a single sentence. The revised version reports its results directly in the text. On 96 successfully attacked Qwen2.5-VL samples, the adversarial image lowers the NLL of the target false-premise spans by 0.075 on average, with the largest decrease on VD samples (0.122). The conclusion that THSR increases monotonically from TD to TL to VD is now supported by specific numbers (e.g., from 39.1% to 46.7% and 66.3% on Qwen2.5-VL).

Q2: Iteration count, the meaning of T, and the perturbation budget

Meaning and value of TT. TT is the maximum number of PGD iterations, not the perturbation budget. We use T=120T=120 (Section 4.1 of the revised version), and the perturbation budget is a separate parameter, ϵ=16/255\epsilon=16/255. The phrase "PGD budget TT" could be confused with the perturbation budget, so the input line of Algorithm 1 in the revised version now reads "maximum number of PGD iterations TT, perturbation budget ϵ\epsilon". The algorithm now runs over t=0,…,T−1t=0,\ldots,T-1, uses qadv(t+1)q_{adv}^{(t+1)} for the question after selection in the current iteration, and aligns the time indices of image updates and returned values with Sections 3.2 and 3.3 (Appendix A.4, Algorithm 1).

Effect of the number of iterations. In our method, iterations alternate between question rewriting and image optimization. The question is rewritten every fixed number of steps, and the image then continues to be optimized for the new question. With a fixed total number of steps, a smaller rewrite interval gives more alternation rounds but fewer image optimization steps after each question change.

(1) Question rewriting frequency (Appendix C.1, Figure 3). On 431 A-OKVQA samples with Qwen2.5-VL, we fix the maximum number of image optimization steps at 120 and vary the question rewriting interval to compare no rewriting with 1, 2, 4, and 6 rewrite rounds. The interval is the number of image optimization steps between consecutive rewrite rounds. The table also lists the corresponding number of rewrite rounds for each setting.

Rewrite interval (image optimization steps)No rewriting8550 (default)2515
Corresponding rewrite rounds01246
ASR (%)83.892.895.191.291.9

ASR is 83.8% without question rewriting, rises to 92.8% with one rewrite round, and increases further to 95.1% with two rounds. Shortening the interval to allow four or six rounds instead lowers ASR to 91.2% and 91.9%, respectively. Under the same image optimization budget, two rewrite rounds outperform a single round, while more frequent rewriting brings no further improvement.

(2) Image optimization steps. The main experiments record the step at which each sample stops early. The table below gives, from the same runs, the proportion of samples that have hit the target by each step. The last column is the final THSR in Table 1 (Qwen2.5-VL, %):

DatasetBy step 95By step 105By step 115Final THSR (Table 1)
A-OKVQA78.289.595.097.7
ScienceQA65.074.280.284.1
MathVista40.754.362.766.3
MathVerse20.439.449.955.0

The early-stop check is performed only in the late stage of optimization, so the first column gives the cumulative hit proportion at the first check. The last rewrite occurs at step 50 for both task types, and only the image is optimized afterwards. From the first check to the end at step 120, the hit proportion rises by a further 19.5, 19.1, 25.6, and 34.6 points on the four datasets, so image optimization after the last rewrite still yields many additional target hits. LLaVA-OneVision and InternVL3.5 show the same trend.

Reviewer_HX8MOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an attack approach that refines the reasoning path to guide the selection and optimization of multimodal inputs (text and image). Specifically, the authors design the text input to direct the model's attention toward the image modality, apply a PDG attack with the proposed Key Reasoning Atom Weighting (KRAW) method, and incorporate a contrastive loss. The experimental results across four benchmarks demonstrate improved attack success rates (ASR).

Critical strengths
  • One critical strength of this paper is its idea of "polluting the reasoning path". I find this interesting for two reasons: (1) using the reasoning path to guide the attack optimization is a novel approach, and (2) most user interfaces (UIs), including chatbots and agents, collapse the reasoning path by default, which may make such attacks harder to detect.
Critical weaknesses
  • The method section is hard to follow. The paper introduces many concepts, but several are not clearly explained in the main text. For example, the paper mentions a "fake chain" without explaining how it is constructed. It may not be necessary to include all the implementation details (e.g., the prompts or models used to generate it), but the authors should at least clarify how or where it is obtained. Currently, the paper only describes it as a "generated fake chain." Additionally, Figure 2 is difficult to read, especially the text portion. The input on the right includes "two images," a fake chain, and text. Why are there two images? Also, why are there more than two candidates when the question set Q\mathcal{Q} in the paper contains only two? I would recommend revising the overall method section and improving the figure to make the proposed approach easier to understand.
  • The attack setup may be impractical in real-world scenarios. The final step of the proposed approach is to prepare a polluted text-image pair. However, I have two major concerns:
    1. Most widely used AI models are closed-source, which limits the applicability of the proposed attack. Although the authors discuss transferability in Appendix C.8, the results do not clearly demonstrate its effectiveness. In addition, the Square Attack approach (non-gradient attack) also appears to perform poorly, as shown in Appendix C.6.
    2. What happens if the text-image pair is separated? For example, if only the image is obtained, would the attack still succeed?

These limitations raise concerns about the practical applicability of the proposed attack and weaken the overall contribution of the paper.

Questions for authors
  • Please revise the method section to improve its clarity and readability. If time does not allow for a revised manuscript during the rebuttal period, please at least explain what changes the authors plan to make to address these concerns.
  • Please provide further clarification on how the proposed approach can be applied to closed-source models, particularly considering the practical limitations discussed in the second weakness.
6 comments

Thank you for pointing out the specific issues in the method description and Figure 2 that hinder understanding. Below we explain where the fake chain comes from, the input relations in the figure, and how the attack applies under different input and model-access conditions. Text revisions are highlighted in blue.

W1 / Q1: Clarity of the method section and Figure 2

In response to your comments, we have revised Section 3 and Figure 2. We first answer your two questions and then list the other changes to Section 3.

1. Where the fake chain comes from. Section 3.1 of the revised version now describes how the fake chain is generated. Before the attack, the attack-side VLM reads the clean image, the question with its options, and the correct answer. It selects a wrong option as AfakeA_{fake} and writes reasoning RfakeR_{fake} that builds on a false visual premise and leads to this option. The fake chain is generated once per sample and kept fixed during the attack. The full prompt requirements and quality control are given in Appendix A.6.

2. The two images in Figure 2, and the candidate questions versus Q\mathcal Q. The two images are the clean image xx and the current adversarial image xadv(t)x_{adv}^{(t)}, which play different roles. xx is fed to the attack-side VLM together with YfakeY_{fake} and the current attack question qadv(t)q_{adv}^{(t)} to generate rewrite candidates, and it is also the center of the perturbation ball. xadv(t)x_{adv}^{(t)} is fed to the victim to score the candidates by fake-path loss, and it is updated by PGD. The candidate questions and Q\mathcal Q are two different sets. Candidate questions are used only for selection at each text-side update. Each round keeps only the question with the lowest fake-path loss as qadv(t+1)q_{adv}^{(t+1)} and discards the rest. The subsequent image optimization uses Q={q,qadv(t+1)}\mathcal Q=\{q,q_{adv}^{(t+1)}\}, or Q={q}\mathcal Q=\{q\} if the original question is selected. The final polishing phase uses only qadv(t+1)q_{adv}^{(t+1)}. Consequently, the number of candidates can exceed two, whereas Q\mathcal Q contains at most two questions. These two points are now stated in Sections 3.2 and 3.3 of the revised version.

Changes to Figure 2. In the original figure, the left part of the Text-Side panel drew the fake chain, the two images, and the question in a single column, which made it unclear which model receives each input. The revised figure splits them into two input boxes. The green box contains YfakeY_{fake}, xx, and qadv(t)q_{adv}^{(t)}, which are fed to the attack-side VLM to generate candidate questions. The purple box contains xadv(t)x_{adv}^{(t)} and YfakeY_{fake}, which are fed to the victim to compute the fake-path loss. The separately drawn clean image on the right of the Image-Side panel has been removed. Candidate questions and the Active Question Set Q\mathcal Q are drawn separately. The update from qadv(t)q_{adv}^{(t)} to qadv(t+1)q_{adv}^{(t+1)} is explicitly marked, and the candidates not selected after reranking are labeled Discarded. All images, questions, and chains in the figure are labeled with the symbols used in the text (xx, xadv(t)x_{adv}^{(t)}, xadvx_{adv}, qq, qadv(t)q_{adv}^{(t)}, qadv(t+1)q_{adv}^{(t+1)}, qadvq_{adv}, Q\mathcal Q, YfakeY_{fake}, YtrueY_{true}), and the names are consistent with the text. The caption explains that the scores are fake-path losses under the current xadv(t)x_{adv}^{(t)} and how the selected qadv(t+1)q_{adv}^{(t+1)} enters Q\mathcal Q.

3. Other changes to Section 3.

  • Consistent naming: the introduction, method, and appendix consistently use the two names fake chain YfakeY_{fake} and true chain YtrueY_{true}.
  • Section 3.2: organized according to the three parts of Figure 2. It states the alternation schedule (the image is updated at every step, and the question is rewritten every KK steps), distinguishes qadv(t)q_{adv}^{(t)} before selection from qadv(t+1)q_{adv}^{(t+1)} after selection, and specifies qadv(t+1)=qadv(t)q_{adv}^{(t+1)}=q_{adv}^{(t)} when no rewriting occurs. It also defines the final polishing phase. The descriptions of the optimization procedure are now consolidated in this section.
  • Section 3.3: rewritten in the order "generate candidates → score and select → form Q\mathcal Q". It explains the scoring criterion (a lower fake-path loss means that the model is more likely to generate the fake chain) and adds a paragraph on the Active Question Set.
  • Section 3.4: before defining KRAW and true-path suppression, we first explain their roles. We rewrote the CE example to show why key content needs to be weighted, and compressed the loss implementation details into one sentence, with the segment-wise definition in Appendix A.7.

W2 / Q2: Practicality: closed-source models and separated text-image pairs (Part 1/2)

I. Applicability to closed-source models

We adopt a fixed-victim white-box setting, targeting public assistants built on open-weight models (Sections 1 and 3.1). In this scenario, the attacker can obtain the same weights as the deployed model. In Table 1, the three attacks of Stop Reasoning (Stop Reasoning! When Multimodal LLM with Chain-of-Thought Reasoning Meets Adversarial Image, COLM 2024) and QAVA (QAVA: Query-Agnostic Visual Attack to Large Vision-Language Models, NAACL 2025), like ours, use the victim model's gradients to optimize the adversarial image under the same perturbation budget (Section 4.1, Appendix D). To ensure a comparison under the same setting, the main experiments are conducted on open-weight models, and we do not evaluate closed-source models. We discuss applicability to closed-source models below, based on the transfer and gradient-free experiments in the appendix.

1. Transfer of our method (Appendix C.7 of the revised version, Table 13). For a closed-source model, one can first generate an adversarial image-question pair on an open-weight source model and submit it to the target model through its API, without access to the target model's weights or gradients. To test this transfer protocol without using target-model gradients, we conducted experiments among three open-weight models (Appendix C.7 of the revised version, Table 13). The transfer experiment uses 810 A-OKVQA samples that are successfully attacked on all three source models with the same target answer. We feed the full image-question pair generated on the source model directly to the target model, without re-optimizing for the target model. Across all six directions among the three models, the target model outputs the specified wrong answer in 16.23%–26.25% of the samples. Using only the rewritten question with the clean image, the rate is 10.16%–20.35%, and adding the adversarial image further increases it by 2.37–7.67 points in all six directions. Without the target model's weights or gradients, and without re-optimizing for it, the image-question pair can still hijack a portion of the samples toward the specified target, and both the image and the text contribute.

2. Dedicated transfer attacks are of limited effectiveness on our task (Appendix C.9 of the revised version, Table 15). M-Attack (A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1, NeurIPS 2025), FOA-Attack (Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment, NeurIPS 2025), and V-Attack (V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs, CVPR 2026) generate adversarial images with ensembles of CLIP surrogate models and transfer them to closed-source models such as GPT-4o. They are evaluated under fixed prompts that describe or identify the image, and success is judged by whether the target semantics appear in the output. We used their official code and settings (ε=16/255, 300 steps), fixed a sample of 200 A-OKVQA questions, removed the samples that each model answers incorrectly on clean inputs, and evaluated them on our three victim models:

MethodQwen2.5-VL-7BInternVL3.5-8BLLaVA-OV-1.5-8B
M-Attack, ASR29.4%30.8%25.3%
FOA-Attack, ASR30.6%29.6%24.7%
V-Attack, ASR35.0%30.8%23.6%
V-Attack, hits the specified wrong option23.3%24.3%14.9%

The FOA-Attack paper reports success rates of 52.6% and 70.7% for M-Attack and FOA-Attack on Qwen2.5-VL-7B in its description task. On our multi-step reasoning QA, their answer flip rates on the same model are only 29.4% and 30.6%. Even when V-Attack sets the target concept to the pre-selected wrong option, only 14.9%–24.3% of the samples output that option.

3. The transfer results on open-source models do not imply that closed-source models are harder to attack. The works above report results on both open-source and closed-source models, and on some closed-source models the success rates are higher than on open-source ones. The success rates of FOA-Attack and M-Attack on GPT-4o (75.1% and 60.3%) are both higher than those on Qwen2.5-VL-7B (70.7% and 52.6%). The average score of V-Attack on GPT-4o in the description task (0.668) is also higher than on the three open-source models it evaluates (0.504–0.560). These works show that closed-source models are not necessarily harder to attack through transfer than open-weight models. Transfer rates between open-weight models alone therefore do not determine the difficulty of attacks in a closed-source setting.

W2 / Q2 (continued, Part 2/2)

4. Gradient-free optimization (Appendix C.5 of the revised version, Table 12). This experiment replaces PGD with Square Attack (Square Attack: A Query-Efficient Black-Box Adversarial Attack via Random Search, ECCV 2020) in Answer Attack (an image attack that only flips the answer), and it is not an evaluation of our method. Under the same perturbation budget, with about 1,300–1,400 queries per image on average, Square Attack achieves a success rate of 2.9%–11.5%, compared with 51.1%–92.2% for the gradient-based method. Even for the simpler goal of only flipping the answer, gradient-free search has difficulty finding an effective perturbation within this budget, which motivates our use of white-box gradient optimization.

II. When the image-question pair is separated

In our threat model, the image and the question are submitted together to the assistant by the same attacker, and those affected are other users who see the assistant's public answer and reasoning (Sections 1 and 3.1). The harm comes from this public reply and does not require other users to reuse the image, so the pair is not separated during the attack. If the pair is separated, two situations can be distinguished.

(1) Using only the adversarial image with the original question. The w/o question rewriting setting in Table 2 only optimizes the image without modifying the text (the adversarial image is optimized separately for the original question), and the victim receives the original question. On Qwen2.5-VL, THSR is 86.9%, 63.7%, 56.4%, and 50.1% on the four datasets, showing that the attack remains effective when the attacker modifies only the image and keeps the original question. Adding question rewriting in the full method further raises THSR to 97.7%, 84.1%, 66.3%, and 55.0%.

(2) Without the original question (no question, or a different question). All four benchmarks are multiple-choice, and the attack target is to make the model choose the specified wrong option among the given options and give reasoning that supports it (Section 3.1). Without the original question, the options and the correct answer disappear or change, so the originally specified wrong option no longer applies. In research on adversarial attacks against VLMs, attacks are usually constructed and evaluated under a known question or prompt. AttackVLM (On Evaluating Adversarial Robustness of Large Vision-Language Models, NeurIPS 2023) fixes the input text to "what is the content of this image?" and specifies that the attacker does not modify the text and only perturbs the image. Robust CLIP (Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models, ICML 2024) constructs per-sample attacks on VQAv2 and TextVQA using the dataset questions, with the ground-truth answers as attack labels. Only works that specifically study cross-prompt or unknown-question settings treat the question as unknown. CroPA (An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Models, ICLR 2024) jointly optimizes the adversarial image over a set of questions and task prompts. QAVA optimizes with randomly sampled VQA v2 questions as surrogate questions when the question is unknown, and its goal is only to make the answer wrong, without specifying a wrong answer. Constructing attacks under a known question is therefore the standard setting in this line of work.

In summary, our setting corresponds to a practical scenario: the attacker submits the image-question pair together to a public assistant deployed on an open-weight model, whose weights the attacker can also obtain. In this scenario the pair is not separated, and even when only the adversarial image is used with the original question, the attack remains effective.

Reviewer_jLkuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the reasoning hijack on MLLMs. Specifically, the attack wants the MLLM to 1) generate targeted wrong answer and 2) generate valid explanations for the wrong answer. The attack optimizes the input question and the image together for each sample under a white-box setting via update the prompt text and PGD attack on image pixels. The experiments cover 3 models and 4 datasets and the method shows the best attack success rate with lowest text detection rate.

Critical strengths

The attack goal is meaningful. Compared to only generated wrong answer, generating the valid explanations corresponding to the wrong answer makes the attack more difficult to detect. The experiments are relatively comprehensive and the results are promising. The optimization method is well designed to make sure the reasoning follows the wrong answer.

Critical weaknesses
  1. The experiments lack validations on more up-to-date MLLMs with better performance and reasoning abilities. The models used in this studies are published in 2025 or earlier. Will recent MLLMs be more vulnerable to the attack or more robust to the attack?
  2. The generalization ability of the proposed method can be further strengthened. Current setup optimizes independently for each question-image pair for a given white-box MLLM. I wonder whether there can be some cross-sample, cross-model generalization analysis for this setting. For instance, by keeping the original question, how about updating a prompt prefix along with image modification given a set of samples, so that the prefix and the pixel modification can lead to successful attacks across samples.
  3. One comparison to highlight the advantage of the reasoning hijack needs more elaboration: whether the Answer Attack can also achieve valid rational given the wrong answers, as the Answer Attack has no prompt optimization modules, such analysis will highlight the necessity of the reasoning hijack.
Questions for authors

Pls see the weaknesses above.

6 comments

Thank you for recognizing the significance of reasoning hijacking, the effectiveness of our attack, and the design of its optimization. We respond below to your three questions on newer models, generalization, and the explanation quality of Answer Attack. Text revisions are highlighted in blue.

W1: Evaluation on newer MLLMs

We have added experiments on Qwen3.5-4B across all four datasets. Qwen3.5-4B is a native multimodal model released in 2026. Its publicly reported MMMU and MathVista (mini) scores are 77.6 and 85.1, compared with 58.6 and 68.2 for Qwen2.5-VL-7B. We compare with Answer Attack, the strongest baseline overall in Table 1, on the same samples:

DatasetAnswer Attack ASRVL-ReasonHijack ASR
A-OKVQA70.68%90.50%
ScienceQA44.19%59.04%
MathVista42.56%50.47%
MathVerse33.45%42.42%

Compared with the three models in Table 1, both attacks obtain lower ASR on Qwen3.5-4B on all four datasets. For example, on A-OKVQA, Answer Attack drops from 91.1%–93.4% to 70.7%, and VL-ReasonHijack drops from 98.1%–99.5% to 90.5%. This indicates that Qwen3.5-4B is more robust. Nevertheless, VL-ReasonHijack outperforms Answer Attack by 7.9–19.8 points on all four datasets, comparable to its margin in Table 1 (4.9–21.3 points), demonstrating that VL-ReasonHijack remains effective and maintains a clear advantage. These results have been added to Appendix C.8 of the revised version.

W2: Cross-sample and cross-model generalization

Cross-sample. You suggest jointly optimizing a shared prefix and image perturbation over a set of samples so that they also work on new samples. This setting differs from our task mainly in the form of the target. In shared attacks that generalize across samples, such as Image Hijacks (Image Hijacks: Adversarial Images Can Control Generative Models at Runtime, ICML 2024) and GCG (Universal and Transferable Adversarial Attacks on Aligned Language Models, arXiv 2023), the target is either the same for all inputs or derivable from the input itself. Image Hijacks induces the same specified string for different inputs, and GCG targets an affirmative response that begins with "Sure, here is" and restates the user's request. Our work follows the per-instance setting used in targeted vision-language attacks, where a target is specified for each input and the corresponding adversarial input is optimized separately. Earlier VQA attacks specify a target answer for each image-question pair (Fooling Vision and Language Models Despite Localization and Attention Mechanism, CVPR 2018). Show-and-Fool (Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning, ACL 2018) assigns a target caption to each image, and AttackVLM (On Evaluating Adversarial Robustness of Large Vision-Language Models, NeurIPS 2023) assigns a target text response to each image. Following this setting, we define the target for each sample as a pre-selected wrong option together with a false visual premise specific to that image, neither of which can be derived from the image-question input itself. For example, the target of one sample may be option (B) with the premise "the object in the image is red", while that of another sample is option (D) with the premise "the angle is 30°". Optimizing a shared prefix and perturbation can yield an attack strategy that is reusable across samples. Our task aims to steer each sample toward an additionally specified wrong answer and visual premise, evaluating per-instance targeted control. Your proposed shared attack raises the further question of how to retain control over each new sample's specified target while reusing the same prefix and perturbation. This differs from our evaluation protocol, in which adversarial inputs are constructed and optimized per sample.

Regarding the setting in which the original question is kept: the w/o question rewriting row in Table 2 does not modify the question text and only optimizes the image per sample. Its THSR is still 86.9%, 63.7%, 56.4%, and 50.1% on the four datasets, so our method remains effective when the question is left unchanged.

Cross-model. Appendix C.7 (Table 13) tests all six transfer directions among the three models on 810 A-OKVQA samples that are successfully attacked on all three source models with the same target answer. The full image-question pair generated on the source model is fed directly to the target model, without re-optimization for the target model. The target model outputs the specified wrong answer in 16.23%–26.25% of the samples. Both components of the pair contribute to this effect. Using only the rewritten question with the clean image yields 10.16%–20.35%, and adding the adversarial image further increases the rate by 2.37–7.67 points in all six directions, so the image perturbation retains a cross-model effect even though it is not optimized for the target model.

W3: Can Answer Attack also produce explanations supporting wrong answers?

Answer Attack can also produce reasoning consistent with its wrong answer on some successful samples, but the proportion is lower than that of VL-ReasonHijack. We first present the measured results and then explain the reason.

1. Does the reasoning of Answer Attack support its wrong answer? On each of the four datasets with Qwen2.5-VL, we sampled up to 200 successfully attacked outputs per method and removed the final answer sentence. An LLM judge (DeepSeek-V4.1-Flash) then identified the option supported by the remaining reasoning, using only the question, the options, and the reasoning. We report the Wrong-Answer Support Rate, defined as the percentage of sampled outputs for which the judge-identified option matches the model’s final wrong answer:

DatasetAnswer AttackVL-ReasonHijack
A-OKVQA79.5%86.0%
ScienceQA57.0%79.0%
MathVista62.5%70.5%
MathVerse56.0%60.0%

VL-ReasonHijack achieves a higher Wrong-Answer Support Rate than Answer Attack on all four datasets. Table 1 provides complementary evidence on text-side detectability: VL-ReasonHijack has a lower Detection Rate than Answer Attack on all 12 model-dataset pairs, indicating fewer detectable textual anomalies in its successful attack outputs.

2. Difference in the optimization target. Under the reason-then-answer setting, the model's final answer usually follows the reasoning it generates. Answer Attack constrains only the final answer and leaves the reasoning unconstrained. The generated reasoning may still partly support the correct option, which makes the answer harder to change, or it may become inconsistent with the answer after the answer changes. This accords with its lower Wrong-Answer Support Rate. VL-ReasonHijack optimizes a target chain consisting of both the reasoning and the answer, so that both point to the same wrong option. This is also in line with its higher ASR. In Table 1, Rationale Attack, which only pushes the reasoning away from its original form without specifying where it goes, has lower ASR than Answer Attack in 11 of the 12 pairs.

Thank you for your time during the discussion period. When convenient, we would appreciate your thoughts on our responses and additional results above. Regarding your interest in cross-model generalization, we would like to provide some further context for interpreting the transfer results.

Appendix C.9 of the revised manuscript evaluates three attacks specifically designed for black-box transfer. We use their official code and settings on 200 randomly sampled A-OKVQA questions, excluding those that each model answers incorrectly on clean inputs. The victim receives the adversarial image with the original question:

Method and metricQwen2.5-VL-7BInternVL3.5-8BLLaVA-OV-1.5-8B
M-Attack, ASR29.4%30.8%25.3%
FOA-Attack, ASR30.6%29.6%24.7%
V-Attack, ASR35.0%30.8%23.6%
V-Attack, target-hit rate23.3%24.3%14.9%

Even when V-Attack's target concept is set to the pre-selected wrong option, its target-hit rate is 14.9%–24.3%, illustrating the challenge of targeted transfer in our multi-step reasoning QA task. In Appendix C.7, we evaluate transfer on 810 samples that are successfully attacked on all three source models with the same target answer. Without re-optimizing for the target model, VL-ReasonHijack achieves a target-hit rate of 16.23%–26.25%. Adding the adversarial image also improves the results in all six transfer directions. We hope this additional context helps clarify the value of our cross-model results.

We would be happy to clarify any remaining questions about our previous responses or this additional information, and would greatly appreciate your feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.