- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper aims to hijack the reasoning chain of LVLMs rather than merely manipulating their final outputs. Previous adversarial attack methods mainly target the final prediction, which may lead to inconsistency between the intermediate reasoning evidence and the final answer. To address this issue, the authors propose VL-REASONHIJACK, a two-stage white-box cross-modal adversarial generation framework. Specifically, VL-REASONHIJACK first rewrites the text-side question and then introduces KRAW to optimize the visual input. Experiments on three LVLMs and four datasets demonstrate that the proposed method achieves state-of-the-art attack performance.
Critical strengths
-
The paper explores cross-modal optimization for attacking the reasoning process of LVLMs, rather than focusing only on the final output. This setting is potentially more challenging than conventional output-targeted attacks.
-
The proposed method achieves strong attack performance and is also shown to bypass existing defense mechanisms, suggesting that reasoning-level attacks may expose limitations that are not fully captured by conventional adversarial evaluation.
Critical weaknesses
- lack representative works as comparisons:
[1] Li et al. A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1. NeurIPS 2025.
[2] Jia et al. Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment. NeurIPS 2025.
[3] Nie et al. V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs. CVPR 2026.
These works are closely related to adversarial attacks on MLLMs/LVLMs and should either be included as baselines or explicitly discussed. Without such comparisons, it is difficult to say previous works can not affect the reasoning chain.
- writing issues: There are so many Appendix.XX in the Experiment Section. In particular, in the MAIN RESULTS section, there are 4 experiments and analyses all in the supplementary material, causing the experimental section to look confusing and not sufficiently support the experiments.
I strongly suggest reorganizing the experimental section and moving the most important results and analyses into the main paper.
Questions for authors
-
Why does the paper not discuss or compare against representative recent adversarial attacks on LVLMs/MLLMs, such as the works listed above? Is the reason that the adversarial examples generated by these methods mainly manipulate the final output while their reasoning chains remain benign or semantically consistent? If so, I think this distinction should be explicitly discussed and empirically demonstrated.
-
What is the number of iterations used in the proposed method? 1) As the authors describe the optimization as being conducted “in an iterative adversarial manner,” I believe the effect of the iteration number on attack performance is an important experiment, particularly for supporting the claimed “collaborative” nature of the method. 2) I checked the pseudocode, and T seems to be the number of iterations (t = 1 to T), but in the Input part it is written as “PGD budget T,” which I suppose should be the perturbation budget ϵ? Please clarify the exact meaning of T and provide further discussion or experiments on how the number of iterations affects the attack results.
7 comments
W1 / Q1 (continued, Part 2/2)
2. Measurements on our task (Appendix C.9 of the revised version, Table 15). We evaluated the three methods with their official code on our three victim models. The evaluation uses a fixed random sample of 200 A-OKVQA questions, excluding those that each model answers incorrectly on clean inputs. The attacks follow the official settings: perturbation budget ε=16/255, 300 steps, and transfer from ensembles of CLIP surrogate models. The target concept of V-Attack is set to our pre-selected wrong option.
| Access | Method | Metric | Qwen2.5-VL-7B | InternVL3.5-8B | LLaVA-OV-1.5-8B |
|---|---|---|---|---|---|
| Black-box transfer | M-Attack | ASR | 29.4% | 30.8% | 25.3% |
| FOA-Attack | ASR | 30.6% | 29.6% | 24.7% | |
| V-Attack | ASR | 35.0% | 30.8% | 23.6% | |
| V-Attack | THSR | 23.3% | 24.3% | 14.9% | |
| White-box | VL-ReasonHijack | ASR | 98.1% | 98.3% | 99.5% |
| VL-ReasonHijack | THSR | 97.7% | 97.8% | 98.9% |
VL-ReasonHijack results are from Table 1.
The FOA-Attack paper reports that, on its image description task, M-Attack and FOA-Attack achieve success rates of 52.6% and 70.7% on the same Qwen2.5-VL-7B. On our task, their answer flip rates are only 29.4% and 30.6%. Replacing the semantics of an entity in the image does not necessarily change the answer to a question that requires multi-step reasoning, and it is difficult for such attacks to steer the answer reliably to a specified wrong option. Even when V-Attack sets the target concept to the wrong option, only 14.9%–24.3% of the samples output that option.
3. Revision. Section 2 of the revised version adds M-Attack and FOA-Attack and explains how the three works differ from ours in threat model, task, and success criterion. The Baselines paragraph in Section 4.1 also states why the three works are reported separately rather than in the main table. V-Attack is also discussed in Section 2 and Table 6 in Appendix B.1 of the revised version. The measurements above are added to the revised version as Appendix C.9, and the entries of the three works in Table 6 are updated accordingly.
W2: Reorganizing the experimental section and moving key evidence into the main paper
We have reorganized the experimental section. Of the four appendix analyses cited in the original Main Results, THSR has been moved into the main table, the significance tests remain in the appendix, and the two supplementary diagnostics, the image audit and cross-model transfer, are no longer cited in the main results. The five consecutive appendix references in the experimental setup have also been merged into a single reference, reducing the appendix references in the experimental section from 19 to 11.
1. Main Results (Section 4.2). The revised section presents the following results and analyses:
- THSR is merged into Table 1. Each VL-ReasonHijack cell reports ASR / THSR / Detection Rate, and the separate THSR table in the original appendix has been removed. The text now reports the difference between ASR and THSR directly. On A-OKVQA and ScienceQA, THSR is within 0.9 points of ASR for every victim model, while the gap is larger on MathVista and especially MathVerse, e.g., 63.0% ASR versus 55.0% THSR for Qwen2.5-VL on MathVerse. The text also gives the ASR improvement of VL-ReasonHijack over the strongest baseline (4.9–15.0 points).
- The image audit and cross-model transfer are supplementary diagnostics and are no longer cited in this section. They remain in Appendices C.4 and C.7.
- The significance tests remain in Appendix C.2, and the text states the conclusion: VL-ReasonHijack achieves significantly higher ASR in all 36 comparisons with the three Stop Reasoning settings.
- A one-sentence result on the newer Qwen3.5-4B is added (in response to another reviewer), with details in Appendix C.8.
2. Section order. The ablation study (Section 4.3, Table 2) now precedes the human evaluation of question rewriting (Section 4.4, Table 3), so the text proceeds as "overall effectiveness → component contributions → rewrite validity → visual dependence analysis". The ablation section also directly reports the sensitivity results. On 431 A-OKVQA samples, disabling question rewriting lowers ASR from 95.1% to 83.8%, and the default rewrite interval of 50 and suppression weight of 0.3 achieve the highest ASR among the tested settings.
3. Visual dependence analysis (Section 4.5, Table 4). The original version pointed to the likelihood analysis in the appendix with a single sentence. The revised version reports its results directly in the text. On 96 successfully attacked Qwen2.5-VL samples, the adversarial image lowers the NLL of the target false-premise spans by 0.075 on average, with the largest decrease on VD samples (0.122). The conclusion that THSR increases monotonically from TD to TL to VD is now supported by specific numbers (e.g., from 39.1% to 46.7% and 66.3% on Qwen2.5-VL).
Q2: Iteration count, the meaning of T, and the perturbation budget
Meaning and value of . is the maximum number of PGD iterations, not the perturbation budget. We use (Section 4.1 of the revised version), and the perturbation budget is a separate parameter, . The phrase "PGD budget " could be confused with the perturbation budget, so the input line of Algorithm 1 in the revised version now reads "maximum number of PGD iterations , perturbation budget ". The algorithm now runs over , uses for the question after selection in the current iteration, and aligns the time indices of image updates and returned values with Sections 3.2 and 3.3 (Appendix A.4, Algorithm 1).
Effect of the number of iterations. In our method, iterations alternate between question rewriting and image optimization. The question is rewritten every fixed number of steps, and the image then continues to be optimized for the new question. With a fixed total number of steps, a smaller rewrite interval gives more alternation rounds but fewer image optimization steps after each question change.
(1) Question rewriting frequency (Appendix C.1, Figure 3). On 431 A-OKVQA samples with Qwen2.5-VL, we fix the maximum number of image optimization steps at 120 and vary the question rewriting interval to compare no rewriting with 1, 2, 4, and 6 rewrite rounds. The interval is the number of image optimization steps between consecutive rewrite rounds. The table also lists the corresponding number of rewrite rounds for each setting.
| Rewrite interval (image optimization steps) | No rewriting | 85 | 50 (default) | 25 | 15 |
|---|---|---|---|---|---|
| Corresponding rewrite rounds | 0 | 1 | 2 | 4 | 6 |
| ASR (%) | 83.8 | 92.8 | 95.1 | 91.2 | 91.9 |
ASR is 83.8% without question rewriting, rises to 92.8% with one rewrite round, and increases further to 95.1% with two rounds. Shortening the interval to allow four or six rounds instead lowers ASR to 91.2% and 91.9%, respectively. Under the same image optimization budget, two rewrite rounds outperform a single round, while more frequent rewriting brings no further improvement.
(2) Image optimization steps. The main experiments record the step at which each sample stops early. The table below gives, from the same runs, the proportion of samples that have hit the target by each step. The last column is the final THSR in Table 1 (Qwen2.5-VL, %):
| Dataset | By step 95 | By step 105 | By step 115 | Final THSR (Table 1) |
|---|---|---|---|---|
| A-OKVQA | 78.2 | 89.5 | 95.0 | 97.7 |
| ScienceQA | 65.0 | 74.2 | 80.2 | 84.1 |
| MathVista | 40.7 | 54.3 | 62.7 | 66.3 |
| MathVerse | 20.4 | 39.4 | 49.9 | 55.0 |
The early-stop check is performed only in the late stage of optimization, so the first column gives the cumulative hit proportion at the first check. The last rewrite occurs at step 50 for both task types, and only the image is optimized afterwards. From the first check to the end at step 120, the hit proportion rises by a further 19.5, 19.1, 25.6, and 34.6 points on the four datasets, so image optimization after the last rewrite still yields many additional target hits. LLaVA-OneVision and InternVL3.5 show the same trend.
Thank you for pointing out the issues regarding the comparison with recent representative methods, the organization of the experimental section, and the description of the iteration settings. We have added evaluations of the three attack methods you suggested and clarify each point below. Text revisions are highlighted in blue.
W1 / Q1: Comparison and discussion of recent adversarial attacks (Part 1/2)
We did not compare with these three works in the submission mainly because the attack settings differ. Our method and the main baseline Stop Reasoning (Stop Reasoning! When Multimodal LLM with Chain-of-Thought Reasoning Meets Adversarial Image, COLM 2024) both optimize the adversarial image directly with the victim model's gradients. M-Attack (A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1, NeurIPS 2025), FOA-Attack (Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment, NeurIPS 2025), and V-Attack (V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs, CVPR 2026) instead generate adversarial images with CLIP surrogate models and transfer them to closed-source models. In response to your comment, we evaluated them on our task with their official code, and their answer flip rates range from 23.6% to 35.0%. By changing image semantics, these methods influence model outputs and may also change the generated reasoning. Our distinction lies in explicit control over the attack target: we form a target chain from an image-specific false visual premise, reasoning that supports a specified wrong option, and the final answer, and optimize the image-question pair toward this chain.
1. Settings and attack targets.
M-Attack and FOA-Attack make the adversarial image close to a target image in the feature space of surrogate vision encoders, and V-Attack replaces one object concept in the image with another through value features. Their attack targets are defined only by image semantics and are independent of the specific question, options, and reasoning. Our work studies multi-step reasoning QA, including the knowledge-intensive A-OKVQA and ScienceQA and the visual mathematical reasoning benchmarks MathVista and MathVerse (Section 4.1). In these tasks: