- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies multimodal attacks where an image provides information needed for a harmful answer. It introduces Visual Exclusivity, a benchmark called VE-Safety, and an attack framework called MM-Plan. The framework uses reinforcement learning to generate complete multi-turn plans with text queries and image operations. Experiments evaluate the attacks across eight multimodal models.
Critical strengths
-
The paper highlights a useful distinction between images that carry written instructions and images that provide task-specific information. This offers a valuable perspective on multimodal safety.
-
The proposed planner coordinates text queries and image operations across multiple turns. This is a meaningful extension beyond static image attacks.
-
The method outperforms the compared baselines across eight target models. Improvements remain when the final FigStep operation is removed, which strengthens the empirical contribution.
Critical weaknesses
-
The main success metric does not verify image-specific correctness. The final-response judge receives the harmful goal and response, but not the original image. A generic harmful answer or an answer with incorrect visual details could therefore count as successful. This weakens the connection between the main results and Visual Exclusivity.
-
The formal definition is stronger than its practical validation. Human screening and selected caption tests provide useful evidence, but cannot establish that every allowed description misses necessary information. The paper acknowledges this limitation, yet the benchmark’s connection to the formal criterion remains incomplete.
-
The role of visual operations is not fully explained. Cropping, masking, and blurring may improve attacks by hiding harmful context or by presenting useful visual details. The current ablations do not clearly separate these effects.
-
The generalization evidence remains limited in scope. The paper tests unseen queries, new images, and cross-model transfer, which is useful. However, it does not systematically hold out safety categories, image types, or task templates. Broader generalization is therefore still uncertain.
Questions for authors
-
Among successful attacks, what fraction of final answers are both factually correct and dependent on details in the original image?
-
Have you tested swapped or irrelevant images while keeping the attack text fixed? Such results would help determine whether success depends on the intended visual information.
4 comments
From a reviewer's perspective, the paper still has a few key limitations:
-
The idea of using images to express the attack content is not novel.
-
The core premise of the approach is "Image-as-Basis," meaning it relies on images to supply visual details. This renders the method highly constrained in applicability. For more complex attacks that images can not precisely express, the framework's effectiveness may drop significantly. This suggests it may address only a narrow slice of multimodal safety rather than serving as a general-purpose multimodal jailbreak paradigm.
-
The authors employ Group Relative Policy Optimization (GRPO) to train an open-weight model (Qwen3-VL-4B) in learning long-horizon multimodal attack plans. Reinforcement learning in multi-turn dialogues and long-horizon planning frequently encounters issues such as reward hacking (difficulties in reward function design) and training instability. Relying on a judge model to provide composite rewards may introduce judge bias, and the computational overhead for multi-turn agent training is substantial.
I would like to maintain the score to weak accept.
We thank the reviewer for maintaining the positive assessment. We share our perspective on each point below and will reflect them in the revision.
-
Novelty. We agree that image-based attacks are not new. Our claim, however, was never the use of images. The contribution is the representation-relative criterion (Definition 2.1), the VE-Safety benchmark screened to it, and the grounding-aware evaluation that verifies a success actually uses the image. To our knowledge no prior benchmark screens each instance for this information gap or audits grounding, so the measurement framework is new even though image attacks are not.
-
Scope. We agree Image-as-Basis is a targeted setting, and we do not claim a general-purpose jailbreak paradigm. We see this focus as deliberate. Image-as-Basis is precisely the case that OCR and caption based filters miss by design and that prior benchmarks did not isolate, so a diagnostic for this specific and previously unmeasured blind spot is useful even though it is not universal, and it complements rather than replaces Image-as-Wrapper work. We will state this scope explicitly.
-
Reinforcement learning challenges. These are general concerns for multi-turn RL, and the paper already provides evidence on each.
- Stability: the progress and goal-drift shaping terms stabilize training, and variance across three seeds is small (Table 16, standard deviation 1.0 and 0.8).
- Judge bias: cross-judge agreement is high and the human audit corroborates the automated judge (Table 4), and the grounding re-evaluation in our rebuttal uses an independent model family.
- Overhead: we deliberately use a small open-weight 4B planner at roughly 30 dollars per target, which is modest for this class of method.
- Reward hacking: the goal-drift penalty together with the human and grounding audits guard against degenerate strategies.
We appreciate the constructive review and are happy to take any further questions during the discussion period.
Revised PDF uploaded
Dear Reviewer cJHt,
We have uploaded the revised PDF. All promised results and statements are incorporated, with the new additions marked in blue. The table below maps each point to its location.
| Point | Change incorporated | Location in revised PDF |
|---|---|---|
| W1 and Q1. Correct image use among successes | Grounded fraction of judge positive successes (92.5% Qwen, 95.9% Sonnet) | Table 5 grounded fraction column (Sec 4.2); Appx D.15 |
| W3 and Q2. Swapped image and visual operations | Unrelated scene control, and the crop confound stated explicitly | Table 6 (Sec 4.2); Sec 4.3 |
| W2. Formal criterion and its validation | Caption conditions test the specified family; screening certifies the procedure and the tested representations, not every description | Sec 2; caption evaluation in Appx B.2 |
| W4. Generalization | Categories absent from training, unseen goals, cross target, and cross benchmark, with the hold out limit stated | Table 23 and Table 22 (Appx D.4); Table 20 and Table 21; HarmBench Table 19 (Appx D.3); Limitations (Appx E) |
Response to Reviewer cJHt
We thank the reviewer for asking about the answers that are counted as successes. We checked those answers against their source images and changed the task image while holding the user messages fixed.
W1 and Q1. Correct use of the image among successes
The image aware judge accepted 161 of 174 original MM-Plan successes on Qwen and 142 of 148 on Sonnet. The fractions you ask about are 92.5 on Qwen and 95.9 on Sonnet.
GPT-6 Astra reviewed each original success using the source image together with the goal and the final answer. An answer had to use an instance specific visual fact correctly to address the goal. Two annotators independently labeled the same 200 responses on that criterion. GPT-6 Astra agreed with the human labels in 98.5 percent of cases.
Table A. Image aware re-evaluation on VE-Safety.
W3 and Q2. Replacing the image with an unrelated scene
Yes. We replaced the task image with an unrelated scene while holding every user message fixed. Each condition used a fresh conversation and newly generated target replies. We reran the original image control in the same evaluation.
Table B. MM-Plan with the user messages held fixed.
With an unrelated scene the ordinary ASR was 8.4 on Qwen and 4.4 on Sonnet. None of those answers passed the check against the original image and goal. The grounded success that remains is tied to the original image. The visual operation ablation in Table 6 addresses a different part of your question. Disabling all image operations lowered Qwen ASR from 54.4 to 36.0. Cropping changes both the visible detail and the surrounding context. The relative size of these two effects remains open and we state this.
W2. The formal criterion and its validation
Definition 2.1 is relative to a family of descriptions and a budget. Our screening and the caption conditions in Table B test that specified family. The 512 token caption yields zero grounded success on both targets within that budget. This evidence certifies the screening procedure and the tested representations. It does not certify every allowed description. We make this limit explicit in Section 2. Other descriptions could preserve more information and VE status can change with the family and the budget.
W4. Generalization to held out categories, image types, and templates
The training set is 80 instances and the 320 test instances are held out, which is 73 percent of VE-Safety. We report generalization along several axes.
Categories absent from training. Because the split is random by instance, four categories with 46 instances happen to receive no training or validation exposure. On these 46 instances MM-Plan reaches 52.2 on Qwen3-VL-8B and 43.5 on Claude 4.5 Sonnet, compared with 54.7 and 46.7 on the remaining 274 instances. The planner transfers to categories it never saw in training.
Unseen goals and images. Table 19 evaluates 214 test instances whose goals and images never appear in training. MM-Plan reaches 53.3 on Qwen3-VL-8B and 45.3 on Claude 4.5 Sonnet. The drop from the seen split is 3.3 and 2.8 points.
Cross target transfer. Table 18 trains on one victim and attacks another without retraining. MM-Plan reaches 50.6 when trained on Claude 4.5 Sonnet and evaluated on Qwen3-VL-8B, and 29.7 in the reverse direction.
Cross benchmark. On the HarmBench multimodal subset we trained on 30 instances and tested on 30 held out instances, which measures in domain performance after training (Appendix D.3). MM-Plan reached 63.3 on Qwen3-VL-8B and 36.7 on Claude 4.5 Sonnet.
A systematic hold out of image types and task templates remains future work.
We will add these results and the tables above to the revised PDF shortly.