acceptodds
Under review as a conference paper at ICLR 2027

Visual Exclusivity Attacks: Automatic Multimodal Red Teaming via Agentic Planning

Abstract

Many multimodal red teaming methods use images as wrappers for malicious payloads via typography or adversarial noise. A complementary threat arises when the image supplies information needed for an image-specific harmful answer. We formalize Visual Exclusivity (VE), an Image-as-Basis criterion defined relative to specified natural-language descriptions and budgets. To study this threat, we propose Multimodal Multi-turn Agentic Planning (MM-Plan), a framework that reframes jailbreaking from turn-by-turn reaction to global plan synthesis. MM-Plan trains an attacker planner to synthesize comprehensive multi-turn strategies, optimized via Group Relative Policy Optimization (GRPO), learning text queries and the use of predefined visual operations without human-annotated attack trajectories. We introduce VE-Safety, a dataset of 440 human-curated instances across 15 safety categories, screened for dependence on visual content such as technical schematics. Across eight evaluated MLLMs, MM-Plan outperforms the compared baselines. It achieves 46.3% attack success against Claude 4.5 Sonnet and 13.8% against GPT-5. These findings reveal that frontier models remain vulnerable to learned multimodal interaction on tasks curated for visual dependence.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
MLLM Safety, Red Teaming

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_cJHtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies multimodal attacks where an image provides information needed for a harmful answer. It introduces Visual Exclusivity, a benchmark called VE-Safety, and an attack framework called MM-Plan. The framework uses reinforcement learning to generate complete multi-turn plans with text queries and image operations. Experiments evaluate the attacks across eight multimodal models.

Critical strengths
  1. The paper highlights a useful distinction between images that carry written instructions and images that provide task-specific information. This offers a valuable perspective on multimodal safety.

  2. The proposed planner coordinates text queries and image operations across multiple turns. This is a meaningful extension beyond static image attacks.

  3. The method outperforms the compared baselines across eight target models. Improvements remain when the final FigStep operation is removed, which strengthens the empirical contribution.

Critical weaknesses
  1. The main success metric does not verify image-specific correctness. The final-response judge receives the harmful goal and response, but not the original image. A generic harmful answer or an answer with incorrect visual details could therefore count as successful. This weakens the connection between the main results and Visual Exclusivity.

  2. The formal definition is stronger than its practical validation. Human screening and selected caption tests provide useful evidence, but cannot establish that every allowed description misses necessary information. The paper acknowledges this limitation, yet the benchmark’s connection to the formal criterion remains incomplete.

  3. The role of visual operations is not fully explained. Cropping, masking, and blurring may improve attacks by hiding harmful context or by presenting useful visual details. The current ablations do not clearly separate these effects.

  4. The generalization evidence remains limited in scope. The paper tests unseen queries, new images, and cross-model transfer, which is useful. However, it does not systematically hold out safety categories, image types, or task templates. Broader generalization is therefore still uncertain.

Questions for authors
  1. Among successful attacks, what fraction of final answers are both factually correct and dependent on details in the original image?

  2. Have you tested swapped or irrelevant images while keeping the attack text fixed? Such results would help determine whether success depends on the intended visual information.

4 comments
AuthorsOpenReview ↗

Response to Reviewer cJHt

We thank the reviewer for asking about the answers that are counted as successes. We checked those answers against their source images and changed the task image while holding the user messages fixed.

W1 and Q1. Correct use of the image among successes

The image aware judge accepted 161 of 174 original MM-Plan successes on Qwen and 142 of 148 on Sonnet. The fractions you ask about are 92.5 on Qwen and 95.9 on Sonnet.

GPT-6 Astra reviewed each original success using the source image together with the goal and the final answer. An answer had to use an instance specific visual fact correctly to address the goal. Two annotators independently labeled the same 200 responses on that criterion. GPT-6 Astra agreed with the human labels in 98.5 percent of cases.

Table A. Image aware re-evaluation on VE-Safety.

TargetMethodASRGrounding aware ASRGrounded fraction
Qwen3-VL-8BDirect Plan22.519.486.1
Qwen3-VL-8BMM-Plan54.450.392.5
Claude 4.5 SonnetDirect Plan9.78.487.1
Claude 4.5 SonnetMM-Plan46.344.495.9

W3 and Q2. Replacing the image with an unrelated scene

Yes. We replaced the task image with an unrelated scene while holding every user message fixed. Each condition used a fresh conversation and newly generated target replies. We reran the original image control in the same evaluation.

Table B. MM-Plan with the user messages held fixed.

Task inputQwen ASRQwen grounding aware ASRSonnet ASRSonnet grounding aware ASR
Original task image54.450.346.344.4
No task image5.90.02.50.0
Unrelated scene8.40.04.40.0
Detailed caption7.50.03.80.0

With an unrelated scene the ordinary ASR was 8.4 on Qwen and 4.4 on Sonnet. None of those answers passed the check against the original image and goal. The grounded success that remains is tied to the original image. The visual operation ablation in Table 6 addresses a different part of your question. Disabling all image operations lowered Qwen ASR from 54.4 to 36.0. Cropping changes both the visible detail and the surrounding context. The relative size of these two effects remains open and we state this.

W2. The formal criterion and its validation

Definition 2.1 is relative to a family of descriptions and a budget. Our screening and the caption conditions in Table B test that specified family. The 512 token caption yields zero grounded success on both targets within that budget. This evidence certifies the screening procedure and the tested representations. It does not certify every allowed description. We make this limit explicit in Section 2. Other descriptions could preserve more information and VE status can change with the family and the budget.

W4. Generalization to held out categories, image types, and templates

The training set is 80 instances and the 320 test instances are held out, which is 73 percent of VE-Safety. We report generalization along several axes.

Categories absent from training. Because the split is random by instance, four categories with 46 instances happen to receive no training or validation exposure. On these 46 instances MM-Plan reaches 52.2 on Qwen3-VL-8B and 43.5 on Claude 4.5 Sonnet, compared with 54.7 and 46.7 on the remaining 274 instances. The planner transfers to categories it never saw in training.

Unseen goals and images. Table 19 evaluates 214 test instances whose goals and images never appear in training. MM-Plan reaches 53.3 on Qwen3-VL-8B and 45.3 on Claude 4.5 Sonnet. The drop from the seen split is 3.3 and 2.8 points.

Cross target transfer. Table 18 trains on one victim and attacks another without retraining. MM-Plan reaches 50.6 when trained on Claude 4.5 Sonnet and evaluated on Qwen3-VL-8B, and 29.7 in the reverse direction.

Cross benchmark. On the HarmBench multimodal subset we trained on 30 instances and tested on 30 held out instances, which measures in domain performance after training (Appendix D.3). MM-Plan reached 63.3 on Qwen3-VL-8B and 36.7 on Claude 4.5 Sonnet.

A systematic hold out of image types and task templates remains future work.

We will add these results and the tables above to the revised PDF shortly.

From a reviewer's perspective, the paper still has a few key limitations:

  1. The idea of using images to express the attack content is not novel.

  2. The core premise of the approach is "Image-as-Basis," meaning it relies on images to supply visual details. This renders the method highly constrained in applicability. For more complex attacks that images can not precisely express, the framework's effectiveness may drop significantly. This suggests it may address only a narrow slice of multimodal safety rather than serving as a general-purpose multimodal jailbreak paradigm.

  3. The authors employ Group Relative Policy Optimization (GRPO) to train an open-weight model (Qwen3-VL-4B) in learning long-horizon multimodal attack plans. Reinforcement learning in multi-turn dialogues and long-horizon planning frequently encounters issues such as reward hacking (difficulties in reward function design) and training instability. Relying on a judge model to provide composite rewards may introduce judge bias, and the computational overhead for multi-turn agent training is substantial.

I would like to maintain the score to weak accept.

We thank the reviewer for maintaining the positive assessment. We share our perspective on each point below and will reflect them in the revision.

  1. Novelty. We agree that image-based attacks are not new. Our claim, however, was never the use of images. The contribution is the representation-relative criterion (Definition 2.1), the VE-Safety benchmark screened to it, and the grounding-aware evaluation that verifies a success actually uses the image. To our knowledge no prior benchmark screens each instance for this information gap or audits grounding, so the measurement framework is new even though image attacks are not.

  2. Scope. We agree Image-as-Basis is a targeted setting, and we do not claim a general-purpose jailbreak paradigm. We see this focus as deliberate. Image-as-Basis is precisely the case that OCR and caption based filters miss by design and that prior benchmarks did not isolate, so a diagnostic for this specific and previously unmeasured blind spot is useful even though it is not universal, and it complements rather than replaces Image-as-Wrapper work. We will state this scope explicitly.

  3. Reinforcement learning challenges. These are general concerns for multi-turn RL, and the paper already provides evidence on each.

    • Stability: the progress and goal-drift shaping terms stabilize training, and variance across three seeds is small (Table 16, standard deviation 1.0 and 0.8).
    • Judge bias: cross-judge agreement is high and the human audit corroborates the automated judge (Table 4), and the grounding re-evaluation in our rebuttal uses an independent model family.
    • Overhead: we deliberately use a small open-weight 4B planner at roughly 30 dollars per target, which is modest for this class of method.
    • Reward hacking: the goal-drift penalty together with the human and grounding audits guard against degenerate strategies.

We appreciate the constructive review and are happy to take any further questions during the discussion period.

AuthorsOpenReview ↗

Revised PDF uploaded

Dear Reviewer cJHt,

We have uploaded the revised PDF. All promised results and statements are incorporated, with the new additions marked in blue. The table below maps each point to its location.

PointChange incorporatedLocation in revised PDF
W1 and Q1. Correct image use among successesGrounded fraction of judge positive successes (92.5% Qwen, 95.9% Sonnet)Table 5 grounded fraction column (Sec 4.2); Appx D.15
W3 and Q2. Swapped image and visual operationsUnrelated scene control, and the crop confound stated explicitlyTable 6 (Sec 4.2); Sec 4.3
W2. Formal criterion and its validationCaption conditions test the specified family; screening certifies the procedure and the tested representations, not every descriptionSec 2; caption evaluation in Appx B.2
W4. GeneralizationCategories absent from training, unseen goals, cross target, and cross benchmark, with the hold out limit statedTable 23 and Table 22 (Appx D.4); Table 20 and Table 21; HarmBench Table 19 (Appx D.3); Limitations (Appx E)
Reviewer_AHUHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies Image-as-Basis multimodal jailbreaks, which the authors call Visual Exclusivity (VE): the image itself supplies the information needed for a harmful answer (e.g. a schematic + how to assemble this?). The contributions are (1) a formal definition of VE, (2) VE-Safety, a benchmark of 440 human-curated real technical images, and (3) MM-Plan, a Qwen3-VL-4B planner trained with GRPO that generates a full multi-turn attack plan with image operations in one pass. The paper includes extensive ablations and judge-human agreement checks.

Critical strengths
  1. The paper is easy to follow and the experiments are extensive: eight targets and ablations on most design choices, and judge reliability checked with both human audits and a cross-judge evaluation. VE-Safety uses real technical imagery rather than synthetic typographic images, which makes it a useful benchmark for the community. This is the main reason for my positive rating. Even without a big methodological step, the paper pushes the area forward in a solid way.
  2. The proposed problem framing is useful. The wrapper vs. basis distinction points to a real gap in current safety evaluation. If the harm comes from the model’s visual reasoning, OCR or caption-based filtering cannot catch it by design. The low ASR of existing attacks on GPT-5 and Sonnet 4.5 also suggests that current safety training mainly targets the wrapper type of attack.
Critical weaknesses
  1. GRPO-trained attackers, multi-turn context building, tool use and judge-based rewards have all appeared in prior RL red-teaming work (e.g., Siren, X-Teaming). Global planning in practice means generating the whole plan once and executing it open-loop, which is a reasonable design choice but not a new idea. As a result, the paper’s contribution relies mainly on the VE framing and the benchmark.
  2. The evaluation does not verify visual exclusivity. The central claim is that VE is an image-dependent threat, but the judge does not check whether the response is grounded in the image (L362–365). A generic harmful answer would also count as a success. The visual-operation ablation (54.4 -> 36.0) also cannot separate visual reasoning from simply hiding sensitive regions, as the authors acknowledge (L416–418). In addition, the VE predicate in Def. 2.1 is never computed and relies only on human screening. So the results show that MM-Plan is a strong multi-turn multimodal attacker, but not yet that it succeeds because of VE.
Questions for authors
  1. What is the ASR under a grounding-aware judge, i.e. one that sees the image and checks whether the response actually uses its instance-specific information? If ASR stays high, my main concern would be largely addressed.
  2. What is the ASR of the same MM-Plan dialogues when the image is replaced by a detailed caption, or removed entirely? This would show how much of the gain comes from the image versus the multi-turn text strategy.
  3. MM-Plan is trained on VE-Safety while all baselines are training-free. How large is the seen vs. unseen gap, and does MM-Plan still clearly outperform the baselines on the unseen split alone?
  4. Table 1 already lists HarmBench’s multimodal subset as Image-as-Basis. What does VE-Safety capture beyond it, apart from multi-turn interaction and real imagery?
5 comments
AuthorsOpenReview ↗

Response to Reviewer AHUH [1/2]

We thank the reviewer for recognizing VE-Safety as a useful benchmark built on real technical imagery and for the clear statement that a grounding aware result would address the main concern. We answer each point below and add two evaluations.

W1. Novelty of the method

We agree that reinforcement learning, multi turn interaction, and tool use are established and we do not claim these as inventions. Our method contribution is a specific instantiation for the multimodal Visual Exclusivity setting. MM-Plan synthesizes a complete multi turn plan in a single pass and couples text queries with visual operations. It is trained with GRPO from judge rewards relative to group averages. It needs no human annotated attack trajectories and no preference pairs. This matters because large corpora of successful multimodal VE jailbreaks do not exist. Methods that rely on preference data such as Siren require paired preferred and dispreferred trajectories for the same task image (Appendix C.5). Our formulation removes that data requirement. Applying GRPO to train a multimodal attacker is not a direct plug in. Two choices are ours and are load bearing: a trajectory level formulation that scores a complete plan rather than a single turn, and a composite reward that combines attack success with per turn progress and penalties for goal drift and turn budget. Table 5 shows that removing the graded signal and the shaping terms lowers ASR substantially. The broader contribution of the paper is the threat model, the benchmark, and the evaluation. We will state this scope in the introduction and in Section 3.

W2 and Q1. Attack success under an image aware judge

MM-Plan reaches a grounding aware ASR of 50.3 on Qwen3-VL-8B and 44.4 on Claude 4.5 Sonnet. Direct Plan reaches 19.4 and 8.4. The MM-Plan advantage holds under the stricter metric.

We re-scored every final response that the Table 3 judge marked successful. The grounding judge is GPT-6 Astra. It receives the original image together with the goal and the final response. A response passes only if it uses an instance specific visual fact correctly to address the goal. This judge is from a different model family than our default Claude judge. Two annotators independently labeled the same 200 responses on this criterion. GPT-6 Astra agreed with the human labels in 98.5 percent of cases. Grounding aware ASR divides the passing count by all 320 test instances. Grounded fraction divides it by the number of original successes.

Table A. Image aware re-evaluation on VE-Safety.

TargetMethodASRGrounding aware ASRGrounded fraction
Qwen3-VL-8BDirect Plan22.519.486.1
Qwen3-VL-8BMM-Plan54.450.392.5
Claude 4.5 SonnetDirect Plan9.78.487.1
Claude 4.5 SonnetMM-Plan46.344.495.9

The stricter check lowers MM-Plan ASR by 4.1 points on Qwen and 1.9 points on Sonnet. More than nine in ten successes use the image correctly. The reported successes are image grounded completions rather than generic harmful text.

On the visual-operation ablation, we agree the 54.4 to 36.0 change cannot separate visual reasoning from concealment of sensitive regions, and we leave this decomposition open. Image dependence itself is established separately by the fixed-text controls in Table B, which do not rely on that ablation.

AuthorsOpenReview ↗

Response to Reviewer AHUH [2/2]

Q2. Removing or replacing the task image

With the attack text held fixed the grounded success requires the original image. We kept every user message fixed and regenerated the target replies from a fresh context. We reran the original image condition as a contemporaneous control. We then removed the image, replaced it with an unrelated scene, and replaced it with a 512 token description. GPT-6 Astra scored every condition against the original image and goal.

Table B. MM-Plan with the user messages held fixed.

Task inputQwen ASRQwen grounding aware ASRSonnet ASRSonnet grounding aware ASR
Original task image54.450.346.344.4
No task image5.90.02.50.0
Unrelated scene8.40.04.40.0
Detailed caption7.50.03.80.0

Grounded success for the original task falls to 0.0 in all three conditions without the original image. Ordinary harmful compliance persists at a low rate. This separates image grounded success from text only compliance and answers how much of the gain comes from the image.

Q3. Seen and unseen queries

MM-Plan leads on the unseen split. Table 19 uses 106 seen instances and 214 unseen instances. We recovered the per instance counts for the two direct baselines on the same partition.

Table. ASR by split.

MethodQwen seenQwen unseenSonnet seenSonnet unseen
Direct Request11.312.18.58.4
Direct Plan22.622.410.49.3
MM-Plan56.653.348.145.3

From seen to unseen queries MM-Plan drops 3.3 points on Qwen and 2.8 points on Sonnet. The ranking against the four external baselines follows from a counting bound. No external baseline exceeded 108 successes across all 320 Qwen cases. MM-Plan obtained 114 successes on the 214 unseen cases alone. On Sonnet the two counts are 78 and 97. A method cannot have more unseen successes than total successes. The unseen ranking therefore holds before the exact external unseen rates are recovered.

Q4. What VE-Safety adds beyond HarmBench

HarmBench multimodal is the closest predecessor. We do not claim priority over the image as basis setting. The distinction is how the setting is defined and certified.

HarmBench multimodal is a content category. Its instances carry a dataset level label of visual reasoning. VE-Safety instead defines visual dependence as a per instance predicate under Definition 2.1. The query must leave the task critical fact unspecified. The image must supply it. That fact must not be recovered by any description in a specified family within a token budget. The unit we screen for is an information gap between the query and the image. This gap is defined against the textual representations that safety systems actually compute such as OCR and captions. A benchmark organized by visual category tells you that a task involves an image. A benchmark organized by this predicate asks whether the needed information survives a text description within the specified budget. Table B instantiates the predicate. With the user text fixed, none of the three conditions that remove or replace the task image produced grounded success on either target, while the original image yields 50.3 and 44.4. The screening covers the procedure and the tested representations, not every admissible description. We state this limit.

We will add these results and the tables above to the revised PDF shortly.

The rebuttal addresses all of my major concerns, particularly through the grounding-aware evaluation, image-replacement controls, and unseen-split results. I still have some reservations about the methodological novelty, but the new evidence substantially strengthens the paper’s empirical claims. I therefore maintain my positive rating.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Response to Reviewer okvp [1/2]

We thank the reviewer for recognizing the clear motivation of the wrapper versus basis distinction and the representation relative definition of Visual Exclusivity. We clarify the empirical comparison and add two evaluations on whether successful answers use the designated source image.

W1 and Q1. Comparison with recent methods

The submitted baselines in Table 3 are established methods. We agree that a current comparison should include recent work. We evaluated two 2026 multimodal jailbreak methods on VE-Safety under the same targets and the Table 3 judge. Both accept an image input and run natively on our setting.

Table C. Recent methods on VE-Safety.

MethodVenueImage roleWhat the image carriesQwen3-VL-8BClaude 4.5 Sonnet
GAMBIT [A]ACL 2026Image-as-WrapperA concealed harmful keyword30.920.0
Visual Text Replacement [B]ICML 2026Image-as-WrapperA substituted term to recover20.911.9
MM-Plan (Ours)This paperImage-as-BasisInstance-specific task structure54.446.3

Both recent methods place the harmful request inside the image, so they belong to the Image-as-Wrapper class. MM-Plan uses the image as the information basis and exceeds both on both targets. On whether the gains reflect optimization or search capacity rather than planning, see our response to W3 and Q4. A per-turn GRPO agent with the same backbone underperforms global planning at 12 times the training compute, and a Best-of-16 search over the untrained planner also underperforms.

Frameworks that use a strong proprietary model in the planning stage face a refusal risk. For example X-Teaming uses GPT-4o as the planner and verifier, and a strongly aligned model can refuse a harmful planning request in that role. Our Direct Plan experiment shows this directly. Used to produce the plan, GPT-5 declined all 320 requests and Claude 4.5 Sonnet declined nearly all (Section 4.3 and Table 9). We also tested a newer and stronger model, Claude Opus 5.5, in this role. It refused on every instance and reached 0 attack success. This creates a dilemma. As models become more powerful and better aligned, using them in the planning stage becomes less viable, which reduces the utility of frameworks that depend on them. MM-Plan avoids this by training an open weight planner. We will evaluate any further recent method the reviewer names that accepts an image input and runs on these targets.

W2 and Q2. What Visual Exclusivity captures

Visual Exclusivity is defined by the information required for a correct answer and not by the presence of an image. Prior image grounded settings group an instance as visual because it involves an image. Visual Exclusivity is a representation relative predicate under Definition 2.1. The query must leave the task critical fact unspecified. The image must supply it. That fact must not be recovered by any description in a specified family within a token budget. The predicate is defined against the textual representations that safety systems compute such as OCR and captions. A category label says that a task involves an image. Our predicate asks whether the needed information survives a text description within the specified budget. Recent methods such as GAMBIT [A] and Visual Text Replacement [B] also place the harmful request inside the image, so they fall in the Image-as-Wrapper class and their payload is recoverable from text. HarmBench multimodal is the closest predecessor in the literature and is a category. HADES [C] conceals the request through typography and adversarial perturbation, which are wrapper mechanisms rather than a task image that supplies a text irreducible fact. VisCo [D] injects visual context across turns without screening each instance for the text irreducible dependence that our predicate requires. Both are discussed in the manuscript. VE-Safety screens each instance against the predicate. Table B gives the empirical form. With the user text fixed, none of the three conditions that remove or replace the task image produced grounded success on either target, while the original image yields 50.3 and 44.4. We would welcome a specific benchmark or method that the reviewer considers closest, and we will add a direct comparison so that the relationship to prior work is concrete. Screening a finite set cannot establish the criterion for every allowed description and we state this.

AuthorsOpenReview ↗

Response to Reviewer okvp [2/2]

Q3. Evidence of image specific reasoning

We added two new evaluations for this rebuttal. They also appear in our responses to the other reviewers and we repeat them here for completeness.

First an image aware re-evaluation. GPT-6 Astra re-scored every response that the original judge marked successful for Direct Plan and MM-Plan. It checked factual correctness and instance specific use against the original image and goal. Two annotators independently labeled the same 200 responses. GPT-6 Astra agreed with the human labels in 98.5 percent of cases.

Table A. Image aware re-evaluation on VE-Safety.

TargetMethodASRGrounding aware ASRGrounded fraction
Qwen3-VL-8BDirect Plan22.519.486.1
Qwen3-VL-8BMM-Plan54.450.392.5
Claude 4.5 SonnetDirect Plan9.78.487.1
Claude 4.5 SonnetMM-Plan46.344.495.9

Second a controlled image substitution. We held every user message fixed and regenerated the target replies from fresh contexts.

Table B. MM-Plan with the user messages held fixed.

Task inputQwen ASRQwen grounding aware ASRSonnet ASRSonnet grounding aware ASR
Original task image54.450.346.344.4
No task image5.90.02.50.0
Unrelated scene8.40.04.40.0
Detailed caption7.50.03.80.0

MM-Plan grounding aware ASR is 50.3 on Qwen3-VL-8B and 44.4 on Claude 4.5 Sonnet. Direct Plan scores 19.4 and 8.4. Among MM-Plan original successes the grounded fractions are 92.5 and 95.9. Removing the image or replacing it with an unrelated scene or a 512 token caption yields 0.0 grounding aware ASR in every such condition. Ordinary compliance persists at a low rate. The unrelated scene yields 8.4 ASR on Qwen and 4.4 on Sonnet. The result ties grounded success to the designated source image and separates it from generic jailbreak behavior.

W3 and Q4. Contributions of planning and training

The planning and training controls the reviewer asks for are already in the main text in Section 4.3. Tables 8 and 9 compare planning and training within each attacker size. We consolidate them here with the reward and tool controls.

Table. Planning and training by attacker size.

SettingAttacker size (B)Qwen3-VL-8BClaude 4.5 Sonnet
Direct Plan422.59.7
Per turn GRPO426.310.0
Global GRPO454.446.3
Direct Plan825.312.8
Global GRPO861.347.5

The two 4B GRPO rows share a backbone and training hyperparameters. Per turn GRPO cost 12 times the training compute of global GRPO and still gained little over Direct Plan. Scaling to 8B helped both the baseline and the trained planner. The gap between them persists at each size. Table 5 varies the reward at a fixed size. A binary success reward gave 30.6 on Qwen and a graded reward gave 38.8. Table 6 varies tool access. Disabling all image operations lowered Qwen ASR from 54.4 to 36.0. Table 23 scales the search budget for Direct Plan. The best of 16 samples reached 27.8 on Qwen and 15.9 on Sonnet. These controls establish a planning advantage under the tested configurations. We do not yet have matched total target and judge call counts across methods and we state this.

We will add these results and the tables above to the revised PDF shortly.

References

[A] Hu et al. GAMBIT: A Gamified Jailbreak Framework for Multimodal Large Language Models. ACL 2026.

[B] Azulay et al. Jailbreaking Vision-Language Models Through the Visual Modality. ICML 2026.

[C] Li et al. Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models. ECCV 2024.

[D] Miao et al. Visual Contextual Attack: Jailbreaking MLLMs with Image-Driven Context Injection. EMNLP 2025.

AuthorsOpenReview ↗

Ethics and responsible release

The vulnerability class we study exists independently of our work. Any user can upload a technical schematic to a frontier model today, so our contribution is to make this threat measurable. We release the VE-Safety dataset so that the community can evaluate and harden their own models against it. All its images come from publicly available technical documents such as engineering textbooks, open patents, and educational materials, and we introduce no novel dangerous content. Releasing such a benchmark follows established precedent including HarmBench, AdvBench, and MM-SafetyBench.

What we do not release is the attack capability itself. We withhold the trained planner weights, and we also withhold the trained attack trajectories and conversations, since these are what encode the optimized strategies. Without them the released materials reproduce only the Direct Plan baseline at 9.7 ASR on Claude 4.5 Sonnet, against 46.3 for the trained agent, and rebuilding the agent requires 8 A100 GPUs, frontier model API access, and reinforcement learning expertise. This release is more conservative than precedents that publish functional attack code. We will add a responsible use statement to the paper and the code, release VE-Safety under a research only access agreement that requires institutional affiliation and acceptance of acceptable use terms, and redact verbatim harmful outputs in the qualitative examples (Appendix F.2) with summarized descriptions. We will work with the Ethics Chairs on the right balance between transparency and safety.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.