acceptodds
Under review as a conference paper at ICLR 2027

GPO-V: Jailbreak Diffusion Vision Language Model by Global Probability Optimization

Abstract

Diffusion Vision-Language Models (dVLMs), built upon the non-causal foundations of Diffusion Large Language Models (dLLMs), have demonstrated remarkable efficacy in multimodal tasks by departing from the traditional autoregressive generation paradigm. While dVLMs appear inherently robust against conventional jailbreak tactics, which we categorize as **Fixed Prefix Optimization** (FPO) (e.g., anchoring responses with "Sure, here is"), this perceived resilience is deceptive. Our investigation into the safety landscape of dVLMs reveals a unique refusal pattern: **Immediate Refusal** and **Progressive Refusal**. We find that while FPO-based attacks often fail by triggering the latter, the progressive refinement process itself uncovers a novel, latent attack surface. To exploit this vulnerability, we propose Global Probability Optimization (GPO), a general jailbreak paradigm designed specifically for the denoising trajectory of masked diffusion models. Unlike prefix-based methods, GPO manipulates the global generative dynamics to bypass guardrails in diffusion language models. Building on this, we introduce **GPO-V, the first visual-modality jailbreak framework tailored for dVLMs**. Empirical results demonstrate that GPO-V produces stealthy perturbations with exceptional cross-model transferability, revealing a critical security gap in non-sequential generative architectures. Our findings underscore the critical urgency of addressing safety alignment in dVLMs. These results necessitate an immediate and fundamental re-evaluation of current defense paradigms to mitigate the unique risks of diffusion-based generation. Our code is available at: https://anonymous.4open.science/r/GPO-V-0250

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Diffusion Vision Language Model, Jailbreak Attack, AI Safety

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VpbWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates jailbreak attacks against diffusion-based vision-language models (dVLMs). The authors identify two distinct refusal patterns in dVLMs, the Immediate Refusal and the Progressive Refusal, which make fixed-prefix attacks ineffective. To address this, the authors propose GPO-V, a method that optimizes image perturbations to suppress the probability of refusal-related tokens across the entire output sequence, while simultaneously steering the model toward affirmative anchor tokens at various positions.

Critical strengths
  1. The observations regarding refusal behaviors in dVLM are valuable. The authors distinguish between two types of refusal patterns, noting that the iterative refinement process in diffusion models can effectively change their minds during generation. This explains why prefix-based attacks commonly used against autoregressive models are largely ineffective against dVLM.

  2. The attack method is fit to the mechanism of diffusion models. The authors leverage the fact that dVLM predicts all positions in parallel, and they suppress refusal-related tokens across the entire sequence while simultaneously establishing anchors at multiple positions.

Critical weaknesses
  1. The distinctions from prior work are not clear. While the paper cites DSN (Zhou et al., 2025) and PAD (Zhang et al., 2025b), it does not discuss the differences between GPO and these approaches. DSN also achieves jailbreaking by suppressing refusal tokens, and PAD employs multi-position guidance for outputs in diffusion-based language models.

  2. The experimental comparisons might not be fair. The FPO baseline is constructed by artificially fixing the prefix probability at 100%, which does not represent a standard baseline for image-based attacks.

Questions for authors
  1. What is the fundamental methodological difference between the proposed method GPO and DSN or PAD?

  2. Could you provide a comparison with standard image-based attack baselines, using the same perturbation budget?

  3. Could you complete the FPO results in Table 3 and specify the values ​​used for ε?

Reviewer_2Yx7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the security vulnerabilities of Diffusion Vision-Language Models and identifies two distinct refusal patterns: Immediate Refusal and Progressive Refusal. It further proposes Global Probability Optimization and its visual-modality jailbreak framework, GPO-V, to improve jailbreak success rates by globally suppressing the probabilities of negative primitives and optimizing positive anchor tokens.

Critical strengths

1.The research question is of clear academic value. Security research on dVLMs remains relatively limited compared with that on conventional autoregressive VLMs. By examining refusal behavior during the diffusion generation process, this paper investigates why FPO fails in this setting and, based on this analysis, proposes a novel attack strategy. The distinction between Immediate Refusal and Progressive Refusal offers a valuable perspective for studying model safety. Although this classification requires more rigorous empirical validation, it provides a useful framework for understanding the safety behavior of these dVLMs and offers testable hypotheses for future defense research. 2.GPO-V achieves strong attack effectiveness, supported by ablation studies that validate its core design. Table 1 shows that GPO-V substantially outperforms FPO. On AdvBench, with two anchors, GPO-V achieves attack success rates of 93.2% and 85.6% on LLaDA-V and LaViDA, respectively, compared to only 9.8% and 1.4% for FPO. Moreover, the ablation study in Figure 6 demonstrates that combining Global Loss and Anchor Loss significantly improves attack effectiveness. Appendices A.3 and A.4 further validate the effectiveness of different anchor templates and pixel-space attacks. These results provide solid empirical evidence for the effectiveness of GPO-V and its core design choices. 3.The paper explores attack transferability, generation configurations, and defense mechanisms. In addition to LLaDA-V and LaViDA, the paper includes the MMaDA and PAD baselines in the appendix. Table 2 tests different block lengths and Fast-dLLM acceleration configurations; Table 8 reports black-box transfer results on MMaDA; and Appendix A.6 evaluates potential mitigation methods using Token Injection. These experiments broaden the scope of the research. In particular, the defense experiments provide valuable insights into model safety, making this work more than just proposing an attack algorithm.

Critical weaknesses

1.The attack configuration and perturbation constraints are not sufficiently clear. Section 3.5 introduces a perturbation bound δ, but the exact value used in the experiments is not clearly specified. Although Appendix A.4 evaluates both latent-space and pixel-space attacks, the corresponding perturbation constraints are not fully explained. Providing these implementation details would improve reproducibility and help readers better understand the attack configurations. 2.The evaluation of attack transferability could be presented more clearly. Table 3 reports promising transfer success rates, but the source and target models for each experiment are not explicitly identified. Although Appendix A.10 provides additional transfer results on MMaDA, a clearer source-to-target comparison would make the findings easier to interpret. Including the corresponding model pairs and baseline results would strengthen the evidence for attack transferability. 3.The evaluation protocol requires further clarification. Table 1 reports attack success rates along with both human-annotated and AI-assessed hazard scores. However, the evaluation protocol is not described in sufficient detail, particularly regarding the sample size, the human annotation procedure, and the criteria used to determine attack success. Although Appendix A.8 provides a scoring rubric, clarifying these details would improve the transparency and reliability of the reported results.

Questions for authors

1.Could the authors clarify the perturbation budget ((\epsilon)) used in the experiments, particularly for latent-space and pixel-space attacks? 2.Could the authors specify the source and target models used in the cross-model transfer experiments in Table 3? 3.Could the authors provide more details about how the Jailbreak Success Rate is calculated, including the number of evaluated samples and the success criterion?

Reviewer_vDb6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies jailbreak attacks against diffusion vision-language models. The authors identify two refusal patterns, Immediate Refusal and Progressive Refusal, and propose GPO-V to overcome the limitations of conventional fixed-prefix optimization.

Critical strengths

The paper investigates a timely and relatively underexplored problem in diffusion vision-language models. The discussion of different refusal patterns provides some useful insights. The proposed method can jailbreak dVLMs via only modifying images, which is an underexplored aspect to the best of my knowledge.

Critical weaknesses

Overall, I think this paper is well-written, and the insights of the two distinct refusal patterns are interesting. The authors also introduce a new attack approach that demonstrates promising results. However, my main concern is that the methodological contribution may be somewhat incremental. Based on the current materials, the proposed method appears to primarily involve applying gradient-based adversarial optimization to visual inputs. I am not yet fully convinced that the method offers a distinctive advantage over existing approaches at a fundamental or mechanistic level, even when taking into account the authors' interesting observations regarding the two refusal patterns. I would appreciate further clarification of what makes the proposed optimization strategy fundamentally different from prior methods. Additional concerns are detailed below.

Questions for authors
  1. Could the authors clarify how the refusal tokens are selected? It would be helpful to understand whether the selected tokens are sufficiently generalizable, or whether the attack performance depends on including a large number of refusal-related tokens. An ablation study on the number of refusal tokens required to achieve effective attacks would be appreciated.

  2. Could the authors provide more details on the magnitude of pixel-level perturbations required for the proposed method to be effective? It would also be helpful to include visualizations of the adversarial images to better assess their perceptual quality.

  3. It would be interesting to further investigate the robustness of the anchor-based mechanism. For instance, if a defender prevents the specific anchor tokens sequences such as "Sure", "First", and "Second" to be appeared simutaneously, would the attack performance be significantly affected? If so, is it possible that the defender can prevent such attack by simutaneously preventing the appearance of a reasonable amount of possible anchor tokens (not necessarily ALL anchor tokens)? Could the authors discuss whether anchor tokens can be effectively selected automatically rather than manually? An additional evaluation against adaptive defenses would help clarify the robustness of the proposed approach.

  4. The appendix includes an additional comparison with PAD and emphasizes that PAD requires direct token overwriting. However, from the perspective of adversarial attack mechanisms, I am not convinced that this distinction necessarily implies a fundamental methodological difference. More importantly, what are the underlying mechanistic differences and advantages of GPO-V over PAD in addressing the two identified refusal patterns?

    Although the authors emphasize that PAD requires token-overwriting privileges, the main experiments of GPO-V primarily rely on latent-space optimization. Directly modifying internal visual features also requires a higher level of model access than simply uploading an adversarial image. Therefore, I believe the distinction in intervention space alone may not be sufficient to establish the methodological novelty of GPO-V.

    Furthermore, could similar attack performance be achieved by applying global probability optimization directly to token representations, rather than visual features? If so, it may suggest that the effectiveness of GPO-V may primarily stem from a different optimization objective applied to a similar underlying attack mechanism. This raises a more fundamental question: Does GPO-V introduce a genuinely different mechanism for overcoming Immediate Refusal and Progressive Refusal compared to PAD, or does it mainly achieve similar effects through a different optimization formulation? I would appreciate further analysis to clarify this distinction.

Comments on the paper

3 comments

Dear Reviewers,

This is a gentle reminder to review the authors’ rebuttal and actively participate in the discussion on OpenReview. Your timely input is important for having an interactive discussion and reaching a well-informed assessment.

Thank you for your time and contribution!

Best,

Your AC

Dear all reviewers,

The authors have responded to your concerns for a while. Please go through the response below and the revised submission to check whether your concerns have been adequately addressed, and if there are any follow-up questions where applicable. Thank you!

Best, AC

Dear all reviewers,

The authors have responded to your concerns for a while. Please go through the response below and the revised submission to check whether your concerns have been adequately addressed, and if there are any follow-up questions where applicable. Thank you!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.