- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates jailbreak attacks against diffusion-based vision-language models (dVLMs). The authors identify two distinct refusal patterns in dVLMs, the Immediate Refusal and the Progressive Refusal, which make fixed-prefix attacks ineffective. To address this, the authors propose GPO-V, a method that optimizes image perturbations to suppress the probability of refusal-related tokens across the entire output sequence, while simultaneously steering the model toward affirmative anchor tokens at various positions.
Critical strengths
-
The observations regarding refusal behaviors in dVLM are valuable. The authors distinguish between two types of refusal patterns, noting that the iterative refinement process in diffusion models can effectively change their minds during generation. This explains why prefix-based attacks commonly used against autoregressive models are largely ineffective against dVLM.
-
The attack method is fit to the mechanism of diffusion models. The authors leverage the fact that dVLM predicts all positions in parallel, and they suppress refusal-related tokens across the entire sequence while simultaneously establishing anchors at multiple positions.
Critical weaknesses
-
The distinctions from prior work are not clear. While the paper cites DSN (Zhou et al., 2025) and PAD (Zhang et al., 2025b), it does not discuss the differences between GPO and these approaches. DSN also achieves jailbreaking by suppressing refusal tokens, and PAD employs multi-position guidance for outputs in diffusion-based language models.
-
The experimental comparisons might not be fair. The FPO baseline is constructed by artificially fixing the prefix probability at 100%, which does not represent a standard baseline for image-based attacks.
Questions for authors
-
What is the fundamental methodological difference between the proposed method GPO and DSN or PAD?
-
Could you provide a comparison with standard image-based attack baselines, using the same perturbation budget?
-
Could you complete the FPO results in Table 3 and specify the values used for ε?
Dear Reviewers,
This is a gentle reminder to review the authors’ rebuttal and actively participate in the discussion on OpenReview. Your timely input is important for having an interactive discussion and reaching a well-informed assessment.
Thank you for your time and contribution!
Best,
Your AC