- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the problem of multi-turn jailbreak attacks on diffusion LLMs (dLLMs). Different from autoregressive based LLM (arLLMs), dLLMs have two attack surfaces: the prompt surface and the canvas surface. The prompt surface is shared with arLLMs that refer to messages that the attacker sends. The canvas surface is exclusive to dLLMs where the response positions can be filled before denoising. The author studies 10 attacks from prior work and combined ones. Key results show that multi-turn attacks are weaker in dLLMs than in arLLMs, and multi-turn escalation stops the dLLM from refusing, but the model then gives a vacuous answer instead of truly harmful content.
Critical strengths
1.The paper studies a timely and important multi-turn safety problem.
2.The paper is well-written and it is easy to follow the problem formulation.
3.The paper evaluates multiple attacker methods and LLMs.
Critical weaknesses
1.The comparison between dLLMs and arLLMs is not very fair from two angles. First, in the paper, the authors only test prompt-only attacks for arLLMs and for dLLMs, they add additional canvas attack, which shows dLLMs look more vulnerable. However, canvas attacks are similar to prefilling attacks in arLLMs, and a fair comparison should be testing prefill-based attacks on arLLMs. Second, the paper claims that multi-turn attacks are weaker on dLLMs because the response opening constrains the rest less than on arLLMs. A fair comparison should be using a same opening. For example, add same opening such as “Sure, here is …” for both models and compare the harmful content.
2.The authors claim that dlLMs are harder to attack with multi-turn attacks coming from the bidirectional architecture. But this may not be the only reason. dLLMs and arLLMs may have different capabilities. A weak model does not refuse and gives a vague answer will lead to low ASR. Dream-7B initialized from Qwen2.5-7B is a good point to compare and study the reason. In Fig. 2(left), Dream-7B and Qwen2.5-7B seem to reach similar Crescendo ASR around turn 10. This suggests that the gap may come from model difference?
3.The paper does not give much new insights on the core question that this paper studies, which is whether the way dLLMs generate text brings new safety challenges compared with arLLMs. The canvas attacks are already shown by DIJA and PAD. The new point is that dLLMs not refusing does not lead to harmful answers, but the mechanism is not fully studied. Also, I suggest the authors test existing guardrails and defense mechanisms on the proposed attacks.
Questions for authors
See Weaknesses.