acceptodds
Under review as a conference paper at ICLR 2027

DRIFT: Diffusion-LLM Red-Teaming Investigation for Multi-Turn Safety

18 views

Abstract

Diffusion language models (dLLMs) reach users through the same multi-turn chat interfaces as autoregressive models (arLLMs), where multi-turn escalation is one of the strongest jailbreak attacks. How multi-turn attacks perform on a dLLM has not been evaluated. We ask that question, and ask what inside the model accounts for the answer. A dLLM exposes two attack surfaces, the prompt surface of messages an attacker sends and the canvas surface of response positions it can fill before denoising, and we describe ten attacks, five from prior work and five we build, as instances of one procedure over both. Evaluated on four open-weight dLLMs and three arLLMs across six representative red-teaming benchmarks, all five attacks we build outperform every prior attack, with Crescendo PAD reaching 78.2% attack success rate (ASR). Decomposing that result shows the two surfaces contribute unequally. Multi-turn escalation alone is weak on a dLLM, reaching 29.7% against 39.9% on arLLMs and falling below the 61.2% of the strongest single-turn dLLM-specific attack, because it evades the refusal decision at the response opening, taking refusal mass from at least 0.65 to at most 0.05, without filling the positions that decision leaves open. The infill attack supplies the level a pairing reaches and the multi-turn half lifts it by 1.4 to 17 points. Evading a refusal and producing a compliant answer are therefore separate steps on a dLLM, and evaluating dLLM multi-turn safety calls for attacks that are aware of both surfaces.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Diffusion language models, multi-turn jailbreak, red-teaming, safety alignment, evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qkunOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problem of multi-turn jailbreak attacks on diffusion LLMs (dLLMs). Different from autoregressive based LLM (arLLMs), dLLMs have two attack surfaces: the prompt surface and the canvas surface. The prompt surface is shared with arLLMs that refer to messages that the attacker sends. The canvas surface is exclusive to dLLMs where the response positions can be filled before denoising. The author studies 10 attacks from prior work and combined ones. Key results show that multi-turn attacks are weaker in dLLMs than in arLLMs, and multi-turn escalation stops the dLLM from refusing, but the model then gives a vacuous answer instead of truly harmful content.

Critical strengths

1.The paper studies a timely and important multi-turn safety problem.

2.The paper is well-written and it is easy to follow the problem formulation.

3.The paper evaluates multiple attacker methods and LLMs.

Critical weaknesses

1.The comparison between dLLMs and arLLMs is not very fair from two angles. First, in the paper, the authors only test prompt-only attacks for arLLMs and for dLLMs, they add additional canvas attack, which shows dLLMs look more vulnerable. However, canvas attacks are similar to prefilling attacks in arLLMs, and a fair comparison should be testing prefill-based attacks on arLLMs. Second, the paper claims that multi-turn attacks are weaker on dLLMs because the response opening constrains the rest less than on arLLMs. A fair comparison should be using a same opening. For example, add same opening such as “Sure, here is …” for both models and compare the harmful content.

2.The authors claim that dlLMs are harder to attack with multi-turn attacks coming from the bidirectional architecture. But this may not be the only reason. dLLMs and arLLMs may have different capabilities. A weak model does not refuse and gives a vague answer will lead to low ASR. Dream-7B initialized from Qwen2.5-7B is a good point to compare and study the reason. In Fig. 2(left), Dream-7B and Qwen2.5-7B seem to reach similar Crescendo ASR around turn 10. This suggests that the gap may come from model difference?

3.The paper does not give much new insights on the core question that this paper studies, which is whether the way dLLMs generate text brings new safety challenges compared with arLLMs. The canvas attacks are already shown by DIJA and PAD. The new point is that dLLMs not refusing does not lead to harmful answers, but the mechanism is not fully studied. Also, I suggest the authors test existing guardrails and defense mechanisms on the proposed attacks.

Questions for authors

See Weaknesses.

Reviewer_8bhpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates multi-turn jailbreak attacks on diffusion language models (dLLMs). Unlike autoregressive models (arLLMs), multi-turn escalation attacks are not comprehensively evaluated. The author reveals two attack surfaces for dLLMs, including prompt surface that is the generated conversation and canvas surface that can be filled prior denoising. The author evaluates single-turn and multi-turn attacks built for dLLM and arLLM. More importantly, the author combines existing multi-turn attacks with attacks for dLLMs and finds great improvement compared to the original arLLM-based attacks that are less effective on dLLMs.

Critical strengths
  1. This paper proposes two distinct attack surfaces, prompt and canvas surfaces, and propose five attacks that exploit both by combining existing multi-turn jailbreak and dLLM-targeted attacks. The experiments demonstrate that combining conversational escalation with diffusion-specific infilling improves ASR over existing single-turn and multi-turn attacks. The strongest method, Crescendo × PAD, achieves 78.2% average ASR.
Critical weaknesses
  1. Although the author proposes two attack surfaces and combines model-agnostic and dLLM-specific attacks, the novelty of method section is relatively limited. Out of the five attack methods proposed by the author, the combination of existing methods have higher attack success rate (ASR) in most of the cases than Ratchet, which is the new method proposed by the author.
  2. The experiments cannot sufficiently isolate the contributions of the two attack surfaces or establish the causal explanations claimed for their interaction. The authors claim that the infilling component determines the overall effectiveness of hybrid attacks, while the multi-turn component determines the additional improvement. However, the experiments primarily compare standalone attacks with their combinations, without sufficiently isolating how the two components interact. Furthermore, the improvements vary substantially across attack combinations and target models, with some combinations even underperforming their standalone infilling components. I suggest that the authors conduct more controlled experiments to investigate how conversational context affects diffusion-specific infilling and provide stronger evidence for the proposed explanations.
Questions for authors
  1. Could the authors explain why Dream exhibits different behavior from the other three models in Table 2? For example, under the PAIR attack, the other three models have relatively high attack success rates, whereas Dream has a much lower success rate.
  2. Have the authors considered combining single-turn model-agnostic attacks with dLLM-specific attacks, given that PAIR already achieves better performance than the multi-turn model-agnostic attacks?
Reviewer_8vNNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multi-term generic attacks to diffusion language models(dLLMs). The paper identifies two attacked surface.the prompt surface and the canvas surface. This work propose five dLLMs specific multi-term attacks. Experimental study the benchmarks show that traditional multi-turn attacks are less effective on dLLMs, while attacks use both surface achieve higher attack success rates.

Critical strengths
  • This paper studies multi term safety on dLLMs. The proposed framework combines the prompt surface and canvas surface.providing a unified way to study different jailbreak attacks in dLLMs.
  • The experiments covered multiple models and benchmarks, and the results show that the proposed method can improve attack success rates.
Critical weaknesses
  • limitation of novelty. Four of the five proposed attacks directly combine existing methods, Crescendo or AMA for multi-turn escalation, and DIJA or PAD for infilling. These components have already studied in prior works.A main difference is applying the existing in filling attacks after several conversation turns.Well, it's improves ASR, but the paper does not clearly explain what new attack mechanisms is introduced beyond combining these methods. Ratchet has a different design but reports lower performance than Crescendo x PAD.

  • The experiments does not fully establish the benefit of combining the two surface.The hybrid attacks use both multi-term conversations and in filling, but the paper does not compare them with repeated single term in filling attacks under the same query budget.One example is that Crescendo x PAD achives 78.2% ASR compared with 61.2% for PAD However, it is unclear whether this improvement Comes from the interaction between the two attack surface or simply from additional model queries and conversation context.

  • compare between dLLMs and arLLMs is not sufficient. The paper reports that traditional multi-turn attacks achieve lower ASR on dLLMs than on arLLMs related to this difference to their decoding mechanisms. However, the model also differ in training data, capability and safety alignment. For example, PAIR also performs much worse on dLLMs than on arLLMs, even though it is not a multi-turn attack. Therefore, it is unclear how much of the observed difference is caused by diffusion decoding rather than difference between model themselves.

Questions for authors
  • Section 3.3 Since four proposed attacks directly combined existing multi term and infilling methods, could the authors clarify the novelty beyond combining these existing approaches?
  • section 3.1-3.3 The proposed attacks require control over selected response positions.Could the author clarify which practical LLMs interface support this capability.
  • section 4.2 Could authors explain whether the improvements of a hybrid attacks come from combining the two attack surface or from additional model queries. how would the methods compare with separated PAD or DIJA under the same query budget?
  • section 4.2 Why do AMA-based hybrid attacks perform worse than their corrsponding single-turn infiling attacks on MMaDA?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.