- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper compares compliant-prefix attacks on open-source diffusion language models and autoregressive language models. The authors claim to identify a safety phenomenon (BSC) in DLMs where the model redirects the harmful intent toward benign content in early denoising steps. Based on this finding on the output side, the authors study the internal mechanism and identify a sparse set of attention heads causally related to the BSC. The authors further show that causally disrupting these heads can bypass the benign redirection and increase attack success rates on DLLMs.
Critical strengths
- This paper identifies the attention heads that are causally associated with DLMs’ redirection of harmful intent towards safety content, which is a new finding associated with the safety of DLMs
- The evaluation of robustness to prefix-based attacks covers different model families and sizes of ARMs and DLMs, making the empirical claim that DLMs are more robust to prefix-based attacks solid.
Critical weaknesses
- The paper is not very explicit about its scope and contribution, and risks overclaiming. For scope, it focuses on prefix-based jailbreak attacks instead of the full landscape of jailbreak attacks. This should be stated explicitly as a limitation. For contribution, it is established by previous works (such as DiffuGuard) that the early decoding steps of DLMs are important for safety behavior, and the proposed steering method is essentially transferring ARM steering to DLMs by identifying specific heads, which is conducted across all steps, making it less DLM specific. Additionally, this paper did not identify the exact mechanism that produces this difference in DLM and ARM’s different behavior to prefix-based attacks. The authors find that DLMs are more robust through evaluation, and identify the attention heads that are causally associated with this behavior. This does not substantiate the stronger claim of DLM specificity.
- The evaluation of prefix attacks on ARMs and DLMs covers a wide range of models. However, the steering experiment is mainly done on LLaDA-1.5 and LLaDA2.0-mini, with the improvement on LLaDA-1.5 very modest.
Questions for authors
- Could you clarify, or perhaps downscope the claims, what the novel mechanistic insight is here in this paper, beyond identifying some heads that are causally associated with this safety behavior of DLMs? Since the paper compares DLMs with ARMs, it remains unclear what unique characteristic of DLMs produces the BSC behavior. Perhaps a matched intervention analysis on ARM would be valuable to study the difference between these two model families.
- Can you expand the steering experiments to more model families?
7 comments
Response to Reviewer VSSZ (Part 2)
Applicability of the steering intervention across DLM families. (Weakness 2 & Question 2)
Our steering intervention targets the key attention heads mediating BSC, with a steering direction from BSC toward harmful compliance (HC). Therefore, it can only be meaningfully applied to models that exhibit identifiable BSC behavior. Although our evaluation covers a wide range of DLMs, only LLaDA-1.5 and LLaDA2.0-mini exhibit clear BSC tendencies under compliant-prefix attacks (Figure 1b). Moreover, as shown in Figure 1b, BSC is less dominant in LLaDA-1.5, which explains its more modest ASR improvement: the intervention specifically disrupts the BSC behavior, and thus has a more limited effect when this behavior is less prevalent.
To further address this concern, we additionally evaluate TESS 2, a DLM from another model family. TESS 2 remains substantially more robust than the evaluated ARMs under compliant-prefix attacks, achieving 28.00% ASR on HarmBench and 22.33% on HEx-PHI. However, its early semantic trajectories are dominated by EHC (82.00%), with only 2.00% EBS, resembling instruction-tuned DLMs more than explicitly aligned DLMs. Since TESS 2 rarely exhibits BSC behavior, there is also no sufficiently clear BSC-related direction to identify and use for targeted steering. Thus, the applicability of our steering intervention depends on whether a model exhibits BSC, rather than merely whether it belongs to the DLM family.
Thanks for the rebuttal. I understand this work focuses on prefix-based attacks, which can be seen as one kind of jailbreak. In my view this scope is a bit limited as it does not fully capture the broader landscape of jailbreak attacks. While the rebuttal addressed some of the concerns, I am still not convinced on the following points.
- Prior works already investigated safety recovery during denoising steps such as "Step-Wise Refusal Dynamics in Autoregressive and Diffusion Language Models" (arxiv2602.02600) and the context nesting paper, which are somewhat related to BSC.
- The authors frame the head-specific steering intervention as a major contribution (L111), but as the authors said during rebuttal, it can only be meaningfully applied to models that exhibit identifiable BSC behavior, and that only LLaDA-1.5 and LLaDA2.0-mini exhibit clear BSC tendencies, with improvement on LLaDA-1.5 very modest. Thus, the empirical evidence seems limited.
Could the authors further address these points? Given the rebuttal period timeframe, I think additional experiments are not necessary and some explanation would suffice.
Response to Reviewer VSSZ (Part 1)
We appreciate the reviewer's thoughtful feedback. Below, we address each of the main concerns raised in the "Critical Weaknesses" and "Questions For Authors" sections.
The scope and DLM-specific contributions of our work. (Weakness 1 & Question 1)
Regarding the scope of our work, our study explicitly focuses on DLM safety behaviors under adversarial prefixes, as clearly stated in both the title and abstract. Compliant-prefix attacks constitute our primary evaluation setting, while Appendix E further examines the generalization of BSC to the structurally different Context Nesting attack. Therefore, the scope of our study is already clearly defined in the submission.
Regarding the DLM-specific contribution, we do not claim novelty in the head-wise probing or activation steering techniques themselves, which are standard interpretability tools applicable to transformer-based models. Our contribution lies in discovering BSC, a distinctive safety behavior observed in the aligned DLMs we evaluate, and investigating how this behavior is supported by the internal dynamics of diffusion-based generation. In particular, our mechanistic analysis goes beyond identifying a sparse set of attention heads. We further investigate how BSC forms and is maintained through the DLM-specific iterative denoising process, using fine-grained interventions across denoising steps and generation blocks (Appendix D.3 and Section 5.2.2). These analyses provide DLM-specific insights into how BSC emerges, persists, and can be disrupted or recovered across denoising steps and generation blocks. Thus, although our default intervention spans all denoising steps and blocks to prevent subsequent safety recovery, our analysis explicitly examines the temporal and block-wise dynamics underlying this intervention, rather than simply transferring a static ARM steering procedure to DLMs.
Regarding the exact origin of the behavioral difference between ARMs and DLMs, head-level probing and steering identify which internal components mediate a behavior within a model, but do not by themselves isolate the complete architectural and post-training factors responsible for differences across generation paradigms. This is a broader limitation of existing interpretability tools rather than a drawback specific to our work. Nevertheless, we still investigate this divergence from DLM-specific perspectives. As stated in Section 3.2 (~line 252), the main purpose of our subsequent analysis is to explain how DLM's iterative denoising process supports this distinctive behavior observed in aligned DLMs but not in the evaluated ARMs.
To further examine whether safety-oriented post-training alone can reproduce this behavior, we additionally conduct standard DPO post-training on Ling-mini-2.0, an ARM with comparable model scale and general capability to LLaDA2.0-mini.
After DPO, its EHC rate remains high (66.00% to 69.00%), substantially differing from LLaDA2.0-mini. Together with our temporal intervention results, these findings suggest that the behavioral divergence between ARMs and DLMs likely stems from the DLM-specific iterative denoising process, rather than safety-oriented preference optimization alone. Regarding the suggested matched intervention on ARMs, such a comparison is not feasible because ARMs do not exhibit a sufficiently prevalent BSC tendency to identify a corresponding BSC-to-HC steering direction.
Overall, our contribution is the discovery and characterization of BSC in aligned DLMs, together with its mechanistic analysis through sparse attention heads and the distinctive temporal dynamics of iterative denoising.