- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes SafeBasin, a training-free inference-time defense against jailbreak attacks on dLLMs. The central observation behind SafeBasin is that, before harmful content appears in the output, successful attacks already cause the predictive distributions along the denoising trajectory to “concentrate”: update directions stay consistently aligned, the step-to-step variation shrinks, and entropy drops rapidly. Based on this, the authors design TGM, which standardizes three statistics against benign trajectories at the same denoising progress to obtain a concentration score, with the trigger threshold calibrated to a target benign trigger rate. Whenever the score exceeds the threshold, ADC intervenes in the decoding process, providing a targeted defense against jailbreaks. SafeBasin is evaluated across multiple dLLMs and attack methods and demonstrate that it substantially reduces ASR while maintaining a low false refusal rate and preserving general capabilities.
Critical strengths
-
The paper addresses a well-motivated problem that is specific to dLLMs and distinguishes them from autoregressive LLMs. The experiments are comprehensive: they report ASR and also false refusal rates and general capabilities.
-
Because the method is training-free, it applies broadly and can be easily combined with training-based defenses such as A2D and MOSA.
Critical weaknesses
-
It is not convincingly established that the detection signal captures safety-relevant behavior. SafeBasin performs well on PAD and DIJA, two structurally similar attacks that both rely on structural anchors and input masks, but the paper does not rule out the effect of this input-mask structure itself. Constrained infilling may by itself produce aligned, low-entropy convergence, regardless of whether the content is harmful. The authors should add experiments that present benign prompts to the model in the same input-mask format to rule out this confound.
-
The writing, especially in the method and theory sections, lacks a clear and coherent storyline, which makes the logical relationships between sections hard to follow. For example, Section 2.2 formally defines the basin and the method is named after it, but the concept is not used again in the algorithm, the propositions, or the experiments. Prop. 1 concerns perturbation contraction between two trajectories, whereas TGM measures directional persistence, variation, and entropy along a single trajectory; the authors do not explain how the two are connected, and is never estimated. In addition, key details are scattered across multiple appendices, so it is difficult for readers to determine exactly what the paper’s core claims are and which evidence supports each of them.
Questions for authors
-
What decoding configurations (e.g., temperature, number of steps, and generation length) were used for the general capability evaluations on MMLU, GSM8K, and HumanEval? What is the actual trigger rate of SafeBasin on these benchmarks?
-
Appendix E.8 defines a detector-aware adaptive attack objective with , but the paper does not appear to report results under this setting. Could the authors explain why?
-
The code in the supplementary material differs from the method described in the main text in several places. For example, is computed without standardization, the code includes neither the benign calibration nor the quantile-based threshold, and the ADC noise is not scaled by . Could the authors clarify whether this is the latest version of the code?