acceptodds
Under review as a conference paper at ICLR 2027

-Monitor: ynamic Safety Monitoring for iffusion LLMs via Hesitation Signals

Abstract

Despite the emergence of diffusion large language models (D-LLMs) as an alternative to autoregressive large language models (AR-LLMs), safety monitoring for D-LLMs remains largely unexplored. Unlike AR-LLMs, D-LLMs generate text through a multi-step denoising process, exposing intermediate hidden representations that may contain safety-relevant information unavailable in standard single-step monitoring setups. Motivated by the suitability of lightweight probes for always-on monitoring, we analyze which trajectory-level signals best indicate when such probes are likely to struggle. We find that the most informative signal is safety hesitation: intermediate hidden states repeatedly falling within a small margin of the probe's decision boundary. The number of such hesitation steps in D-LLM's trajectory predicts probe failure effectively, providing a proxy for sample difficulty. Building on this analysis, we propose -Monitor, in which a lightweight probe runs on every sample, producing a base prediction and a hesitation estimate that decides whether to escalate the sample to a more expressive probe. That probe is trained on trajectories with at least one hesitation step, and reads only the minimal span of those steps. Evaluated on 3 datasets (WildGuardMix, ToxicChat, OpenAI-Moderation) across 4 D-LLMs, -Monitor achieves state-of-the-art performance with a compact parameter footprint ( 0.93M parameters), and exhibits the best trade-off between effectiveness and efficiency relative to 8 baselines.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
safety monitoring, probing classifiers, interpretability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FkfWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a safety monitoring method for diffusion LLMs. In particular, the proposed method uses a lightweight linear probe to classify whether a given prompt is safe or unsafe based on intermediate hidden representations obtained during the denoising process. The authors observe that the probe may repeatedly produce predictions that are close to its decision boundary, which they refer to as a “safety hesitation”. Based on the number of such hesitation steps (denoising steps where the probe is close to the decision boundary) the method decides whether to use a more expressive probe, that is trained on hesitation examples. This framework allows computational resources to be allocated to more difficult inputs. The proposed method is evaluated on three safety datasets and four diffusion LLMs, outperforming the considered probe- based baselines.

Critical strengths
  • The proposed method is conceptually simple and does not require any modifications to the underlying diffusion LLM.
  • Despite its relatively simple design, the method outperforms more complex probe-based baselines, such as temporal attention and LSTM probes.
Critical weaknesses
  • The paper motivates a multi-step routing, but it never compares it against routing on a single uncertainty score from the first-stage probe, as in existing cascades that the authors cite (McKenzie et al., 2025; Cunningham et al., 2026). Since the authors present this multi-step routing as a key difference from prior cascades (Sec. 2, Sec. 4.1), such a comparison would have been helpful to understand the benefit of the proposed multi-step signal over the prior work.
  • All four models come from the LLaDA lineage. Since the paper’s claims concern D-LLMs in general, and the hesitation signal depends on the model’s denoising dynamics, it is unclear whether the findings carry over to other D-LLMs (e.g., Dream-7B).
Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer FkfW

We are grateful for the reviewer’s careful assessment and suggestions for additional comparisons. We also appreciate the recognition of our method’s simplicity and its ability to monitor D-LLMs without modifying the underlying model.


W1: Multi-step hesitation versus simpler uncertainty-based routing

A1: Thank you for suggesting this comparison. To address your concern, we compare D²-Monitor's routing rule, the number of hesitation steps (multi-step, ours), against the final-step margin (single-step), an adaptation of the routing rule used in existing cascades to D-LLMs. Following Reviewer dEuR's suggestion, we also include two multi-step alternatives that route on the minimum or average margin across denoising steps.

We conduct this comparison on ToxicChat using LLaDA-8B-Instruct, keeping the trained probes and margin threshold τ\tau fixed. Let ∣ds∣|d_s| denote the absolute decision margin at denoising step ss, with s=1s=1 denoting the final step. The alternatives route an example to the second-stage probe when:

  • Final-step: ∣d1∣<τ|d_1|<\tau;
  • Multi-Step Minimum: min⁡s∣ds∣<τ\min_s|d_s|<\tau;
  • Multi-Step Average: 1S∑s=1S∣ds∣<τ\frac{1}{S}\sum_{s=1}^{S}|d_s|<\tau.

D²-Monitor instead routes when nτ>λn_\tau>\lambda, where nτ=∑s=1S1[∣ds∣<τ]n_\tau=\sum_{s=1}^{S}\mathbf{1}[|d_s|<\tau], retaining its original operating point. The results are shown below (F1: mean ± SD over seeds 0, 1, 2).

Routing ruleMLP E[P]MLP F1 (%)TimeAttn E[P]TimeAttn F1 (%)
Final-step0.32M89.2±0.30.49M88.8±0.6
Minimum margin0.56M89.2±0.30.84M89.0±0.3
Average margin0.20M90.2±0.30.30M89.9±0.2
D²-Monitor (Ours)0.16M90.3±0.20.37M90.1±0.2

D²-Monitor improves mean F1 over final-step routing by 1.1 (MLP) and 1.3 (TimeAttn) percentage points while using fewer expected parameters. This indicates that multiple denoising steps contribute meaningfully. D²-Monitor also outperforms the other multi-step rules, minimum-margin and average-margin routing. We have added this comparison to Appendix D.5 (Table 8), with a summary in Section 5.3.


W2: Generalization beyond the LLaDA family

A2: We appreciate the suggestion to evaluate generalization beyond the LLaDA family. We additionally evaluate D²-Monitor on Dream-v0-Instruct-7B [1] using ToxicChat, following the same experimental protocol as for the other models (Appendix C).

The table below reports accuracy and F1 as mean ± SD over seeds 0, 1, 2. Best results are in bold, and second-best are underlined‾\underline{\mathrm{underlined}}.

MethodE[P]Acc (%)F1 (%)
LP (Last Step)3.6×10−33.6\times10^{-3}M95.7±0.485.5±0.5
MLP (Last Step)0.92M96.5±0.287.0±0.1
LP (MV)3.6×10−33.6\times10^{-3}M95.8±0.185.8±0.1
LP (Mean)3.6×10−33.6\times10^{-3}M95.9±0.386.1±0.3
MLP (MV)0.92M96.5±0.287.1±0.0
MLP (Mean)0.92M96.7±0.187.0±0.1
TimeAttn1.39M96.9±0.188.2±0.0
LSTM2.30M96.6±0.287.4±0.1
D²-MLP (Ours)0.27M97.3±0.2‾\underline{97.3±0.2}90.0±0.7‾\underline{90.0±0.7}
D²-TimeAttn (Ours)0.37M97.4±0.090.0±0.1

On Dream, both D²-Monitor variants outperform the strongest baseline (TimeAttn) by 1.8 F1 points with 3.8–5.1× fewer expected parameters, which shows that our approach generalizes beyond the LLaDA family. We have added the Dream results to Appendix D.7 (Table 10) and introduced this additional evaluation in Section 5.1.


[1] Jiacheng Ye et al. Dream 7B: Diffusion Large Language Models. arXiv:2508.15487, 2025.

AuthorsOpenReview ↗

Summary of Revisions

The revised manuscript includes the following additions in response to your feedback:

  • Added the routing-rule comparison to Appendix D.5 (Table 8), with a summary in Section 5.3.
  • Added the evaluation on Dream-v0-Instruct-7B to Appendix D.7 (Table 10), and introduced this additional evaluation in Section 5.1.
Reviewer_FkfWOpenReview ↗

Response to authors' rebuttal

[W1] Thank you for the routing-rule comparison. The numbers look good to me, and your method seems to outperform final-step routing. Would it be feasible within the discussion period to run the same comparison on WildGuardMix? This should only require test-time evaluation, right? Also, since average-margin routing performs within seed variation of D2D^2-Monitor on ToxicChat, it would be interesting to see whether the two separate on WildGuardMix.

[W2] Thank you for adding the Dream results. The numbers address my concern about generalization beyond the LLaDA family.

Reviewer_bAKqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Inspired by the observation that D-LLMs have fluctuating outputs during mathematical reasoning, this paper finds a similar phenomenon occur while using a linear probe for safety monitoring. It is observed that when it comes to samples that are difficult for a linear probe to classify, intermediate hidden states fluctuate close to the decision boundary across denoising steps. Following the discovery, the paper proposes a two-stage safety monitoring pipeline for D-LLMs, including using a lightweight linear probe that makes predictions for the easier samples and routes the difficult samples that require a more complex advanced probe. Experiments show their method achieves best performance compared to non-LLM based monitors while remaining computationally efficient.

Critical strengths

Safety monitoring for D-LLMs is relateively less studied. Using LLMs as monitor is an approach, but this usually requires high computational cost. This work proposes a highly efficient cascaded detection approach that utilizes the hidden representations over diffusion steps to evaluate the safetiness of outputs. The paper evaluates the method against multiple D-LLMs and datasets, and also cross dataset performance.

Critical weaknesses

While the paper focuses mainly on empirical evaluation, it provides limited explanation of the mechanism underlying hesitation. The observation that the linear probe score oscillates between the decision boundary is somewhat expected from the training of the probe, as it utilizes the temporal mean of hidden representations over the diffusioon trajectory. It is unclear whether the hesitation signal comes from the representations at the masked positions or unmasked positions; an ablation that separates these representations could provide more insights in this direction.

Questions for authors
  • Does this method allow for early detection before the denoising process completes?
  • How do different D-LLM decoding schemes affect the performance of your method?
  • As mentioned in weaknesses, the linear probe is trained on the temporal mean of hidden representations, but the hesitation is observed with signals from individual denoising steps. Will the phenomenon still occur if linear probe is trained directly on step-wise representations?
  • Since the method mean-pools hidden representations across all token positions, does the hesitation signal mainly arise from masked or unmasked positions?

Minor issues:

  • Which layer is used for probing in your main evaluation?
  • Figure 3 (a) and (b) lack x-axis titles
4 comments
AuthorsOpenReview ↗

Response to Reviewer bAKq

We appreciate the reviewer’s recognition of our method’s efficiency and broad evaluation, and the suggestions for further analysis.


W1 & Q3: Does temporal-mean training itself produce hesitation?

A1: Thank you for suggesting this control. We compare three ways of training the first-stage linear probe: (i) Temporal mean: train one probe on the temporally averaged representation of each example; (ii) Shared step-wise: train one shared probe on individual-step representations, averaging the classification loss across steps; (iii) Separate step-wise: train a separate probe for each denoising step.

Results on LLaDA-8B-Instruct / ToxicChat (mean ± SD over seeds 0, 1, 2). Predictions use majority voting across denoising steps. Error AUROC measures how well hesitation severity nτn_\tau identifies incorrect predictions. The last two columns show mean hesitation severity for correct and incorrect predictions, respectively.

First-stage trainingError AUROC (%)nˉτ,correct\bar n_{\tau,\mathrm{correct}}nˉτ,incorrect\bar n_{\tau,\mathrm{incorrect}}
Temporal mean83.8 ± 0.25.9 ± 0.421.8 ± 0.5
Shared step-wise84.5 ± 1.17.4 ± 0.424.3 ± 0.1
Separate step-wise88.0 ± 0.63.4 ± 0.117.2 ± 0.1

Across all three settings, incorrect predictions show substantially more hesitation, suggesting that it is not merely an artifact of temporal-mean training. We use temporal-mean training for its lower probe-training cost: it uses one averaged representation per example, whereas shared step-wise training evaluates all SS step representations per example, and separate step-wise training fits SS distinct probes, where SS denotes the number of denoising steps. We have added these results to Appendix E.6 (Table 12) and a summary to Section 3.3.


Q4: Does the signal come from masked or unmasked tokens?

A2: We appreciate the suggestion to examine masked and unmasked representations separately. We compare three token-pooling variants on LLaDA-8B-Instruct using ToxicChat: All, Masked-only, and Unmasked-only. Mask status is determined at the input to each forward pass. We retrain D²-Monitor for each alternative pooling under the original protocol.

Results (F1: mean ± SD over three seeds):

PoolingMLP E[P]MLP F1 (%)TimeAttn E[P]TimeAttn F1 (%)
All (Ours)0.16M90.3±0.20.37M90.1±0.2
Masked-only0.13M89.8±0.10.21M89.4±0.4
Unmasked-only0.22M89.7±0.20.34M89.8±0.2

All-token pooling achieves the highest mean F1 for both probe architectures. We have added these results and threshold-sweep curves to Appendix E.7 (Table 13 and Fig. 8), and clarified the pooling choice in Section 3.1. The curves show this advantage at every evaluated threshold over λ∈4,…,27\lambda\in\\{4,\ldots,27\\}, supporting our original pooling choice. LLaDA’s bidirectional attention allows information exchange across masked and unmasked positions [1]. Hesitation-related signals may therefore be distributed across both groups, potentially explaining the benefit of all-token pooling.


Q2: Robustness to different decoding schemes

A3: Thank you for asking about decoding robustness. Our original submission evaluates D²-Monitor across step lengths, generation lengths, and remasking strategies (Section 5.3, Figure 3). The probes were trained under a single configuration with low-confidence remasking and kept fixed throughout evaluation. The results demonstrate robustness across all evaluated decoding configurations.


Q1: Can monitoring detect unsafe content before denoising finishes?

A4: We appreciate this suggestion for extending our work to early detection. Our work is an initial exploration of safety monitoring for D-LLMs, and the current implementation makes its decision using the full denoising trajectory. It does not yet support online routing or early stopping. Our current design improves monitoring efficiency by invoking the advanced probe only for routed hard samples, keeping the expected test-time parameter count at ≤ 0.93M while achieving the best effectiveness–efficiency trade-off relative to eight baselines (Fig. 1). We see online routing and early stopping as promising extensions for earlier intervention and further efficiency gains. We have added this discussion to the Limitations paragraph in Section 6.


Minor comments. Probing layer and Figure 3 axis labels

A5: We have clarified the use of last-layer hidden states in Appendix E.5. We have added the missing x-axis labels to Figure 3(a,b) in the latest version: (a) step length, the number of tokens unmasked per denoising step, with generation length fixed at 128 tokens; (b) generation length, the number of generated tokens, with step length fixed at 1.


[1] Shen Nie et al. Large Language Diffusion Models. NeurIPS, 2025.

AuthorsOpenReview ↗

Summary of Revisions

We have updated the manuscript to address your suggestions:

  • Added the comparison of first-stage probe training schemes to Appendix E.6 (Table 12), with a summary in Section 3.3.
  • Added the token-pooling comparison and threshold-sweep curves to Appendix E.7 (Table 13 and Figure 8), and clarified all-token pooling and its empirical support in Section 3.1.
  • Added a discussion of online routing and early stopping to the Limitations paragraph in Section 6.
  • Clarified the use of last-layer hidden states in Appendix E.5 (layer sensitivity analysis).
  • Added the missing x-axis labels to Figure 3(a,b): Step length and Generation length.

Thank you for the detailed response and the additional experiments. In particular, the new ablations comparing temporal-mean training against shared and separate per-step probes directly addressed my main concern that trajectory hesitation could be an artifact of the training objective. I am now much more convinced by the proposed method and have no further questions.

Thank you for your response! We are glad that the additional experiments have effectively addressed your concern. Your constructive suggestions helped us strengthen the analysis and improve the manuscript.

Reviewer_dEuROpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper explores diffusion LLMS as safety monitors. The authors first introduce a hesitation metric, defined as the number of denoising steps where a linear safety probe's prediction is close to its decision boundary. The authors demonstrate that this measure is associated with classification difficulty. Based on those observations, the paper proposes D2-Monitor: a lightweight, always-on probe that produces a base prediction and a hesitation estimate. The hesitation estimate is used to decide whether to pass a sample to a more expressive probe. The method is evaluated on three safety datasets and four diffusion LLMs.

Critical strengths
  • The paper studies safety monitoring for diffusion LLMs. The literature on autoregressive LLMs is already quite extensive. This paper addresses a relatively important and unexplored gap.

  • The proposed D2-Monitor shows consistent improvements over the evaluated baselines.

  • The experiments are broad: four diffusion LLMs and three safety datasets. Additionally, the paper includes useful ablations on routing, hesitation-based training, and robustness.

Critical weaknesses
  • The novelty of the proposed method is unclear. In Sec. 3.3, the hesitation measure is defined as the number of denoising steps in which the linear probe's prediction is close to its decision boundary. The paper does not directly compare its routing method against simpler approaches based on the linear probe's uncertainty, such as the final denoising step margin. The fact that the hesitation signal calculation involves multiple denoising steps is the diffusion-specific aspect of the method, but I am unsure if it is really meaningfully contributing to the performance of D2-monitor.

  • The idea of passing uncertain samples to a stronger classifier has already been studied in previous work

  • The paper reports strong cross-dataset results. However, the cross-dataset evaluation relies on target-dataset validation samples (App C.1)

Questions for authors
  • How does the current hesitation measure (based on decision margins of multiple denoising steps) compare to a simple measure based on the final step only? Additionally, the authors could also compare against minimum decision margin and average decision margin approaches.

  • Can the authors report cross-dataset results without tuning tau and lambda on labeled validation samples? For example: by doing the tau and lambda calibration on another, similar dataset / a general dataset? Or assuming some reasonable default values?

The rebuttal has addressed my concerns. Now I feel more confident with the Weak Accept score; thus, I increase my confidence to make this a stronger recommendation.

3 comments
AuthorsOpenReview ↗

Response to Reviewer dEuR (1/2)

We thank the reviewer for the thoughtful feedback and for recognizing the safety motivation, consistent improvements, and breadth of our evaluation. We address your concerns and present additional experimental results below.


W1 & Q1: Comparison with simpler uncertainty scores

A1: Thank you for suggesting a direct comparison with simpler uncertainty scores. We compare routing rules on ToxicChat using LLaDA-8B-Instruct, keeping the trained probes and margin threshold τ\tau fixed.

Let ∣ds∣|d_s| denote the absolute decision margin at denoising step ss, with s=1s=1 denoting the final step. The three alternatives route an example to the second-stage probe when

  • Final-step: ∣d1∣<τ|d_1|<\tau;
  • Multi-step Minimum: min⁡s∣ds∣<τ\min_s|d_s|<\tau;
  • Multi-step Average: 1S∑s=1S∣ds∣<τ\frac{1}{S}\sum_{s=1}^{S}|d_s|<\tau.

D²-Monitor instead routes when nτ>λn_\tau>\lambda, where nτ=∑s=1S1[∣ds∣<τ]n_\tau=\sum_{s=1}^{S}\mathbf{1}[|d_s|<\tau], retaining its original operating point. The results are shown below (F1: mean ± SD over seeds 0,1,2).

Routing ruleMLP E[P]MLP F1 (%)TimeAttn E[P]TimeAttn F1 (%)
Final-step0.32M89.2±0.30.49M88.8±0.6
Minimum margin0.56M89.2±0.30.84M89.0±0.3
Average margin0.20M90.2±0.30.30M89.9±0.2
D²-Monitor (Ours)0.16M90.3±0.20.37M90.1±0.2

D²-Monitor improves mean F1 over final-step routing by 1.1 (MLP) and 1.3 (TimeAttn) percentage points while using fewer expected parameters. This indicates that multiple denoising steps contribute meaningfully. D²-Monitor also outperforms the other multi-step rules, minimum-margin and average-margin routing. Finally, it is easier to calibrate than all three alternatives: its threshold is a count of uncertain steps rather than a raw margin value. We have added this comparison to Appendix D.5 (Table 8), with a summary in Section 5.3.


W2: Connection to previous work on cascaded monitoring

A2: Thank you for the opportunity to clarify our contributions in relation to prior work on cascaded monitoring. We agree that the general cascade design is well established. In Section 2 (lines 144–153), we discuss this literature, including McKenzie et al. [1] and Cunningham et al. [2], and explicitly state in line 153: “We do not claim the bi-level design itself as a novelty.” Our contribution focuses on the diffusion-specific hesitation signal across denoising steps and its use for routing and curating trajectories to train the second-stage probe.


[1] Alex McKenzie et al. Detecting High-Stakes Interactions with Activation Probes. NeurIPS, 2025.

[2] Hoagy Cunningham et al. Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreaks. ICLR, 2026.

AuthorsOpenReview ↗

Response to Reviewer dEuR (2/2)

W3 & Q2: Transfer without labeled target-domain calibration

A3: To address your concern, we remove samples that overlap with the source training or target datasets, then calibrate D²-Monitor on the resulting 1,031-example Aegis validation set [3]. We choose Aegis because its safety-labeled user prompts align with ToxicChat’s conversational toxicity detection [4] and OpenAI-Moderation’s identification of harmful content across safety categories [5]. Following the operating-point selection rule in Appendix C.1, we obtain the following results:

We keep the probes trained on WildGuardMix fixed, select thresholds on Aegis, and evaluate on the full ToxicChat and OpenAI-Moderation test sets. We report F1 (%) as mean ± SD over seeds 0, 1, 2; baseline results are taken from Table 3 of the paper. Best results are in bold, and second-best are underlined‾\underline{\mathrm{underlined}}.

Base and Instruct refer to LLaDA-8B-Base and LLaDA-8B-Instruct, respectively.

MethodBase / ToxicChatBase / OpenAI-ModerationInstruct / ToxicChatInstruct / OpenAI-Moderation
LP (Last)62.5±0.561.4±0.566.5±2.266.4±0.7
MLP (Last)65.1±1.361.2±1.467.8±2.665.0±1.9
LP (MV)65.8±1.359.5±3.567.1±0.763.5±0.8
LP (Mean)66.5±1.459.9±3.567.8±0.563.9±1.0
MLP (MV)62.3±1.360.8±2.170.5±1.362.2±0.6
MLP (Mean)65.4±1.362.1±2.071.1±1.162.7±0.7
TimeAttn67.7±1.565.4±1.972.2±1.167.1±1.8
LSTM66.1±3.065.5±2.072.2±2.167.8±2.2‾\underline{67.8±2.2}
D²-MLP69.6±3.1‾\underline{69.6±3.1}64.9±1.172.9±1.1‾\underline{72.9±1.1}67.6±0.6
D²-TimeAttn69.7±2.165.4±2.9‾\underline{65.4±2.9}73.7±1.368.5±0.4

D²-Monitor maintains strong cross-dataset performance when calibrated on Aegis, without tuning on either target dataset.

We have added these results to Appendix D.6 (Table 9) and summarized them in Section 5.2.


[3] Shaona Ghosh et al. AEGIS2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails. NAACL, 2025.

[4] Zi Lin et al. ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation. Findings of EMNLP, 2023.

[5] Todor Markov et al. A Holistic Approach to Undesired Content Detection in the Real World. AAAI, 2023.

AuthorsOpenReview ↗

Summary of Revisions

We have made the following revisions in response to your feedback:

  • Added the routing-rule comparison to Appendix D.5 (Table 8), with a summary in Section 5.3.
  • Added the independent Aegis calibration experiment to Appendix D.6 (Table 9), including accuracy, F1, and expected test-time parameters, with a summary in Section 5.2.

Comments on the paper

1 comment

Dear all reviewers,

The authors have responded to your concerns for a while. Please go through the response below and the revised submission to check whether your concerns have been adequately addressed, and if there are any follow-up questions where applicable. Thank you!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.