acceptodds
Under review as a conference paper at ICLR 2027

Detecting and Stopping NSFW Generation via Denoising Dynamics

Abstract

Text-to-image diffusion models can synthesize not-safe-for-work (NSFW) content. Existing detection methods primarily assess the content of prompts, completed images, or intermediate states. However, prompt filters are vulnerable to obfuscated intent, while post-generation classifiers incur the full generation cost before detection. In contrast, in-generation detectors inspect intermediate features but rely primarily on static cues from the semantic content of individual states, leaving activation dynamics across denoising steps underexplored. We therefore model harmful dynamics, i.e., temporal patterns of activation motion associated with unsafe content, as a safety signal. Based on this signal, we formulate detection as a constrained sequential decision problem, aiming to stop unsafe generations early while balancing detection errors and sampling cost. Our approach follows an “observe, summarize, and stop” workflow: it observes activation motion during denoising, summarizes its history into risk scores, and stops generation based on accumulated risk. We conduct comprehensive experiments across four diffusion backbones and three benchmarks. Our approach significantly improves in-domain and cross-domain detection, with relative AUC gains of up to 14.7% over the strongest baseline, demonstrating the value of denoising dynamics as a safety signal. In addition, dynamics-based monitoring generalizes to unseen prompt distributions across different diffusion architectures and enables early intervention in unsafe generation with minimal overhead.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
NSFW Detection, In-Generation Detection, Diffusion Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aK3dOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The method selects blocks within model, averages theirs activations across all tokens, and computes angles and distances between consecutive steps over denoising steps. These features are then used for classification (safe or unsafe image). Experiments on several diffusion models show better detection than earlier in-generation methods and some savings from stopping early.

Critical strengths
  1. The idea of watching how the model's internal activations move during denoising (instead of judging a single step) is interesting and seems new to me.
  2. The method is lightweight and adds almost no cost to generation.
  3. The experiments cover several model families, and the early-stopping evaluation is close to how such a tool would actually be used.
Critical weaknesses
  1. The method averages activations across tokens at selected blocks and computes turning angles between consecutive activation displacements and step lengths during denoising. I struggle to find what meaningful information these features capture or why they should distinguish unsafe from benign generation. Although the authors provide quantitative plots, these results do not sufficiently explain why the features are effective.

  2. Benign samples come from COCO captions, while positives come from I2P etc. These differ heavily in prompt style , so activation trajectories could separate them for reasons unrelated to unsafe content.

  3. Some important Post-generation baseline is missing, such as Q16 and MHSC.

Questions for authors
  1. Why the Denoising Dynamics work, what are the extracted feature capture? (W1)
  2. The shortcut issue in W2.
  3. Compare with Q16 and the Multi-Headed Safety Classifier.
1 comment
AuthorsOpenReview ↗

Response to Reviewer aK3d (1/2)

(W1+Q1) Why the Denoising Dynamics work, what are the extracted feature capture?

Response:

We sincerely apologize for this confusion. In short, the two motion features describe "how much" the activations of the whole layer change and "whether the change keeps going in the same direction". We believe they can separate unsafe from benign generations because the formation of unsafe content often comes with larger adjustments whose direction is less consistent. We would like to explain this in three points:

1. What the features capture. At each denoising step, we average the activations of the same block over spatial positions (or image tokens), which gives one response value per channel. The difference between two consecutive steps is then the change in channel responses (averaged over all positions). Pooling just discards where the change happens, but keeps how the response of the whole layer changes. Step length is the size of this change. Turning angle is how well this change agrees in direction with the previous one.

2. Why they can separate unsafe from benign generations. Prior work shows that semantic attributes in diffusion models form within short time windows (Biroli et al., 2024; Li & Chen, 2024). Such transitions in the sampling dynamics are related to the formation of semantic structure (Ramachandran et al., 2025; Handke et al., 2026). Our hypothesis is that, when a generation moves toward unsafe content, the model makes a larger adjustment, with a change of direction. We propose the step length and turning angle measure this adjustment.

Our observations are consistent with this hypothesis. Specifically, we generate images from 300 I2P prompts with SD 1.4, and manually audit the images. Among these images, 67 contain NSFW content and 233 do not. In the middle of denoising, the median turning angle is 23.3° for the NSFW group and 19.5° for the non-NSFW group. The median step lengths are 0.875 and 0.780, respectively. That is, with the same prompt source, the trajectories that finally produce NSFW content change more in the middle of denoising, and their consecutive updates agree less in direction.

3. Why we need to combine the features and the history. For a single feature, the two groups still overlap clearly, so it is hard to judge a single image from one angle at one step. Therefore, the monitor uses the observations from several layers together with the changes at earlier steps. We show in the manuscript that the two kinds of features are complementary: in the I2P evaluation on SD1.4, the AUC is 0.869 with turning angles only, 0.866 with step length only, and 0.907 with both.

Reviewer_ugdEOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Dynamics-based Risk Monitoring (DRM), a framework designed to detect and halt NSFW image generation in text-to-image diffusion models in real time. Instead of analyzing the static content of an intermediate image, DRM focuses on the trajectory of the model's internal activations across denoising steps. By extracting geometric features (how the activations change direction and magnitude), a temporal self-attention monitor evaluates the ongoing risk. The system continuously accumulates this risk, comparing it against a normal baseline, and triggers an early stop to the generation process the moment the accumulated risk exceeds a safe threshold.

Critical strengths

The paper creatively shifts from static text/image detection to analyzing denoising trajectory dynamics. Extracting geometric features (turning angles and step lengths) to model "activation motion" is an innovative and effective approach. Furthermore, the framework formulates early intervention as a sequential decision problem consisting of an observe, summarize, and stop workflow. The risk accumulation and abort gate effectively stop unsafe generation early, saving sampling costs.

A highly consequential strength of this method is its practical deployability. By utilizing read-only probes to extract features from frozen backbones, the monitor requires absolutely no extra forward passes. This enables real-time, in-generation intervention with a negligible runtime overhead, providing a massive advantage for real-world application. This efficiency is further backed by extensive experiments across 4 backbones (U-Net and DiT) and multiple benchmarks (I2P, Ring-A-Bell, T2I-RiskyPrompt)

Critical weaknesses

In the introduction, the paper explicitly claims that shifting the focus from the state content to the denoising dynamics provides the key evidence for detecting unsafe generation. However, the paper lacks a controlled baseline that feeds the raw static content features into the exact same temporal monitor. The ablation studies in Table 4 only test different history aggregation methods while keeping the motion features fixed. Without directly comparing the proposed motion features against the raw content features under identical conditions, the core methodological claim remains unverified.

DRM computes its detection metrics (AUC/F1) using the final risk score at step 48 (total is 50 DDIM steps), observing virtually the entire generated image trajectory. In contrast, baselines such as IGD and FlowGuard_Single operate on step 6, while FlowGuard uses steps 20-30. Consequently, Tables 1 and 2 compare a full-trajectory observer against early single-step observers. This does not prove that motion dynamics are inherently superior to state content, but rather reflects an advantage in sequence length.

The benign dataset consists of COCO captions, whereas the unsafe benchmarks (I2P, Ring-A-Bell, and T2I-RiskyPrompt) differ in prompt length and style. This raises concerns that the detector may exploit dataset-specific distributional cues rather than learn generalizable signals of unsafe content. Notably, a simple keyword filter achieves an AUC of 0.879 on Ring-A-Bell (Table 2a), suggesting that text-level cues alone are highly discriminative. Given that DRM achieves an AUC near 0.99 using only four scalar features per step, further evaluations on style-matched benign prompts or hard negatives are needed to verify robust safety detection beyond dataset biases.

Questions for authors
  1. What are the detection results if the raw static content features are fed directly into the identical temporal monitor, rather than the proposed motion features?
  2. Since baselines like FlowGuard and IGD are constrained to early steps, is DRM's performance gain primarily driven by its full-trajectory access (up to step 48)? Could you evaluate DRM when restricted strictly to the same early budget steps (e.g., step 6 or 20) to ensure a fair comparison?
  3. Could you evaluate DRM on style-matched benign prompts (e.g., long, complex, but safe prompts) or hard negatives to prove the model isn't just exploiting the prompt-length dataset bias?
Reviewer_JXkGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Dynamics-based Risk Monitoring (DRM), which detects the generation of NSFW images by probing internal activations of text-to-image diffusion models. Unlike prior in-generation detectors that classify content based on individual intermediate states, DRM monitors how internal activations move during denoising. At each timestep, DRM first observes selected backbone blocks during generation and extracts turning angles between consecutive activation displacements and the step length of each displacement. Next, DRM summarizes the history of these observations into a risk score at each step. It then accumulates how much these scores exceed the average score of benign generations and aborts sampling once this accumulated value crosses a threshold calibrated on benign trajectories. Across four backbones (SD1.4, SD1.5, SDXL, PixArt-α), DRM achieves higher AUC and F1 than in-generation baselines in all settings, and outperforms pre-generation and post-generation baselines in most settings. DRM also achieves lower Miss than FlowGuard at comparable FA, as well as lower Miss on I2P and lower FA and Miss on Ring-A-Bell than IGD-PCA, which presents effective early intervention across in-domain and cross-domain settings.

Critical strengths

1. Denoising dynamics as a safety signal

Unlike prior in-generation detectors, which classify the content of individual intermediate states (the noise prediction in IGD and the decoded latent in FlowGuard), DRM uses the motion of internal activations across denoising steps. DRM can detect NSFW generation using only four values per step derived from existing model activations. The ablations show that both turning angles and step lengths contribute to detection performance, while using the full observation history further improves it. I think this is a cheap and promising direction that has not been explored much before.

2. Explicit formulation of the stopping problem

Rather than evaluating detection accuracy at a fixed timestep, DRM considers when to stop generation and evaluates false aborts, misses, and sampling cost separately. Since the paper applies this protocol to the baselines as well, it is not tied to DRM and provides a practical evaluation framework for in-generation detection.

Critical weaknesses

1. Differences in evaluation settings

Different evaluation steps. In Tables 1–2, DRM is evaluated using rTr_T, which uses all 48 observation steps. In contrast, IGD and FlowGuard_Single use step 6, while FlowGuard takes the maximum over steps {20, 22, 24, 25, 27, 30} (App. B.4). The reason for this choice of steps is not explained. As a result, DRM has access to substantially more information at evaluation time. Fig. 5a partly addresses this concern by showing that DRM reaches 0.991 AUC on Ring-A-Bell by step 12. However, this result is limited to SD1.4, does not include baseline curves, and uses a setting where the baselines already perform relatively well. A comparison at matched timesteps would help clarify how much of DRM's improvement comes from its proposed features.

Limited post-generation baselines. The post-generation baselines are Falconsai and SD Safety Checker, both relatively old methods.

2. The central claim is not directly tested

The paper argues that denoising dynamics provide a useful safety signal beyond the static content of intermediate states. However, the ablations (Tables 3–4) only compare variants of the motion features. No experiment trains the same monitor on content features (e.g., the pooled activations vtv_t from the same blocks and steps) instead of motion features. So it remains unclear whether DRM's gains come from using dynamics or from the monitor aggregating information over many steps.

3. Generalization to other unsafe categories.

All unsafe candidates are screened using NudeNet, so the evaluation focuses on sexual content. Since DRM relies on distinctive activation motion rather than directly modeling semantic content, it is unclear whether the learned patterns generalize to other unsafe categories, such as violence, or are specific to nudity. Evaluating DRM on a violence detection task would help assess the generality of the proposed signal.

Minor

Model and sampler coverage. All evaluated backbones use DDIM sampling, while flow-matching models such as SD3 and Flux and other samplers are not considered. In addition, probe blocks are selected separately for each backbone, leaving the robustness of the method to different model architectures unclear.

Benign set. The benign class consists only of COCO captions. Evaluating on more diverse benign prompt sets would make the false-abort results more convincing.

Questions for authors

1. Recent post-generation detectors. How does DRM compare with recent VLM-based safety checkers such as LlavaGuard or ShieldGemma 2? Although these methods operate after generation, they would provide a useful reference for assessing DRM's detection performance against modern image safety classifiers.

2. Choice of the conditional branch. DRM uses only conditional-branch activations under classifier-free guidance (Sec. 3.2). What motivates this choice? Since these activations depend on the prompt embedding, the motion features may partly reflect prompt information rather than the evolving image. How does DRM perform when using the unconditional branch or the difference between the two branches?

3. Content-feature control. Could the authors train the same temporal monitor using content features, such as the pooled activations vtv_t from the same blocks and steps, instead of motion features? This would help isolate the contribution of activation dynamics from that of temporal aggregation.

4. Other unsafe categories. Does DRM generalize to other unsafe categories, such as violence (e.g., the violence subset of Ring-A-Bell)?

5. Generalization to new backbones. Can DRM be extended to other architectures, such as flow-matching models like SD3?

6. Failure cases. Could the authors show some examples of false aborts and misses, and comment on whether these failures share common patterns?

4 comments

Thank you for the detailed response and the additional experiments. The content-feature control and the matched-step comparison address my main concerns, and the unconditional-branch result (Table R9) is also informative.

I have one remaining question about the post-generation detectors. The labels are defined from the final images, so post-generation checkers should act as an upper bound. Yet ShieldGemma 2 and LlavaGuard score well below DRM. Were these models prompted with a policy that matches the paper's label definition, i.e., asking whether specific body parts are exposed, as NudeNet does? If their default policies were used, they may target a different notion of unsafe content (e.g., sexually explicit content rather than nudity), which may partly explain the gap. Both models accept custom policies, so results with an aligned policy would make the comparison more informative.

Dear Reviewer JXkG,

Thank you very much for the follow-up. We are glad that the content-feature control and the matched-step comparison address your main concerns.

Policies of the post-generation detectors. In Table R2, both detectors used their default policies. ShieldGemma 2 used its default sexually explicit policy, and LlavaGuard used its default nine-category policy.

Following your suggestion, we re-evaluate both detectors with an aligned policy that asks whether specific body parts are exposed. Specifically, the policy lists five body parts: female genitalia, male genitalia, female breasts (including nipples), anus, and buttocks. For ShieldGemma 2, we replace the text of the default policy with these five categories and keep the probability of "Yes" as the score. For LlavaGuard, the policy asks the model to output "Unsafe" if the image meets any of the five conditions, and "Safe" otherwise. Its score is the softmax probability of Unsafe over the logits of the Unsafe and Safe tokens at the rating position. The F1 thresholds are re-computed under the aligned policy, using the median I2P-validation score for each backbone. All other settings follow Table R2.

Table R10. Post-generation detectors with default and aligned policies. We report the mean AUC and F1 across the 12 backbone–dataset combinations.

MethodsPolicyAUCF1
ShieldGemma 2Default0.7030.644
ShieldGemma 2Aligned0.9160.843
LlavaGuardDefault0.8060.726
LlavaGuardAligned0.9390.878
DRM–0.9250.854

With the aligned policy, the mean AUC of ShieldGemma 2 rises from 0.703 to 0.916, and that of LlavaGuard rises from 0.806 to 0.939. This confirms that the policy mismatch accounts for the gap in Table R2. Under the aligned policy, the two post-generation detectors and DRM perform at a comparable level. Both detectors inspect the completed image with a vision-language model, and thus serve as a reference for what is achievable after generation. DRM reaches a comparable level during denoising using four values per step, and can stop unsafe generations before the image is completed.

We sincerely appreciate this suggestion, which makes the comparison with post-generation detectors more informative.

Thank you for the additional results. They address my concerns, and I have raised my score.

One suggestion for the revision: since aligned post-generation checkers reach a similar AUC to DRM's final-step score, the main practical benefit of DRM lies in early stopping. It would strengthen the paper to compare DRM with an aligned checker such as LlavaGuard applied to the VAE-decoded x^0\hat{x}_0 (via Tweedie's formula) at intermediate steps (e.g., steps 20 and 30), in terms of both accuracy and cost.

Thank you very much for your thoughtful feedback and for taking the time to review our additional results! We sincerely appreciate your suggestion on comparing DRM with post-generation checkers at intermediate steps and will add it in the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.