- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes Safe Latent Control Flow (SLCF), a training-free method for steering diffusion models away from unsafe outputs. The authors formulate safe sampling as staying close to the pretrained sampling process in path measure KL while limiting the probability that the final output exceeds a threshold on a learned safety score. In practice, the method applies guidance through a CLIP-based safety probe evaluated on the predicted clean sample. It then switches off guidance separately for each sample using thresholds calibrated on uncontrolled trajectories. Under assumptions on the pretrained model’s continuation probabilities, the authors bound the surrogate violation rate among samples whose guidance switches off. Experiments cover nudity suppression, inappropriate content on CoPro, and memorization on Imagenette, with additional toy-model results.
Critical strengths
- The safety and image-quality results are strong, particularly on MMA-Diffusion. SLCF achieves an ASR of 0.181 compared with 0.297 for SLD + SGF and 0.423 for SAFREE + SGF, while its COCO CLIPScore remains close to the base model. The additional evaluation with a second detector is useful for checking whether these gains extend beyond the detector used to fit the safety probe.
- The paper reports more than the final unsafe rate. Table 4 includes the fraction of samples whose guidance switches off, their surrogate violation rate, and the disagreement between the surrogate and the evaluator. This helps distinguish satisfying the learned safety criterion from satisfying the external evaluator, which matters for interpreting the guarantee.
Critical weaknesses
- The novelty of the stated “key insight” is limited. Sec. 3 describes the insight as follows: once control stops, the sample follows the pretrained model, and its future depends on the current state rather than on the control applied before it. This is the Markov property of the sampling process, and it also applies to existing fixed-window methods such as Safe Denoiser and SGF. The contribution appears to be estimating when to stop and calibrating that decision, rather than the observation itself. Similarly, Eq. 8 applies the gradient of a classifier evaluated at the predicted clean sample directly to the latent. This is related to training-free guidance with off-the-shelf classifiers, such as Universal Guidance [1], which the paper does not discuss. The closest safety-specific method is STG [2], which also evaluates a safety function on the predicted clean sample but updates the text embedding instead of the latent. The paper discusses STG but reports only its runtime (Table 10). A safety and quality comparison would help establish the contribution relative to this closely related approach.
[1] Bansal, A., Chu, H.-M., Schwarzschild, A., Sengupta, S., Goldblum, M., Geiping, J., & Goldstein, T. Universal Guidance for Diffusion Models. ICLR 2024
[2] Na, B., Kang, M., Kwak, J., Park, M., Shin, J., Jun, S., Lee, G., Kim, J.-H., & Moon, I.-C. Training-Free Safe Text Embedding Guidance for Text-to-Image Diffusion Models. NeurIPS 2025.
Questions for authors
- How much of the improvement comes from the adaptive stopping rule rather than the CLIP-probe guidance? In Table 11, the fixed-step arm reaches a similar MMA-Diffusion ASR (0.185 vs. 0.181) with similar FID and CLIPScore. Could the authors clarify where adaptive stopping helps most beyond the surrogate violation rate, and whether it also improves an existing method such as SGF or Safe Denoiser?
- What does mean in practice for a user choosing a safety level? Proposition 1 bounds violations of among samples whose guidance switches off. The bound under the external evaluator additionally depends on , , and the stopping coverage, which are only known after generating and labeling outputs. What safety statement can a user make on a new prompt set before obtaining evaluator labels for those outputs?
- The guarantee assumes that from any state where SLCF stops, the base model rarely ends up unsafe. This is calibrated on uncontrolled trajectories, while SLCF stops at states reached under guidance. The violation rates among stopped samples in Table 4 are higher than the calibrated levels in Table 7. Could the authors comment on this gap, and possibly check the assumption by running the base model forward from the states where SLCF stops?
- Could the authors report safety and image-quality results for STG on the nudity and CoPro benchmarks, alongside the runtime in Table 10?
Other notes:
- Section 2.2 assumes a fair amount of stochastic control background. A short intuitive explanation would help readers from the diffusion-safety side.
- Table 4 has a formatting issue around the “Unsafe” headers at line 437.
8 comments
Response to Q1: contribution of the safety function and adaptive stopping
[Q1-1] Contribution of the safety function and the control (continued)
Either component alone falls short, as the probe under SGF's guidance (B) stays at SGF's level and MMD under SLCF's guidance (C, E) is less safe than SGF. Quality tracks the guidance, not . Every row with SLCF's guidance (C–F) keeps quality close to no control, whatever the safety function and the stopping rule, while both rows with SGF's guidance (A, B) lose quality in every metric (F against A, paired for HPS and 19.9 for PickScore). Safety requires the combination. Under SGF's guidance, swapping MMD for the probe does not move ASR, since SGF's push does not follow . Under SLCF's guidance, the same swap lowers ASR from 0.860 to 0.195, because SLCF moves along the gradient of and benefits from a more accurate . With the safety function held fixed, the improvement persists. Keeping the probe and replacing SGF's guidance by SLCF's (B → F) lowers ASR on all three attack sets and improves quality, and keeping MMD (A → C, E) still yields the gain in quality.
[Q1-2] Benefit of adaptive stopping
We first clarify that the fixed stop in Table 11 is the mean stopping time of , so it is available only after is solved. Even so, neither fixed stop is certified at on hold, while is (Table R4 below).
Advantage over fixed stopping.
Without , a fixed stop must be searched for each task, as fixed-window methods do by publishing a setting per task, and a task without one, such as CoPro for SGF, needs a new search (Appendix D). Adaptive stopping sets the stopping time of each task from calibration data alone. Even within one task, prompts need different stopping times, and adaptive stopping sets one per prompt. Grouping the prompts by their stopping time under , the fixed stop at 0.46 stops the prompts that holds longer too early, leaving 0.526 of them above against 0.132 for . Meanwhile, stops the 118 prompts that can go early at the first admissible time. We add this grouping to Appendix F.6.
[Table R3. on the first 200 MMA-Diffusion prompts, grouped by the stopping time under .]
| Stopping time under | Prompts | Fixed | |
|---|---|---|---|
| (first admissible time) | 118 | 0.085 | 0.093 |
| 41 | 0.098 | 0.195 | |
| 38 | 0.132 | 0.526 |
This per-prompt timing also gives higher quality (HPS +0.0016, paired , FID 72.05 against 72.44, CLIPScore 31.25 against 31.17) and a lower unsafe rate under Falconsai, a classifier that never saw (0.041 against 0.051).
Comparison of stopping rules.
To see the effect of adaptive stopping, we compared five stopping rules on the same seeds, namely no control (), fixed stops at (the end of SGF's window) and (the mean stopping time of ), the adaptive stopping , and control to the end (). We report ASR on the three attack sets, the violation rate with its hold certificate, the path cost, wall-clock time, and quality on COCO-1k (FID, CLIPScore, HPS and PickScore). Section 6.4 of the revision reports this comparison in Table 5.
[Table R4. Stopping rules with the probe and the SLCF control fixed (nudity). Violation and time on MMA-Diffusion, Hold UCB on the 200 hold prompts. Bold: .]
| Stop | MMA | RAB | UD | Hold UCB | Cost (nats) | Time (s) | FID | CLIP | HPS | Pick | |||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| No control | 0.967 | 0.823 | 0.802 | – | – | 0.989 | – | 0 | 3.25 | 71.24 | 31.46 | 0.2593 | 21.61 |
| 0.195 | 0.063 | 0.121 | 1.000 | 0.220 | 0.220 | 0.181 | 4.28e4 | 5.26 | 71.86 | 31.18 | 0.2567 | 21.43 | |
| 0.185 | 0.076 | 0.138 | 1.000 | 0.179 | 0.179 | 0.148 | 4.10e4 | 5.54 | 72.44 | 31.17 | 0.2559 | 21.40 | |
| (ours) | 0.181 | 0.051 | 0.103 | 0.990 | 0.099 | 0.106 | 0.095 | 4.33e4 | 5.60 | 72.05 | 31.25 | 0.2576 | 21.42 |
| (never stop) | 0.154 | 0.051 | 0.121 | 0.000 | – | 0.008 | 0.031 | 4.22e4 | 8.06 | 72.05 | 31.16 | 0.2560 | 21.38 |
As the reviewer notes, ASR barely changes with the stopping rule (0.154–0.195), and the rules differ instead in the certified level, the time and the quality. Of the rules that stop early, only is certified at on hold (0.095), while neither fixed stop is (0.181 at 0.40 and 0.148 at 0.46), even the one with the same mean stopping time as . Control to the end is also certified (0.031) but runs 1.44 times longer (8.06 against 5.60 s). Compared with it, adaptive stopping also has higher HPS (+0.0016, paired ) and PickScore (+0.040, ) at the same FID.
Response to Q1: Benefit of adaptive stopping
[Q1-2] Benefit of adaptive stopping (continued)
Advantage over control to the end.
Stopping at different certified levels. Stopping pushes only as far as requires. We varied in one calibration, keeping the calibration data, , , prompts and seeds, in the two-dimensional model and in Stable Diffusion.
[Table R5. The two-dimensional model, one calibration, same prompts and noise for every arm (N = 2000). Near-arc columns: the 465 samples nearest the unsafe arc. Displacement: distance from the uncontrolled output of the same noise.]
| Rule | Mean stopping time | Path KL | Displacement | Near arc: | Near arc: median displacement | |
|---|---|---|---|---|---|---|
| 0.000 | 0.96 | 3.15 | 0.33 | 0.914 | 0.92 | |
| 0.012 | 0.81 | 2.82 | 0.27 | 0.729 | 0.81 | |
| 0.040 | 0.27 | 2.40 | 0.19 | 0.609 | 0.67 | |
| 0.053 | 0.24 | 2.25 | 0.16 | 0.544 | 0.61 | |
| 0.069 | 0.20 | 1.99 | 0.12 | 0.402 | 0.50 | |
| Control to | 0.000 | 1.00 | 3.16 | 0.33 | 0.923 | 0.93 |
In the two-dimensional model, a larger stops earlier and keeps the output closer to the pretrained model (path KL 3.15 → 1.99, displacement 0.33 → 0.12), with the violation rate below at every level. Control to the end keeps pushing samples that already meet the criterion, pushing 92% of those near the unsafe region below (61% at ) at a path KL 32% above that of . We add a figure to Appendix H.1 showing that stopped outputs stay just outside the unsafe region while outputs controlled to the end are pushed further away.
In SD, a larger also stops earlier (0.46 → 0.37), and keeps higher HPS (0.2573 against 0.2560, paired ) and PickScore (21.43 against 21.38, ) than control to the end.
Two branches from the same stopping state. Continuing the control after the stopping state lowers further but adds almost no safety under the evaluators and barely changes the content. We completed the stopping states of one trajectory controlled to the end with the pretrained model under the same subsequent noise, on the 197 MMA-Diffusion prompts and 96 COCO captions stopped by . We add this comparison to Appendix F.6.
[Table R6. Branches of one controlled trajectory from the same stopping states.]
| Branch | NudeNet unsafe | Falconsai unsafe | LPIPS to (MMA / COCO) | HPS (COCO) | CLIPScore (COCO) | |
|---|---|---|---|---|---|---|
| Uncontrolled | 0.995 | 0.985 | 0.944 | 0.707 / 0.365 | 0.2621 | 31.42 |
| Fixed | 0.218 | 0.213 | 0.066 | 0.024 / 0.023 | 0.2583 | 31.32 |
| Fixed | 0.198 | 0.203 | 0.051 | 0.014 / 0.018 | 0.2582 | 31.32 |
| 0.096 | 0.208 | 0.041 | – | 0.2581 | 31.25 | |
| Control to | 0.000 | 0.193 | 0.025 | 0.013 / 0.013 | 0.2572 | 31.24 |
The stopped branch already meets the certified level (0.096), and continuing lowers it to 0 but leaves the labels of NudeNet and of Falconsai statistically unchanged (McNemar ) and the content nearly unchanged (LPIPS 0.013, 2–4% of the change made by the whole control).
Attaching the stopping rule to SGF.
To test whether the stopping rule also improves SGF, we kept SGF unchanged and replaced only the end of its window by read on the probe , continuing SGF's push to the last noisy step for samples not stopped.
[Table R7. attached to SGF (nudity).]
| Guidance (push) | Stopping | MMA | RAB | UD | FID | CLIP | HPS | Pick | |
|---|---|---|---|---|---|---|---|---|---|
| SGF (MMD) | window | 0.621 | 0.190 | 0.431 | 1.000 | 76.40 | 30.77 | 0.2477 | 20.71 |
| SGF (MMD) | (probe ) | 0.883 | 0.696 | 0.793 | 0.037 | 75.95 | 30.74 | 0.2406 | 20.69 |
| SLCF (probe) | (probe , ours) | 0.181 | 0.051 | 0.103 | 0.990 | 72.05 | 31.25 | 0.2576 | 21.42 |
The stopping rule does not carry over to SGF, since it needs a safety function it can calibrate and a control that lowers it. SGF's push does not lower the probe below the threshold, so stops only 3.7% of samples, and the rest receive SGF's re-noising to the last noisy step. This raises ASR (0.621 → 0.883 on MMA-Diffusion) and lowers quality (HPS 0.2406), as the SGF paper also reports for guidance beyond its published window.
Revision.
Section 6.4 compares the stopping rules, Appendices F.6 and H.1 add the branches and the two-dimensional comparison, and Appendix H.8 shows examples.
Response to W1 & Q4, Q1: Key insight, comparison with STG and contribution of the safety function and the control
We thank the reviewer for the thoughtful review. We added a calibration procedure whose guarantee covers the deployed rule, a direct comparison with STG, and ablations of the stopping rule and the safety function. We have uploaded a revised paper, with the changes marked in purple. We address each point in detail below.
[W1-1] Novelty of the key insight
By "key insight" we meant the observation that motivates the calibration, not a claimed contribution. Once the control stops, the risk under the pretrained model depends on the state alone, so it can be measured by pretrained-model completions from that state. This is what lets us calibrate when to stop, per time and per sample, and certify the deployed rule. Fixed-window methods share the same property but have no such criterion for when to stop. The contribution is this calibration and its certification.
[W1-2, Q4] UG and STG
We agree that our control is related to training-free guidance such as Universal Guidance. The submission discussed only the control view, since the stopping rule and its probabilistic guarantee follow from the control formulation. We add the relation to Universal Guidance in Section 3.1 and Appendix D.3.
STG. STG addresses the same task but needs access to a differentiable evaluator during generation, while SLCF and our baselines assume that only reference data are available, so STG was not in our main tables. Such access is often unavailable in practice, as our NudeNet classifier is distributed in ONNX format. STG utilizes the gradient of a differentiable NudeNet detector, which differs from the NudeNet classifier of Table 2, and measures DSR with a detector of the same family. We ran STG at its released setting with the same prompts and seeds and scored it with both evaluators, with quality measured on COCO-1k and time per image on MMA-Diffusion. STG's released code provides only the nudity reward, so on the 994 CoPro prompts we ran it as released.
[Table R1. Nudity and CoPro. Classifier: the classifier of Table 2. 1 − DSR: the evaluator of the STG paper. Q16: CoPro unsafe rate. Bold: best per column among STG and SLCF.]
STG scores well under its own detector but poorly under the classifier of Table 2, with a gap of 0.53–0.73 between the two, whereas SLCF stays low under both, with a gap of at most 0.124. STG lowers the score of its own guidance detector without removing the content, as that detector flags 1.6% of its outputs on MMA-Diffusion while the classifier flags 88.1%. In quality, SLCF has higher HPS and PickScore than STG (paired z 8.4, 7.0). STG takes 2.4 times as long as SLCF, since it differentiates the detector through the denoiser. On CoPro, STG stays at the base model's level (0.207 against 0.217), since extending it to a new concept needs a differentiable evaluator of that concept, while SLCF needs only new reference data. We add the comparison with STG in Appendix H.7.
[Q1-1] Contribution of the safety function and the control
The gain comes from the probe Φ together with the SLCF guidance that pushes along it. To separate the safety function from the control, we tested combinations of three components, the safety function (SGF's MMD or our CLIP probe), the guidance (SGF's re-noising repellency or SLCF's control) and the stopping (the end of SGF's window, t=0.40, or the calibrated y∗). For SGF with the probe, we replaced only the direction of its push by −∇Φ and kept the step norm of SGF's MMD push. All rows use the same calibration, sampler and seeds, with quality on COCO-1k. We add this matrix to Appendix G.5.
[Table R2. Safety function × guidance and stopping (nudity). With MMD, adaptive stopping never fires. Bold: best among A–F.]