- Rating
- 3
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper investigates safety alignment for text-to-image diffusion models, focusing on the limitations of image-level preference optimization. The authors propose SaTPO, which constructs step-level safety preferences using shared denoising states and noise, together with multi-scale drift regularization and benign replay. Experiments on SD v1.5 and SDXL demonstrate improved safety performance while largely preserving benign generation quality.
Critical strengths
- The proposed method is well-motivated and technically sound. By constructing preference pairs from shared denoising states and noise, SaTPO reduces trajectory-induced variations and provides more controlled supervision for safety alignment.
- The experimental results are convincing. SaTPO demonstrates better safety performance across two diffusion backbones, multiple evaluation datasets, and safety metrics. The improvements also extend to adversarial prompts, supporting the effectiveness of the proposed approach.
- This paper is well-written and easy to follow.
Critical weaknesses
- The safety relevance of the constructed transition preferences could be better demonstrated. While shared-state comparison eliminates trajectory-induced variations, it remains unclear whether the resulting differences primarily reflect safety-related changes rather than unrelated semantic variations. Visualizing the preferred (safe) and rejected (unsafe) branches at different denoising steps could provide more direct insights.
- The contributions of individual drift regularization components remain unclear. The proposed method combines stepwise and trajectory-level drift penalties, but the ablation studies do not independently evaluate their effects. This makes it difficult to determine their respective contributions to the reported perfromance.
- The experiments are limited to SD v1.5 and SDXL, both based on U-Net architectures. Evaluating SaTPO on DiT-based models such as SD v3 would help demonstrate its applicability to more recent text-to-image generation architectures.
Questions for authors
Could the authors clarify why the unsafe-conditioned trajectory is chosen as the shared-state anchor, rather than the safe-conditioned trajectory?
7 comments
Official Comment by Authors(W2 - Part1/2)
W2 — Step-wise and common drift
“The proposed method combines stepwise and trajectory-level drift penalties, but the ablation studies do not independently evaluate their effects.”
We have added separate and joint removals of the two drift terms, retaining benign replay and fixing the data and update budget across the three controls. Following reviewer itda’s suggestion, we also assess benign attribute retention with TIFA [1], which asks visual questions about the benign content explicitly requested in unsafe prompts. MUSIQ [2] complements this evaluation by assessing perceptual image quality without a reference image, including sensitivity to blur and other visual distortions.
Local drift constrains each candidate’s transition log ratio, while common drift constrains their accumulated signed shift across steps. The table therefore examines both individual transitions and their accumulated change, with a separate replay removal for context. Unsafe rates and TIFA are reported as percentages.
\begin{array}{lllllllll} \hline \rlap{\smash{\raise{-0.65em}{\hspace{0em}\textbf{SDXL configuration}}}}\hspace{10.04262em}\vphantom{\textbf{PV}} & \rlap{\hspace{8.62492em}\textbf{I2P-5000}}\rlap{\raise{-0.45em}{\hspace{0.25em}\rule{21.22644em}{0.04em}}}\hspace{2.47767em}\vphantom{\textbf{PV}} & \hspace{2.83754em} & \hspace{2.27483em} & \hspace{5.39917em} & \hspace{4.73723em} & \rlap{\hspace{0.67994em}\textbf{COCO-5000}}\rlap{\raise{-0.45em}{\hspace{0.25em}\rule{6.93081em}{0.04em}}}\hspace{3.5671em}\vphantom{\textbf{PV}} & \hspace{2.86371em} & \rlap{\hspace{0em}\textbf{I2P-1000}}\rlap{\raise{-0.45em}{\hspace{0.25em}\rule{3.97659em}{0.04em}}}\hspace{4.47659em}\vphantom{\textbf{PV}} \\\\ \hspace{10.04262em} & \ORcell{0}{0}{\textbf{PV ↓}} & \ORcell{0}{0}{\textbf{Q16 ↓}} & \ORcell{0.11505}{0.11505}{\textbf{IP ↓}} & \ORcell{0}{0}{\textbf{NudeNet ↓}} & \ORcell{0}{0}{\textbf{MUSIQ ↑}} & \ORcell{0}{0}{\textbf{CLIP ↑}} & \ORcell{0}{0}{\textbf{FID ↓}} & \ORcell{0.41331}{0.41331}{\textbf{TIFA ↑}} \\\\ \hline \ORcell{0}{6.80813}{\text{SaTPO}} & \ORcell{0.21756}{0.21756}{\mathbf{9.30}} & \ORcell{0.3975}{0.3975}{\mathbf{4.10}} & \ORcell{0.11614}{0.11614}{\mathbf{5.12}} & \ORcell{1.8119}{1.8119}{1.04} & \ORcell{1.05999}{1.05999}{\mathbf{65.99}} & \ORcell{0.3964}{0.3964}{0.2602} & \ORcell{0.29444}{0.29444}{27.51} & \ORcell{0.92967}{0.92967}{\mathbf{63.63}} \\\\ \ORcell{0}{4.78521}{\text{− local drift}} & \ORcell{0.10142}{0.10142}{12.60} & \ORcell{0.53108}{0.53108}{7.94} & \ORcell{0.24973}{0.24973}{8.86} & \ORcell{1.8119}{1.8119}{1.02} & \ORcell{1.2312}{1.2312}{63.54} & \ORcell{0.3964}{0.3964}{0.2643} & \ORcell{0.29444}{0.29444}{25.13} & \ORcell{1.10088}{1.10088}{62.20} \\\\ \ORcell{0}{3.11998}{\text{− common drift}} & \ORcell{0.35115}{0.35115}{9.98} & \ORcell{0.53108}{0.53108}{5.26} & \ORcell{0.24973}{0.24973}{6.04} & \ORcell{1.67831}{1.67831}{\mathbf{0.84}} & \ORcell{1.2312}{1.2312}{61.77} & \ORcell{0.18757}{0.18757}{\mathbf{0.2662}} & \ORcell{0.12323}{0.12323}{\mathbf{24.35}} & \ORcell{1.10088}{1.10088}{60.13} \\\\ \ORcell{0}{2.08508}{\text{− both drift terms}} & \ORcell{0.35115}{0.35115}{2.24} & \ORcell{0.53108}{0.53108}{2.36} & \ORcell{0.24973}{0.24973}{2.46} & \ORcell{1.8119}{1.8119}{0.10} & \ORcell{1.2312}{1.2312}{73.96} & \ORcell{0.3964}{0.3964}{0.2621} & \ORcell{0.29444}{0.29444}{30.18} & \ORcell{1.10088}{1.10088}{40.98} \\\\ \ORcell{0}{3.18541}{\text{− benign replay}} & \ORcell{0.10142}{0.10142}{13.98} & \ORcell{0.53108}{0.53108}{8.94} & \ORcell{0}{0}{10.12} & \ORcell{1.8119}{1.8119}{1.38} & \ORcell{1.2312}{1.2312}{62.07} & \ORcell{0.3964}{0.3964}{0.2483} & \ORcell{0.29444}{0.29444}{29.43} & \ORcell{1.10088}{1.10088}{62.49} \\\\ \hline \end{array}$$Official Comment by Authors(W2 - Part2/2)
The individual removals support both drift scales and benign replay. Removing local drift weakens safety and reduces TIFA and MUSIQ. Removing common drift gives the largest TIFA reduction among the single removals, from 63.63% to 60.13%, and lowers MUSIQ from 65.99 to 61.77. This directly supports common drift’s contribution to retaining requested benign attributes and perceptual quality. Removing replay also lowers TIFA and MUSIQ while worsening safety and COCO FID.
Removing both drift terms reduces TIFA to 40.98%, and the displayed I2P outputs collapse into recurring flat, colorful illustrated portraits, substantially impairing the preservation of requested benign attributes. Together with the individual removals, this result supports the complementary roles of local and common drift in maintaining content and generation quality during safety alignment. The collapsed variant is excluded from ranking. All loss-removal controls use 14,790 updates.
Qualitative examples are shown in Figure 9.
We have revised the manuscript to include these component ablations and the attribute evaluation protocol in Section 5.3, Table 4, and Appendices C.11 and C.13.
References
- [1] Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna and Noah A Smith, TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering, ICCV 2023
- [2] Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar and Feng Yang, MUSIQ: Multi-Scale Image Quality Transformer, ICCV 2021
Official Comment by Authors(W1)
We sincerely thank you for the positive feedback and thoughtful suggestions. Below, we address the raised concerns, with details in the revised version.
W1 — Safety relevance of intermediate branches
Following your suggestion, we have visualized both candidate branches at multiple denoising steps and measured their safety with two common readouts. We use continuation to the final image with zero conditioning (∅) and LCM lookahead with zero conditioning. Latent Consistency Models (LCM) [1] distill denoising into a few steps. We use an LCM-LoRA [2] accelerated model to preview intermediate branches in one step, following LCM-Lookahead [3], making local safety and semantic differences visible in image space. Each comparison starts from candidates with the same parent and noise.
Furthermore, Figure 2(b) provides an insight of safety preference at intermediate steps. The mean SafetyLRM gap, defined as the safe rewrite branch score minus the unsafe prompt branch score, is positive early in denoising, when it is largest, then approaches zero toward the end. This indicates that the intermediate branches contain a safety preference signal consistent with their prompt roles.
\hline \textbf{Common readout} & \begin{array}{c} \textbf{Safe branch} \\\\ \textbf{PV (\\%, ↓)} \end{array} & \begin{array}{c} \textbf{Unsafe branch} \\\\ \textbf{PV (\\%, ↓)} \end{array} & \textbf{PV gap (pp)} & \begin{array}{c} \textbf{Mean Q16} \\\\ \textbf{probability gap (pp)} \end{array} \\\\ \hline \text{∅ continuation} & \mathbf{49.38} & 57.86 & 8.48 & 7.66 \\\\ \text{LCM readout} & \mathbf{47.14} & 52.68 & 5.54 & 5.37 \\\\ \hline \end{array}$$ Each readout covers 140 prompt pairs across seven risk categories and eight timesteps, yielding 1,120 paired branches from frozen SDXL Base. Gaps are the unsafe branch minus the safe branch. Both readouts give the safe branch lower average unsafe scores. We display six key timesteps, t = 951, 851, 751, 651, 351, and 101, with denser coverage of early denoising, where the visible branch differences are more pronounced, and later views showing how they evolve. This selection is consistent with SaTPO’s greater weighting of earlier steps, where Figure 2(b) shows stronger safety separation. Figure 10 (p. 29) shows portraits that retain the subject while reducing graphic facial details, and street scenes that retain the crowd and plaza while changing harmful foreground content. Qualitative examples are shown in [Figure 10](https://anonymous.4open.science/r/satpo-rebuttal-figures-92B8/images/figure10_intermediate_branches.png). The TIFA \[4] evaluation in Appendix C.13 provides complementary evidence after training, using visual question answering to check whether generated images retain benign attributes specified in the prompts. We have revised the manuscript to include the readout protocol and results at individual timesteps in Appendix C.14 and Table 21 (pp. 26–27). **References** * \[1] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li and Hang Zhao, Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference, arXiv preprint arXiv:2310.04378, 2023 * \[2] Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li and Hang Zhao, LCM-LoRA: A Universal Stable-Diffusion Acceleration Module, Technical report, arXiv:2311.05556, 2023 * \[3] Rinon Gal, Or Lichter, Elad Richardson, Or Patashnik, Amit H Bermano, Gal Chechik and Daniel Cohen-Or, LCM-Lookahead for Encoder-based Text-to-Image Personalization, ECCV 2024 * \[4] Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna and Noah A Smith, TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering, ICCV 2023