acceptodds
Under review as a conference paper at ICLR 2027

Steer and Pace: Joint Safety Guidance for Text-to-Image Generation

25 views

Abstract

Text-to-image (T2I) generative models produce high-quality images from free-form prompts, but can also generate unsafe content. Existing sampling-time safeguards often aggregate multiple unsafe categories into one condition or handle them one at a time, limiting their ability to control the contributions of different categories jointly. A further challenge is determining how to allocate correction strength across sampling stages that play different roles in image formation. To address these challenges, we propose SafeConductor, a training-free method that combines joint correction with phase-aware allocation for safe text-to-image generation. At each iteration, SafeConductor formulates joint correction as minimizing changes to the current guidance under separate non-positive projection constraints for all categories. The resulting high-dimensional problem reduces to a nonnegative quadratic program with one coefficient per category, accounting for interactions among unsafe directions. We approximately solve this problem using cyclic coordinate descent. The correction is then applied with a smooth sinusoidal allocation that emphasizes the intermediate stage and reduces intervention near the trajectory endpoints, aiming to preserve image composition and visual details. Extensive experiments across four T2I safety benchmarks demonstrate that SafeConductor achieves state-of-the-art harmful content mitigation on both SD1.5 and FLUX.2 while maintaining competitive prompt alignment and generation quality. Our code is available in the supplementary material.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Text-to-image generation, AI safety, Classifier-free guidance

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_6sSeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SafeConductor, a training-free sampling-time safeguard for text-to-image models with classifier-free guidance. For each of seven unsafe categories, it builds a direction from the difference between predictions under paired unsafe and safe descriptions. It then corrects the prompt guidance by finding the vector closest to it with non-positive projections onto all category directions, which reduces to a nonnegative quadratic program over one coefficient per category, solved by cyclic coordinate descent. The correction is scaled by a sinusoidal schedule that emphasizes the intermediate sampling stage. The method is evaluated on SD1.5 and FLUX.2 (and FLUX.1-dev in the appendix) using ASR on I2P, T2VSafetyBench, Unsafe Diffusion, and CoProV2, FID and CLIP score on MS-COCO, a combined SQA score, and adversarial prompts from P4D, Ring-A-Bell, and MMA-Diffusion.

Critical strengths
  • SafeConductor achieves the lowest ASR on all four safety benchmarks on both SD1.5 and FLUX.2 (Table 1), and compared with the sampling-time baselines SLD-Max and CASG, it also improves FID and CLIP score on both backbones. The results hold on FLUX.1-dev (Table 7) and across all seven I2P categories (Table 8), and SafeConductor also achieves the lowest average ASR on adversarial prompts (Table 4). These consistent results across backbones and benchmarks are the main reason for my positive assessment.

  • The joint correction is a reasonable extension of single-direction correction. Rather than correcting only the category direction most aligned with the current guidance, as in CASG, it determines the coefficients of all category directions together, and Table 2 shows lower ASR than independent correction (5.38% vs. 9.04%) with similar FID and CLIP score. The method requires no training and applies to both noise and velocity predictions.

Critical weaknesses
  • Each of the two main components is ablated on only one backbone: the joint correction on SD1.5 (Table 2) and the phase-aware allocation on FLUX.2 (Table 3), without explanation or corresponding cross-backbone ablations in the appendix. Since the paper claims architectural transferability across noise and velocity prediction, it is unclear whether the benefit of each component holds on the other backbone. In addition, the paired unsafe and safe descriptions, which are intended to reduce the influence of shared semantic content (Section 3.2), are not evaluated separately; the gap between the joint correction and the full SafeConductor in Table 2 (5.38% vs. 4.81% ASR) combines their effect with that of the sinusoidal allocation.

  • The phase-aware allocation is one of the two main contributions, but it is compared only with constant and single-window schedules of the proposed correction (Table 3). Dynamic Negative Guidance (Koulischer et al., 2025), which the paper discusses in Section 2.2 as modulating negative guidance according to sampling time and state, is not included as a baseline or as an alternative allocation. It is therefore unclear whether the sinusoidal allocation offers an advantage over such adaptive time-dependent guidance.

Questions for authors
  • (Clarification) Why is the joint correction ablated only on SD1.5 and the phase-aware allocation only on FLUX.2? How do both ablations look on the other backbone?
  • (Additional result) What is the effect of the paired unsafe and safe descriptions alone, for example, compared with directions computed against the empty condition under the same setting?
  • (Additional result) How does the sinusoidal allocation compare with Dynamic Negative Guidance applied to the same joint correction?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 6sSe

We sincerely appreciate your careful review and your recognition of our method’s consistent improvements across backbones and benchmarks. In response to your valuable suggestions, we provide additional ablations, comparisons with alternative strength allocation methods, and further analysis of the contribution of paired descriptions below.

W1 Component ablations across architectures

Thank you for suggesting these complementary ablations. We provide the comparison of strategies for handling concurrent unsafe concepts on FLUX.2 in Table 1 and the comparison of correction strength allocations across sampling stages on SD1.5 in Table 2. ASR is evaluated on I2P, while FID and CLIP are evaluated on COCO, following the sampling settings used in the main paper.

Table 1. Comparison of strategies for handling concurrent unsafe concepts on FLUX.2.

MethodASR (%) ↓FID ↓CLIP ↑SQA ↑
Native34.1572.2432.1579.41
SLD-Max16.3189.2129.4484.80
CASG16.6795.2928.6282.60
Equal-weight mean17.9587.1129.7684.68
Independent correction18.5081.5830.0885.98
Joint correction (Ours)15.1881.5330.0387.77
SafeConductor (Ours, full)7.7075.6930.3993.62

On FLUX.2, we follow the experimental design of Table 2 in the main paper and evaluate the strategies without paired unsafe–safe descriptions or sinusoidal strength allocation. Among these correction strategies, joint correction achieves the lowest ASR and highest SQA while maintaining competitive image quality and prompt alignment. These results support the value of jointly optimizing category coefficients to account for interactions among unsafe directions. We also include the full SafeConductor method, which further improves safety, image quality, and prompt alignment.

Table 2. Comparison of correction strength allocations across SD1.5 sampling stages.

MethodASR (%) ↓FID ↓CLIP ↑SQA ↑
Native42.5069.8131.5973.02
Peak-matched constant3.2792.2824.9885.95
Mean-matched constant7.0473.7830.0793.92
Early-only9.27111.5423.5477.76
Intermediate-only11.4271.0830.1592.51
Late-only27.0369.8431.3784.24
Sinusoidal (Ours)4.8172.1229.5195.13

On SD1.5, we follow the experimental design of Table 3 in the main paper, retaining paired descriptions and joint correction while comparing different strength allocations. Maintaining the peak strength throughout sampling provides the strongest suppression but substantially degrades image quality and prompt alignment. Among the window controls, early-only correction provides the strongest suppression but causes the greatest quality loss, whereas intermediate-only and late-only correction preserve quality better but offer weaker suppression of unsafe content. Sinusoidal allocation achieves lower ASR than the mean-matched constant and all three windows while avoiding the substantial quality and alignment losses of the peak-matched constant. The original and additional ablations support the effectiveness of joint correction and phase-aware strength allocation on both SD1.5 and FLUX.2.

In addition, to address your concern about the role of unsafe–safe description pairs, we present further experimental results in Table 3. Both variants retain joint correction and sinusoidal strength allocation, using the sampling and evaluation settings reported in the main paper. ASR is evaluated on I2P, while FID and CLIP are evaluated on COCO.

Table 3. Comparison of SafeConductor with and without unsafe–safe description pairs.

BackboneConfigurationASR (%) ↓CLIP ↑FID ↓SQA ↑
SD1.5SafeConductor (w/o paired descriptions)8.8729.6871.5393.38
SD1.5SafeConductor (full, w/ paired descriptions)4.8129.5172.1295.13
FLUX.2SafeConductor (w/o paired descriptions)19.8830.3785.4184.44
FLUX.2SafeConductor (full, w/ paired descriptions)7.7030.3975.6993.62

As shown in Table 3, the full configuration with unsafe–safe description pairs achieves lower ASR and higher SQA on both backbones. The improvement is especially substantial on FLUX.2, where ASR decreases from 19.88% to 7.70%, a relative reduction of approximately 61%, while FID improves from 85.41 to 75.69. These results demonstrate the value of the full paired-description configuration in achieving stronger safety.

AuthorsOpenReview ↗

Response to Reviewer 6sSe

W2 Alternative allocation functions and adaptive guidance

Thank you for highlighting the need to compare our sinusoidal allocation with other strength allocation methods. Beyond the constant and single-window controls in the main paper, we compare it with alternative functions on FLUX.2 in Table 4. We follow the experimental settings of Table 3 in the main paper, retaining paired descriptions and joint correction and changing only the allocation function. All continuous functions are normalized to a peak of one and use the same strength parameter γ=3\gamma=3. ASR is evaluated on I2P, while FID and CLIP are evaluated on COCO.

Let τi=i/(T−1)\tau_i=i/(T-1) denote the normalized sampling progress at iteration ii, where TT is the number of sampling iterations. We define the correction strength as si=γf(τi)s_i=\gamma f(\tau_i), where ff is a strength allocation function and γ\gamma controls the overall scale of the correction strength. Besides the sinusoidal function fSinusoidal(τ)=sin⁡(πτ)f_{\mathrm{Sinusoidal}}(\tau)=\sin(\pi\tau), we compare the sine-squared function fSine-squared(τ)=sin⁡2(πτ)f_{\mathrm{Sine\text{-}squared}}(\tau)=\sin^2(\pi\tau), the parabolic function fParabolic(τ)=4τ(1−τ)f_{\mathrm{Parabolic}}(\tau)=4\tau(1-\tau), and a Gaussian function centered at τ=0.5\tau=0.5 with width parameter σ=0.20\sigma=0.20. We also consider two functions derived from the Beta distribution: fBeta(2,3)(τ)=274τ(1−τ)2f_{\mathrm{Beta}(2,3)}(\tau)=\frac{27}{4}\tau(1-\tau)^2, fBeta(3,2)(τ)=274τ2(1−τ)f_{\mathrm{Beta}(3,2)}(\tau)=\frac{27}{4}\tau^2(1-\tau). Their continuous peaks occur at τ=1/3\tau=1/3 and τ=2/3\tau=2/3, respectively, allowing us to examine allocations that place the strongest correction earlier or later in the sampling trajectory.

Table 4. Comparison of correction strength allocation functions on FLUX.2.

Allocation functionASR (%) ↓FID ↓CLIP ↑SQA ↑
Native34.1572.2432.1579.41
Sine-squared11.8073.0531.2492.85
Parabolic6.6677.9829.7692.97
Gaussian13.5273.1531.3591.94
Beta(2,3)7.8779.8029.9691.98
Beta(3,2)16.0173.3931.7090.68
Sinusoidal (Ours)7.7075.6930.3993.62

As shown in Table 4, sinusoidal allocation achieves a relatively low ASR and the highest SQA among the compared functions. The parabolic function achieves a lower ASR but degrades image quality and prompt alignment. At the same peak parameter, the parabolic function has a larger area and a higher sum of strength coefficients than the sinusoid, which may help explain its stronger suppression but worse image quality and prompt alignment. Sine-squared and Gaussian functions better preserve image quality and prompt alignment but provide weaker suppression of unsafe content. These results support our choice of sinusoidal allocation to balance safety, image quality, and prompt alignment.

Regarding your suggestion to compare with adaptive guidance, We further compare sinusoidal allocation with DNG weighting on SD1.5 in Table 5. DNG adaptively scales negative guidance using an estimated posterior of the undesired condition. Following its formulation for diffusion models, we evaluate it on SD1.5, retaining unsafe–safe description pairs and joint correction while replacing sinusoidal allocation with DNG weighting. The comparison follows the SD1.5 sampling and evaluation settings used in Table 1 of the main paper. ASR is evaluated on I2P, while FID and CLIP are evaluated on COCO.

Table 5. Comparison of sinusoidal allocation and DNG weighting for joint correction on SD1.5.

MethodASR (%) ↓FID ↓CLIP ↑SQA ↑
Native42.5069.8131.5973.02
DNG39.8769.9431.4775.02
Sinusoidal (Ours)4.8172.1229.5195.13

As shown in Table 5, the DNG variant preserves image quality and prompt alignment well, but its ASR remains close to Native, indicating limited suppression of unsafe content. A possible explanation is that its posterior is estimated from the aggregated negative condition and the user’s positive condition, whereas the resulting coefficient scales a joint correction constructed from seven unsafe–safe description pairs. A low posterior for the aggregated condition does not necessarily mean that the individual category constraints require little correction. The resulting weighting may therefore apply insufficient correction. In comparison, sinusoidal allocation achieves substantially stronger safety, reducing ASR from 39.87% to 4.81% and increasing SQA from 75.02 to 95.13, while maintaining competitive generation quality.

AuthorsOpenReview ↗

Response to Reviewer 6sSe

Q1 Ablations on both backbones

Thank you for this question. We address it in W1 by providing the additional comparison of correction strategies on FLUX.2 in Table 1 and strength allocations on SD1.5 in Table 2. On FLUX.2, joint correction achieves lower ASR than averaging and independent correction, supporting the benefit of accounting for interactions among unsafe directions. On SD1.5, sinusoidal allocation achieves lower ASR than the mean-matched constant and all three window controls, while avoiding the substantial image-quality and prompt-alignment losses of the peak-matched constant. These additional results extend the original ablations to the other backbone and support the effectiveness of joint correction and phase-aware strength allocation on SD1.5 and FLUX.2.

Q2 Contribution of paired descriptions

Thank you for highlighting the need to examine the contribution of paired descriptions more directly. We provide the additional comparison in Table 3 of our response to W1. The full configuration with paired descriptions achieves lower ASR and higher SQA on both backbones, with particularly clear improvements in safety and image quality on FLUX.2. These results support our use of paired descriptions in the full configuration for stronger safety.

Q3 Comparison of sinusoidal allocation and DNG weighting

We appreciate your suggestion to compare sinusoidal allocation with DNG, which dynamically adjusts guidance strength during sampling. We provide this comparison in Table 5 of our response to W2, retaining unsafe–safe description pairs and joint correction in both variants. On SD1.5, DNG weighting achieves an ASR of 39.87%, close to Native’s 42.50%, indicating limited suppression of unsafe content in this adaptation. Sinusoidal allocation reduces ASR to 4.81% and achieves an SQA of 95.13, compared with 75.02 for DNG weighting. Although the DNG variant preserves image quality and prompt alignment better in this experiment, sinusoidal allocation provides substantially stronger safety while maintaining competitive generation quality. These results support sinusoidal allocation as the more effective choice for balancing safety, image quality, and prompt alignment.

Reviewer_Ki3MOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors identify two limitations in existing sampling-time T2I safeguards: (1) they either aggregate multiple unsafe categories into one condition or handle them one at a time, and (2) they lack an effective strategy for allocating correction strength across sampling stages that play different roles in image formation. To address these challenges, the authors propose SafeConductor, a training-free method that combines joint correction with phase-aware allocation for safe T2I generation. SafeConductor formulates joint correction as minimizing changes to the current guidance under separate non-positive projection constraints for all categories. The resulting correction is then applied with a smooth sinusoidal allocation that emphasizes the intermediate stage and reduces intervention near the trajectory endpoints, aiming to preserve image composition and visual details. Extensive experiments across four T2I safety benchmarks demonstrate that SafeConductor achieves state-of-the-art harmful content mitigation on both SD1.5 and FLUX.2 while maintaining competitive prompt alignment and generation quality.

Critical strengths
  • This paper is well written and easy to follow.
  • The authors identify two problems in existing safety guidance methods for T2I generation: how to combine guidance from multiple unsafe categories and how to allocate correction strength across sampling stages. To address these problems, the authors propose two reasonable solutions: joint correction under separate non-positive projection constraints for all categories and a smooth sinusoidal allocation of correction strength.
  • The experimental results demonstrate consistent performance improvements across different datasets and both U-Net- and DiT-based architectures (i.e., SD1.5 and FLUX.2).
Critical weaknesses
  • The authors compare strategies for handling concurrent unsafe concepts only on SD1.5 (Table 2), while evaluating correction strength allocations only on FLUX.2 (Table 3). It remains unclear whether the conclusions of these ablation studies hold across both U-Net- and DiT-based architectures. Could the authors provide corresponding cross-architecture ablation results?
  • The main experiments do not include representative activation steering-based methods, such as CASteer (ICLR 2026) [1], which is an important baseline for safe T2I generation. A comparison with such methods would help establish the effectiveness of SafeConductor.
  • The inference efficiency of SafeConductor is not compared with SLD-based methods, such as SLD and CASG. Given the additional conditional predictions and optimization at each sampling step, it would be helpful to report the inference time and computational overhead.
  • The proposed sinusoidal allocation relies on a hand-crafted schedule. Why do the authors choose a sinusoidal function rather than other functions with a similar small-large-small pattern? For example, the authors could compare it with squared-sine, Gaussian, or parabolic schedules. Such comparisons would help justify the specific choice of the sinusoidal function.
  • It remains unclear whether the proposed sinusoidal allocation is a generalizable strategy that could also improve existing methods, such as SLD and CASG, or whether its benefits are specific to SafeConductor. It would be helpful to evaluate the proposed schedule on these baselines.

[1] CASteer: Cross-Attention Steering for Controllable Concept Erasure. ICLR 2026

Questions for authors

Please see the Weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Ki3M

We appreciate your thoughtful review and insightful comments, as well as your recognition of our clear presentation, the motivation and design of our method, and its consistent improvements across architectures. In response to your suggestions, we provide further experiments and analyses below to clarify the effectiveness, efficiency, and scope of our method.

W1 Ablations across architectures

Thank you for suggesting these complementary ablations. We provide the comparison of strategies for handling concurrent unsafe concepts on FLUX.2 in Table 1 and the comparison of correction strength allocations across sampling stages on SD1.5 in Table 2. ASR is evaluated on I2P, while FID and CLIP are evaluated on COCO, following the sampling settings used in the main paper.

Table 1. Comparison of strategies for handling concurrent unsafe concepts on FLUX.2.

MethodASR (%) ↓FID ↓CLIP ↑SQA ↑
Native34.1572.2432.1579.41
SLD-Max16.3189.2129.4484.80
CASG16.6795.2928.6282.60
Equal-weight mean17.9587.1129.7684.68
Independent correction18.5081.5830.0885.98
Joint correction (Ours)15.1881.5330.0387.77
SafeConductor (Ours, full)7.7075.6930.3993.62

On FLUX.2, we follow the experimental design of Table 2 in the main paper and evaluate the strategies without paired unsafe–safe descriptions or sinusoidal strength allocation. Among these correction strategies, joint correction achieves the lowest ASR and highest SQA while maintaining competitive image quality and prompt alignment. These results support the value of jointly optimizing category coefficients to account for interactions among unsafe directions. We also include the full SafeConductor method, which further improves safety, image quality, and prompt alignment.

Table 2. Comparison of correction strength allocations across SD1.5 sampling stages.

MethodASR (%) ↓FID ↓CLIP ↑SQA ↑
Native42.5069.8131.5973.02
Peak-matched constant3.2792.2824.9885.95
Mean-matched constant7.0473.7830.0793.92
Early-only9.27111.5423.5477.76
Intermediate-only11.4271.0830.1592.51
Late-only27.0369.8431.3784.24
Sinusoidal (Ours)4.8172.1229.5195.13

On SD1.5, we follow the experimental design of Table 3 in the main paper, retaining paired descriptions and joint correction while comparing different strength allocations. Maintaining the peak strength throughout sampling provides the strongest suppression but substantially degrades image quality and prompt alignment. Among the window controls, early-only correction provides the strongest suppression but causes the greatest quality loss, whereas intermediate-only and late-only correction preserve quality better but offer weaker suppression of unsafe content. Sinusoidal allocation achieves lower ASR than the mean-matched constant and all three windows while avoiding the substantial quality and alignment losses of the peak-matched constant.

The original and additional ablations support the effectiveness of joint correction and phase-aware strength allocation on both SD1.5 and FLUX.2.

AuthorsOpenReview ↗

Response to Reviewer Ki3M

W2 Comparison with CASteer

We sincerely appreciate your suggestion to compare our method with CASteer, which allows us to extend our comparison to a representative activation steering-based method and provide further evidence of SafeConductor’s effectiveness. We additionally evaluate CASteer on SD1.5 using the sampling and evaluation settings of Table 1 in the main paper. We follow the authors’ seven-category averaging scheme, estimating steering directions on SD1.5 and using their recommended steering strength of β=2\beta=2.

Table 3. Quantitative comparison of safety, image quality, and prompt alignment on SD1.5.

MethodI2P ASR (%) ↓T2VSB ASR (%) ↓UD ASR (%) ↓CoProV2 ASR (%) ↓FID ↓CLIP ↑SQA ↑
Native42.5057.2553.9827.7069.8131.5970.67
ESD (ICCV'23)30.2640.8932.4120.4067.3629.9480.76
SLD-Max (CVPR'23)10.9926.4615.276.9078.1029.2087.89
UCE (WACV'24)32.6846.3339.7119.3078.8431.0776.93
RECE (ECCV'24)26.7333.6328.8716.7077.6629.3581.48
MACE (CVPR'24)18.5816.3215.2714.8073.7228.1487.60
SafetyDPO (ICCV'25)14.1223.0017.704.3076.1630.6589.52
SAFREE (ICLR'25)21.2442.5525.6613.5074.1030.5083.50
CASteer (ICLR'26)28.1943.8632.7413.1069.6631.4782.65
CASG (CVPR'26)9.259.279.624.0073.8829.1692.67
SafeConductor (Ours)4.816.538.412.5072.1229.5194.76

Table 3 presents the updated comparison including CASteer on SD1.5. Although CASteer preserves image quality and prompt alignment well, it provides weaker suppression of unsafe content. SafeConductor achieves substantially lower ASR across all four safety benchmarks and the highest SQA of 94.76 among the compared methods, demonstrating its strong performance in safe text-to-image generation.

W3 Inference cost

Thank you for raising this concern. We acknowledge that the additional conditional predictions and joint optimization increase inference cost. However, our additional experiments show that even without paired descriptions, SafeConductor achieves a lower I2P ASR than all compared baselines at an inference cost comparable to CASG on SD1.5. We adopt paired descriptions in the full configuration to further improve safety, achieving the lowest ASR and highest SQA. Table 4 summarizes these cost and performance comparisons on SD1.5, which we will also include in the revised appendix.

We would first like to emphasize that SafeConductor requires no additional training or fine-tuning, yet achieves the lowest ASR across all four safety benchmarks on both backbones in our main comparison. We compare SafeConductor with other training-free, inference-time safeguards on SD1.5 in Table 4, measuring latency and peak GPU memory on a single NVIDIA A800 80GB GPU. ASR is evaluated on I2P, while FID and CLIP are evaluated on COCO.

Table 4. Comparison of inference cost and overall performance on SD1.5.

MethodInference time (s/sample) ↓Peak GPU memory (GiB) ↓ASR (%) ↓FID ↓CLIP ↑SQA ↑
Native3.82 (1.00×)4.9542.5069.8131.5973.02
SLD-Max5.51 (1.44×)4.9510.9978.1029.2089.94
CASG15.70 (4.11×)5.289.2573.8829.1692.05
SafeConductor (w/o paired descriptions)15.83 (4.14×)5.288.8771.5329.6893.38
SafeConductor (full)26.38 (6.91×)6.184.8172.1229.5195.13

On SD1.5, SafeConductor without unsafe–safe description pairs runs at nearly the same speed as CASG and has almost identical peak GPU memory usage, but achieves better safety, image quality, and prompt alignment. We adopt paired descriptions in the full configuration to further improve safety: it reduces I2P ASR by approximately 48% relative to CASG and achieves the highest SQA among the configurations shown.

On FLUX.2, SLD-Max and CASG exhibit substantial image-quality degradation, as reflected in the qualitative comparisons and FID/CLIP scores, making inference cost alone insufficient to assess practical utility. SafeConductor achieves the lowest ASR across all four safety benchmarks and the highest SQA on FLUX.2 in our main comparison. We believe this is a valuable trade-off for applications that prioritize safety, particularly as it requires no model retraining or fine-tuning.

AuthorsOpenReview ↗

Response to Reviewer Ki3M

W4 Choice of the sinusoidal function

Thank you for suggesting these comparisons. We additionally evaluate alternative strength allocation functions on FLUX.2, as reported in Table 5. We follow the experimental settings of Table 3 in the main paper, retaining paired descriptions and joint correction and changing only the allocation function. All continuous functions are normalized to a peak of one and use the same strength parameter γ=3\gamma=3. ASR is evaluated on I2P, while FID and CLIP are evaluated on COCO.

Let τi=i/(T−1)\tau_i=i/(T-1) denote the normalized sampling progress at iteration ii, where TT is the number of sampling iterations. We define the correction strength as si=γf(τi)s_i=\gamma f(\tau_i), where ff is a strength allocation function and γ\gamma controls the overall scale of the correction strength. Besides the sinusoidal function fSinusoidal(τ)=sin⁡(πτ)f_{\mathrm{Sinusoidal}}(\tau)=\sin(\pi\tau), we compare the sine-squared function fSine-squared(τ)=sin⁡2(πτ)f_{\mathrm{Sine\text{-}squared}}(\tau)=\sin^2(\pi\tau), the parabolic function fParabolic(τ)=4τ(1−τ)f_{\mathrm{Parabolic}}(\tau)=4\tau(1-\tau), and a Gaussian function centered at τ=0.5\tau=0.5 with width parameter σ=0.20\sigma=0.20. We also consider two functions derived from the Beta distribution: fBeta(2,3)(τ)=274τ(1−τ)2f_{\mathrm{Beta}(2,3)}(\tau)=\frac{27}{4}\tau(1-\tau)^2, fBeta(3,2)(τ)=274τ2(1−τ)f_{\mathrm{Beta}(3,2)}(\tau)=\frac{27}{4}\tau^2(1-\tau). Their continuous peaks occur at τ=1/3\tau=1/3 and τ=2/3\tau=2/3, respectively, allowing us to examine allocations that place the strongest correction earlier or later in the sampling trajectory.

Table 5. Comparison of correction strength allocation functions on FLUX.2.

Allocation functionASR (%) ↓FID ↓CLIP ↑SQA ↑
Native34.1572.2432.1579.41
Sine-squared11.8073.0531.2492.85
Parabolic6.6677.9829.7692.97
Gaussian13.5273.1531.3591.94
Beta(2,3)7.8779.8029.9691.98
Beta(3,2)16.0173.3931.7090.68
Sinusoidal (Ours)7.7075.6930.3993.62

As shown in Table 5, sinusoidal allocation achieves a relatively low ASR and the highest SQA among the compared functions. The parabolic function achieves a lower ASR but degrades image quality and prompt alignment. At the same peak parameter, the parabolic function has a larger area and a higher sum of strength coefficients than the sinusoid, which may help explain its stronger suppression but worse image quality and prompt alignment. Sine-squared and Gaussian functions better preserve image quality and prompt alignment but provide weaker suppression of unsafe content. These results support our choice of sinusoidal allocation to balance safety, image quality, and prompt alignment.

Reviewer_AUmgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SafeConductor, which is a training-free concept removal method for K different categories. Using an optimization technique, SafeConductor finds the optimal guidance direction that ensures all categories remain safe.

Critical strengths

[S1] Tackles the critical challenge of mitigating KK toxic concepts simultaneously.

[S2] Mathematically rigorous formulation based on KKT conditions.

[S3] Based on KKT conditions, if certain categories are already safe, the corresponding λ\lambda becomes 0, allowing the original semantic direction to remain unchanged.

Critical weaknesses

[W1] The method relies on fixed, paired prompts to obtain qq, which vary by model architecture and seem heuristic. Additionally, these fixed paired prompts potentially make some generated safe images look similar as gamma increases. (see Q2)

[W2] Novelty: The main contribution of this paper appears to be an optimization technique for combining different vectors, rather than a novel safety mechanism alone. As I understand it, this method can apply to other safe directions rather than a naive derivation of qq, serving as a plug-and-play method that guarantees safety across KK categories. Adopting stronger baseline methods for generating qq could potentially yield further improvements.

[W3] Computation cost: Calculating predictions for 2K+22K+2 conditions per sampling step causes large computation cost, while the other methods only need to compute at most KK.

[W4] The paper claims state-of-the-art multi-category robustness on adversarial prompts, yet the adversarial benchmarks are single-category: based on my understanding, when K=1, it is the same as simply using qq as a guidance.

[Minor] notations: Some notations are confusing. For example, defining AA as a matrix containing qq as its kk-th row is slightly unintuitive. Consider modifying the notation (e.g., using QQ for a matrix comprising qq rows).

Questions for authors

[Q1] How did you conduct the experiments for the baseline methods that only provide one concept removal at a time? Sequentially K times? If so, does the performance vary depending on the order of removal?

[Q2] Following W2, is there a specific reason for using the paired prompt? I think ablation on ‘Joint correction’ shows that this method performs well even without using the paired prompts. The marginal gain might not justify the massive computational overhead of evaluating the safe description aka_k

[Q3] Why is the ASR of Native already small in Table 1, while the ASR of Native on adversarial prompts is high (Table 4)?

17 comments
AuthorsOpenReview ↗

Response to Reviewer AUmg

We sincerely thank you for recognizing the importance of addressing multiple unsafe categories and the rigor of our formulation. We appreciate your constructive questions and clarify below the role of paired descriptions, inference cost, and other points raised in your review.

W1 Fixed descriptions and image diversity

We would first like to clarify that although the descriptions are fixed, the resulting directions are recomputed at each sampling step from the current latent state and sampling time: hi,k=vθ(xi,ti,ck)−vθ(xi,ti,ak)h_{i,k}=v_\theta(x_i,t_i,c_k)-v_\theta(x_i,t_i,a_k). As the latent evolves under the user’s prompt guidance, these directions can vary across sampling steps and prompt-conditioned generation trajectories. The joint correction further depends on the current prompt guidance gig_i. Fixed description pairs therefore do not mean that the same correction is applied to every prompt. We discuss the purpose of paired descriptions and the corresponding ablation results in Q2.

W2 Novelty

Thank you for this comment. Our contribution is to mathematically formulate and solve the problem of jointly correcting prompt guidance under multiple category constraints, rather than to propose a rule for combining vectors. At each sampling step, we construct the unsafe directions from the current latent state and model time. We then formulate an optimization problem over the corrected guidance yy: it should satisfy all category constraints while remaining as close as possible to the current prompt guidance gig_i. Using Lagrange duality, we transform this high-dimensional problem into a dual optimization over KK nonnegative coefficients, which we solve jointly. The expression yi⋆=gi−∑k=1Kλi,k⋆qi,ky_i^\star=g_i-\sum_{k=1}^{K}\lambda_{i,k}^\star q_{i,k} follows from the optimality conditions. The weighted correction is therefore the derived solution structure of the problem we model, rather than a combination assumed in advance.

We additionally introduce sinusoidal strength allocation to reduce unnecessary disruption to image quality, considering the different roles of sampling stages in structure formation and detail refinement.

We also appreciate your suggestion to explore alternative ways of constructing unsafe directions. Our current construction computes unsafe directions from the latent state and time step, allowing them to vary across sampling steps and generation trajectories guided by different prompts. We agree that exploring alternative constructions and further refining these directions may lead to additional improvements in safety guidance.

AuthorsOpenReview ↗

Response to Reviewer AUmg

W3 Inference cost

Thank you for raising this concern. We agree that using unsafe–safe description pairs introduces additional inference cost, with 2K+22K+2 condition predictions required by the formulation. However, our additional experiments show that even without paired descriptions, SafeConductor achieves a lower I2P ASR than all compared baselines at an inference cost comparable to CASG on SD1.5. We adopt paired descriptions in the full configuration to further improve safety, achieving the lowest ASR and highest SQA. Table 1 summarizes these cost and performance comparisons on SD1.5, which we will also include in the revised appendix. We would also like to clarify that CASG needs the user-prompt and unconditional predictions in addition to its KK unsafe-category predictions, giving K+2K+2 necessary condition predictions. We analyse below the results of these cost and performance comparisons.

We would first like to emphasize that SafeConductor requires no additional training or fine-tuning, yet achieves the lowest ASR across all four safety benchmarks on both backbones in our main comparison. We compare SafeConductor with other training-free, inference-time safeguards on SD1.5 in Table 1, measuring latency and peak GPU memory on a single NVIDIA A800 80GB GPU. ASR is evaluated on I2P, while FID and CLIP are evaluated on COCO.

Table 1. Comparison of inference cost and overall performance on SD1.5.

MethodInference time (s/sample) ↓Peak GPU memory (GiB) ↓ASR (%) ↓FID ↓CLIP ↑SQA ↑
Native3.82 (1.00×)4.9542.5069.8131.5973.02
SLD-Max5.51 (1.44×)4.9510.9978.1029.2089.94
SAFREE5.55 (1.45×)4.9521.2474.1030.5086.27
CASG15.70 (4.11×)5.289.2573.8829.1692.05
SafeConductor (w/o paired descriptions)15.83 (4.14×)5.288.8771.5329.6893.38
SafeConductor (full)26.38 (6.91×)6.184.8172.1229.5195.13

On SD1.5, SafeConductor without unsafe–safe description pairs runs at nearly the same speed as CASG and has almost identical peak GPU memory usage, but achieves better safety, image quality, and prompt alignment. We adopt paired descriptions in the full configuration to further improve safety: it reduces I2P ASR by approximately 48% relative to CASG and achieves the highest SQA among the configurations shown.

On FLUX.2, SLD-Max and CASG exhibit substantial image-quality degradation, as reflected in the qualitative comparisons and FID/CLIP scores, making inference cost alone insufficient to assess practical utility. SafeConductor achieves the lowest ASR across all four safety benchmarks and the highest SQA on FLUX.2 in our main comparison. We believe this is a valuable trade-off for applications that prioritize safety, particularly as it requires no model retraining or fine-tuning.

AuthorsOpenReview ↗

Response to Reviewer AUmg

W4 Adversarial robustness evaluation

Thank you for this comment. We would like to clarify that our paper does not claim state-of-the-art multi-category robustness on adversarial prompts. What we report is the “strongest aggregate adversarial robustness among the compared safeguards.” Here, “aggregate” refers to the mean ASR across the three evaluated adversarial prompt sets.

Regarding the K=1K=1 case you mentioned, we would like to clarify that KK denotes the number of category directions configured in the safeguard, not the number of categories targeted by an attack. Even when an attack targets a single category, our evaluation retains K=7K=7 and jointly considers all seven directions. This is important because correcting the guidance away from violence, for example, may increase its alignment with sexual content when the two directions conflict. Our joint correction accounts for such interactions, as illustrated in Figure 1 of the main paper and discussed in Section 3.2. For a given backbone, the same safety configuration is applied to different prompts without retraining or selecting a separate model for each category. This allows SafeConductor to protect incoming prompts without assuming that their risk categories are known in advance.

In addition, safeConductor’s performance across multiple unsafe categories is reported in Appendix Table 8, and we reproduce it below for convenience. We compute ASR separately for the seven I2P prompt groups using the same SafeConductor configuration on SD1.5. SafeConductor achieves the lowest ASR among the compared methods in all seven I2P prompt groups.

Table 2. ASR (%) across I2P categories on SD1.5.

MethodHateHarassmentViolenceSelf-harmSexualShockingIllegalOverall
Native41.9931.4344.8438.9545.9753.9737.2842.50
ESD25.9720.6334.7934.0828.1438.5526.6930.26
SLD-Max12.1210.4411.645.8716.7614.025.5010.99
UCE38.9629.3734.5229.3430.9343.1126.9632.68
RECE25.5425.8530.8226.2222.6634.2323.6626.73
MACE17.3216.1422.8819.8518.3717.8717.7418.58
SafetyDPO12.5513.1116.1411.8617.5115.079.7714.12
SAFREE16.0215.5323.0217.1025.5629.2116.2321.24
CASG8.6611.1714.294.496.9812.976.339.25
SafeConductor (Ours)5.635.345.293.374.516.783.444.81

Matrix notation

We appreciate your suggestion to make the notation more intuitive. We will use QiQ_i for the matrix whose kk-th row is qi,k⊤q_{i,k}^{\top}, replacing AiA_i consistently throughout the main text and appendix, including the derivation and algorithm.

Reviewer_g3RjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SafeConductor, a training-free method for safe text-to-image generation. It combines joint optimization of multiple unsafe directions with phase-aware correction strength allocation. Experiments on SD1.5 and FLUX.2 demonstrate improved safety while maintaining competitive generation quality.

Critical strengths
  1. Important and well-motivated problem. The paper discusses the challenge of handling multiple unsafe concepts jointly in sampling-time safety guidance, which is an important and practical problem.

  2. Strong experimental results. Extensive experiments across multiple safety benchmarks and different generative architectures demonstrate the effectiveness of the proposed method.

Critical weaknesses
  1. Significant inference overhead. SafeConductor requires (2K) additional network predictions per sampling step (14 when (K=7)), potentially introducing substantial inference latency and GPU memory overhead. However, the paper lacks efficiency comparisons with existing methods, which are important for evaluating its practical applicability.
  2. Insufficient baseline implementation details. The paper does not clearly specify the target concepts and key configurations used for the baselines and SafeConductor in Tables 1 and 4. These details are essential for assessing the fairness of the comparisons.
  3. Some symbols lack sufficient explanation, such as the distinction between y and g_i in Section 3.2.
Questions for authors

See Weaknesses. I think this paper discusses an important problem in sampling-time safety guidance, how to handle multiple unsafe concepts jointly. However, the proposed method introduces substantial computational overhead. If the authors can adequately address this concern, I would be willing to raise my score.

4 comments
AuthorsOpenReview ↗

Response to Reviewer g3Rj

Thank you for your thoughtful review and for recognizing the importance of jointly handling multiple unsafe concepts and the strength of our experiments.

W1. Inference cost

In response to your concern about inference cost, we provide additional efficiency comparisons below. We will include these results in the revised appendix and discuss the additional inference cost alongside the gains in safety, image quality, and prompt alignment.

SafeConductor requires no additional training or fine-tuning and achieves the lowest ASR across all four safety benchmarks on both backbones in our main comparison. To assess the inference cost associated with these safety gains, we compare SafeConductor with other training-free, inference-time safeguards on SD1.5 in Table 1, measuring latency and peak GPU memory on a single NVIDIA A800 80GB GPU. ASR is evaluated on I2P, while FID and CLIP are evaluated on COCO.

Table 1. Comparison of inference cost and overall performance on SD1.5.

MethodInference time (s/sample) ↓Peak GPU memory (GiB) ↓ASR (%) ↓FID ↓CLIP ↑SQA ↑
Native3.82 (1.00×)4.9542.5069.8131.5973.02
SLD-Max5.51 (1.44×)4.9510.9978.1029.2089.94
SAFREE5.55 (1.45×)4.9521.2474.1030.5086.27
CASG15.70 (4.11×)5.289.2573.8829.1692.05
SafeConductor (w/o paired descriptions)15.83 (4.14×)5.288.8771.5329.6893.38
SafeConductor (full)26.38 (6.91×)6.184.8172.1229.5195.13

On SD1.5, SafeConductor without unsafe–safe description pairs runs at nearly the same speed as CASG and has almost identical peak GPU memory usage, but achieves better safety, image quality, and prompt alignment. Our full configuration with paired descriptions prioritizes stronger safety: it reduces I2P ASR by approximately 48% relative to CASG, and achieves the highest SQA among the configurations shown.

On FLUX.2, SLD-Max and CASG exhibit substantial image-quality degradation, as reflected in the qualitative comparisons and FID/CLIP scores, making inference cost alone insufficient to assess practical utility. SafeConductor achieves the lowest ASR across all four safety benchmarks and the highest SQA on FLUX.2 in our main comparison. We believe this is a valuable trade-off for applications that prioritize safety, particularly as it requires no model retraining or fine-tuning.

AuthorsOpenReview ↗

Response to Reviewer g3Rj

W2. Baseline implementation details

Thank you for emphasizing the importance of these details. We agree that the original submission should have explained the baseline configurations more clearly. We clarify the unsafe concepts and key baseline settings below. These details will also be included in the revised appendix to support fair comparisons and reproducibility.

On SD1.5, our baseline experiments build on publicly released implementations and reported experimental settings. SLD-Max retains its default 20-keyword safety condition, and CASG retains its default seven keyword groups; both use the published Max preset. For ESD, we adopt the SLD 20-keyword vocabulary evaluated in its paper for multiple unsafe concepts and train ESD-u for 1,000 updates at a learning rate of 10−510^{-5}. SAFREE uses the same vocabulary through the baseline implementation released with CASG. UCE, RECE, and MACE target seven unsafe concepts covering the evaluated risks. UCE jointly edits 32 key and value projection matrices in the cross-attention layers, with a regularization coefficient of 0.1. RECE likewise edits the cross-attention key and value projections, using joint UCE initialization followed by two refinement rounds with a representation regularization coefficient of 0.1. MACE follows the official configuration for explicit content, training a rank-1 LoRA for each concept for 120 updates and combining the adapters using the authors’ closed-form fusion procedure. SafetyDPO directly uses the authors’ released safe checkpoint without additional training.

On FLUX.2, SLD-Max and CASG retain the same target texts and Max settings as on SD1.5. ESD builds on the official FLUX.2 implementation and uses the same 20-keyword safety vocabulary as in our SD1.5 experiments. UCE and RECE retain the same seven target terms as on SD1.5, with their editing operations adapted to the text-conditioning input projection. For EraseAnything, we train a separate LoRA with a rank of 32 for each of the seven unsafe categories, using 1,000 training steps per concept. We then combine the seven parameter updates with equal weights of 1/71/7, following the normalized composition proposed in its paper.

We evaluate adversarial robustness using released adversarial prompt sets from P4D (272 universal prompts), Ring-A-Bell (285 prompts generated through model-agnostic offline search), and MMA-Diffusion (a text-attack benchmark containing 1,000 prompts). We retain the checkpoints and defense settings from Table 1, with the same image resolution, sampler, sampling steps, and safety evaluators. All defenses use the same prompts. For each prompt, all defense methods use the same random seed and CFG scale; for P4D prompts, these are the seeds and CFG values provided in the released dataset.

W3. Explanation of symbols

Thank you for your suggestion to explain these symbols more clearly. In Section 3.1, Eq. (2) defines gi=vir−vi∅g_i=v_i^r-v_i^{\varnothing} as the original prompt guidance at sampling iteration ii. Section 3.2 introduces yy as a candidate corrected guidance vector in the same prediction space. Thus, Eq. (6) holds gig_i fixed and optimizes yy to find the closest vector yi⋆y_i^\star to gig_i that satisfies all category constraints. While these definitions are provided in the main text, we agree that their relationship could be explained more explicitly. We will revise the explanation preceding Eq. (6) to clearly distinguish the fixed input gig_i, the optimization variable yy, and the resulting solution yi⋆y_i^\star.

Thank the authors for the detailed responses.

The additional efficiency comparisons help clarify the trade-off between inference cost and safety improvements (W1). However, the full SafeConductor still incurs substantial inference overhead, and the computational cost on FLUX.2 remains unclear, potentially raising further concerns about its practical applicability. I encourage the authors to explicitly discuss these limitations in the revised paper.

The clarifications regarding baseline configurations (W2) and notation (W3) adequately address my remaining concerns. Please incorporate these details into the revised manuscript.

Overall, I appreciate the authors' efforts in addressing my concerns. However, given the remaining concerns about computational efficiency and practical applicability, I will maintain my original rating.

AuthorsOpenReview ↗

Response to Reviewer g3Rj

We sincerely thank you for your response. We guarantee that the discussion of inference cost, the baseline configuration details, and clearer explanations of the notation will be incorporated into the revised paper.

To address your concern about inference cost on FLUX.2, Table 2 reports inference time and peak allocated GPU memory alongside safety and generation performance, using the same design as our earlier Table 1.

Table 2. Comparison of inference cost and overall performance on FLUX.2.

MethodInference time (s/sample) ↓Peak GPU memory (GiB) ↓ASR (%) ↓FID ↓CLIP ↑SQA ↑
Native3.36 (1.00×)15.5534.1572.2432.1579.41
SLD-Max4.90 (1.46×)15.5516.3189.2129.4484.80
CASG14.40 (4.29×)15.6416.6795.2928.6282.60
SafeConductor (w/o paired descriptions)13.52 (4.02×)15.6419.8885.4130.3784.44
SafeConductor (full)23.72 (7.06×)15.747.7075.6930.3993.62

On FLUX.2, the SLD-Max and CASG adaptations exhibit severe image-quality degradation, including visible distortions and disrupted scene structure, as illustrated in Figure 3 of the main paper and Appendix Figures 11–13 and reflected in their FID and CLIP scores. Under such degradation, ASR alone is insufficient to demonstrate practically useful safety protection. Among the training-free, inference-time safeguards compared on FLUX.2, SafeConductor without paired descriptions has inference cost comparable to CASG but achieves better image quality and prompt alignment. Our full configuration achieves the ASR of 7.70%, representing relative reductions of 77.45% compared with Native and 52.73% compared with RECE, the strongest baseline in the main comparison. These safety gains come at approximately 1.65× CASG’s inference time, with nearly the same peak GPU memory. We consider this strong safety performance meaningful and practically valuable for applications with stringent safety requirements.

We wish to reiterate that, on SD1.5, SafeConductor without paired descriptions has inference cost comparable to CASG but achieves better safety, image quality, and prompt alignment. This variant already achieves the lowest ASR and the highest SQA compared with all SD1.5 baselines in Table 1 of the main paper. The full configuration achieves a further safety improvement, reducing ASR by approximately 88.68% relative to Native and 48% relative to CASG, the strongest baseline. These results demonstrate both the value of our method at comparable inference cost and the additional safety benefits available from the full configuration.

We also hope to convey the broader significance of our work. Many representative safety and concept-erasure studies published in 2026, including CASG (CVPR 2026) on SD1.5 [1] and GenErase (CVPR 2026) on SD1.4 [2], continue to use U-Net-based Stable Diffusion models as their main experimental backbones. SafeConductor is the first training-free safeguard for general multi-category protection on FLUX.2. SafeConductor applies to both U-Net-based latent diffusion models and Transformer-based rectified flow models. Without additional training or fine-tuning, SafeConductor achieves the lowest ASR across all four safety benchmarks and the highest SQA among the compared methods on both SD1.5 and FLUX.2. We consider these benefits highly meaningful for applications with stringent safety requirements.

In summary, we would like to highlight three points for your consideration:

  1. The strongest safety performance across backbones.

  2. The variant without paired descriptions can achieve the strongest safety performance on SD1.5 at a similar inference cost to the strongest baseline, CASG.

  3. The first training-free safeguard for general multi-category protection on FLUX.2.

We would be very grateful if you could consider these contributions together in your overall assessment of our work.

References

[1] Yongli Xiang, Ziming Hong, Zhaoqing Wang, Xiangyu Zhao, Bo Han, and Tongliang Liu. When Safety Collides: Resolving Multi-Category Harmful Conflicts in Text-to-Image Diffusion via Adaptive Safety Guidance. CVPR, 2026.

[2] Korada Sri Vardhana and Soma Biswas. GenErase: Generalizable and Semantically-Aware Concept Erasure in Diffusion Models. CVPR, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.