acceptodds
Under review as a conference paper at ICLR 2027

PaLo-Steer: Nuisance-Reduced Signal Construction for Text-to-Image Safety Steering

Abstract

Text-to-image (T2I) diffusion models offer powerful generation capabilities but can also produce harmful content. To suppress harmful content, *activation-steering* methods construct signals from harmful-safe activation contrasts and apply them to intermediate activations during denoising. The effectiveness of these methods depends on how well the constructed signals capture harmful-relevant activation changes while minimizing nuisance. However, we identify two sources of nuisance in existing methods that can weaken safety steering: *Trajectory-induced confounding* mixes harmful-concept effects with differences in intermediate image states; *Aggregation-induced dilution* weakens the harmful-relevant contrast by pooling irrelevant image tokens. To address these issues, we propose *PaLo-Steer*, which constructs cleaner steering signals through state-controlled activation comparison and concept-aware spatial aggregation. (i) **Pa**rallel contrastive activation collection isolates the harmful-concept effect by collecting harmful- and safe-conditioned activations at shared intermediate states along a single harmful trajectory. (ii) Spatially **Lo**calized activation selection uses model-intrinsic cross-attention maps to locate image tokens relevant to harmful content. It then aggregates activation contrasts only over these tokens to reduce concept dilution. PaLo-Steer supports both vector- and transport-map-based steering. Across multiple safety benchmarks, it achieves state-of-the-art harmful-content suppression among parameter-free methods, reducing harmful rates by up to 12.0% over existing activation-steering methods while preserving benign generation quality.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
T2I Safety, Activation Steering

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_Aac4OpenReview ↗
Rating
3
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PaLo-Steer, a training-free framework for improving activation-based safety steering in text-to-image diffusion models. The method addresses two sources of interference in steering-signal construction: trajectory-induced confounding and aggregation-induced dilution. It introduces parallel contrastive activation collection and spatially localized activation selection to construct cleaner steering signals. Experiments across different safety benchmarks demonstrate improvements over existing activation-steering methods while largely preserving benign generation quality.

Critical strengths

Clear motivation and problem formulation. The paper identifies two specific sources of nuisance in existing activation-steering methods and provides intuitive explanations and empirical analyses to motivate the proposed solution.

Critical weaknesses
  1. Limited methodological novelty. The two main components rely on relatively straightforward techniques: fixed-state contrastive activation collection and cross-attention-based spatial selection. The paper could better clarify its methodological novelty beyond combining these existing ideas.
  2. The paper primarily relies on MMD separability and downstream harmful rates to support its motivation. However, improved activation separability does not necessarily indicate that steering directions contain less irrelevant information. More direct analyses of steering-vector specificity and stability would strengthen the central claim.
Questions for authors
  1. Can the authors provide more direct evidence that PaLo-Steer reduces nuisance information rather than simply improving activation separability?
  2. How reliably does cross-attention localization handle harmful concepts that involve multiple objects or global scene semantics?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Aac4

Dear Reviewer Aac4,

We appreciate your careful review and constructive suggestions, which we address in detail below.

W1: Methodological novelty

R1: We clarify our contributions in problem identification and method design below.

  • Finding: We first identify two sources of nuisance in signal construction that can weaken steering effectiveness: trajectory confounding introduces differences between independently evolved states, while concept dilution introduces harmful-irrelevant spatial activations into aggregation. Our controlled analyses support the importance of addressing these construction effects for harmful-content suppression.

  • Method: PaLo-Steer directly addresses the two identified sources of nuisance: shared-state collection removes the state mismatch underlying trajectory confounding, while localized aggregation reduces concept dilution. Specifically:

    • Contrast at shared intermediate states. Our contrastive collection shares the intermediate state at each comparison, beyond merely fixing the initial noise. We construct the safe-conditioned reference at states actually visited by the harmful trajectory, so the contrast directly captures how those states respond to safer conditioning without mixing in trajectory-state differences.

    • Spatial selection adapted to signal construction. Our contribution lies in adapting cross-attention localization to the requirements of activation-steering construction. These cross-attention maps provide layer- and timestep-specific relevance cues without an external localizer. Their coarse harmful-relevance cues remain useful before visual details emerge, allowing us to filter harmful-irrelevant activations during aggregation throughout denoising.

Component ablations support both design choices, and improvements with CASteer and AcT demonstrate their applicability to vector- and transport-based steering.

W2.1 / Q1: Direct evidence of specificity and nuisance reduction

R2.1: We agree that direct measures of concept specificity and nuisance reduction would strengthen the evidence for our construction. To compute such reference-based measures, we need a nuisance-free direction that separates target-relevant changes from irrelevant components. However, obtaining a nuisance-free reference direction is difficult even with carefully controlled harmful-safe prompt pairs and identical initial noise. Despite these controls, nuisance can still arise from trajectory-state mismatch and harmful-irrelevant spatial aggregation, as identified in our work.

Given this difficulty, we combine MMD-based representation analysis with downstream harmful rates (Table 1). Steering relies on harmful-safe activation differences to identify how activations should be modified for suppression. When harmful-irrelevant variation obscures this contrast, the resulting signal can become less informative about the intended intervention. MMD serves as a diagnostic of this contrast under controlled construction changes, while downstream ablations evaluate the resulting improvement in harmful-content suppression.

W2.2: Steering-direction stability

R2.2: Appendix G.4 evaluates robustness using different construction-prompt sets, harmful-keyword settings, and activation-collection seeds. PaLo+CASteer consistently achieves lower harmful rates than CASteer across these settings, supporting the stability of its safety gains under construction variability.

To assess steering-vector stability directly, we also compare PaLo+CASteer directions constructed under different prompt sets and activation-collection seeds with the default direction. Table D1 shows high cosine similarity to the default across all tested variants, supporting the consistency of the constructed directions under these changes.

Table D1: Cosine similarity of PaLo+CASteer steering directions to the default construction setting.

Setting variedVariantCosine similarity vs. default
Construction-prompt set104 prompts0.9950
Construction-prompt set50 prompts0.9857
Activation-collection seed420.9658
Activation-collection seed430.9598

Q2: Localization of multi-object and global harmful concepts

R3: Thank you for referring to these scenarios. Cross-attention assigns relevance scores to individual image tokens, so localization is not restricted to a single object or contiguous region. We have added two qualitative examples in Appendix F.1 (Figure 13). In the multi-person example, attention highlights regions across several subjects. In the violence-related scene, responses extend across spatially distributed crowd and street regions. These examples illustrate how localization can accommodate multiple subjects and scene-level content.

Dear Reviewer Aac4, thank you for your time and effort in reviewing our paper. Your constructive feedback helped us strengthen our analyses of steering-direction stability and localization in complex scenes. We appreciate your suggestions for improving the evaluation.

Reviewer_3XwmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines two sources of error in activation-steering methods for text-to-image diffusion safety: activation differences that conflate text-conditioning effects with differences between denoising trajectories, and the dilution of localized harmful signals through global pooling. To address these issues, the authors propose PaLo Steer. The method collects contrastive activations by evaluating harmful and safe text conditions at shared intermediate states along a harmful-conditioned denoising trajectory. It then uses the model’s cross-attention maps to select spatial locations for activation aggregation. Experiments across multiple safety benchmarks show promising results.

Critical strengths
  1. The distinction between conditioning-induced and trajectory-induced activation differences provides a clear motivation for paired activation collection. The discussion of signal dilution under global pooling also motivates spatially selective aggregation. The presentation is logically organized and easy to follow.
  2. The method appears compatible with existing activation-steering approaches and achieves substantial improvements across several evaluation datasets. The ablations and visualizations help explain the roles of paired collection and spatial selection.
Critical weaknesses
  1. Evaluating both text conditions at the same intermediate state does remove differences caused by using separate trajectories. However, those states are sampled exclusively from a harmful-conditioned trajectory and may not be representative of states encountered under safe conditioning. The resulting difference is therefore a conditional contrast at harmful-trajectory states, rather than necessarily a general harmful-versus-safe activation direction. This does not invalidate the approach, but it leaves its sensitivity to the reference trajectory unresolved. Comparisons using safe-conditioned or mixed reference trajectories would help establish whether the extracted direction is stable.
  2. Figure 6 indicates that cross-attention is less informative during early, high-noise denoising steps. It is unclear how the method handles these steps or prevents diffuse attention from selecting irrelevant spatial tokens. As a result, the proposed aggregation may still suffer from the signal dilution it is intended to address. A timestep-stratified analysis, together with an ablation that excludes early steps or weights selection by attention confidence, would clarify this limitation.
  3. FID and CLIP-based scores provide useful measures of overall generation quality and text alignment, but they may obscure failures on benign prompts that share vocabulary or visual concepts with unsafe prompts. Stronger safety performance could partly reflect broad suppression of related content. The paper needs targeted benign-prompt evaluations to show that the method suppresses harmful content without unnecessarily removing legitimate concepts.
Questions for authors
  1. How sensitive are the steering directions and downstream results to the choice of reference trajectory? Do safe-conditioned or mixed trajectories produce comparable results, and how do the authors justify interpreting contrasts collected only along harmful-conditioned trajectories?
  2. How are early timesteps with diffuse or unreliable cross-attention handled? Does excluding these steps, or introducing confidence-weighted aggregation, improve safety performance or preserve benign content more effectively?
  3. Can the authors report benign-content preservation on prompts closely related to the targeted unsafe concepts? A safety–utility curve across steering strengths, supplemented by prompt-level failure rates or human evaluation, would make the trade-off clearer.
6 comments
AuthorsOpenReview ↗

Response to Reviewer 3Xwm (1/2)

Dear Reviewer 3Xwm,

We thank you for your thoughtful feedback and provide detailed responses to your concerns below.

W1 / Q1: Sensitivity to the reference trajectory

R1: Thank you for raising this question. For harmful-content suppression, activation steering aims to steer harmful activations toward safer responses at the states where harmful generation unfolds. The “conditional contrast at harmful-trajectory states” serves this objective by directly comparing harmful and safe responses at the states targeted by the intervention.

Using a harmful-conditioned trajectory provides a more direct reference for the intervention because the contrast starts at harmful-generation states. A safe-conditioned reference instead starts at safe-generation states. Reversing the safe-to-harmful direction does not change its reference states, so it is not equivalent to a contrast computed at harmful-generation states. We report the cosine similarity ranges across layers and timesteps below, with both directions oriented from harmful to safe. The results show that the two reference choices produce steering directions that are not identical.

Table C1: Direction similarity between harmful and safe references.

Direction comparisonViolenceSexIllegal
cosine similarity range0.84-1.000.71-1.000.73-1.00
Proportion with cosine similarity < 0.9518.6%31.4%14.3%

We further added an ablation study of the reference-trajectory choice (Section 5.3, Table 4) to examine its effect on harmful-content suppression. Using shared image states from a harmful-conditioned rather than a safe-conditioned trajectory reduces harmful rates across categories, both with and without localized aggregation. These ablation results support our choice of the harmful-conditioned trajectory.

Table C2: Effect of reference trajectory.

Reference trajectoryLocalizedHate (%) ↓Violence (%) ↓Sex (%) ↓Illegal (%) ↓
Safe×28.121.318.913.1
Harmful×27.718.813.912.8
Safe✓24.217.114.713.8
Harmful✓17.714.811.211.0

W2 / Q2: Unreliable localization at early denoising steps

R2: We appreciate your concern about early localization. Although cross-attention maps are coarser at early denoising steps, they can still provide useful spatial cues. Figures 6 and 12 show attention maps at steps 0 and 10 of the 50-step sampling process. High-attention regions roughly indicate where harmful content later emerges, while substantial background regions receive lower scores. These examples suggest that spatial cues remain useful for filtering harmful-irrelevant tokens at early denoising stages, even before fine image details emerge. Therefore, we apply the same localization procedure from the first denoising step.

To test the contribution of early localization to harmful-content suppression, we disable localization for the first 20 denoising steps while retaining steering and evaluate harmful rates on UD and CoProV2. Applying localization from the first denoising step lowers harmful rates on both benchmarks, supporting our choice to retain localization throughout denoising.

Table C3: Effect of early localization.

MethodUD (%) ↓CoProV2 (%) ↓
Localization ≥ step2020.010.7
PaLo-Steer18.8 (-1.2)9.0 (-1.7)
AuthorsOpenReview ↗

Response to Reviewer 3Xwm (2/2)

W3 / Q3: Preservation of benign content related to unsafe concepts

R3: Thank you for highlighting this important evaluation setting. We examine benign scenarios that share visual concepts with harmful content. Knife-related scenarios provide a concrete example: knives can appear in violence or legitimate activities such as cutting vegetables in a kitchen. We construct 50 diverse harmful prompts depicting knife-related violence and 50 diverse benign prompts depicting legitimate uses of knives, generating each prompt with three seeds. We compare CASteer and PaLo+CASteer at steering strengths of 1.5, 2.0, and 2.5.

Table C4: Safety and benign generation across steering strengths in knife-related scenarios.

MethodStrengthHarmful rate (%) ↓Benign knife presence (%) ↑Benign CLIP ↑
SD-v1.5-91.391.384.43
CASteer1.527.3 (-64.0)88.0 (-3.3)83.42
PaLo+CASteer1.524.7 (-66.6)86.7 (-4.6)83.65
CASteer2.012.7 (-78.6)85.3 (-6.0)83.77
PaLo+CASteer2.06.7 (-84.6)86.0 (-5.3)83.36
CASteer2.56.0 (-85.3)84.7 (-6.6)83.45
PaLo+CASteer2.52.0 (-89.3)86.0 (-5.3)83.04

Across the tested strengths, PaLo+CASteer consistently achieves lower harmful rates than CASteer. Increasing steering strength improves harmful-content suppression for both methods, while benign knife-presence rates remain high overall (84.7%-88.0%). At strengths 2.0 and 2.5, PaLo+CASteer achieves lower harmful rates than CASteer while maintaining comparable benign-content preservation and text-image alignment. These results support stronger harmful-content suppression without broadly suppressing related benign content. We've added these results and the corresponding safety-utility curve to Appendix G.7 (Table 20 and Figure 17) of the revised manuscript.

Overall, the authors have addressed my main concerns in a targeted manner and strengthened the paper through additional ablation studies and quantitative analyses. First, the authors provide reference-trajectory comparisons and ablation results to justify the choice of the harmful-conditioned trajectory. In the context of the original paper, the method aims to extract local activation differences induced by changes in text conditioning at fixed harmful-generation states, rather than to recover responses along an actual safe-conditioned trajectory. The authors' explanations and additional experiments have largely alleviated my concern. Although direct evidence is still lacking as to whether the resulting contrast sufficiently excludes harmful-irrelevant semantic variation, the current results provide support for this design choice. Second, the ablation results on early-stage localization indicate that applying localization from the beginning of denoising helps improve downstream harmful-content suppression. However, they do not directly evaluate the reliability of cross-attention-based localization under high-noise conditions or quantify the extent of aggregation-induced dilution at different denoising stages. Given that mitigating aggregation-induced dilution is one of the paper's core contributions, I suggest further comparing the effects of different localization thresholds across denoising stages on the final performance. This would provide a more direct assessment of the effectiveness of early-stage localization and its ability to mitigate aggregation-induced dilution. Finally, by introducing harmful and benign scenarios that share visual concepts and evaluating the trade-off between harmful-content suppression and benign-content preservation across different steering strengths, the authors have largely addressed my previous concern regarding the insufficient evaluation of over-suppression. Overall, while the rebuttal provides stronger empirical support for the proposed method and alleviates some of my concerns, I believe that the central methodological concern has not yet been sufficiently resolved. Therefore, I have decided to maintain my original rating.

Reviewer_QWEAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PaLo-Steer, a method for improving activation-steering signals for text-to-image safety without updating the diffusion model’s parameters. It identifies two potential sources of nuisance in existing signal construction: differences between independently evolved harmful and safe trajectories, and dilution from globally pooling irrelevant spatial tokens. PaLo addresses these through shared-state activation comparisons and cross-attention-guided spatial selection. The resulting representations support both steering vectors and activation transport maps. Experiments show consistent improvements over CASteer and AcT, with additional evaluations on SD-v3, adversarial prompts, and cross-version transfer.

Critical strengths

A clear and technically coherent intervention. The method directly modifies the stages implicated in its motivation. Comparing text conditions at the same intermediate state eliminates the state-mismatch term from the local activation contrast, while using identical spatial support preserves correspondence between paired activations. The construction procedure is straightforward and compatible with two existing steering formulations.

Consistent improvements supported by useful controls. PaLo+CASteer reduces harmful rates by 3.9–12.0 percentage points across the four main benchmarks, and PaLo+AcT also improves over AcT on all four. Table 2 shows contributions from both components. The matched-token-count comparison in Table 5 is particularly useful: random selection yields a harmful rate of 25.4%, whereas attention-guided selection achieves 21.6%, compared with 25.7% for global pooling.

Substantial supplementary evaluation and practical applicability. The appendix examines construction seeds, prompt counts, keyword settings, localization quantiles, SD-v3, adversarial prompts, and transfer across model versions. These experiments provide evidence that the improvements extend beyond a single configuration. Because the modifications occur during offline signal construction, PaLo retains the underlying method’s inference procedure and reported generation cost.

Critical weaknesses

W1. The mechanistic explanation is less firmly established than the empirical improvement.

The Gaussian-noise experiment shows that corrupting construction activations can reduce MMD separability and weaken steering. However, it does not establish how closely this corruption represents the structured variation caused by trajectory mismatch or irrelevant spatial tokens. Moreover, MMD measures distributional differences rather than the specificity of the estimated direction to harmful content. Its association with harmful rate therefore supports a diagnostic relationship, but does not establish that increased separability explains the gains. Shared-state comparison removes state mismatch by construction, yet the removed component can contain target-concept information as well as nuisance.

W2. The justification for intrinsic localization lacks a controlled comparison with external localization.

Appendix C provides an illustrative Grounded-SAM failure during early denoising and discusses representation and computational limitations. These observations motivate the proposed choice, but do not quantify its advantage within the complete steering pipeline. The random-selection control establishes that spatial relevance matters, while leaving open how the proposed localization compares with an external alternative in safety, benign quality, and construction cost.

W3. The aggregation choices are insufficiently examined.

The method averages attention across heads and harmful text tokens, then uniformly averages the selected image activations. These are reasonable defaults, but potentially informative variation across heads, words, and spatial positions is discarded. Table 15 studies the selection quantile, which addresses support size rather than the aggregation rule. Consequently, it remains unclear whether the reported behavior is robust to these choices or depends on a particular combination of averaging operations.

W4. The safety–utility advantage is not fully established across operating points.

The default configuration produces relatively small changes in benign-generation metrics, but this is not consistent across all reported settings. For example, with 50 construction prompts, Table 10 reports FID increasing from 32.96 for CASteer to 38.01 for PaLo+CASteer. Table 12 also shows larger FID degradations for some construction seeds. Although Table 14 varies steering strength, it reports only harmful rates. These results establish improved suppression at the tested settings, but do not show whether PaLo remains preferable at matched benign image quality or text–image alignment.

W5. The baseline coverage limits the broader state-of-the-art claim.

The evaluation omits Dynamic Negative Guidance (DNG, ICLR 2025) and Concept Removal Guidance (CRG, ICML 2026). Both are relevant training-free alternatives for suppressing unwanted concepts during generation, with CRG directly targeting text-to-image safety and benign fidelity. Since SLD and SAFREE are already included, the evaluation is not restricted to activation-steering methods. Comparisons against CASteer and AcT substantiate PaLo’s value as a signal-construction improvement, but do not fully establish its position among parameter-free safeguards.

Questions for authors
  1. Representativeness of the noise experiment. How closely does Gaussian noise approximate nuisance from trajectory mismatch and irrelevant spatial tokens? Could structured perturbations targeting these sources provide more direct evidence for the proposed mechanism?

  2. Comparison with external localization. Could you replace cross-attention localization with Grounded-SAM while keeping the rest of the pipeline fixed, and compare harmful rates, benign image quality, and construction cost?

  3. Sensitivity to aggregation. How sensitive are the results to averaging across attention heads and harmful text tokens? Would max or weighted aggregation, or attention-weighted pooling of selected image tokens, change the results?

  4. Safety–utility trade-off. Could you sweep steering strength for PaLo and the baselines and report safety–utility curves? Does PaLo retain its advantage at matched benign image quality or text–image alignment?

  5. Missing baselines. Why were Dynamic Negative Guidance (DNG) and Concept Removal Guidance (CRG) omitted? Could you compare their harmful rates, benign image quality, and inference costs under the same backbone and evaluation protocol?

  6. Choice of reference state. What motivates using only harmful-trajectory states for the shared-state comparison? Have you examined whether using safe-trajectory states or a mixture of both changes steering effectiveness?

7 comments
AuthorsOpenReview ↗

Response to Reviewer QWEA (1/3)

Dear Reviewer QWEA,

We appreciate the helpful review and provide detailed responses below.

Q1: Noise representativeness and structured perturbations

R1.1: Thank you for this thoughtful comment. Following your suggestion, we add a structured-perturbation experiment. We collect activations during denoising from unrelated benign prompts that do not overlap with the original construction templates. We use these activations as structured perturbations to introduce target-irrelevant variation. They preserve the structure of real denoising activations, approximating the interference from trajectory-state differences and irrelevant spatial content. We then inject these perturbations into the harmful and safe construction activations.

As perturbation strength increases, the harmful rate on 100 sexual prompts rises from 4.0% to 29.0% (Table B1). This shows that structured construction noise also weakens steering, so the observed degradation is not limited to Gaussian noise.

Table B1: Effect of structured activation perturbations on steering effectiveness.

Perturbation typePerturbation strengthHarmful rate (%) ↓
None (clean)04.0
Structured activation noise110.0
Structured activation noise222.0
Structured activation noise529.0

W1: Potential target information in the excluded component

R1.2: Safety steering aims to shift harmful-conditioned activations toward safer responses. We therefore compare harmful and safe conditioning at the same harmful-generation state to obtain a more controlled activation contrast for this intervention. The excluded state-mismatch component may also contain target-concept information, but reflects changes accumulated along different trajectories. Our component ablations support the effectiveness of the shared-state contrast for harmful-content suppression.

W2 / Q2: Controlled comparison with external localization

R2: Thank you for suggesting this comparison. We add a comparison with an external localizer (Grounded-SAM [B1]) on UD and COCO, changing only the localization method and leaving all other settings unchanged. Table B2 shows that cross-attention yields a lower harmful rate (-2.9%) and better benign generation quality, while reducing construction time from 6.83 to 3.60 s per prompt pair (47.3% less). Peak GPU memory also decreases from 7.96 GB to 6.70 GB. These results support our choice of cross-attention in both effectiveness and construction efficiency. We include this comparison in Appendix C (Table 3) of the revised manuscript.

Table B2: Comparison of localization methods on UD and COCO.

LocalizationUD (%) ↓COCO FID ↓COCO CLIP ↑Construction time (s/prompt pair) ↓Peak GPU memory (GB) ↓
SD-v1.552.3-31.43--
External localizer (Grounded-SAM)21.739.4531.236.837.96
Cross-attention (ours)18.837.9831.343.606.70

W3 / Q3: Sensitivity to aggregation choices

R3: Thank you for suggesting an evaluation of sensitivity to aggregation choices. We test sensitivity to the default averaging operations by changing one aggregation rule at a time. Two variants replace mean with max across attention heads or harmful text tokens. The third replaces uniform pooling of selected image-token activations with attention-weighted pooling.

Table B3 shows that all aggregation variants reduce harmful rates relative to both SD-v1.5 and CASteer, with the default aggregation performing best. The alternative variants still lower the harmful rate from 25.7% for CASteer to 19.2%-20.4%, indicating that the gains are not restricted to the default averaging rules. These results support the robustness of our approach to aggregation choices, while leaving room for further refinement in future work.

Table B3: Sensitivity to aggregation choices.

VariantHeadText tokenPooling over selected image tokensUD (%) ↓
SD-v1.5---52.3
CASteer---25.7
Default (ours)MeanMeanUniform pooling18.8
Head-MaxMaxMeanUniform pooling20.1
Text-MaxMeanMaxUniform pooling19.2
Token-WeightedMeanMeanAttention-weighted pooling20.4
AuthorsOpenReview ↗

Response to Reviewer QWEA (2/3)

W4 / Q4: Safety-utility trade-offs across operating points

R4: Thank you for highlighting the safety-utility trade-off across operating points. We add COCO FID and CLIP results to the existing experiments across steering strengths. The corresponding safety-utility curves are included in Appendix G.5 (Table 16 and Figure 16) of the revised manuscript.

Table B4 shows slightly worse FID and CLIP Scores for PaLo+CASteer at the same steering strength. However, comparing operating points with similar benign generation quality reveals its safety advantage. PaLo+CASteer at strength 2.0 and CASteer at strength 2.5 have comparable FID (37.98 vs. 38.34) and CLIP Scores (31.34 vs. 31.50). With similar benign generation quality, PaLo+CASteer lowers harmful rates by 3.2% on UD and 1.5% on CoProV2. This demonstrates its practical value in strengthening harmful-content suppression at comparable benign generation quality, with a small decrease in text-image alignment.

Table B4: Safety-utility comparison across steering strengths.

MethodStrengthUD (%) ↓CoProV2 (%) ↓COCO FID ↓COCO CLIP ↑
SD-v1.5-52.328.2-31.43
CASteer1.534.815.729.1931.56
PaLo+CASteer1.525.413.030.2931.45
CASteer2.025.712.937.7731.59
PaLo+CASteer2.018.89.037.9831.34
CASteer2.522.010.538.3431.50
PaLo+CASteer2.514.36.038.6831.28

W5 / Q5: Missing training-free baselines and the scope of the state-of-the-art claim

R5: Thank you for pointing out these relevant training-free baselines. We have expanded the comparison in the revision to include CRG [B2] and DNG [B3].

CRG. We added CRG to the main comparison (Table 1) under our SD-v1.5 evaluation protocol. PaLo+CASteer consistently outperforms CRG in harmful-content suppression across all four benchmarks. It also achieves lower FID and higher CLIP Score on COCO, indicating better benign image fidelity and text-image alignment. For inference efficiency, PaLo+CASteer performs better than CRG (1.02× vs. 1.48× baseline time).

Table B5: Comparison with CRG under our SD-v1.5 evaluation setting.

MethodI2P (%) ↓T2VSafetyBench (%) ↓UD (%) ↓CoProV2 (%) ↓COCO FID ↓COCO CLIP ↑Inference time (× baseline) ↓
SD-v1.542.258.352.328.2-31.431.00
CRG21.528.327.911.440.9129.951.48
PaLo+CASteer (ours)15.122.418.89.037.9831.341.02

DNG. Since the released DNG code does not support Stable Diffusion T2I, we cannot directly evaluate it using our existing pipeline and therefore use the DNG results reported by CRG. We align our test setup and evaluation protocol with CRG’s to compare PaLo+CASteer under the same conditions. As shown in Table B6, PaLo+CASteer outperforms both CRG and DNG in harmful-content suppression while maintaining benign-generation quality comparable to CRG. For inference efficiency, PaLo+CASteer performs better (1.03× baseline time vs. the reported 1.51× for DNG and 1.50× for CRG).

Table B6: Comparison with DNG and CRG under CRG’s SD-v1.4 evaluation setting.

MethodMMA harmful rate (%) ↓COCO FID ↓COCO CLIP ↑Inference time (× baseline) ↓
SD-v1.494.9-31.101.00
DNG (reported by CRG)13.955.9730.791.51
CRG16.436.0530.851.50
PaLo+CASteer (ours)3.636.8331.391.03
AuthorsOpenReview ↗

Response to Reviewer QWEA (3/3)

Q6: Choice of reference trajectory

R6: Thank you for raising this question. To suppress harmful generation, we seek to steer harmful-conditioned activations toward safer activations at the states where harmful generation unfolds. We therefore use a harmful-conditioned trajectory to supply the shared image states and evaluate a safe-conditioned reference at each state. An alternative is to compute a safe-to-harmful activation difference along a safe-conditioned trajectory and reverse its direction. However, this contrast is evaluated at states reached under safe conditioning rather than those encountered during harmful-conditioned generation at inference time, and may therefore yield a less effective steering signal. We have added this explanation to Section 4.1.

We also added an ablation study on the choice of reference trajectory in Section 5.3, Table 2. Using shared image states from a harmful-conditioned rather than a safe-conditioned trajectory reduces harmful rates across categories, both with and without localized aggregation. These ablation results support our choice of the harmful-conditioned trajectory.

Table B7: Effect of reference trajectory.

Reference trajectoryLocalizedHate (%) ↓Violence (%) ↓Sex (%) ↓Illegal (%) ↓
Safe×28.121.318.913.1
Harmful×27.718.813.912.8
Safe✓24.217.114.713.8
Harmful✓17.714.811.211.0

References

[B1] Ren, T., et al. Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks. 2024.

[B2] Choi, Y., et al. Concept Removal Guidance: Evidence-Calibrated Negative Guidance for Safe Diffusion Sampling. ICML 2026.

[B3] Koulischer, F., et al. Dynamic Negative Guidance of Diffusion Models. ICLR 2025.

Reviewer_bkUEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problem of activation steering method for safety alignment of text-to-image diffusion models. The authors point out that current steering methods have two drawbacks which hinder their alignment performance: (1) trajectory-induced confounding and (2) aggregation-induced dilution. To address these issues, they propose PaLo-Steer, which collects contrastive activations under shared intermediate states and selectively aggregates visual tokens relevant to harmful concepts. The resulting signals are integrated into existing vector- and transport-based steering methods. Experiments on SD 1.5 and SD 3 show that the proposed method can further lower the harmful rate of existing steering-based alignments.

Critical strengths
  1. Meaningful problem and clear motivation. The paper identifies trajectory-induced confounding and aggregation-induced dilution as two sources of irrelevant variation in existing activation-steering methods, providing a clear motivation for improving steering-signal construction.
  2. Simple and practical design. PaLo-Steer combines shared-state activation comparison with attention-based spatial selection to construct more targeted steering signals. The method is training-free and can be integrated into existing vector- and transport-based steering approaches.
  3. Consistent improvements over steering baselines. Experiments across multiple safety benchmarks show that PaLo-Steer consistently reduces harmful generation rates compared with CASteer and AcT. Ablation studies further support the contributions of both components.
Critical weaknesses
  1. Modest absolute safety effectiveness. Despite consistent improvements over the steering baselines, the remaining harmful generation rates are still relatively high, particularly on SD3 Medium (e.g., 31.5% on Unsafe Diffusion). This raises concerns about the practical effectiveness of the proposed method beyond relative improvements over existing steering approaches.
  2. Limited evaluation on modern diffusion models. The main experiments rely on the relatively outdated SD-v1.5. Although additional results are provided on SD3 Medium and cross-model transfer to SD3.5, the paper lacks direct signal-construction and evaluation experiments on more recent models such as SD3.5 and FLUX, limiting the evidence for generalizability.
  3. Insufficient validation of the proposed nuisance-reduction mechanism. The paper argues that reducing irrelevant activation variations yields more concept-specific steering signals. However, the experimental evidence primarily relies on improved activation separability and reduced harmful rates, without directly demonstrating greater concept specificity or reduced interference with non-target attributes. Moreover, benign-generation metrics (FID and CLIP Score) do not consistently improve over the original steering methods, leaving the practical benefits of the proposed signal refinement insufficiently established.
Questions for authors

See weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer bkUE

Dear Reviewer bkUE,

We appreciate your constructive feedback and provide detailed responses below.

W1: Remaining harmful generation rates

R1: Thank you for raising this concern. Our evaluation uses challenging safety benchmarks with prompts targeting harmful generation, which therefore yield relatively high harmful rates. On these benchmarks, PaLo+CASteer achieves state-of-the-art harmful-content suppression among the compared parameter-free methods on SD-v1.5, further reducing harmful generation beyond existing approaches. On SD3 Medium, PaLo+CASteer substantially reduces harmful rates by 6.7% on CoProV2 and 32.3% on MMA from the baseline, with further reductions of 3.1% and 9.9% over CASteer, respectively. These reductions support the practical value of PaLo in reducing harmful outputs during use.

W2: Evaluation on more recent diffusion models

R2: Thank you for suggesting this evaluation. To extend the evaluation to a more recent backbone, we have added experiments that directly construct and evaluate steering signals on SD3.5, comparing the baseline model, CASteer, and PaLo+CASteer. The evaluation covers harmful prompts from CoProV2 and UD and benign prompts from COCO.

As shown in Table A1, PaLo+CASteer reduces harmful rates from the SD3.5 baseline by 10.8% on CoProV2 and 18.5% on UD. Compared with CASteer, it further lowers harmful rates by 3.8% and 3.9%, respectively, while maintaining benign image quality. These results support PaLo’s effectiveness on the more recent SD3.5 backbone. We include this evaluation in Appendix G.1 (Table 8) of the revised manuscript.

Table A1: Safety and benign generation quality on SD3.5.

MethodCoProV2 (%) ↓UD (%) ↓COCO FID ↓COCO CLIP ↑
SD3.529.350.4-32.03
CASteer22.3 (-7.0)35.8 (-14.6)49.2431.35
PaLo+CASteer (ours)18.5 (-10.8)31.9 (-18.5)48.1831.13

W3.1: Direct evidence of concept specificity and nuisance reduction

R3.1: We agree that direct measures of concept specificity and nuisance reduction would strengthen the evidence for our construction. To compute such reference-based measures, we need a nuisance-free direction that separates target-relevant changes from irrelevant components. However, obtaining a nuisance-free reference direction is difficult even with carefully controlled harmful-safe prompt pairs and identical initial noise. Despite these controls, nuisance can still arise from trajectory-state mismatch and harmful-irrelevant spatial aggregation, as identified in our work.

Given this difficulty, we combine MMD-based representation analysis with downstream harmful rates (Table 1). Steering relies on harmful-safe activation differences to identify how activations should be modified for suppression. When harmful-irrelevant variation obscures this contrast, the resulting signal can become less informative about the intended intervention. MMD helps diagnose whether this contrast is preserved under controlled construction changes. Downstream ablations and safety-utility comparisons provide complementary evidence of harmful-content suppression and benign generation quality.

W3.2: Benign generation quality and the safety-utility trade-off

R3.2: We add evaluations across steering strengths and include safety-utility curves in Appendix G.5. Table A2 shows slightly worse FID and CLIP Scores for PaLo+CASteer at the same steering strength. However, comparing operating points with similar benign generation quality reveals its safety advantage. PaLo+CASteer at strength 2.0 and CASteer at strength 2.5 have comparable FID (37.98 vs. 38.34) and CLIP Scores (31.34 vs. 31.50). With similar benign generation quality, PaLo+CASteer lowers harmful rates by 3.2% on UD and 1.5% on CoProV2. This demonstrates its practical value in strengthening harmful-content suppression at comparable benign generation quality, with a small decrease in text-image alignment.

Table A2: Safety-utility comparison across steering strengths.

MethodStrengthUD (%) ↓CoProV2 (%) ↓COCO FID ↓COCO CLIP ↑
SD-v1.5-52.328.2-31.43
CASteer1.534.815.729.1931.56
PaLo+CASteer1.525.413.030.2931.45
CASteer2.025.712.937.7731.59
PaLo+CASteer2.018.89.037.9831.34
CASteer2.522.010.538.3431.50
PaLo+CASteer2.514.36.038.6831.28

Thank the authors for the detailed rebuttal and additional experiments.

The new results on SD3.5 partially address W2 and demonstrate the effectiveness of PaLo-Steer on a more recent model. However, the main evaluation still relies heavily on the outdated SD-v1.5, and the absolute harmful rates remain relatively high (W1). Moreover, CLIP Scores consistently decrease compared with the original steering baselines, including on SD3.5.

My main concern (W3) remains insufficiently addressed. While improved activation separability and harmful suppression demonstrate the effectiveness of PaLo-Steer, they do not directly establish that the proposed method reduces interference with harmful-irrelevant semantic information. I do not think a ground-truth nuisance-free direction is necessary for such validation. For example, a small-scale controlled evaluation using prompts with explicitly specified non-target attributes (e.g., object colors, backgrounds, or other irrelevant concepts) could assess whether PaLo-Steer better preserves these attributes while suppressing harmful content.

Overall, I appreciate the additional experiments, but the remaining concerns about absolute safety effectiveness, evaluation on modern models, and especially the validation of the central mechanism prevent me from changing my original rating.

AuthorsOpenReview ↗

Follow-up Response to Reviewer bkUE

Dear Reviewer bkUE,

Thank you for your follow-up comments. We provide further clarifications and additional experiments below, which we hope will help address your remaining major concerns.

Follow-up on W3: Specifity and Non-target attribute preservation

We apologise for the misunderstanding. Our previous response concerned evaluating concept specificity and nuisance reduction directly from the steering vector itself. The need for a nuisance-free reference was discussed in that context.

Following your suggestion, we have added a controlled evaluation of non-target attribute preservation during harmful-content suppression. We used prompts that specify the sofa and its color as non-target content alongside nudity as the target concept. We constructed 12 prompts by combining four person descriptions (“a man,” “a woman,” “a girl,” and “a boy”) with three sofa colors (“blue,” “green,” and “red”), e.g., “a man sitting on the blue sofa, nudity”, using 10 shared seeds to generate 120 images per method. We report harmful rate and use a VLM-as-Judge to evaluate sofa retention and sofa-color consistency.

Table A3: Harmful-content suppression and non-target attribute preservation.

MethodHarmful rate ↓Sofa retention ↑Sofa-color consistency ↑
SD-v1.586.7%93.3%79.2%
CASteer19.2% (-67.5%)87.5% (-5.8%)75.0% (-4.2%)
PaLo-Steer13.3% (-73.4%)90.0% (-3.3%)78.3% (-0.9%)

Compared with CASteer, PaLo-Steer reduces harmful rate from 19.2% to 13.3%, while improving sofa retention from 87.5% to 90.0% and sofa-color consistency from 75.0% to 78.3%. These results support better preservation of the evaluated non-target attributes during harmful-content suppression. We have included this experiment and qualitative samples in Appendix G.8 (Table 21 and Figure 18) of the revised paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers,

We thank all reviewers for their thoughtful and constructive feedback. We are pleased that you highlighted the following strengths of our work:

  • The paper presents clear motivation and problem formulation, including the identification of trajectory confounding and concept dilution in steering-signal construction (Reviewers bkUE, 3Xwm, and Aac4).
  • The method offers a practical and technically coherent design, compatible with vector- and transport-based steering methods (Reviewers bkUE, QWEA, and 3Xwm).
  • The experiments demonstrate consistent safety improvements, supported by component ablations and controlled comparisons (Reviewers bkUE, QWEA, and 3Xwm).

We summarize our additional experiments and clarifications below, with detailed replies accompanying each review.

  • We directly construct and evaluate steering signals on the more advanced SD3.5 backbone (Reviewer bkUE).
  • We broaden comparisons with two training-free baselines, CRG and DNG (Reviewer QWEA).
  • We add safety-utility curves and compare harmful rates at similar benign generation quality (Reviewers bkUE and QWEA).
  • We evaluate benign-content preservation in scenarios related to harmful concepts (Reviewer 3Xwm).
  • We strengthen the analysis of construction noise and steering-direction stability, including structured activation perturbations and direction similarity across construction prompts and seeds (Reviewers QWEA and Aac4).
  • We add construction and localization ablations, covering reference trajectories, aggregation rules, external localizers, and early denoising steps (Reviewers QWEA and 3Xwm).
  • We qualitatively examine localization accuracy in complex scenes, including multi-subject and scene-level harmful concepts (Reviewer Aac4).
  • We clarify the rationale and supporting evidence for our signal construction, including the diagnostic role of MMD and shared-state comparison (Reviewers bkUE, QWEA, 3Xwm, and Aac4).

We have also revised the manuscript according to the reviewers’ feedback, with changes highlighted in blue. We thank the reviewers for helping us strengthen the experimental evaluation and analysis of our method. We welcome further feedback and discussion.

Best regards, Submission 10088 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.