PaLo-Steer: Nuisance-Reduced Signal Construction for Text-to-Image Safety Steering

Text-to-image (T2I) diffusion models offer powerful generation capabilities but can also produce harmful content. To suppress harmful content, *activation-steering* methods construct signals from harmful-safe activation contrasts and apply them to intermediate activations during denoising. The effectiveness of these methods depends on how well the constructed signals capture harmful-relevant activation changes while minimizing nuisance. However, we identify two sources of nuisance in existing methods that can weaken safety steering: *Trajectory-induced confounding* mixes harmful-concept effects with differences in intermediate image states; *Aggregation-induced dilution* weakens the harmful-relevant contrast by pooling irrelevant image tokens. To address these issues, we propose *PaLo-Steer*, which constructs cleaner steering signals through state-controlled activation comparison and concept-aware spatial aggregation. (i) **Pa**rallel contrastive activation collection isolates the harmful-concept effect by collecting harmful- and safe-conditioned activations at shared intermediate states along a single harmful trajectory. (ii) Spatially **Lo**calized activation selection uses model-intrinsic cross-attention maps to locate image tokens relevant to harmful content. It then aggregates activation contrasts only over these tokens to reduce concept dilution. PaLo-Steer supports both vector- and transport-map-based steering. Across multiple safety benchmarks, it achieves state-of-the-art harmful-content suppression among parameter-free methods, reducing harmful rates by up to 12.0% over existing activation-steering methods while preserving benign generation quality.