Where and How Much to Edit? Localized Concept Suppression in Diffusion Models

Inference-time concept suppression in text-to-image diffusion models aims to remove unwanted concepts while preserving the rest of the generation. Existing inference-time steering methods often rely on fixed-strength, spatially global interventions, which induce a suppression–preservation trade-off: stronger interventions improve target removal but increasingly distort unrelated semantic and visual content. We introduce GeoGRAFT, a localized inference-time concept-suppression framework for text-to-image diffusion models. GeoGRAFT keeps the diffusion model fixed and trains sparse autoencoders (SAEs) once on generic U-Net residual representations; concept-specific calibration then identifies target-associated sparse features. At inference time, these features produce a spatial evidence map from which GeoGRAFT extracts connected regions likely to express the target concept. GeoGRAFT then performs region-localized donor grafting, interpolating activations only within these selected regions toward spatially corresponding activations from a paired concept-free generation. To avoid unnecessarily strong edits, GeoGRAFT separates intervention proposal from downstream validation: it first selects the minimum graft strength that satisfies a local sufficiency criterion, and then retains the intervention only if it produces the desired downstream effect in the frozen generator. An independent target-presence (no-op) gate further prevents unnecessary editing when the target concept is absent. Across I2P, Ring-A-Bell, MMA-Diffusion, P4D, and UnlearnDiffAtk, GeoGRAFT reduces the mean unsafe-content attack success rate of SDXL-Turbo from 29.4% to 1.07%. On UnlearnCanvas objects, it achieves 85.3% target-removal accuracy while preserving 90.8% intra-domain and 82.6% cross-domain retaining accuracy. Matched-suppression comparisons and controlled component ablations show the value of combining sparse spatial localization, region-wise minimum-sufficient dose selection, and downstream validation for selective inference-time concept control.