acceptodds
Withdrawn from ICLR 2027

Where and How Much to Edit? Localized Concept Suppression in Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Inference-time concept suppression in text-to-image diffusion models aims to remove unwanted concepts while preserving the rest of the generation. Existing inference-time steering methods often rely on fixed-strength, spatially global interventions, which induce a suppression–preservation trade-off: stronger interventions improve target removal but increasingly distort unrelated semantic and visual content. We introduce GeoGRAFT, a localized inference-time concept-suppression framework for text-to-image diffusion models. GeoGRAFT keeps the diffusion model fixed and trains sparse autoencoders (SAEs) once on generic U-Net residual representations; concept-specific calibration then identifies target-associated sparse features. At inference time, these features produce a spatial evidence map from which GeoGRAFT extracts connected regions likely to express the target concept. GeoGRAFT then performs region-localized donor grafting, interpolating activations only within these selected regions toward spatially corresponding activations from a paired concept-free generation. To avoid unnecessarily strong edits, GeoGRAFT separates intervention proposal from downstream validation: it first selects the minimum graft strength that satisfies a local sufficiency criterion, and then retains the intervention only if it produces the desired downstream effect in the frozen generator. An independent target-presence (no-op) gate further prevents unnecessary editing when the target concept is absent. Across I2P, Ring-A-Bell, MMA-Diffusion, P4D, and UnlearnDiffAtk, GeoGRAFT reduces the mean unsafe-content attack success rate of SDXL-Turbo from 29.4% to 1.07%. On UnlearnCanvas objects, it achieves 85.3% target-removal accuracy while preserving 90.8% intra-domain and 82.6% cross-domain retaining accuracy. Matched-suppression comparisons and controlled component ablations show the value of combining sparse spatial localization, region-wise minimum-sufficient dose selection, and downstream validation for selective inference-time concept control.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unleanring, Editing

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_wxo3OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes GeoGRAFT, an inference-time method for suppressing concepts in one-step diffusion models (SDXL-Turbo). The generator weights are left unchanged. Instead, the method trains SAEs on U-Net residual activations, selects target-related features using paired target and concept-free prompts, and uses those features to build a spatial evidence map. Patches above a calibrated threshold are grouped into connected regions. Within each region, activations are interpolated toward a same-seed concept-free generation. The interpolation strength is set to the smallest value in a fixed set that lowers the SAE evidence below a threshold, and an edit is kept only if it moves the final noise prediction toward the concept-free branch. An external detector (NudeNet or a classifier) skips editing when the target is not present in the original output. On nudity benchmarks, the method reduces the mean ASR of SDXL-Turbo from 29.3% to 1.07%. On UnlearnCanvas objects, it reports 85.3% UA with 90.8% IRA and 82.6% CRA, and compares favorably to global SAE steering at similar suppression levels.

Critical strengths
  1. Practical deployment setting. The generator weights are never modified. The SAEs are trained once on generic captions and are not tied to any concept, so adding a new concept only requires a small calibration step with paired prompts. This makes the method easy to apply to an existing deployed model.

  2. Well-structured method. The pipeline separates three questions: where to edit (SAE evidence and connected regions), how strongly to edit (the smallest graft strength that meets a local evidence criterion), and whether the edit works (a check on the frozen U-Net's output). Using a same-seed concept-free branch as the donor keeps the edited activations close to the model's normal distribution, which avoids the artifacts that direct SAE steering is known to cause.

  3. Broad and careful evaluation. Results cover five nudity attack benchmarks, UnlearnCanvas objects, and styles. All baselines use the same backbone, prompts, seeds, and evaluators. The ablations remove each component in turn and are complemented by threshold sensitivity, layer choice, and transfer to SD-Turbo.

Critical weaknesses
  1. Limited novelty, and the SAE appears unnecessary. Every component has a direct precedent:

Using SAE features for detection and then replacing the detected patches is the approach of PER [1].

Region-restricted intervention with adaptive strength for safe generation appears in Detect-and-Guide [2], SP-Guard [3], and SafeCtrl [4]. Gated localized concept erasure appears in GLoCE [5].

Blending activations from a paired generation under a spatial mask is standard in image editing [6, 7].

Apart from PER, none of these are cited or compared, so the paper's main claim, "where and how much to edit", is not positioned against the work that already addresses it. The SAE is also not shown to be needed. The method already produces a same-seed concept-free branch, so the activation difference between the two branches would localize the target without any SAE. Cross-attention maps, as in [2], are another existing option. Neither is compared, yet the SAE must be retrained for every backbone and layer. The ablations add to this concern: under the same grafting mechanism, the global graft reaches 83.9/84.3 against 86.3/87.0 for the full method, so localization contributes little.

  1. The donor is already a safe image. Each input requires a concept-free prompt and at least one extra generation, sometimes several (Appendix A.4). The paper does not explain how these prompts are produced, particularly for nonsensical adversarial prompts such as those in MMA-Diffusion and Ring-A-Bell. Two obvious baselines are missing: returning the donor image directly, and prompt rewriting alone, each reported with ASR and DINO/LPIPS. Without them, it is unclear how much of the gain comes from grafting rather than from simply having a safe generation available. Runtime is also not reported.

  2. The gate detector is also the evaluator. For nudity, NudeNet decides whether to edit and also computes ASR. For objects, the gate appears to be the same classifier that computes UA. Any output the evaluator already considers clean is left unedited, which gives GeoGRAFT information the baselines do not have and weakens the headline 1.07% ASR. The no-gate ablation covers objects only. The authors should report nudity results with a gate detector that differs from the evaluator, or give the baselines the same gate.

References

[1] E. Cassano, R. Renzulli, R. Ahmed, M. Grangetto, S. Alaniz. Look But Don't Touch with Sparse Autoencoders for Unlearning in Diffusion Models. ECCV 2026. arXiv:2606.31699. [2] Li et al. Detect-and-Guide: Self-regulation of Diffusion Models for Safe Text-to-Image Generation via Guideline Token Optimization. CVPR 2025. arXiv:2503.15197. [3] S. Yu et al. SP-Guard: Selective Prompt-adaptive Guidance for Safe Text-to-Image Generation. arXiv:2511.11014, 2025. [4] SafeCtrl: Region-Aware Safety Control for Text-to-Image Diffusion via Detect-Then-Suppress. arXiv:2604.03941, 2026. [5] B. H. Lee, S. Lim, S. Y. Chun. Localized Concept Erasure for Text-to-Image Diffusion Models Using Training-Free Gated Low-Rank Adaptation. CVPR 2025. arXiv:2503.12356. [6] A. Hertz et al. Prompt-to-Prompt Image Editing with Cross-Attention Control. ICLR 2023. [7] O. Avrahami, O. Fried, D. Lischinski. Blended Latent Diffusion. ACM TOG (SIGGRAPH) 2023.

Questions for authors

Please see Critical Weaknesses.

Reviewer_MmSXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a concept-removal framework for images generated by diffusion modes to remove unwanted concepts, while preserving content. Specifically, the authors train a SAE on U-net outputs and learn disentangled concepts, then identify the concept they want to erase and which SAE features are associated with. To zero a concept out they construct a spatial evidence map to localize target-associated patches and connect regions, and then they mix activations with concept-free activations (derived from features of U-Net with a prompt that does not contain the concept) inside selected regions.

Critical strengths
  • The challenge in concept suppression is identified, many works on concept removal suffer from preservation of semantics
  • The distinction between where to intervene, how strongly to intervene, and whether intervention is necessary is nice. I like the idea of mixing between two activations, and from where to mix and where to mix at.
  • The paper is well presented and easy to undertand
Critical weaknesses
  • I think there are very important comparisons the paper misses out completely. [R1, R2, R3] . R1 does exactly the same thing, just in a different way. It is also training-free, no SAEs. R2 also performs concept editing by simply switching the prompt during the generation with the same random seed. Both these methods are training-free, and much more simpler. No comparisons with these methods are made, and they are not even cited. There are many other training-free methods, these are just what I can remember. So if I can do concept erasing in a training-free manner, why would I train an SAE, and perform this engineered and computationally expensive pipeline that the authors propose?
  • Computation overhead is not justified. The method requires now an additional SAE, two passes through U-Net (one additional pass to get the concept-free features - and with classifier free guidance that makes it 4 passes), an external detector and multiple SAE passes. This also increases the memory. I am sure the memory and computation overhead goes up by quite a large number.
  • I think U-net diffusion models are pretty obsolete by now. No experiments on DiT-based diffusion models are shown (e.g., FLUX, PixArt-alpha)
  • The proposed method assumes that a suitable alternative prompt and corresponding activation can be obtained. In reality, how is this constructed? Would you need an LLM/small prompt re-writer to do this? This just means even more extra-head.
  • The evlauation is wrong. NudeNet is used both for the presence gate and as the benchmark evaluator. In other words you are using the same detector to detect and evaluate, so logically you would get higher scores.

[R1] Localizing Knowledge in Diffusion Transformers
[R2] Temporal Concept Dynamics in Diffusion Models via Prompt-Conditioned Interventions
[R3] SPEED: Scalable, Precise, and Efficient Concept Erasure for Diffusion Models

Questions for authors

The paper is interesting, but it has major concerns. See weaknesses. Therefore, my decision will be sadly negative.

Reviewer_Vp4UOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces GeoGRAFT, an inference-time method for suppressing specific concepts in one-step text-to-image diffusion models while preserving unrelated image content. GeoGRAFT trains sparse autoencoders on generic U-Net residual representations, then applies concept-specific calibration to identify target-associated features and localize the regions where the target appears. For each detected region, it interpolates the target activation toward the corresponding activation from a paired, concept-free generation. The method selects the smallest tested graft strength that satisfies a local evidence criterion, and keeps the edit only if downstream validation confirms movement toward the concept-free prediction. A target-presence gate prevents editing generations that do not contain the target. On SDXL-Turbo, the authors report reducing the mean unsafe-content attack success rate from 29.34% to 1.07%. On UnlearnCanvas object suppression, they report 85.3% unlearning accuracy, with 90.8% intra-domain and 82.7% cross-domain retaining accuracy.

Critical strengths
  1. The paper presents a clear and well-motivated approach to the suppression–preservation trade-off in diffusion models. Its central design choice, using SAE features to localize target-associated regions and restricting interventions to those regions, s intuitive and practically relevant.
Critical weaknesses
  1. The method relies on SAE to localize the concept. However, the SAE inputs lack motivation.
  2. GeoGRAFT performs additional inference-time computation, including generating a paired concept-free branch, evaluating candidate graft strengths, and validating proposed edits through downstream U-Net computation.
  3. The practical advantage of GeoGRAFT over simpler prompt-level interventions is not clearly established. Since the method leaves the model weights unchanged and adds inference-time processing, why not rewrite the prompt directly?
  4. The evaluation is not comprehensive. The paper only focuses on nudity. But for the I2P benchmark, there are also other concepts such as hate, harassment, violence, self-harm, sexual content, shocking images, and illegal activity. These concepts should also be included.
  5. The generalization evaluation is limited to SDXL-Turbo and SD-Turbo, two closely related one-step Stable Diffusion models with U-Net backbones. It therefore remains unclear whether GeoGRAFT generalizes to conventional multi-step sampling or to newer architectures such as SDXL 1.0 with multi-step sampling [1], Stable Diffusion 3.5 Large (MMDiT) [2], PixArt-Σ (Diffusion Transformer) [3], and FLUX.1-schnell (rectified-flow transformer) [4].

[1]. https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0

[2]. https://huggingface.co/stabilityai/stable-diffusion-3.5-large

[3]. https://huggingface.co/PixArt-alpha/PixArt-Sigma

[4]. https://huggingface.co/black-forest-labs/FLUX.1-schnell

Questions for authors
  1. The method’s spatial localization depends on SAE features extracted from a particular U-Net block. The main experiments use up.0.0, while the appendix compares it with up.0.1 (a near one). What evidence motivated the choice of up.0.0, and how sensitive are the localization maps and suppression results to the choice of intervention block?
  2. In particular, do SAEs trained on different blocks identify consistent target regions, or can they produce substantially different, even contradictory, spatial localization?
  3. What is the end-to-end latency, throughput, and computational overhead relative to unmodified SDXL-Turbo and competing suppression methods? Does the overhead grow with the number of detected regions, and does the method remain practical in latency-sensitive settings?
  4. GeoGRAFT processes candidate regions greedily and may apply multiple region-wise grafts to a single generation, but the paper does not report how many regions are typically considered or accepted per image, how many dose candidates are evaluated, or how many downstream validation passes are required.
  5. Considering the practical scenario mentioned in CCRT[1] and UCE[2], could the method remove multiple concepts?

[1].Han, Tingxu, et al. "Continuous concepts removal in text-to-image diffusion models." Advances in Neural Information Processing Systems 38 (2026): 88343-88372.

[2]. Gandikota, Rohit, et al. "Unified concept editing in diffusion models." 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.