- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes GeoGRAFT, an inference-time method for suppressing concepts in one-step diffusion models (SDXL-Turbo). The generator weights are left unchanged. Instead, the method trains SAEs on U-Net residual activations, selects target-related features using paired target and concept-free prompts, and uses those features to build a spatial evidence map. Patches above a calibrated threshold are grouped into connected regions. Within each region, activations are interpolated toward a same-seed concept-free generation. The interpolation strength is set to the smallest value in a fixed set that lowers the SAE evidence below a threshold, and an edit is kept only if it moves the final noise prediction toward the concept-free branch. An external detector (NudeNet or a classifier) skips editing when the target is not present in the original output. On nudity benchmarks, the method reduces the mean ASR of SDXL-Turbo from 29.3% to 1.07%. On UnlearnCanvas objects, it reports 85.3% UA with 90.8% IRA and 82.6% CRA, and compares favorably to global SAE steering at similar suppression levels.
Critical strengths
-
Practical deployment setting. The generator weights are never modified. The SAEs are trained once on generic captions and are not tied to any concept, so adding a new concept only requires a small calibration step with paired prompts. This makes the method easy to apply to an existing deployed model.
-
Well-structured method. The pipeline separates three questions: where to edit (SAE evidence and connected regions), how strongly to edit (the smallest graft strength that meets a local evidence criterion), and whether the edit works (a check on the frozen U-Net's output). Using a same-seed concept-free branch as the donor keeps the edited activations close to the model's normal distribution, which avoids the artifacts that direct SAE steering is known to cause.
-
Broad and careful evaluation. Results cover five nudity attack benchmarks, UnlearnCanvas objects, and styles. All baselines use the same backbone, prompts, seeds, and evaluators. The ablations remove each component in turn and are complemented by threshold sensitivity, layer choice, and transfer to SD-Turbo.
Critical weaknesses
- Limited novelty, and the SAE appears unnecessary. Every component has a direct precedent:
Using SAE features for detection and then replacing the detected patches is the approach of PER [1].
Region-restricted intervention with adaptive strength for safe generation appears in Detect-and-Guide [2], SP-Guard [3], and SafeCtrl [4]. Gated localized concept erasure appears in GLoCE [5].
Blending activations from a paired generation under a spatial mask is standard in image editing [6, 7].
Apart from PER, none of these are cited or compared, so the paper's main claim, "where and how much to edit", is not positioned against the work that already addresses it. The SAE is also not shown to be needed. The method already produces a same-seed concept-free branch, so the activation difference between the two branches would localize the target without any SAE. Cross-attention maps, as in [2], are another existing option. Neither is compared, yet the SAE must be retrained for every backbone and layer. The ablations add to this concern: under the same grafting mechanism, the global graft reaches 83.9/84.3 against 86.3/87.0 for the full method, so localization contributes little.
-
The donor is already a safe image. Each input requires a concept-free prompt and at least one extra generation, sometimes several (Appendix A.4). The paper does not explain how these prompts are produced, particularly for nonsensical adversarial prompts such as those in MMA-Diffusion and Ring-A-Bell. Two obvious baselines are missing: returning the donor image directly, and prompt rewriting alone, each reported with ASR and DINO/LPIPS. Without them, it is unclear how much of the gain comes from grafting rather than from simply having a safe generation available. Runtime is also not reported.
-
The gate detector is also the evaluator. For nudity, NudeNet decides whether to edit and also computes ASR. For objects, the gate appears to be the same classifier that computes UA. Any output the evaluator already considers clean is left unedited, which gives GeoGRAFT information the baselines do not have and weakens the headline 1.07% ASR. The no-gate ablation covers objects only. The authors should report nudity results with a gate detector that differs from the evaluator, or give the baselines the same gate.
References
[1] E. Cassano, R. Renzulli, R. Ahmed, M. Grangetto, S. Alaniz. Look But Don't Touch with Sparse Autoencoders for Unlearning in Diffusion Models. ECCV 2026. arXiv:2606.31699. [2] Li et al. Detect-and-Guide: Self-regulation of Diffusion Models for Safe Text-to-Image Generation via Guideline Token Optimization. CVPR 2025. arXiv:2503.15197. [3] S. Yu et al. SP-Guard: Selective Prompt-adaptive Guidance for Safe Text-to-Image Generation. arXiv:2511.11014, 2025. [4] SafeCtrl: Region-Aware Safety Control for Text-to-Image Diffusion via Detect-Then-Suppress. arXiv:2604.03941, 2026. [5] B. H. Lee, S. Lim, S. Y. Chun. Localized Concept Erasure for Text-to-Image Diffusion Models Using Training-Free Gated Low-Rank Adaptation. CVPR 2025. arXiv:2503.12356. [6] A. Hertz et al. Prompt-to-Prompt Image Editing with Cross-Attention Control. ICLR 2023. [7] O. Avrahami, O. Fried, D. Lischinski. Blended Latent Diffusion. ACM TOG (SIGGRAPH) 2023.
Questions for authors
Please see Critical Weaknesses.