- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes SurgUn, a post-hoc concept unlearning method for text-to-image diffusion models that targets the erase–retain trade-off. First, under the target prompt, it pushes the model away from reconstructing target images while training it to reconstruct a fixed, diverse set of 100 distractor images, so that outputs spread across many non-target modes rather than collapsing to one anchor. Second, it trains LoRA on each candidate attention block separately and selects the block with the best unlearning–retention trade-off for all subsequent updates. This is based on the observation that suppression is achievable from most blocks, while retention depends strongly on which block is updated. Experiments on five benchmarks and three backbones show a better erase–retain balance than prior methods.
Critical strengths
-
An insightful block-wise empirical observation. The paper offers an interesting finding at the level of individual blocks: training each attention block on its own, almost every block can suppress the target concept well, but retention of non-target concepts changes a lot from block to block. This suggests that where an unlearning update is placed should be decided by its effect on retention, not only by how sensitive a location is to the target. That is a useful and practical insight for future localization-based unlearning methods.
-
A comprehensive and diverse evaluation. The method is tested across a wide range of challenging settings: object and style unlearning, IP-character erasure, compositional unlearning, sequential and hierarchical unlearning, and adversarial prompt recovery. It is also tested on both U-Net and flow-matching transformer backbones. This breadth gives a fairly complete picture of how the method behaves beyond the direct target prompt.
-
A lightweight and efficient design. All updates are restricted to a LoRA module on one selected block, so the per-concept checkpoint is very small and training stays cheap even on larger backbones such as SDXL and SANA-1.5.
Critical weaknesses
-
Limited novelty and unclear attribution of gains. The method largely combines existing ingredients: gradient ascent on the target (ESD, EraseDiff), non-target anchoring against over-erasure (Concept Ablation, here with a fixed pool of 100 anchors), and localized updates (SalUn). The retroactive-interference framing is, as the authors acknowledge, an analogy rather than a mechanism. More importantly, all baselines use global updates, while SurgUn updates only a carefully selected block, so it is unclear whether the gains come from the proposed objective or simply from localization.
-
Insufficient robustness evaluation and an unsupported dispersion claim. Robustness is tested only with Ring-A-Bell and paraphrased prompts. Stronger recovery attacks are missing, such as UnlearnDiffAtk, P4D and embedding-level attacks based on textual inversion (e.g., CCE), which are the standard test of whether a concept is truly removed from the weights.
-
Overstated superiority and an uneven comparison. In Table 1, SAeUron outperforms SurgUn on all three style metrics (95.8/99.1/99.4 vs. 93.8/95.3/90.8), and SurgUn's FID (63.55) is only mid-range (e.g., MiM-MU 49.14, CA 54.21). Moreover, baselines are reported only on SD v1.5, while SDXL and SANA-1.5 results (Tables 1 and 2) cover SurgUn alone, so its advantage on these backbones cannot be verified.
-
Incomplete accounting of computational cost. The reported runtime (e.g., 150 s on SD v1.5) seems to cover only the final unlearning run. It excludes per-block localization training and evaluation, generating the 100 distractor images, and checkpoint calibration. The amortization argument is validated only for objects and styles, and Table 9 appears to list setting-specific blocks, which suggests that localization must be rerun for new settings. A full end-to-end cost comparison is needed.
Questions for authors
Please see Critical Weaknesses.
2 comments
Response to Reviewer wWHF-2
[W3] Clarifying the Erase–Retain Trade-off, Generation Quality, and Cross-Architecture Comparisons
SAeUron achieves stronger style-unlearning performance, it exhibits a less balanced erase-retain-off in object unlearning, achieving only 78.82% UA despite 95.53% RA.
Reporting unlearning accuracy and retention accuracy separately can hide SurgUn's balanced gains. We therefore report their harmonic mean (HM) in Table 3, which penalizes strong performance on one objective at the expense of the other. SurgUn achieves the highest object HM (92.53%) and average style–object HM (92.97%) among compared methods, with competitive style-unlearning performance. We have also included an HM comparison plot in Figure 1 of the main paper.
FID should be considered alongside erasure and retention, not as an independent measure of unlearning effectiveness. Although MiM-MU and CA have lower FID than SurgUn (Table 3), their average HM scores are substantially lower. Thus, lower FID alone does not establish superior performance; erasure, retention, and generation quality must be evaluated jointly.
Our primary comparisons use SD v1.5, where established benchmarks and baseline implementations are available. We additionally evaluate SurgUn against baselines on SD3.5 (Table 4) with results presented in [2]. SurgUn achieves the highest style (80.76%) and object (87.32%) HM scores among compared methods, supporting generalization to newer diffusion transformer architectures.
Table 3: Comparison of harmonic mean (HM) scores across style and object unlearning on SD v1.5. HM measures the balance between UA and RA.
| Method | Style HM ↑ | Object HM ↑ | Average HM ↑ | FID ↓ |
|---|---|---|---|---|
| ESD | 92.69 | 64.83 | 78.76 | 65.55 |
| FMN | 65.25 | 58.65 | 61.95 | 131.37 |
| UCE | 69.70 | 53.16 | 61.43 | 182.01 |
| CA | 73.96 | 60.52 | 67.24 | 54.21 |
| SalUn | 89.38 | 92.11 | 90.75 | 61.05 |
| SEOT | 69.57 | 36.88 | 53.23 | 62.38 |
| SPM | 72.13 | 78.02 | 75.08 | 59.79 |
| EDiff | 89.32 | 78.21 | 83.77 | 81.42 |
| SHS | 75.18 | 77.53 | 76.36 | 119.34 |
| SAeUron | 97.49 | 86.37 | 91.93 | 62.15 |
| TRUST | 82.16 | 75.54 | 78.85 | 55.33 |
| SAFREE | 66.90 | 65.90 | 66.40 | 56.77 |
| MiM-MU | 86.33 | 87.11 | 86.72 | 49.14 |
| SurgUn (Ours) | 93.41 | 92.53 | 92.97 | 63.55 |
Table 4: Style and object unlearning performance on SD3.5
| Method | Style UA ↑ | Style IRA ↑ | Style CRA ↑ | Style HM ↑ | Object UA ↑ | Object IRA ↑ | Object CRA ↑ | Object HM ↑ |
|---|---|---|---|---|---|---|---|---|
| EraseAnything | 62.54 | 64.49 | 87.32 | 68.58 | 65.32 | 88.21 | 64.36 | 70.38 |
| SAeUron | 66.54 | 62.41 | 87.47 | 70.49 | 64.45 | 90.11 | 62.31 | 69.84 |
| AGE | 60.18 | 57.09 | 81.47 | 64.41 | 60.75 | 78.96 | 57.43 | 64.26 |
| ConceptPrune | 52.19 | 50.94 | 76.42 | 57.37 | 57.81 | 72.34 | 48.32 | 59.04 |
| GLoCE | 58.28 | 55.32 | 84.88 | 63.65 | 61.78 | 84.51 | 59.34 | 66.47 |
| AdaVD | 63.71 | 60.11 | 85.41 | 67.93 | 61.44 | 86.61 | 60.49 | 66.95 |
| BLOCK | 69.60 | 67.30 | 92.60 | 74.42 | 68.00 | 93.00 | 67.50 | 73.62 |
| SurgUn | 78.21 | 72.76 | 94.22 | 80.76 | 92.33 | 89.21 | 76.44 | 87.32 |
[W4] Comprehensive Computational Cost Analysis and Amortization of Localization Overhead
We extend our computational cost analysis to include one-time localization and checkpoint calibration costs, alongside per-concept unlearning time, as summarized in Table 5 for SD v1.5 object unlearning.
Table 5: Comparison of unlearning effectiveness and computational efficiency for object unlearning on SD v1.5.
| Method | UA ↑ | RA ↑ | HM ↑ | One-time Setup / Localization ↓ | One-Concept Training ↓ | GPU Memory ↓ |
|---|---|---|---|---|---|---|
| SalUn | 86.91 | 97.97 | 92.11 | — | 11.1 min | 23.40 GB |
| SAeUron | 78.82 | 95.53 | 86.37 | 27 h (SAE training) | 94.03 s | 7.80 GB |
| SurgUn (Ours) | 92.36 | 92.70 | 92.53 | 7 h (Localization and CC) | 150 s | 4.02 GB |
SurgUn achieves the highest object-unlearning harmonic mean (92.53%) among the compared SD v1.5 methods, with a GPU memory footprint of 4.02 GB. Although localization incurs an initial computational cost, it is performed once using representative concepts, and the selected blocks are reused across concepts within the same category, as shown in Table 9. Similarly, the fixed distractor pool is shared across concepts, avoiding repeated generation. Thus, one-time preparation costs are amortized across multiple unlearning tasks, making SurgUn computationally efficient for repeated concept removal.
Reference
[1] Riemer, Matthew, et al. "Learning to learn without forgetting by maximizing transfer and minimizing interference." arXiv preprint arXiv:1810.11910 (2018).
[2] Kumar, Aditya, et al. "Concept Removal for Frontier Image Generative Models." arXiv preprint arXiv:2606.25548 (2026).
Response to Reviewer wWHF-1
We thank the reviewer for recognizing our block-wise analysis of the erase–retain trade-off, the breadth of our evaluation across architectures, and the efficiency of our single-block LoRA-based design. We address the reviewer's concerns and provide additional clarifications and experimental evidence below.
[W1] Methodological Novelty and Disentangling the Contributions of Interference and Localization
We provide comparison of SurgUn's objective with existing unlearning methods in Table 1 below, highlighting the distinctions in how target suppression, distractor conditioning, and localized updates are formulated.
To examine whether SurgUn induces competition between target suppression and distractor reconstruction, we analyze their gradient interactions during training. Following [1], we use cosine similarity to measure gradient alignment, as it captures the relationship between gradient directions independently of their magnitudes. We compute the intended update directions for target suppression and distractor reconstruction with respect to the trainable LoRA parameters. Negative cosine similarity indicates conflicting update directions, whereas positive similarity indicates alignment. We observe a transition from weak alignment to negative similarity as training progresses. This provides gradient-level evidence consistent with the interference mechanism proposed in SurgUn.
Furthermore, we respectfully clarify that not all baselines employ global parameter updates. As discussed in the reviewer's own observation, SalUn uses gradient-saliency-based parameter localization, while SAeUron identifies intervention features through sparse autoencoder activations.
More importantly, Figure 13 (Appendix E) directly examines the contribution of our objective by comparing the complete distractor-conditioned loss against target-only and non-logarithmic variants across candidate attention blocks. The superior forgetting–retention balance achieved by the complete objective demonstrates that the gains are not solely attributable to localization.
[W2] Expanded Adversarial Robustness Evaluation
For adversarial robustness, we extend the comparison to include SalUn, SAFREE, SAeUron, and TRUST (CIP and CSR) across P4D, Ring-A-Bell, and UnlearnDiffAtk. The results are summarized in Table 2 below.
SurgUn achieves the second-lowest ASR on P4D (0.50%) and UnlearnDiffAtk (3.80%), while attaining the joint-lowest ASR on Ring-A-Bell (0.00%) alongside SalUn. Although TRUST achieves lower ASR on P4D and UnlearnDiffAtk, SurgUn remains competitive in adversarial robustness while achieving stronger performance across other unlearning tasks. In particular, SurgUn achieves higher style and object unlearning HM scores (93.41%/92.53% vs. 82.16%/75.54%), highlighting its effectiveness in balancing target suppression and concept preservation.
Table 2: Adversarial robustness against nudity concept recovery. Comparison of unlearning methods under P4D, Ring-A-Bell, and UnlearnDiffAtk attacks.