acceptodds
Withdrawn from ICLR 2027

Forgetting is Competition: Rethinking Unlearning as Representation Interference in Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Deployed text-to-image diffusion models increasingly require post-hoc concept unlearning for copyright claims, artist opt-outs, safety updates, and protected-content mitigation without full retraining. A central challenge is erase–retain imbalance: aggressive updates suppress targets but damage shared capabilities, while conservative or anchor-based updates preserve quality yet leave concepts recoverable through related, compositional, paraphrased, or adversarial prompts. Inspired by retroactive interference, we propose SurgUn, which treats forgetting as controlled competition rather than direct deletion or one-to-one reassignment. SurgUn instantiates retroactive concept interference via distractor-conditioned gradient competition: target-gradient ascent weakens target-conditioned denoising or flow-matching behavior, while descent over a semantically diverse distractor set introduces competing non-target trajectories under the same prompt context. This redistributes outputs across multiple non-target modes instead of collapsing to a single proxy. To limit collateral forgetting through shared pathways, SurgUn adds pixel-grounded weight-space localization, a lightweight diagnostic that selects attention blocks by generated-image erase–retain behavior, exploiting the asymmetry that suppression is broadly achievable whereas retention is block-selective. Across UnlearnCanvas, IP-character erasure, Holistic Unlearning, EraseBench, and Ring-A-Bell on Stable Diffusion v1.5, SDXL, and SANA-1.5, SurgUn achieves a stronger erase–retain balance than baselines. Ablations show that diverse distractors, contrastive competition, and localization are all necessary for robust suppression while preserving related and unrelated concepts.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, Retroactive Interference, Adversarial Robustness, Concept Localization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wWHFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes SurgUn, a post-hoc concept unlearning method for text-to-image diffusion models that targets the erase–retain trade-off. First, under the target prompt, it pushes the model away from reconstructing target images while training it to reconstruct a fixed, diverse set of 100 distractor images, so that outputs spread across many non-target modes rather than collapsing to one anchor. Second, it trains LoRA on each candidate attention block separately and selects the block with the best unlearning–retention trade-off for all subsequent updates. This is based on the observation that suppression is achievable from most blocks, while retention depends strongly on which block is updated. Experiments on five benchmarks and three backbones show a better erase–retain balance than prior methods.

Critical strengths
  1. An insightful block-wise empirical observation. The paper offers an interesting finding at the level of individual blocks: training each attention block on its own, almost every block can suppress the target concept well, but retention of non-target concepts changes a lot from block to block. This suggests that where an unlearning update is placed should be decided by its effect on retention, not only by how sensitive a location is to the target. That is a useful and practical insight for future localization-based unlearning methods.

  2. A comprehensive and diverse evaluation. The method is tested across a wide range of challenging settings: object and style unlearning, IP-character erasure, compositional unlearning, sequential and hierarchical unlearning, and adversarial prompt recovery. It is also tested on both U-Net and flow-matching transformer backbones. This breadth gives a fairly complete picture of how the method behaves beyond the direct target prompt.

  3. A lightweight and efficient design. All updates are restricted to a LoRA module on one selected block, so the per-concept checkpoint is very small and training stays cheap even on larger backbones such as SDXL and SANA-1.5.

Critical weaknesses
  1. Limited novelty and unclear attribution of gains. The method largely combines existing ingredients: gradient ascent on the target (ESD, EraseDiff), non-target anchoring against over-erasure (Concept Ablation, here with a fixed pool of 100 anchors), and localized updates (SalUn). The retroactive-interference framing is, as the authors acknowledge, an analogy rather than a mechanism. More importantly, all baselines use global updates, while SurgUn updates only a carefully selected block, so it is unclear whether the gains come from the proposed objective or simply from localization.

  2. Insufficient robustness evaluation and an unsupported dispersion claim. Robustness is tested only with Ring-A-Bell and paraphrased prompts. Stronger recovery attacks are missing, such as UnlearnDiffAtk, P4D and embedding-level attacks based on textual inversion (e.g., CCE), which are the standard test of whether a concept is truly removed from the weights.

  3. Overstated superiority and an uneven comparison. In Table 1, SAeUron outperforms SurgUn on all three style metrics (95.8/99.1/99.4 vs. 93.8/95.3/90.8), and SurgUn's FID (63.55) is only mid-range (e.g., MiM-MU 49.14, CA 54.21). Moreover, baselines are reported only on SD v1.5, while SDXL and SANA-1.5 results (Tables 1 and 2) cover SurgUn alone, so its advantage on these backbones cannot be verified.

  4. Incomplete accounting of computational cost. The reported runtime (e.g., 150 s on SD v1.5) seems to cover only the final unlearning run. It excludes per-block localization training and evaluation, generating the 100 distractor images, and checkpoint calibration. The amortization argument is validated only for objects and styles, and Table 9 appears to list setting-specific blocks, which suggests that localization must be rerun for new settings. A full end-to-end cost comparison is needed.

Questions for authors

Please see Critical Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer wWHF-1

We thank the reviewer for recognizing our block-wise analysis of the erase–retain trade-off, the breadth of our evaluation across architectures, and the efficiency of our single-block LoRA-based design. We address the reviewer's concerns and provide additional clarifications and experimental evidence below.

[W1] Methodological Novelty and Disentangling the Contributions of Interference and Localization

We provide comparison of SurgUn's objective with existing unlearning methods in Table 1 below, highlighting the distinctions in how target suppression, distractor conditioning, and localized updates are formulated.

ComponentExisting MethodsSurgUn
Gradient ascent on target (ESD, EraseDiff)Suppresses target concepts through erasure-oriented optimization objectives.SurgUn does not treat ascent and descent as independent objectives. Instead, it jointly optimizes target suppression and distractor reconstruction under the same target-conditioning prompt through a unified log-ratio loss, inducing direct competition between target and distractor trajectories.
Non-target anchoring against over-erasure (Concept Ablation)Redirects the target representation toward a predefined anchor concept using distinct target and anchor conditions.SurgUn redistributes target-conditioned outputs across diverse non-target concepts without prescribing a replacement through competition. Figure 37 demonstrates this redistribution, with outputs dispersed across multiple classes rather than concentrated on a few substitutes, supporting more robust unlearning.
Localized updates (SalUn)Existing approaches rely on gradient-based saliency or latent-space activations to identify intervention regions. These proxy signals do not directly measure the impact of interventions on generated images, potentially causing unintended degradation of related and unrelated concepts.SurgUn employs pixel-grounded, retention-aware localization, selecting attention blocks based on their actual forgetting–retention performance in generated images. This enables effective target suppression while better preserving non-target concepts (Figures 4 and 11).

To examine whether SurgUn induces competition between target suppression and distractor reconstruction, we analyze their gradient interactions during training. Following [1], we use cosine similarity to measure gradient alignment, as it captures the relationship between gradient directions independently of their magnitudes. We compute the intended update directions for target suppression and distractor reconstruction with respect to the trainable LoRA parameters. Negative cosine similarity indicates conflicting update directions, whereas positive similarity indicates alignment. We observe a transition from weak alignment to negative similarity as training progresses. This provides gradient-level evidence consistent with the interference mechanism proposed in SurgUn.

Furthermore, we respectfully clarify that not all baselines employ global parameter updates. As discussed in the reviewer's own observation, SalUn uses gradient-saliency-based parameter localization, while SAeUron identifies intervention features through sparse autoencoder activations.

More importantly, Figure 13 (Appendix E) directly examines the contribution of our objective by comparing the complete distractor-conditioned loss against target-only and non-logarithmic variants across candidate attention blocks. The superior forgetting–retention balance achieved by the complete objective demonstrates that the gains are not solely attributable to localization.


[W2] Expanded Adversarial Robustness Evaluation

For adversarial robustness, we extend the comparison to include SalUn, SAFREE, SAeUron, and TRUST (CIP and CSR) across P4D, Ring-A-Bell, and UnlearnDiffAtk. The results are summarized in Table 2 below.

SurgUn achieves the second-lowest ASR on P4D (0.50%) and UnlearnDiffAtk (3.80%), while attaining the joint-lowest ASR on Ring-A-Bell (0.00%) alongside SalUn. Although TRUST achieves lower ASR on P4D and UnlearnDiffAtk, SurgUn remains competitive in adversarial robustness while achieving stronger performance across other unlearning tasks. In particular, SurgUn achieves higher style and object unlearning HM scores (93.41%/92.53% vs. 82.16%/75.54%), highlighting its effectiveness in balancing target suppression and concept preservation.

Table 2: Adversarial robustness against nudity concept recovery. Comparison of unlearning methods under P4D, Ring-A-Bell, and UnlearnDiffAtk attacks.

MethodP4D ↓Ring-A-Bell ↓UnlearnDiffAtk ↓
SalUn1.300.0011.27
SAFREE38.4011.4028.20
SAeUron3.802.8919.70
TRUST0.270.831.18
SurgUn (SD v1.5)0.500.003.80
AuthorsOpenReview ↗

Response to Reviewer wWHF-2

[W3] Clarifying the Erase–Retain Trade-off, Generation Quality, and Cross-Architecture Comparisons

SAeUron achieves stronger style-unlearning performance, it exhibits a less balanced erase-retain-off in object unlearning, achieving only 78.82% UA despite 95.53% RA.

Reporting unlearning accuracy and retention accuracy separately can hide SurgUn's balanced gains. We therefore report their harmonic mean (HM) in Table 3, which penalizes strong performance on one objective at the expense of the other. SurgUn achieves the highest object HM (92.53%) and average style–object HM (92.97%) among compared methods, with competitive style-unlearning performance. We have also included an HM comparison plot in Figure 1 of the main paper.

FID should be considered alongside erasure and retention, not as an independent measure of unlearning effectiveness. Although MiM-MU and CA have lower FID than SurgUn (Table 3), their average HM scores are substantially lower. Thus, lower FID alone does not establish superior performance; erasure, retention, and generation quality must be evaluated jointly.

Our primary comparisons use SD v1.5, where established benchmarks and baseline implementations are available. We additionally evaluate SurgUn against baselines on SD3.5 (Table 4) with results presented in [2]. SurgUn achieves the highest style (80.76%) and object (87.32%) HM scores among compared methods, supporting generalization to newer diffusion transformer architectures.

Table 3: Comparison of harmonic mean (HM) scores across style and object unlearning on SD v1.5. HM measures the balance between UA and RA.

MethodStyle HM ↑Object HM ↑Average HM ↑FID ↓
ESD92.6964.8378.7665.55
FMN65.2558.6561.95131.37
UCE69.7053.1661.43182.01
CA73.9660.5267.2454.21
SalUn89.3892.1190.7561.05
SEOT69.5736.8853.2362.38
SPM72.1378.0275.0859.79
EDiff89.3278.2183.7781.42
SHS75.1877.5376.36119.34
SAeUron97.4986.3791.9362.15
TRUST82.1675.5478.8555.33
SAFREE66.9065.9066.4056.77
MiM-MU86.3387.1186.7249.14
SurgUn (Ours)93.4192.5392.9763.55

Table 4: Style and object unlearning performance on SD3.5

MethodStyle UA ↑Style IRA ↑Style CRA ↑Style HM ↑Object UA ↑Object IRA ↑Object CRA ↑Object HM ↑
EraseAnything62.5464.4987.3268.5865.3288.2164.3670.38
SAeUron66.5462.4187.4770.4964.4590.1162.3169.84
AGE60.1857.0981.4764.4160.7578.9657.4364.26
ConceptPrune52.1950.9476.4257.3757.8172.3448.3259.04
GLoCE58.2855.3284.8863.6561.7884.5159.3466.47
AdaVD63.7160.1185.4167.9361.4486.6160.4966.95
BLOCK69.6067.3092.6074.4268.0093.0067.5073.62
SurgUn78.2172.7694.2280.7692.3389.2176.4487.32

[W4] Comprehensive Computational Cost Analysis and Amortization of Localization Overhead

We extend our computational cost analysis to include one-time localization and checkpoint calibration costs, alongside per-concept unlearning time, as summarized in Table 5 for SD v1.5 object unlearning.

Table 5: Comparison of unlearning effectiveness and computational efficiency for object unlearning on SD v1.5.

MethodUA ↑RA ↑HM ↑One-time Setup / Localization ↓One-Concept Training ↓GPU Memory ↓
SalUn86.9197.9792.11—11.1 min23.40 GB
SAeUron78.8295.5386.3727 h (SAE training)94.03 s7.80 GB
SurgUn (Ours)92.3692.7092.537 h (Localization and CC)150 s4.02 GB

SurgUn achieves the highest object-unlearning harmonic mean (92.53%) among the compared SD v1.5 methods, with a GPU memory footprint of 4.02 GB. Although localization incurs an initial computational cost, it is performed once using representative concepts, and the selected blocks are reused across concepts within the same category, as shown in Table 9. Similarly, the fixed distractor pool is shared across concepts, avoiding repeated generation. Thus, one-time preparation costs are amortized across multiple unlearning tasks, making SurgUn computationally efficient for repeated concept removal.


Reference

[1] Riemer, Matthew, et al. "Learning to learn without forgetting by maximizing transfer and minimizing interference." arXiv preprint arXiv:1810.11910 (2018).

[2] Kumar, Aditya, et al. "Concept Removal for Frontier Image Generative Models." arXiv preprint arXiv:2606.25548 (2026).

Reviewer_88acOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes SurgUn, a concept unlearning method inspired by retroactive interference that treats forgetting as controlled competition rather than direct concept deletion. SurgUn combines gradient ascent on the target concept with gradient descent on semantically diverse distractors to suppress target generation. Additionally, it introduces pixel-grounded weight-space localization to selectively update blocks that provide a favorable balance between concept erasure and non-target preservation.

Critical strengths
  • The paper demonstrates that target suppression can be achieved across many attention blocks, whereas non-target preservation is more block-selective. This provides empirical insights into how the choice of intervention site affects the erasure-preservation trade-off.

  • Extensive evaluation across models and unlearning settings. The paper evaluates SurgUn on SD v1.5, SDXL, and SANA-1.5, covering object, style, and IP-character erasure, as well as compositional, sequential, and hierarchical unlearning.

Critical weaknesses

[W1] Unclear methodological distinction of weight-space localization. The proposed localization procedure appears to independently fine-tune candidate blocks and select the one that achieves the best balance between UA and RA. Although the paper analyzes the effects of interventions across different blocks, it remains unclear how this procedure differs from conventional evaluation-driven hyperparameter search or model selection, and what additional methodological benefits it provides.

[W2] Insufficient baseline comparisons in several major experimental settings. While the paper compares against a wide range of methods for object and style unlearning, the comparisons for nudity erasure and sequential unlearning are relatively limited. In particular, the original papers introducing SAeUron and TRUST report competitive performance in harmful concept erasure and settings involving multiple concept removals, yet these methods are not included in the corresponding evaluations.

[W3] Limited evaluation of adversarial robustness. Adversarial robustness is an important aspect of concept erasure, but the evaluation is limited to Ring-A-Bell. Additional evaluations using MMA-Diffusion[1] and UnlearnDiffAtk[2], which are commonly used to evaluate concept erasure methods, would help further validate the robustness of the proposed method under diverse adversarial settings.


[1] Yang, Yijun, et al. "Mma-diffusion: Multimodal attack on diffusion models." CVPR, 2024.

[2] Zhang, Yimeng, et al. "To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images... for now." ECCV, 2024.

Questions for authors
  1. Do the optimal blocks and checkpoints selected based on UA and RA during block localization and checkpoint calibration also exhibit strong preservation performance on independent benchmarks compared with other candidate configurations?

  2. In some qualitative results (e.g., Figures 24-26), the generated images after target erasure appear noisy or lack meaningful visual structure. Is this behavior intentional? In particular, when a prompt contains both target and non-target concepts, does the proposed method have limitations in selectively removing the target while preserving the remaining concepts?

  3. How are the validation sets constructed for block localization and checkpoint calibration in the nudity and violence erasure settings?

  4. SurgUn selects an optimal checkpoint for each target concept. Does SurgUn still maintain its performance advantage when the existing training-based baselines are given the same or a comparable checkpoint selection procedure?

  5. Tables 1 and 2 show performance differences across model backbones. Are these differences related to model scale or architectural characteristics? Could the authors also evaluate the proposed method on a larger and widely used model such as FLUX.1-dev?

  6. Block localization selects an optimal block for each concept category. How would this procedure be applied when simultaneously erasing concepts from different categories, such as nudity and artistic styles?

  7. In sequential unlearning, how does performance change as the number of erased concepts increases? Is there a scale at which the method begins to degrade substantially? Furthermore, can the proposed method effectively handle simultaneous multi-concept erasure, rather than sequentially removing concepts?

  8. Figure 14 analyzes the stability of block rankings across checkpoints on SDXL. Are similar trends observed on SD v1.5 and SANA-1.5?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 88ac -1

We thank the reviewer for acknowledging our finding that target suppression is achievable across multiple attention blocks, while non-target preservation depends on the choice of intervention site, along with our evaluation across unlearning settings. We provide detailed responses to the reviewer's concerns and questions below.

[W1]: Methodological distinction of weight-space localization.

Existing localization approaches, such as SalUn and SAeUron, rely on gradient-based saliency or latent-space activations to identify intervention regions. However, these proxy signals do not reflect the actual impact of parameter updates on generated images. This leads to effective target suppression but unintended degradation of related and unrelated concepts.

SurgUn addresses this limitation through pixel-grounded localization, selecting blocks based on the observed erase–retain trade-off in generated images (UA and RA), rather than relying solely on internal representations. As shown in Figures 10–12 (Appendix D), target suppression is achievable across multiple blocks, but retention varies substantially, underscoring the importance of pixel-space evaluation. Unlike conventional hyperparameter search, which selects configurations for individual tasks, SurgUn identifies intervention blocks using representative concepts and reuses them across subsequent unlearning requests within the same category, avoiding repeated per-concept searches.

The balanced gains achieved by SurgUn may not be fully apparent when unlearning accuracy and retention accuracy are considered independently. Thus to assess this balance, we report the harmonic mean (HM) of UA and RA, which penalizes disproportionate performance between forgetting and retention.

As shown in Table 1 below, SurgUn achieves an HM above 90% for both object and style unlearning. Moreover, localization is performed once using representative concepts, and the selected block is reused across the corresponding concept category, avoiding repeated per-concept searches. Together, these results highlight SurgUn's pixel-grounded, transferable, and computationally reusable localization strategy.

Table 1: Forgetting–retention performance on SD v1.5 for style and object unlearning. RA = (IRA + CRA)/2; HM = harmonic mean of UA and RA.

MethodStyle UA ↑Style RA ↑Style HM ↑Object UA ↑Object RA ↑Object HM ↑
SalUn86.2692.7389.3886.9197.9792.11
SAeUron95.8099.2597.4978.8295.5386.37
SurgUn93.7993.0493.4192.3692.7092.53

[W2–W3] Expanded Baseline Comparisons and Adversarial Robustness Evaluation

For nudity erasure, we extend the comparison to include SalUn, SAFREE, SAeUron, and TRUST (CIP and CSR) across P4D, Ring-A-Bell, and UnlearnDiffAtk. The available adversarial robustness results are summarized in Table 2 below.

SurgUn achieves the second-lowest ASR on P4D (0.50%) and UnlearnDiffAtk (3.80%), while attaining the joint-lowest ASR on Ring-A-Bell (0.00%) alongside SalUn. Although TRUST achieves lower ASR on P4D and UnlearnDiffAtk, SurgUn delivers higher style and object unlearning HM scores (93.41%/92.53% vs. 82.16%/75.54%), demonstrating a stronger balance between adversarial robustness, target suppression, and retention.

For sequential unlearning, we additionally include SAeUron alongside SalUn and SurgUn under the six-concept UnlearnCanvas setting. We present the expanded comparison in Table 3 below, highlighting the relative performance of these methods across successive unlearning requests. We will incorporate this comparison into the revised manuscript and discuss the observed differences in forgetting and retention performance.

Table 2: Adversarial robustness against nudity concept recovery. Comparison of unlearning methods under P4D, Ring-A-Bell, and UnlearnDiffAtk attacks

MethodP4D ↓Ring-A-Bell ↓UnlearnDiffAtk ↓
SalUn1.300.0011.27
SAFREE38.4011.4028.20
SAeUron3.802.8919.70
TRUST0.270.831.18
SurgUn (SD v1.5)0.500.003.80

Table 3: Sequential unlearning performance after six concept-removal requests. Comparison of Unlearning Accuracy (UA, %, ↑) and Retain Accuracy (RA, %, ↑), averaged across the six target concepts after the final unlearning request.

MethodUA ↑RA ↑
SalUn77.1760.19
SAeUron98.0099.00
SurgUn92.5085.00
AuthorsOpenReview ↗

Response to Reviewer 88ac -2

[Q1] Generalization of Selected Blocks and Checkpoints to Independent Benchmarks

The effectiveness of the selected blocks and checkpoints is further supported by evaluations beyond the localization benchmark. Specifically, Figure 28 (Appendix H.2) demonstrates effective IP-character unlearning using the object-localized configuration while preserving unrelated characters. Similarly, Figure 29 (Appendix H.2) shows that unlearning an artist's style also suppresses the model's ability to recall the artist's associated artworks, even though these artworks were never targeted during training. These results provide additional evidence that the selected configurations generalize beyond the localization benchmark.


[Q2] Selective Target Suppression and Preservation in Compositional Generation

In Figures 24–26, the prompts primarily focus on the unlearning target without specifying additional concrete objects, which explains the lack of meaningful visual structure following target suppression.

To investigate selective unlearning, we conducted additional experiments on SD v1.5, SDXL, SANA 1.5, and SD3.5. Specifically, we used compositional prompts containing Dog (Target) alongside non-target objects, generated 10 images per model, and averaged the target and non-target CLIP similarity scores before and after unlearning.

As shown in Table 4 below, target-concept alignment decreases consistently across all four architectures. Importantly, SDXL, SANA 1.5, and SD3.5 largely preserve non-target alignment, demonstrating selective target suppression while retaining other objects. In contrast, SD v1.5 shows a greater reduction in non-target alignment, which may reflect differences in model capacity and architecture. We will include these results and discuss the observed differences across backbones in the revised manuscript.

Table 4: Selective concept unlearning on compositional prompts across diffusion architectures.

ModelTarget CLIP (Before/After)Non-target CLIP (Before/After)
SD v1.533.09 / 25.1134.71 / 23.88
SDXL32.76 / 25.6533.44 / 30.55
SANA 1.533.20 / 22.8632.19 / 31.45
SD3.534.77 / 23.8034.60 / 32.79

[Q3] Validation set for nudity and violence

For both nudity and violence erasure, we construct the validation sets by selecting 10 prompts for each category from the I2P (Inappropriate Image Prompts) dataset. These prompts are used consistently during both block localization and checkpoint calibration to evaluate target-concept suppression and select the appropriate block and checkpoint configurations.


[Q4] Fair Comparison with Baselines under Checkpoint Calibration

We clarify that most existing unlearning methods also employ hyperparameter tuning to optimize their performance, and our baseline results are taken directly from the UnlearnCanvas benchmark, following its reported evaluation protocol.

To further ensure a fair comparison, we evaluated SalUn on three objects and three styles, training for 10 epochs and selecting the best checkpoint using the forgetting–retention criteria. As shown in Table 5 below, SurgUn maintains higher UA for both style and object unlearning, with competitive retention, demonstrating that its gains are not solely due to checkpoint calibration.

We exclude SAeUron from this experiment because it performs concept removal through inference-time sparse-feature interventions rather than iterative concept-specific model fine-tuning, making the same checkpoint-selection procedure inapplicable.

Table 5: Comparison of SalUn and SurgUn with checkpoint calibration on three objects and three styles.

MethodStyle UA ↑Style IRA ↑Style CRA ↑Object UA ↑Object IRA ↑Object CRA ↑
SalUn + CC87.6091.3994.0888.9194.1598.69
SurgUn + CC95.2192.2293.2194.2693.2395.77
AuthorsOpenReview ↗

Response to Reviewer 88ac -2

[Q5] Understanding Cross-Backbone Performance Differences and Generalization to MMDiT Architectures

Performance variations across backbones may reflect architectural differences in attention mechanisms, parameterization, and concept representations, rather than model scale alone.

To further assess generalization, we extend our evaluation to SD3.5 (MMDiT architecture) and compare SurgUn against multiple baselines, including the recent state-of-the-art method BLOCK [1].

As shown in Table 6 below, SurgUn achieves the highest harmonic mean for both style (80.76%) and object (87.32%) unlearning, demonstrating a superior overall forgetting–retention balance. These results further support SurgUn's effectiveness and generalization to newer diffusion transformer architectures.

Table 6: Style and object unlearning performance on SD3.5-Medium. UA denotes unlearning accuracy, IRA and CRA denote in-domain and cross-domain retain accuracy, respectively. HM is the harmonic mean of UA and RA, where RA = (IRA + CRA)/2. Higher values indicate better performance.

MethodStyle UA ↑Style IRA ↑Style CRA ↑Style HM ↑Object UA ↑Object IRA ↑Object CRA ↑Object HM ↑
EraseAnything62.5464.4987.3268.5865.3288.2164.3670.38
SAeUron66.5462.4187.4770.4964.4590.1162.3169.84
AGE60.1857.0981.4764.4160.7578.9657.4364.26
ConceptPrune52.1950.9476.4257.3757.8172.3448.3259.04
GLoCE58.2855.3284.8863.6561.7884.5159.3466.47
AdaVD63.7160.1185.4167.9361.4486.6160.4966.95
BLOCK69.6067.3092.6074.4268.0093.0067.5073.62
SurgUn78.2172.7694.2280.7692.3389.2176.4487.32

[Q6] Extending Block Localization to Simultaneous Unlearning Across Concept Categories

For simultaneous unlearning across different concept categories, such as nudity and artistic styles, we construct representative prompts and images that cover all target categories and perform block localisation using the same pixel-grounded UA/RA evaluation procedure. This allows us to identify the intervention block that provides the best overall forgetting–retention trade-off across the combined targets. We have already demonstrated this approach in compositional object–style unlearning, where the objective is to erase a specific object–style combination while preserving the individual object and style concepts.


[Q7] Scalability and Stability of Sequential and Simultaneous Multi-Concept Unlearning

Table 19 (Appendix) presents the complete sequential unlearning results across six concept-removal requests. For SD v1.5, SurgUn maintains strong target suppression, with average UA of 92.5% after six removals, while RA gradually declines from 92% to 85%, a drop of 7 percentage points. No abrupt degradation is observed within the six-concept evaluation, indicating that SurgUn maintains a relatively stable forgetting–retention balance as the number of erased concepts increases.

We evaluate simultaneous multi-concept unlearning on SD3.5 by jointly removing 10 object concepts. We present the average of UA and RA and compare our results with LOCOEDIT, UCE, and BLOCK, using the multi-concept object-unlearning results reported in [1]. As shown in Table 7, SurgUn achieves an average score of 71.5%, demonstrating its ability to suppress multiple target concepts simultaneously while maintaining moderate retention performance.

Since SurgUn is an interference-based approach, jointly optimizing multiple targets may introduce competing interference signals, making the balance between unlearning and retention more challenging. These findings motivate further investigation into coordinating interference signals for simultaneous multi-concept unlearning.

Table 7: Simultaneous unlearning of 10 object concepts on SD3.5.

MethodAverage Score (%) ↑
LOCOEDIT52.0
UCE73.0
BLOCK74.0
SurgUn71.5
Reviewer_dUcDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes SurgUn, a diffusion concept unlearning method inspired by retroactive interference. The main idea is to suppress a target concept by combining target-gradient ascent with reconstruction training on a diverse set of distractor images under the same target prompt, encouraging the model to redistribute its outputs across non-target concepts. To reduce unintended changes to other concepts, SurgUn selects an attention block based on its forgetting-retention performance and restricts updates to that block. Experiments across multiple benchmarks and diffusion architectures show promising results in target removal, concept preservation, and robustness

Critical strengths

Extensive experimental evaluation

The paper provides a comprehensive evaluation across multiple unlearning settings, including object and style removal, IP-character erasure, compositional unlearning, sequential unlearning, and adversarial robustness. I particularly appreciate the inclusion of related-concept preservation and sequential unlearning, as these are important challenges for practical concept unlearning. The experiments also cover multiple generative architectures, including SD v1.5, SDXL, and SANA-1.5, with promising forgetting-retention results

Critical weaknesses

1. Unclear contribution of the proposed interference mechanism

My main concern is that it is difficult to understand how much of the reported improvement actually comes from the proposed representation interference mechanism. SurgUn combines distractor-conditioned training with weight-space localization and checkpoint calibration, all of which can affect the forgetting-retention trade-off. I appreciate that the authors provide several ablation studies for these components. However, the experiments mostly examine them separately, making it difficult to disentangle their contributions to the final performance

In particular, the paper attributes its improvements to controlled representation interference, but I am not fully convinced that the experiments establish this mechanism as the main reason for the observed gains. A more direct comparison of the interference objective with and without the additional localization and calibration procedures would help clarify the central contribution of the method.

2. Limited evaluation of adversarial robustness

Although the paper evaluates robustness under Ring-A-Bell attacks and reports promising results, I find the adversarial evaluation somewhat limited. Since the proposed method is motivated by interference between concept representations rather than direct removal, I wonder whether the target concept could still be recovered through other attack strategies. The hierarchical and paraphrase evaluations are useful, but relying on a single adversarial attack framework makes it difficult to assess how robust the method is against different ways of accessing the target concept. I would have liked to see a broader evaluation of adversarial robustness, particularly under stronger or adaptive attacks

Questions for authors
  • The paper evaluates sequential unlearning of multiple concepts. How well would SurgUn extend to simultaneous multi-concept unlearning, where several target concepts are removed within a single optimization process? Would the shared distractor pool and localized attention block remain effective as the number of target concepts increases?
1 comment
AuthorsOpenReview ↗

Response to Reviewer dUcD

We thank the reviewer for recognizing the breadth of our experimental evaluation, particularly the inclusion of related-concept preservation and sequential unlearning across multiple diffusion architectures. We address the reviewer's concerns and questions below.

[W1] Disentangling the Contributions of Representation Interference.

We would like to clarify that the contributions of the individual components are already examined in the Appendix of the paper. Figure 13 (Appendix E) compares the proposed interference objective with target-only and non-log-contrastive objectives. Figure 11 (Appendix D) evaluates weight-space localization, including full U-Net optimization. Table 4 (Appendix D) examines the effect of distractor strength, while Table 9 (Appendix D) quantifies the contribution of checkpoint calibration.

To make these contributions more explicit and facilitate a direct comparison, we present a consolidated component-wise ablation table below, highlighting the individual and complementary effects of the interference objective, weight-space localization, and checkpoint calibration on the forgetting–retention trade-off.

The results in Table 1 below establish the contribution of representation interference, which improves UA, IRA, and CRA over the non-interference baseline even without localization. Localization further improves the erase–retain balance, while checkpoint calibration provides additional gains, confirming the complementary roles of all three components.

Table 1: Disentangling the contributions of representation interference, weight-space localization, and checkpoint calibration .

No.ConfigurationUA ↑IRA ↑CRA ↑
1Best non-interference loss (Ltarget) with localization68.2232.4440.16
2Interference loss (Full U-Net)73.3380.0076.67
3Interference loss (Localized, without CC)93.3392.5089.00
4Full SurgUn (Interference + Localization + CC)94.0092.5091.20

[W2] Additional adversarial robustness

We extend our adversarial robustness evaluation to P4D [1] and UnlearnDiffAtk [2], which provide complementary adversarial prompting strategies for recovering erased concepts. We compare SurgUn against recent unlearning approaches, including SalUn, SAFREE, SAeUron, and TRUST.

We report the corresponding attack success rates in the table below. SurgUn achieves the second-lowest ASR on both P4D (0.50%) and UnlearnDiffAtk (3.80%), outperforming SalUn, SAFREE, and SAeUron. While TRUST achieves lower ASR on P4D and UnlearnDiffAtk, SurgUn remains competitive in adversarial robustness and demonstrates stronger performance across other unlearning tasks. In particular, SurgUn achieves higher style and object unlearning HM scores (93.41%/92.53% vs. 82.16%/75.54%), highlighting its effectiveness in balancing target suppression and concept preservation.

Table 2: Adversarial robustness comparison for nudity concept unlearning on SD v1.5.

MethodP4D ASR ↓ (%)UnlearnDiffAtk ASR ↓ (%)
SalUn1.3011.27
SAFREE38.4028.20
SAeUron3.8019.70
TRUST0.271.18
SurgUn0.503.80

[Q1] Generalization to Simultaneous Multi-Concept Unlearning and Scalability

To further assess the generalizability of SurgUn, we evaluate simultaneous multi-concept unlearning on SD3.5 by jointly removing 10 object concepts. We report average of UA and RA and compare our results with LOCOEDIT, UCE, and BLOCK [3], using the multi-concept object-unlearning results reported in [3].

As shown in Table 3 below, SurgUn achieves an average score of 71.5%, demonstrating its ability to suppress multiple target concepts simultaneously while maintaining moderate retention performance. Since SurgUn is an interference-based approach, jointly optimizing multiple targets may introduce competing interference signals, making the balance between unlearning and retention more challenging. These findings motivate further investigation into coordinating interference signals for simultaneous multi-concept unlearning.

Table 3: Simultaneous unlearning of 10 object concepts on SD3.5.

MethodAverage Score (%) ↑
LOCOEDIT52.0
UCE73.0
BLOCK74.0
SurgUn71.5

References

[1] Zhi-Yi Chin, Chieh-Ming Jiang, Ching-Chun Huang, Pin-Yu Chen. Prompting4debugging: red-teaming text-to-image diffusion models by finding problematic prompts. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.

[2] Yimeng Zhang, Jinghan Jia, Xin Chen, Aochuan Chen, Yihua Zhang, Jiancheng Liu, Ke Ding, and Sijia Liu. To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images ... for now. In Computer Vision – ECCV 2024

[3] Kumar, Aditya, et al. "Concept Removal for Frontier Image Generative Models." arXiv preprint arXiv:2606.25548 (2026).

Comments on the paper

1 comment

We thank all reviewers for their constructive feedback. We appreciate their recognition of our extensive evaluation across unlearning settings and diffusion architectures (dUcD, 88ac, wWHF), the finding that target suppression is achievable across multiple attention blocks while retention depends on the intervention site (88ac, wWHF), and the efficiency of our single-block LoRA-based design (wWHF).

Across our detailed responses, we address the reviewers' concerns and questions. The main additions and clarifications are:

  • Clarifying methodological contributions. We provide a component-wise ablation separating the contributions of representation interference, weight-space localization, and checkpoint calibration. We also clarify how SurgUn differs from existing erase–retain objectives and localization approaches. [Table 5, Appendix, Page 23]

  • Additional evidence for representation interference and localization. We analyze gradient alignment between target suppression and distractor reconstruction, clarify the motivation for pixel-grounded localization, and provide further evidence of preservation across concepts and architectures. [Gradient Interference box, Page 4]

  • Expanded baseline comparisons and robustness evaluation. We extend adversarial evaluation to P4D and UnlearnDiffAtk and compare against recent methods, including TRUST, SAFREE, and SAeUron. [Table 3, Page 9]

  • Generalization to newer architectures and multi-concept unlearning. We extend SurgUn to SD3.5-Medium (MMDiT) and compare against recent baselines, including BLOCK, demonstrating its applicability beyond the original backbones. We also evaluate selective preservation under compositional prompts, sequential unlearning, and simultaneous removal of 10 object concepts, while discussing the challenges of balancing suppression and retention across multiple targets. [Table 2, Page 9]

  • Computational cost and experimental clarifications. We provide a checkpoint-selection comparison, separate one-time localization costs from per-concept unlearning time, clarify how selected blocks and distractor pools are reused, and discuss the trade-offs among unlearning accuracy, retention, and generation quality. [Table 1, Page 8]

Overall, the balanced gains achieved by SurgUn may not be fully apparent when unlearning accuracy and retention accuracy are considered independently. We therefore report their harmonic mean (HM), which provides a single measure of the erase–retain trade-off and penalizes disproportionate performance across the two objectives. We present the results in Figure 1 of the main paper.

On SD v1.5, SurgUn achieves the highest object-unlearning HM (92.53%) and the highest average style–object HM (92.97%) among the compared methods, with competitive style-unlearning performance.

On SD3.5, SurgUn achieves the highest HM for both style (80.76%) and object (87.32%) unlearning among the evaluated baselines, supporting its generalization to newer architectures.

SurgUn also demonstrates competitive adversarial robustness, achieving the second-lowest attack success rates on P4D and UnlearnDiffAtk and the joint-lowest on Ring-A-Bell in the reported main-paper comparison.

Together, these results and additional analyses strengthen the evidence for SurgUn's effectiveness in balancing target suppression, concept preservation, and adversarial robustness. We welcome further questions and would be happy to provide additional clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.