- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper propose Erase-while-Sparsify (EwS), a method that targets a deployment setting in which a T2I model must meet a parameter budget and a safe content at the same time. EwS starts from dense SD 1.4 and lowers density gradually over a single fine-tuning run while also optimizing an erasure loss. The authors show that the proposed joint, gradual schedule beats sequential alternatives at the same sparsity budget.
Critical strengths
- Deployment of generative models that are both tiny and safe is a well-motivated problem
- Overall, the proposed approach is very interesting and the paper well-written
Critical weaknesses
My main concerns regard the evaluation setting:
- It's been a while now since the first unlearning papers came out, and several benchmarks were introduced (unlearncanvas, hub etc). Authors should target those for style/object erasure instead of selecting their own concepts. UnlearnDiffAtk should be also included. Authors should also include comparison with SalUn and more recent SAE-based approaches like SAeUron and SAEmnesia
- It's not clear why sequential pipelines are fine-tuned with also concepts to erase in the recovery phase, while this does not happen with EwS. So I think the comparisons are not fair
Questions for authors
- Can you rerun the sequential baselines with a recovery set that excludes the target concepts (or that includes the erasure objective)?
- Can you include some efficiency measures such as latency, peak memory and storage benefits?
- Can the method be applied to models other than SD 1.4?
4 comments
Response to Reviewer ceuB (2/2)
W1-b. UnlearnDiffAtk
We evaluated UnlearnDiffAtk for explicit-content erasure using 50 fixed prompts and the toolkit's default settings.
| Method | ASR ↓ | ASR excluding pre-attack detections ↓ | FID ↓ |
|---|---|---|---|
| UCE | 100% | 100% | 19.68 |
| ESD | 96% | 94% | 17.88 |
| Shirkavand et al. | 94% | 90% | 18.09 |
| RECE | 92% | 89% | 18.24 |
| EwS | 84% | 79% | 17.58 |
| ConceptPrune | 64% | 62% | 20.89 |
Among methods with FID close to the original model, EwS has the lowest attack success rate. Its differences from ESD, Shirkavand et al., and RECE are not statistically significant at n = 50; it is significantly more robust than UCE (McNemar p = 0.008). ConceptPrune is more robust (p = 0.031) but incurs a larger quality cost (FID 20.89; art 0.040).
W1-c. Standard benchmarks (UnlearnCanvas, HUB) and SAE-based methods
Concept-set provenance. We adopt concept sets and evaluation prompts from prior erasure studies and report all concepts in those sets.
- Art: all 10 artists introduced by ESD and reused by UCE, with their evaluation prompts. These comprise 5 famous artists (Van Gogh, Picasso, Rembrandt, Warhol, Caravaggio) and 5 modern artists (McKernan, Kinkade, Edlin, Kilian Eng, Ajin: Demi-Human). Van Gogh is a commonly studied representative target.
- Objects: all 10 Imagenette classes, the same set used for ESD object erasure.
- Explicit content: the full I2P set (4,703 prompts), Ring-A-Bell, and COCO-10k, as in AdvUnlearn.
EwS has no concept-specific components. Prompts define the erasure and preservation sets, and the same procedure applies across concepts.
UnlearnCanvas and HUB. We agree with the motivation for standardized evaluation in UnlearnCanvas and HUB. Our central comparison concerns erasure and preservation under the same network-wide sparse budget. Extending this comparison to UnlearnCanvas, for example, requires constructing a common sparse model from its base model and reevaluating the methods under matched budgets. In this revision, we retain the existing evaluation settings and strengthen the evaluation with target-excluded recovery reruns, UnlearnDiffAtk, and SalUn comparisons.
SAE-based methods (SAeUron and SAEmnesia). SAeUron and SAEmnesia intervene on activations during generation using trained SAEs. EwS directly modifies base-model weights to achieve erasure and a network-wide sparse budget jointly. Our quantitative comparisons focus on weight-modification approaches. Evaluation of SAE-based methods under the same deployment budget requires validation of their intervention effects on the sparse base, accounting for the SAE's additional parameters and execution cost. We cite both methods in the revised related-work section and explain the differences between the approaches.
Q2. Efficiency metrics: latency, peak memory, and storage
Please see General Response 2. On an A100, latency (1.62 s) and peak memory (4.54 GB) match SD 1.4 because the sparsity is unstructured and execution uses standard dense kernels. With zstd, the EwS storage cost is 1.04 GB at d = 0.75 and 0.69 GB at d = 0.5, compared with 1.35 GB for dense SD 1.4.
Q3. Models beyond SD 1.4
Yes. EwS operates on SDXL at d = 0.8 and 0.6 (Table 27). The new SD3-medium results (General Response 1) extend this to a DiT-based model: EwS achieves the strongest target erasure among the compared methods while retaining other artists at 96% of the original . We added the main results to Section 5.4 (Table 4), with full settings and results in Appendix D.6 (Table 28).
Dear authors,
I truly appreciated your response and the additional experiments, which have strengthened the paper. However, I have decided to maintain my original score. Given how rapidly the literature on unlearning and concept erasure is growing, and given the ICLR venue, I believe new work in this area should be evaluated on established benchmarks and follow standard tasks. This makes it much easier to compare methods and track genuine progress in the field.
Follow-up Response to Reviewer ceuB
Dear Reviewer ceuB,
Thank you for recognizing that our responses and additional experiments have strengthened the paper.
Our evaluation uses established concept-erasure tasks and evaluation practices from prior work, including artistic-style and object-class erasure, I2P for explicit-content evaluation, and COCO for generation-quality evaluation. Our central comparisons use a common evaluation protocol within each task and matched network-wide sparse parameter budgets, directly assessing the erasure–preservation trade-off under the parameter constraints studied in this paper.
To address the reviewers’ concerns, we rebuilt the common sparse model with target concepts excluded from recovery and reran the sequential baselines. We also added comparisons with SalUn and Scissorhands, expanded the SPACE regularization sweep, and assessed susceptibility to adversarial prompts using UnlearnDiffAtk. These experiments examine the effects of recovery-data composition, additional baseline choices, and adversarial prompting on the erasure–preservation trade-off.
Extending this matched-budget comparison to UnlearnCanvas would require constructing a common sparse model from its benchmark-specific fine-tuned base and retraining and evaluating EwS and the sequential baselines at matched budgets.
Taken together, our evaluations examine target suppression, non-target preservation, and susceptibility to adversarial prompts. We hope these results clarify the paper’s contribution to joint sparsification and concept erasure under a prescribed parameter budget.
Response to Reviewer ceuB (1/2)
Thank you for the concrete suggestions, which motivated the additional experiments below. Please also see the General Response.
W2 / Q1. Fairness: the sequential recovery set included target concepts
The original design models a deployment workflow in which a sparse model is prepared before future concept-erasure requests are known. We therefore first constructed a common sparse model, independently of any particular erasure target, to recover generation across multiple domains and reuse the same model across subsequent requests and sequential methods. Recovery served general post-pruning restoration rather than erasure, so its mixed-domain data did not exclude any particular erasure target. Consequently, the recovery set included 293 target images: 99 Van Gogh, 98 tench, and 96 explicit-content images. Since EwS excludes target concepts from its preservation set, we tested whether this difference affected the comparison by removing those images, rebuilding the common model with the same mask, and rerunning all sequential baselines with their original settings. EwS does not use this common model, so its results are unchanged.
The main comparative findings for EwS persist when target concepts are excluded from recovery. Among the methods in the table above, EwS ranks first in Van Gogh Ld and tench retention. For explicit content, only ConceptPrune among the methods in the table has fewer detections. The target-excluded base has COCO metrics comparable to the original base: FID 18.17 versus 18.47 and CLIP 31.20 versus 31.20. We also reran Shirkavand et al.: Van Gogh Ld 0.241 → 0.211, tench 0.75 / 0.768, and I2P images 62 → 58. We exclude this method from the ranking because of content collapse (Appendix D.2.1). Shirkavand et al. jointly optimize erasure and recovery on a pruned model, addressing the reviewer's suggested case of including the erasure objective during recovery.
W1-a. SalUn
We evaluated SalUn with the authors' official code and settings on the target-excluded common sparse model, maintaining the same sparse budget.
SD 1.4 and EwS use the manuscript reference results. The target-excluded common model and SalUn use the additional evaluation results.
SalUn reduces I2P detections to 5 images, but its COCO FID increases from 18.17 for the starting model to 20.17, above EwS at 17.58. This is another instance of the trade-off between strong erasure and generation-quality preservation observed in the existing comparisons. EwS leaves more detections while better preserving generation quality. SalUn uses a random-labeling erasure objective and training on preservation data. For tench, SalUn and EwS both obtain erased-class accuracy 0.00; their retained-class accuracies are 0.842 and 0.774, respectively.
For artistic-style erasure (Van Gogh), we used the SalUn authors' UnlearnCanvas implementation and default settings, with Van Gogh-style and photographic images generated by the same common sparse model.
The SalUn style implementation aligns denoising predictions under the target-style prompt with those under Photo prompts, using the same noisy target-style image. One possible interpretation of the residual target probability (0.841) is that this conditional-alignment objective was insufficient to suppress the style across evaluation prompts under the evaluated sparse budget. EwS achieves greater target–non-target separation (Ld 0.135 versus 0.045), with a larger change on target prompts (Lf 0.310 versus 0.231) and a smaller change on non-target prompts (Lr 0.175 versus 0.185).