acceptodds
Under review as a conference paper at ICLR 2027

Erase-while-Sparsify: Joint Sparse Training and Concept Erasure for Diffusion Models

16 views

Abstract

Text-to-image diffusion models are often deployed under both prescribed parameter budgets and content-policy constraints requiring the suppression of undesirable or restricted concepts. A common approach is sequential: prune and recover the model before erasure, or erase the target concept before sparsification and recovery. Both orders optimize the stages separately. In the former, erasure is performed on a sparse connectivity pattern selected without its objective; in the latter, subsequent pruning weakens the target–retained separation established by erasure, and recovery without the erasure objective partially restores the target concept. We propose Erase-while-Sparsify (EwS), a unified framework that jointly optimizes sparsification and concept erasure. Rather than fixing the sparse topology or imposing the target sparsity at the outset, EwS progressively reduces model density under joint erasure and preservation objectives, so that the weights adapt to both objectives as the sparse budget is imposed. It applies gradual magnitude pruning and gradient-based regrowth to convolutional layers, while using a concept-ratio criterion to prioritize cross-attention key and value connections with high target-relative weight scores for removal. We evaluate EwS on artistic-style, object-class, and explicit-content erasure. Under matched parameter budgets, we compare EwS with existing U-Net erasure methods applied to a common sparse model as sequential baselines. EwS achieves a favorable overall erasure–preservation trade-off relative to these baselines without requiring separate pre-pruning and recovery stages. Across the evaluated target-density settings from 0.9 to 0.5, EwS retains its advantage over the sequential baselines.

Primary area:
generative models
Keywords:
concept erasure, machine unlearning, text-to-image diffusion models, model sparsification, dynamic sparse training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ceuBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose Erase-while-Sparsify (EwS), a method that targets a deployment setting in which a T2I model must meet a parameter budget and a safe content at the same time. EwS starts from dense SD 1.4 and lowers density gradually over a single fine-tuning run while also optimizing an erasure loss. The authors show that the proposed joint, gradual schedule beats sequential alternatives at the same sparsity budget.

Critical strengths
  • Deployment of generative models that are both tiny and safe is a well-motivated problem
  • Overall, the proposed approach is very interesting and the paper well-written
Critical weaknesses

My main concerns regard the evaluation setting:

  • It's been a while now since the first unlearning papers came out, and several benchmarks were introduced (unlearncanvas, hub etc). Authors should target those for style/object erasure instead of selecting their own concepts. UnlearnDiffAtk should be also included. Authors should also include comparison with SalUn and more recent SAE-based approaches like SAeUron and SAEmnesia
  • It's not clear why sequential pipelines are fine-tuned with also concepts to erase in the recovery phase, while this does not happen with EwS. So I think the comparisons are not fair
Questions for authors
  • Can you rerun the sequential baselines with a recovery set that excludes the target concepts (or that includes the erasure objective)?
  • Can you include some efficiency measures such as latency, peak memory and storage benefits?
  • Can the method be applied to models other than SD 1.4?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ceuB (1/2)

Thank you for the concrete suggestions, which motivated the additional experiments below. Please also see the General Response.

W2 / Q1. Fairness: the sequential recovery set included target concepts

The original design models a deployment workflow in which a sparse model is prepared before future concept-erasure requests are known. We therefore first constructed a common sparse model, independently of any particular erasure target, to recover generation across multiple domains and reuse the same model across subsequent requests and sequential methods. Recovery served general post-pruning restoration rather than erasure, so its mixed-domain data did not exclude any particular erasure target. Consequently, the recovery set included 293 target images: 99 Van Gogh, 98 tench, and 96 explicit-content images. Since EwS excludes target concepts from its preservation set, we tested whether this difference affected the comparison by removing those images, rebuilding the common model with the same mask, and rerunning all sequential baselines with their original settings. EwS does not use this common model, so its results are unchanged.

MethodVan Gogh LdL_d ↑ (original → excluded)Tench erased ↓ / retained ↑ (excluded)I2P images ↓ (original → excluded)
Common sparse model, no erasure−0.038 → −0.0110.80 / 0.779201 → 214
UCE0.070 → 0.0470.12 / 0.600113 → 127
RECE0.065 → 0.0640.00 / 0.53126 → 53
ESD0.121 → 0.1070.41 / 0.72465 → 63
ANT0.102 → 0.0790.09 / 0.70958 → 69
ScaPre0.053 → 0.0480.00 / 0.56781 → 103
ConceptPrune−0.031 → −0.0650.00 / 0.60311 → 10
EwS (no common model)0.1350.00 / 0.77421

The main comparative findings for EwS persist when target concepts are excluded from recovery. Among the methods in the table above, EwS ranks first in Van Gogh LdL_d and tench retention. For explicit content, only ConceptPrune among the methods in the table has fewer detections. The target-excluded base has COCO metrics comparable to the original base: FID 18.17 versus 18.47 and CLIP 31.20 versus 31.20. We also reran Shirkavand et al.: Van Gogh LdL_d 0.241 → 0.211, tench 0.75 / 0.768, and I2P images 62 → 58. We exclude this method from the ranking because of content collapse (Appendix D.2.1). Shirkavand et al. jointly optimize erasure and recovery on a pruned model, addressing the reviewer's suggested case of including the erasure objective during recovery.

W1-a. SalUn

We evaluated SalUn with the authors' official code and settings on the target-excluded common sparse model, maintaining the same sparse budget.

MethodParts ↓Images ↓COCO FID ↓CLIP ↑Ring-A-Bell ↓
SD 1.4 (dense)39124117.7631.1087
Common sparse model (target-excluded)34221418.1731.2086
SalUn6520.1730.100
EwS292117.5830.2410

SD 1.4 and EwS use the manuscript reference results. The target-excluded common model and SalUn use the additional evaluation results.

SalUn reduces I2P detections to 5 images, but its COCO FID increases from 18.17 for the starting model to 20.17, above EwS at 17.58. This is another instance of the trade-off between strong erasure and generation-quality preservation observed in the existing comparisons. EwS leaves more detections while better preserving generation quality. SalUn uses a random-labeling erasure objective and training on preservation data. For tench, SalUn and EwS both obtain erased-class accuracy 0.00; their retained-class accuracies are 0.842 and 0.774, respectively.

For artistic-style erasure (Van Gogh), we used the SalUn authors' UnlearnCanvas implementation and default settings, with Van Gogh-style and photographic images generated by the same common sparse model.

MethodLfL_f / LrL_r / LdL_d ↑PtgtP_{tgt} ↓ / PretP_{ret} ↑
Common sparse model (target-excluded)0.170 / 0.181 / −0.0110.913 / 0.950
SalUn0.231 / 0.185 / 0.0450.841 / 0.942
EwS0.310 / 0.175 / 0.135–

The SalUn style implementation aligns denoising predictions under the target-style prompt with those under Photo prompts, using the same noisy target-style image. One possible interpretation of the residual target probability (0.841) is that this conditional-alignment objective was insufficient to suppress the style across evaluation prompts under the evaluated sparse budget. EwS achieves greater target–non-target separation (LdL_d 0.135 versus 0.045), with a larger change on target prompts (LfL_f 0.310 versus 0.231) and a smaller change on non-target prompts (LrL_r 0.175 versus 0.185).

AuthorsOpenReview ↗

Response to Reviewer ceuB (2/2)

W1-b. UnlearnDiffAtk

We evaluated UnlearnDiffAtk for explicit-content erasure using 50 fixed prompts and the toolkit's default settings.

MethodASR ↓ASR excluding pre-attack detections ↓FID ↓
UCE100%100%19.68
ESD96%94%17.88
Shirkavand et al.94%90%18.09
RECE92%89%18.24
EwS84%79%17.58
ConceptPrune64%62%20.89

Among methods with FID close to the original model, EwS has the lowest attack success rate. Its differences from ESD, Shirkavand et al., and RECE are not statistically significant at n = 50; it is significantly more robust than UCE (McNemar p = 0.008). ConceptPrune is more robust (p = 0.031) but incurs a larger quality cost (FID 20.89; art LdL_d 0.040).

W1-c. Standard benchmarks (UnlearnCanvas, HUB) and SAE-based methods

Concept-set provenance. We adopt concept sets and evaluation prompts from prior erasure studies and report all concepts in those sets.

  • Art: all 10 artists introduced by ESD and reused by UCE, with their evaluation prompts. These comprise 5 famous artists (Van Gogh, Picasso, Rembrandt, Warhol, Caravaggio) and 5 modern artists (McKernan, Kinkade, Edlin, Kilian Eng, Ajin: Demi-Human). Van Gogh is a commonly studied representative target.
  • Objects: all 10 Imagenette classes, the same set used for ESD object erasure.
  • Explicit content: the full I2P set (4,703 prompts), Ring-A-Bell, and COCO-10k, as in AdvUnlearn.

EwS has no concept-specific components. Prompts define the erasure and preservation sets, and the same procedure applies across concepts.

UnlearnCanvas and HUB. We agree with the motivation for standardized evaluation in UnlearnCanvas and HUB. Our central comparison concerns erasure and preservation under the same network-wide sparse budget. Extending this comparison to UnlearnCanvas, for example, requires constructing a common sparse model from its base model and reevaluating the methods under matched budgets. In this revision, we retain the existing evaluation settings and strengthen the evaluation with target-excluded recovery reruns, UnlearnDiffAtk, and SalUn comparisons.

SAE-based methods (SAeUron and SAEmnesia). SAeUron and SAEmnesia intervene on activations during generation using trained SAEs. EwS directly modifies base-model weights to achieve erasure and a network-wide sparse budget jointly. Our quantitative comparisons focus on weight-modification approaches. Evaluation of SAE-based methods under the same deployment budget requires validation of their intervention effects on the sparse base, accounting for the SAE's additional parameters and execution cost. We cite both methods in the revised related-work section and explain the differences between the approaches.

Q2. Efficiency metrics: latency, peak memory, and storage

Please see General Response 2. On an A100, latency (1.62 s) and peak memory (4.54 GB) match SD 1.4 because the sparsity is unstructured and execution uses standard dense kernels. With zstd, the EwS storage cost is 1.04 GB at d = 0.75 and 0.69 GB at d = 0.5, compared with 1.35 GB for dense SD 1.4.

Q3. Models beyond SD 1.4

Yes. EwS operates on SDXL at d = 0.8 and 0.6 (Table 27). The new SD3-medium results (General Response 1) extend this to a DiT-based model: EwS achieves the strongest target erasure among the compared methods while retaining other artists at 96% of the original PretP_{ret}. We added the main results to Section 5.4 (Table 4), with full settings and results in Appendix D.6 (Table 28).

Dear authors,

I truly appreciated your response and the additional experiments, which have strengthened the paper. However, I have decided to maintain my original score. Given how rapidly the literature on unlearning and concept erasure is growing, and given the ICLR venue, I believe new work in this area should be evaluated on established benchmarks and follow standard tasks. This makes it much easier to compare methods and track genuine progress in the field.

AuthorsOpenReview ↗

Follow-up Response to Reviewer ceuB

Dear Reviewer ceuB,

Thank you for recognizing that our responses and additional experiments have strengthened the paper.

Our evaluation uses established concept-erasure tasks and evaluation practices from prior work, including artistic-style and object-class erasure, I2P for explicit-content evaluation, and COCO for generation-quality evaluation. Our central comparisons use a common evaluation protocol within each task and matched network-wide sparse parameter budgets, directly assessing the erasure–preservation trade-off under the parameter constraints studied in this paper.

To address the reviewers’ concerns, we rebuilt the common sparse model with target concepts excluded from recovery and reran the sequential baselines. We also added comparisons with SalUn and Scissorhands, expanded the SPACE regularization sweep, and assessed susceptibility to adversarial prompts using UnlearnDiffAtk. These experiments examine the effects of recovery-data composition, additional baseline choices, and adversarial prompting on the erasure–preservation trade-off.

Extending this matched-budget comparison to UnlearnCanvas would require constructing a common sparse model from its benchmark-specific fine-tuned base and retraining and evaluating EwS and the sequential baselines at matched budgets.

Taken together, our evaluations examine target suppression, non-target preservation, and susceptibility to adversarial prompts. We hope these results clarify the paper’s contribution to joint sparsification and concept erasure under a prescribed parameter budget.

Reviewer_Kvb4OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Erase-while-Sparsify (EwS), a unified framework that progressively reduces model density while achieving concept erasure and non-target preservation. EwS uses ERK to determine the target densities of convolutional layers and prescribed densities for K/V projections. During sparsification (mask updates), EwS prunes the smallest-magnitude active weights in convolutional layers and prunes the weights with the largest concept-ratio scores in K/V projections. Inactive weights with the largest gradient magnitudes are activated again. During training, EWS adopts ANT's timestep-dependent supervision strategy and combines it with anchor consistency, non-target preservation, and teacher distillation losses to balance concept erasure and preservation.

Critical strengths
  • The proposed method is effective and achieves on-par or even better concept erasure and non-target preservation than the compared sparsification and concept erasure baselines.
  • The experiments are thorough. EwS is compared with sufficient recent baselines across multiple domains (artistic styles, objects, and nudity). The ablation studies and analyses are also thorough, including the effects of the target density, sparsification timing, and cross-attention K/V sparsity.
  • Overall, Figure 2 is well designed. However, the text and notation are too small, making them difficult to read.
Critical weaknesses
  • The contribution of the proposed method is a concern.
    • Can the proposed method be applied to DiT-based diffusion or flow-matching models? If not, its significance may be limited, as many recent state-of-the-art text-to-image generation models are DiT-based.
    • The 75% density setting seems insufficient to demonstrate the practical benefits of sparsification. It is not very convincing to say that a dense model cannot fit within a given memory or storage budget while a model keeping 75% of its weights can. A 75% dense model is only moderately smaller than the original model, rather than significantly smaller.
  • In Appendix D.2.1, the paper shows coherence statistics and CLIP scores for Shirkavand et al. (2025), but only reports a single mean coherence score for EWS. To support the claim that Shirkavand et al. (2025) has much more severe content-collapse issues than EWS, the full coherence statistics and CLIP scores for EWS should be reported as well.
Questions for authors
  • Why does EWS use ERK to determine the target densities of convolutional layers but prescribe the densities of K/V projections? Couldn't ERK also be used to determine the target densities of K/V projections?
  • How are gradients computed for inactive weights during regrowth? Since these weights are masked out by a binary mask, their gradients would normally be zero.
6 comments
AuthorsOpenReview ↗

Response to Reviewer Kvb4 (1/2)

Thank you for the positive assessment and precise questions. Please also see the General Response.

W1. Applicability to DiT / flow-matching models

Yes. Please see General Response 1. On SD3-medium (MMDiT, a flow-matching model) at d = 0.75, EwS achieves LdL_d 0.066 and PtgtP_{tgt} 0.163 (sequential UCE: 0.028 / 0.219; ESD: 0.023 / 0.391), while retaining PretP_{ret} at 96% of the original. As MMDiT has no separate cross-attention, we apply concept-ratio scoring to text-stream K/V projections. Appendix D.6 describes the settings, including SD3's flow-velocity prediction.

W2. Is d = 0.75 sufficient to demonstrate practical benefits?

Although d = 0.75 is the main setting, the manuscript also reports lower densities, and EwS retains coherent generation down to d = 0.5 (Tables 8–9).

DensityArt LdL_d ↑ (10-artist mean)Object erased ↓ / retained ↑ (10-class mean)I2P parts ↓ / FID ↓File size (bitmask + zstd)
0.750.1540.006 / 0.84129 / 17.581.04 GB
0.50.1010.006 / 0.80619 / 16.110.69 GB

With the same zstd compression, the SD 1.4 U-Net occupies 1.35 GB, compared with 1.04 GB for EwS at d = 0.75 (23% smaller) and 0.69 GB for EwS at d = 0.5 (49% smaller). The uncompressed dense U-Net occupies 1.72 GB. SDXL also operates at d = 0.6 (Table 27). We agree that 0.75 represents a moderate reduction and have included d = 0.5 in the main-text storage table (Section 5.4, Table 5).

W3. Full coherence statistics for EwS (Appendix D.2.1)

We retrained EwS for all 10 artists and report below the full statistics computed with the same metrics as the Shirkavand et al. analysis (Table 24). This separate retraining run yields mean LdL_d 0.150 and Van Gogh LdL_d 0.137, compared with 0.154 and 0.135, respectively, in the manuscript's main experiments.

ArtistLdL_dCoh. meanCoh. minCLIP (SD)CLIP (EwS)
Van Gogh0.1370.9740.78325.327.7
Picasso0.0990.9900.86025.826.1
Rembrandt0.1031.0000.99918.118.4
Warhol0.2320.9850.83420.120.8
Caravaggio0.1540.9910.83824.926.0
McKernan0.1780.9970.96024.126.2
Kinkade0.1650.9990.98316.715.6
Edlin0.1090.9980.96617.317.9
Kilian Eng0.1450.9920.89225.125.0
Ajin0.1800.8650.21418.620.0
Mean0.1500.979–21.622.4
Shirkavand et al. (Table 24)0.2380.823–21.619.3

Content CLIP follows Table 24: image–text similarity to prompts with artist names removed, using ViT-L/14. EwS matches or exceeds the original model's content CLIP for 8 of 10 artists (mean 22.4 versus 21.6), compared with 19.3 for Shirkavand et al.

As in the submission, we use per-artist mean coherence below 0.85 as an operational criterion for content collapse. Seven of 10 artists meet this criterion for Shirkavand et al., whereas all 10 EwS artists remain above it. Ajin has mean coherence 0.865, above the threshold, but a minimum of 0.214. The low scores were observed for prompts requesting abstract or minimalist works. Because the metric assesses the presence of recognizable subjects, the abstract renderings induced by these prompts are a possible contributor to the low scores.

The original submission reported mean coherence 0.987. Following retraining and reevaluation, we updated the EwS mean in revised Table 24 to 0.979.

AuthorsOpenReview ↗

Response to Reviewer Kvb4 (2/2)

Q1. Why prescribe K/V densities instead of allocating them with ERK?

We prescribe a separate K/V budget because ERK allocation alone leaves these projections unpruned in this setting, despite their role in concept erasure. ERK is applied to the entire U-Net, including K/V projections. However, it assigns higher densities to smaller layers, so at d = 0.75 all linear layers, including attention, feed-forward, and cross-attention K/V layers, receive density 1. Pruning therefore occurs only in convolutional layers: 36.5% of convolutional weights and 0% of linear weights are zeroed. Omitting K/V pruning increases target probability from 0.216 to 0.410 for Van Gogh and from 0.155 to 0.546 for tench (Table 15). We therefore supplement the ERK allocation with a separate K/V erasure budget and evaluate its pruning fraction in Table 10.

The separate K/V budget serves target suppression as well as capacity reduction. In diagnostic experiments, ERK-style nonuniform allocation pruned the outer, low-dimensional K layers less and allocated more pruning to other layers. Target suppression induced by pruning the low-dimensional K layers persisted during subsequent training, whereas the effects in deeper layers were largely compensated. Thus, the final erasure effect depended on the layer allocation even at the same total budget. Uniform allocation achieved stronger erasure than the tested nonuniform alternatives: 22 detected body parts versus 38 with ERK-style layer allocation and 40 with a global threshold. We therefore use the same pruning fraction in every K/V layer. This diagnostic used explicit content and a 10% K/V budget with an earlier pruning schedule different from that of the main experiments.

Q2. How are gradients computed for inactive weights?

The mask is not a multiplicative gate in the forward pass. After every optimizer step, masked weights are zeroed in place (p.data.mul_(mask)). Backpropagation therefore computes ordinary dense gradients dL/dw at all weights, including weights currently equal to zero. Regrowth selects inactive positions with the largest |dL/dw|, as in RigL. We clarified this in Section 4.2.

Figure 2. We enlarged the text and notation.

Thank you for the response. My questions and the concern about the coherence statistics have been addressed. The only remaining concern is about the contribution of the proposed method. The proposed method is applied to SD3-medium (MMDiT) on one artistic style erasure only, which is insufficient to demonstrate its generalizability to other model architectures. Experiments on erasing 10 objects should also be reported.

I still maintain my score.

Reviewer_t4NCOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper provides a new method (Erase-While-Sparsify) for joint erasure and sparsification. To achieve this joint goal, rather than sequentially pruning and erasing, it progressively reduces the model density during erasure and preservation. The method is tested from target density settings from 0.9 to 0.5.

Critical strengths
  • The paper proposes a new method for joint erasure and sparsification
Critical weaknesses
  1. The paper does not demonstrate that the true model size decreases, thus not demonstrating the motivation for such a method.
  2. The paper only focuses on U-Net models. In particular, the main part of the paper entirely focuses on SD 1.4. However, the authors should demonstrate the relevance for recent DiT-based text-to-image models
Questions for authors
  • How relevant is the study of removing 25% of parameters? Can the authors show if there is a decrease of model size after such a reduction? Otherwise, what is the advantage of such a sparsity?
  • If there is no reduction of real model size, is there an advantage in using the training+sparsification method proposed? For instance, is your method still leading to a performance improvement compared to baselines? In other words, can you compare the proposed method with other standard erasure methods on the dense model?
  • The authors mention that their method is the first one doing a joint erasure and sparsification. For example, their first main contribution is: " Unified formulation. We formulate sparsification and concept erasure as a unified learning problem that updates sparse connectivity during capacity reduction.". However, this paper is not the first paper formulating a joint erasure and sparsification objective. For instance, [1] tackles the same erasure+sparsification task. However, the paper does not reference, nor compare with recent existing concept erasure work that rely on joint sparsity and erasure. The paper needs to explicitly mention differences and similarities with other sparsity based methods. Another example is [2], as the proposed prune+regrow method is similar to the approach used by Scissorhands.
  • What is the computational complexity of the proposed algorithm? And its comparison against other erasure methods?

[1] Novello, Nicola, and Andrea M. Tonello. "Empty SPACE: Cross-Attention Sparsity for Concept Erasure in Diffusion Models." arXiv preprint arXiv:2605.10198 (2026).
[2] Wu, Jing, and Mehrtash Harandi. "Scissorhands: Scrub data influence via connection sensitivity in networks." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

9 comments
AuthorsOpenReview ↗

Response to Reviewer t4NC (1/3)

Thank you for the detailed questions. Please also see the General Response.

W1 / Q1. Does the actual model size decrease, and is 25% sparsity meaningful?

Yes. At d = 0.75, the number of nonzero U-Net parameters decreases from 859.5M to 642.8M. Storing the nonzero weights together with a bitmask of their positions reduces the actual file size: from 1.35 GB to 1.04 GB with zstd (−23%), and from 1.72 GB to 1.35 GB without compression (−22%). At d = 0.5, the EwS storage cost is 0.69 GB, 49% less than zstd-compressed SD 1.4. The benefit is reduced storage and transmission size. Latency (1.62 s) and peak memory (4.54 GB) are unchanged with dense GPU kernels. We added the storage results to Section 5.4 (Table 5).

Q2. Comparison with standard erasure methods on dense models

Q1 establishes a reduction in actual checkpoint size. In addition, Appendix D.1 (Tables 20–22) compares EwS at d = 0.75 with baselines applied to dense SD 1.4. This comparison tests whether EwS remains competitive with dense erasure methods using fewer nonzero parameters.

  • Objects: EwS achieves the highest retained-class accuracy (0.84 versus 0.45–0.71), with erased-class accuracy 0.01.
  • Explicit content: EwS yields 29 detected body parts and FID 17.58. Dense methods with similar FID have more detections: RECE 36 / 17.73, ESD 32 / 18.46, and UCE 236 / 18.03 (detections / FID).
  • Art: performance is comparable to the dense methods (LdL_d 0.154 versus 0.159–0.170 for UCE / ESD / RECE).

Thus, EwS removes 25% of the nonzero parameters while achieving comparable or better erasure–preservation performance than dense erasure.

AuthorsOpenReview ↗

Response to Reviewer t4NC (2/3)

Q3. Relationship to joint sparsity–erasure methods (SPACE and Scissorhands)

Thank you for pointing us to these studies. We revised the contribution statement to specify the methodological contribution: progressively reaching a prescribed network-wide sparse budget by repeatedly pruning and regrowing connections during joint erasure and preservation training (Section 1). Section 2 now explains how this procedure differs from sparse K/V editing in SPACE and sensitivity-based reinitialization followed by repair in Scissorhands.

  • Empty SPACE (Novello & Tonello, arXiv 2026). SPACE augments a UCE-based cross-attention K/V editing objective with an L1 sparsity penalty and optimizes it using FISTA. Its sparse support results from shrinkage of the edited K/V weights. EwS instead fine-tunes the diffusion model under erasure and preservation losses, progressively reducing density to a prescribed U-Net-wide budget. It repeatedly updates the sparse support through pruning and gradient-based regrowth, using concept-ratio scores for K/V selection. The methods therefore differ in their optimization procedure and connectivity updates, as well as in parameter coverage. With the same 10% K/V sparsity budget and similar I2P erasure (29 versus 30 detected body parts), concept-ratio selection reduces Ring-A-Bell success to 9.3% (10/107), compared with 16.8% (18/107) for magnitude selection and 27.1% without K/V sparsification. This is an internal K/V selection ablation within EwS, not a direct comparison with SPACE. We added it to Appendix C.3 (Table 16).

We also directly evaluated SPACE on the same target-excluded common sparse model, using the authors' official code and settings (λ = 0.006 for SD 1.x, with their concept and preservation configurations). We enforced the common model's mask at every FISTA iteration to maintain the same sparse budget.

MethodVan Gogh LdL_d ↑tench erased ↓ / retained ↑I2P parts / images ↓Ring-A-Bell ↓FID ↓ / CLIP ↑
SPACE (sequential)−0.0120.63 / 0.657125 / 8065/10720.79 / 30.80
EwS0.1350.00 / 0.77429 / 2110/10717.58 / 30.24

SPACE's K/V editing term has the same objective as UCE's closed-form formulation; without the L1 term, its solution coincides with UCE. The additional L1 term shrinks K/V weights irrespective of the erased concept, setting approximately 15% to zero. Erasure does not improve over the UCE reference (Van Gogh LdL_d 0.047; tench erased-class accuracy 0.12), while larger output changes on other prompts reduce preservation (Van Gogh PretP_{ret} 0.950 → 0.891). EwS selects K/V connections for removal using concept-ratio scores and optimizes erasure and preservation jointly within a budget that also includes convolutional weights.

  • Scissorhands (Wu & Harandi, ECCV 2024). Scissorhands uses connection sensitivity (SNIP) on the forgetting data to select weights for a one-time reinitialization, followed by repair through fine-tuning. Reinitialization replaces selected weight values; it does not implement the repeated pruning and regrowth of sparse connectivity used by EwS. Scissorhands does not prescribe a final nonzero-parameter budget. EwS progressively enforces such a budget, repeatedly removes active connections, and regrows inactive connections selected by gradients of the joint erasure and preservation objective.

We directly evaluated Scissorhands on the same target-excluded common sparse model using the authors' code and settings: their SD implementation for explicit content and objects, and the UnlearnCanvas implementation for artistic styles. We enforced the common mask immediately after reinitialization and after every optimizer step to maintain the same sparse budget. For explicit content and objects, we used the checkpoints saved by the authors' code (step 109 for explicit content; step 79 for objects).

MethodVan Gogh LdL_d ↑tench erased ↓ / retained ↑I2P parts / images ↓Ring-A-Bell ↓FID ↓ / CLIP ↑
Scissorhands (sequential)−0.0570.00 / 0.74691 / 5529/10720.04 / 31.05
EwS0.1350.00 / 0.77429 / 2110/10717.58 / 30.24
  • The related-work section also discusses differences from SalUn, SAeUron/SAEmnesia, Shirkavand et al., and ConceptPrune.

Our contribution is to reach a network-wide sparse budget from a dense model during a single erasure fine-tuning run, without a separate pre-pruning and recovery stage.

AuthorsOpenReview ↗

Response to Reviewer t4NC (3/3)

Q4. Computational complexity

EwS uses one fine-tuning run per concept. Masks are updated every 200 steps, and the reported training times include mask selection and update operations. Sequential pipelines first require a pruning–recovery stage to construct the common sparse model, followed by each method's erasure stage. We compared training costs for explicit-content erasure on a single A100, excluding initial data preparation and final evaluation; sampling performed within the erasure algorithm is included.

MethodCommon sparse modelErasure stageTotal
EwS—20 min20 min
UCE (sequential)41 min1 min42 min
RECE (sequential)41 min66 min107 min
SCAPRE (sequential)41 min2 min43 min
ConceptPrune (sequential)41 min9 min50 min
ESD (sequential)41 min28 min69 min
SalUn (sequential)41 min24 min65 min
ANT (sequential)41 min99 min140 min
Shirkavand et al. (sequential)41 min26 min67 min

These are single-concept erasure costs, with common sparse-model construction included in the sequential totals. When the same common model is reused across concepts, sequential pipelines amortize its construction cost over those concepts. EwS jointly achieves the target sparse budget and erasure in one training run, without a separate pruning–recovery stage. Appendix C.5 (Table 19) reports training times for the manuscript's existing comparison methods.

W2. DiT models

Please see General Response 1. On SD3-medium (MMDiT), EwS achieves the strongest Van Gogh erasure among the compared methods (LdL_d 0.066 versus 0.028 / 0.023; PtgtP_{tgt} 0.163 versus 0.219 / 0.391), while retaining other artists at 96% of the original PretP_{ret}.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response (1/3)

We thank all reviewers for their careful reading. We address three shared concerns with new experiments and discuss the remaining points in the individual responses. We have uploaded a revised manuscript; a list of revisions appears at the end of this response.

Changes in the revised manuscript are highlighted in blue.

1. Beyond U-Net: EwS on a DiT model (SD3-medium) (t4NC, Kvb4, ceuB)

We applied EwS to SD3-medium (MMDiT) for Van Gogh erasure at an ERK target density of 0.75, resulting in 25.6% transformer sparsity. The sequential baselines start from a common sparse SD3-medium model obtained by pruning and 2,000 recovery steps on SD3-generated artist images excluding Van Gogh. UCE and ESD adapt their authors' FLUX configurations to SD3. Appendix D.6 describes the SD3 settings.

Method (SD3, d = 0.75)LdL_d ↑PtgtP_{tgt} ↓PretP_{ret} ↑ΔCStgtΔCS_{tgt}ΔCSretΔCS_{ret}
Original SD3 (dense)–0.6720.721––
Common sparse model, no erasure−0.0270.5960.699−0.4%+0.8%
UCE (sequential)0.0280.2190.698−8.7%+0.6%
ESD (sequential)0.0230.3910.649−5.8%−2.6%
EwS0.0660.1630.693−11.0%−1.4%

ΔCS is the relative change in prompt–image CLIP score (ViT-L/14) from original SD3. “tgt” denotes Van Gogh prompts, and “ret” denotes other-artist prompts. EwS achieves the strongest style erasure among these methods (largest LdL_d and lowest PtgtP_{tgt}), while retaining other artists close to the original model (PretP_{ret} at 96% of the original; ΔCSretΔCS_{ret} −1.4% versus ΔCStgtΔCS_{tgt} −11.0%). For context, published erasure methods on FLUX.1 reduce target CLIP scores by 1.6% (UCE), 7.5% (ESD), and 11.5–11.7% (EraseAnything, Uni-AdaVD). These use different models and prompt sets, so we report them only as a reference for the magnitude of the change. MMDiT has no separate cross-attention, so we apply concept-ratio scoring to its text-stream K/V projections (add_k_proj, add_v_proj). We retain the erasure/preservation objectives and gradual sparsification framework. Appendix D.6 specifies the SD3 settings, including flow-velocity prediction and the erasure boundary. We added the main results to Section 5.4 (Table 4), with detailed settings and full results in Appendix D.6 (Table 28).

AuthorsOpenReview ↗

General Response (2/3)

2. Practical benefits of sparsity (t4NC, Kvb4, ceuB)

We evaluated EwS's U-Net storage costs at the corresponding layer-wise sparsity budgets and benchmarked generation on an A100 (fp16 generation, 512 px, 50 steps).

SD 1.4 (dense)EwS (d = 0.75)EwS (d = 0.5)
Nonzero parameters (U-Net)859.5M642.8M428.0M
File size, sparse bitmask format1.72 GB1.35 GB0.93 GB
File size, bitmask + zstd1.35 GB*1.04 GB0.69 GB
Generation latency (bs 1 / bs 4, approx.)1.62 / 4.0 s1.62 / 4.0 s1.62 / 4.0 s
Peak GPU memory (bs 1)4.54 GB4.54 GB4.54 GB

*SD 1.4 uses dense + zstd. Storage measurements use bf16 weights, while inference benchmarks use fp16. We wrote all files to disk, reloaded them, and verified exact recovery of the bf16 weights after conversion; lossless encoding refers to this bf16 representation.

Our deployment setting requires the same model to satisfy both a storage budget and a concept-erasure requirement. Reducing checkpoint size while preserving erasure performance reduces the storage and transfer volume needed to retain and distribute the model. With the same zstd compression, the EwS U-Net at d = 0.75 occupies 1.04 GB instead of 1.35 GB, a 23% reduction; relative to the uncompressed bf16 dense file, the reduction is 40%. These measurements concern the U-Net; the text encoder and VAE are unchanged.

  • Unstructured sparsity provides no latency or memory reduction with standard dense GPU kernels.

We added the storage results to Section 5.4 (Table 5). Appendix C.5 (Tables 18–19) provides the storage protocol and training-time details.

AuthorsOpenReview ↗

General Response (3/3)

3. Fairness of the sequential baselines (ceuB; also relevant to t4NC Q2)

As ceuB noted, the common sparse model's recovery set included target concepts: 99 Van Gogh images, 98 tench images, and 96 explicit-content images, accounting for 7.9% of the set. We removed these 293 images, rebuilt the common sparse model with a bit-identical mask, and reran all sequential baselines on it. EwS does not use this model, so its results are unchanged. The main comparative findings for EwS persist when target concepts are excluded from recovery. The ceuB response below reports method-specific changes and full results, together with SalUn comparisons on the target-excluded base and a separate UnlearnDiffAtk evaluation.

Updated coherence statistics and corrections

  • To provide full per-artist coherence statistics, we retrained and reevaluated EwS for all 10 artists. The mean coherence in this evaluation is 0.979, which is reflected in revised Table 24. The full per-artist statistics appear in the Kvb4 response below. The original submission reported a mean of 0.987.
  • Appendix B: the 571 recovery images in the explicit-content partition comprise 96 explicit-content images and 475 non-explicit preservation images. We corrected the description.

List of revisions

  • Main text
    • Section 1: refined the contribution statement (t4NC Q3).
    • Section 2: cited sparsity-based erasure and SAE-based methods and explained the differences between approaches (t4NC Q3; ceuB W1-c).
    • Section 4.2: clarified mask enforcement and gradients at inactive weights (Kvb4 Q2).
    • Section 5.1: identified the sources of the concept sets (ceuB W1-c).
    • Section 5.4: added storage results at d = 0.75 and 0.5 (Table 5; General Response 2).
    • Section 5.4 (Table 4): added SD3-medium results (General Response 1).
  • Appendix
    • D.6 (Table 28): SD3-medium settings and full results.
    • C.3 (Table 16): K/V selection ablation (concept ratio, magnitude, and no K/V sparsification).
    • C.5 (Tables 18–19): measured file sizes, storage protocol, and training time.
    • Table 24: updated EwS mean coherence to 0.979.
  • Statistical updates and corrections: Table 24 coherence, the Appendix B recovery-set description, and Figure 2 font size.
AuthorsOpenReview ↗

Summary of the Discussion and Key Revisions

We thank the AC and all three reviewers. Your specific questions and suggestions helped us examine the comparison conditions more rigorously, strengthen the evaluation, and clarify the contributions of this work. The additional evidence and revised manuscript make the following three points clearer.

Reviewer Kvb4 has confirmed that the additional experiments addressed their concerns about the method and described EwS as reasonable and effective in this setting.

1. EwS jointly optimizes erasure and preservation under matched sparse budgets.

Our work addresses deployment settings in which a model must satisfy both a parameter budget and a concept-erasure requirement. Instead of applying erasure to a sparse structure selected independently of the erasure objective, EwS progressively reduces model density under joint erasure and preservation objectives.

Following the reviewers' feedback, we excluded target concepts from the recovery data and reran the sequential baselines; the main comparative findings persisted. We broadened the comparisons with SalUn, Scissorhands, and a SPACE lambda sweep, and clarified the differences in optimization procedures. We also added a dense-comparison table and mean and standard deviation results for Van Gogh and tench, allowing both performance and variation to be examined. These additions support the value of jointly allocating limited model capacity to erasure and preservation.

Our evaluation builds on artistic-style, object-class, and explicit-content erasure tasks used in prior work, including I2P for explicit-content evaluation and COCO for generation-quality evaluation, and additionally examines adversarial prompts using UnlearnDiffAtk.

2. The checkpoint-storage benefit does not depend on a specific bitmask setting.

Following the reviewer's suggestion, we analyzed sensitivity to the storage-layout selection threshold and evaluated standard compressors. At both density budgets, storage reductions persisted over a broad threshold range and were also observed when all weights were stored without bitmask encoding and then compressed with zstd, gzip, or xz.

This shows the practical benefit of reducing storage and transfer costs for a model that also satisfies a concept-erasure requirement. The additional analysis clarifies the scope of this benefit. Under a prescribed storage budget, matched-budget comparisons directly assess how effectively concept suppression and preservation can be achieved within the available model capacity. Storage results and the measurement protocol appear in Section 5.4, Table 5, and Appendix C.5.

3. Joint erasure and sparsification extend beyond U-Net to multiple targets and tasks.

In response to questions about architectural generalization, we expanded the SD3-medium evaluation to five artists and all ten Imagenette object classes. For artistic styles, EwS reduces target CLIP scores for all five artists, with a mean reduction of 9.0%, compared with 4.1% for UCE and 3.3% for ESD under matched sparse budgets. Non-target CLIP changes range from -1.4% to +1.4%.

For objects, EwS reduces mean target-class accuracy from 88.4% for original SD3 to 1.1%, while maintaining 87.5% retained-class accuracy, compared with 88.4% for the original model. These results support the applicability of EwS's joint optimization framework to multiple targets and two erasure tasks on an MMDiT-based model. The mean artistic-style comparison appears in Section 5.4, Table 4; per-artist and per-object results and qualitative comparisons appear in Appendix D.6, Tables 28-29 and Figure 8.

The reviewers' feedback helped us strengthen the comparisons and evaluation evidence and present the benefits of EwS more clearly. Changes in the revised manuscript are highlighted in blue. Detailed results and explanations addressing individual questions are provided in our individual responses.

These results support the value of studying deployment budgets and concept-erasure requirements together as a distinct research problem. EwS offers a unified approach to this problem, with evidence of its utility from erasure-preservation performance under matched sparse budgets and its application to a different architecture. We hope that the final assessment will consider this problem formulation and methodological contribution together with the additional evidence developed during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.