acceptodds
Withdrawn from ICLR 2027

Empty SPACE: Cross-Attention Sparsity for Concept Erasure in Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Erasing specific concepts from text-to-image diffusion models is essential for avoiding the generation of copyrighted and explicit content. Closed-form concept erasure methods offer a fast alternative to backpropagation-based techniques, but they become less effective when scaling from smaller models such as Stable Diffusion 1.5 to larger models like Stable Diffusion XL. To maintain erasure effectiveness in these larger-scale architectures, we propose SParse cross-Attention-based Concept Erasure (SPACE). SPACE iteratively modifies the cross-attention parameters of a model with a closed-form update that jointly induces sparsity and erases target concepts. By concentrating the concept mapping to a lower-dimensional subspace, SPACE achieves superior erasure efficacy compared to dense baselines. Extensive experimental results show the erasure effectiveness for different scenarios and a large set of models. Furthermore, SPACE achieves 80%-90% cross-attention sparsity, reducing the storage requirements for saving the modified parameters by 70%, demonstrating its memory efficiency.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
unlearning, erasure, text-to-image diffusion model, safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UjypOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper claims that closed-form concept erasure methods become less effective for larger text-to-image diffusion models and attributes this degradation to redundancy in the cross-attention (CA) parameter space. To address this, the paper proposes SPACE, which builds on UCE's objective function by adding an entry-wise L1 norm on the CA weights to encourage sparsity. Due to the non-smoothness of the L1 norm, SPACE uses the Fast Iterative Shrinkage-Thresholding Algorithm (FISTA) to iteratively update the CA parameters. The gradient is computed in closed form without backpropagation.

Critical strengths
  • The method is sound, fast, storage-efficient, and effective in concept erasure. The paper proves that SPACE will surely be convergent. Also, updating cross-attention weights for 1000 iterations takes less than 2 minutes. Moreover, SPACE achieves over 80% sparsity, which reduces storage requirements significantly after erasure.
  • The paper provides thorough analyses demonstrating the importance of cross-attention sparsity. It shows that increasing the erasure strength in UCE does not improve erasure performance and that sparsity, rather than iterative updates, is the key to SPACE's effectiveness.
  • The paper is well written and easy to follow.
Critical weaknesses
  • The motivation of this paper is not convincing to me. The paper claims that closed-form erasure methods become less effective on larger models due to redundancy in the cross-attention parameter space. Why focus on cross-attention rather than self-attention or feed-forward modules?
  • In artistic style erasure, SPACE performs noticeably worse than UCE and SPEED at preserving non-target concepts. This raises the concern that SPACE erases the target concept so hard that it sacrifices the non-target quality. Could lowering the sparsity value λ\lambda or reducing the number of FISTA iterations improve this trade-off?
  • The sparsity-based unlearning methods mentioned in line 83, such as SalUn and Scissorhands, are not included in the experiments.
  • The paper lacks quantitative results for celebrity erasure; only qualitative results are provided. Also, quantitative results for fictional character erasure using SDXL are also missing.
  • Figure 2 is not well designed. It’s full of math formulas and notation. The math formula in the middle is even displayed vertically.
Questions for authors
  • How does the number of FISTA iterations affect erasure performance?
  • How does λ\lambda affect erasure performance?
Reviewer_3BMZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SPACE, a method designed to mitigate the degradation of existing closed-form concept erasure methods when applied to large-scale text-to-image diffusion models. SPACE extends UCE-based cross-attention weight editing by introducing sparsity regularization and iteratively updates the cross-attention parameters using FISTA while inducing sparsity. The experiments indicate that SPACE improves erasure performance relative to existing closed-form baselines.

Critical strengths
  • The proposed method is simple and intuitive. SPACE extends the UCE formulation with sparsity regularization and optimizes the resulting objective using FISTA.
  • The paper directly addresses an important limitation of existing closed-form concept erasure methods on large-scale models.
  • The effect of sparsity is analyzed from multiple perspectives. The paper considers erasure performance, efficiency-related metrics, and additional analyses, providing empirical support for the role of sparsity.
Critical weaknesses

[W1] The extent of performance improvement achieved by SPACE on large-scale models remains unclear. Although SPACE improves over existing closed-form methods, its absolute erasure and preservation performance can remain limited. For example, in Table 1, the CA for Van Gogh decreases from 100 to 72, while noticeable degradation is also observed for some non-target concepts. In this case, accepting additional computational cost for a stronger training-based method may be a practical alternative.

[W2] The evaluation protocol varies across architectures and experimental settings. SDXL mainly reports I2P results, while adversarial robustness and retention are evaluated on other models. In addition, the set of baselines differs across experiments, which makes it more difficult to form a consistent comparison across settings. A more unified evaluation protocol would enable a more comprehensive assessment of SPACE on the main large-scale setting.

Questions for authors
  1. Does the trend observed for UCE in Fig. 9 also hold for other closed-form baselines? In addition to UCE, could the authors compare SPACE with stronger, well-tuned configurations of more recent methods such as RECE and SPEED by sufficiently adjusting their erasure-strength or other key hyperparameters?

  2. What validation set is used for the λ\lambda grid search, and is it separated from the final evaluation set?

  3. How is the concept set PP used in the preservation loss constructed? Does preservation based on a predefined set of concepts generalize to unseen non-target concepts that are not included in PP? Furthermore, can concept-level preservation also maintain broader compositional generation capabilities, as evaluated by benchmarks such as GenEval[1]?

  4. The paper mentions that performance can be sensitive to the choice of anchor concept in some settings. Could the authors quantitatively report how much erasure and retention performance varies across different anchors?

  5. Could the authors report both adversarial robustness and retention performance on the main SDXL setting? If possible, in addition to black-box attacks, it would be valuable to consider a white-box attack such as UnlearnDiffAtk[2], which directly attacks the final erased model.

  6. Could the authors include more recent training-based baselines? Although CAbl is included as a training-based baseline, it may not sufficiently represent recent training-based concept erasure methods. Comparing SPACE with recent training-based methods in terms of erasure, retention, and computational cost would better clarify both the motivation of the paper and the practical advantage of its closed-form formulation.

  7. Could the authors further analyze the impact of cross-domain joint erasure on preservation performance? In particular, it would be useful to evaluate whether jointly erasing concepts from different domains, such as nudity and Van Gogh, leads to greater non-target degradation than jointly erasing multiple concepts within the same domain.


[1] Ghosh, Dhruba, Hannaneh Hajishirzi, and Ludwig Schmidt. "Geneval: An object-focused framework for evaluating text-to-image alignment." NeurIPS, 2023.

[2] Zhang, Yimeng, et al. "To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images... for now." ECCV, 2024.

Reviewer_zSinOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This work proposes a simple method for concept erasing by employing a closed-form update for CA param updates and jointly inducing sparsity in params with an entry-wise L1 norm (Eq. (5)). This optimization was done with conventional FISTA. Limited comparisons were reported with SDXL , SD1.5, and Juggernaut-XL.

Critical strengths

This work's idea is simple, but elegant for concept erasing. Conventional FISTA was used properly for realizing this idea with closed-form update for cross-attention parameters and L1 norm for sparsity.

Critical weaknesses
  1. While using L1 sparsity seems new, the idea of sparse intervention or selected removal has been investigated in prior works. ConceptPrune, SalUn, Sculpting Memory, SAeUron, SAEmnesia are some of them. Direct comparisons and analyses seem important to demonstrate the novelty of this work over prior works. Gen Li et al., Sculpting Memory: Multi-Concept Forgetting in Diffusion Models via Dynamic Mask and Concept-Aware Optimization, ICCV 2025
  2. The experiments are limited in terms of benchmarks, adversarial attacks and SOTA methods. Table 1 only experimented several style erasures and Table 2 were also limited. Other concept preservation was not sufficiently evaluated. Various attacks were reported in the appendix, but they are not SOTA as compared to recent works. This work also did not evaluate larger models such as SD3.5-L, which makes hard to properly evaluate for modern models.
  3. It is strongly recommended to compare the proposed method over more recent works with similar settings. See below: Chi Zhang et al., Closed-Form Concept Erasure via Double Projections, CVPR 2026 Korada Sri Vardhana, Soma Biswas, GenErase: Generalizable and Semantically-Aware Concept Erasure in Diffusion Models, CVPR 2026 Zhuan Shi et al., Neighbor-Aware Localized Concept Erasure in Text-to-Image Diffusion Models, CVPR 2026 Yuze Cai et al., Prototype-Guided Concept Erasure in Diffusion Models, CVPR 2026 Hoigi Seo et al., Erasing Thousands of Concepts: Towards Scalable and Practical Concept Erasure for Text-to-Image Diffusion Models, CVPR 2026
Questions for authors

While the formulation seems neat, similar ideas have been explored in different forms. Thus, it is important to clearly justify the novelty of this work theoretically and empirically. It is especially important to outperform recent works in erasing, preservation and robustness to attacks. See the above points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.