acceptodds
Under review as a conference paper at ICLR 2027

Concept Unlearning via Cross-Attention Activation Subspaces in Diffusion Models

Abstract

Existing closed-form methods for concept unlearning in text-to-image diffusion models typically derive editing directions from fixed text embeddings, which may not fully capture how concepts are expressed across latent states, timesteps, and layers. To capture this variation, we investigate cross-attention activations collected during denoising. In controlled probing experiments using the same anchor prompts, activation-derived bases achieve approximately five times the recall of text-derived bases on held-out prompts expressing the target concepts. Based on this finding, we propose Cross-Attention Subspace Erasure (CASE), a closed-form method that constructs layer-specific forget and retain subspaces from cross-attention activations. These subspaces define a retain-constrained linear operator incorporated directly into cross-attention weights, requiring no gradient-based fine-tuning or additional inference-time computation. Across ten concepts spanning four categories, CASE achieves the highest harmonic-mean score among evaluated baselines in all four categories, balancing suppression, retention, adversarial robustness, and generation quality. Further experiments demonstrate robustness to recovery attacks and a favorable suppression-retention trade-off when jointly unlearning up to 100 artistic styles. The benefits of activation-derived editing also extend to larger diffusion models, including SDXL and FLUX.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Concept Unlearning, Diffusion Models, Model Editing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yzEvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CASE, a training-free, closed-form weight editing method for concept unlearning in diffusion models. By utilizing SVD on cross-attention activations to construct subspaces, it derives a retain-constrained linear editing operator merged into Key/Value weights for targeted suppression. Evaluated on SD 1.5 across Style, IP, Celebrity, and NSFW tasks with comprehensive metrics and ablations, the method demonstrates the empirical effort.

Critical strengths
  1. An efficient and lightweight method for concept unlearning. CASE achieves concept unlearning by collecting internal activations, constructing subspaces, and performing closed-form weight editing. It eliminates the need for gradient-based fine-tuning and avoids introducing additional inference-time modules, thereby significantly reducing computational costs. This design offers a practical and effective paradigm for concept editing in diffusion models.

  2. Comprehensive evaluation protocol and metrics. The paper systematically evaluates target concept suppression, adjacent concept retention, robustness to adversarial prompts, and overall generation quality. This multi-dimensional analysis, supported by rigorous metrics, goes beyond simple qualitative sample showcases, thereby clearly delineating the strengths and limitations of the proposed method.

Critical weaknesses
  1. Limited algorithmic novelty. The proposed method essentially modifies only the cross-attention key/value weights via a closed-form projection. While the use of activation-space subspaces is empirically motivated, the algorithmic contribution is largely a straightforward adaptation of existing closed-form editing ideas. The paper lacks deeper novelty and in-depth exploration regarding the choice of editing angle.

  2. Moreover, several compared baselines (e.g., UCE, ESD) were proposed relatively early, and the paper does not adequately compare against more recent state-of-the-art methods, which weakens the justification of the claimed advances.

Questions for authors

see the weakness and the following suggestions: 1: This paper uses the same projection operator to simultaneously modify the Key and Value weights in cross-attention. How do the authors demonstrate that this joint editing effectively suppresses the target concept while preserving non-target concepts? ablation studies is promised to be conducted by editing only Key, only Value.

2: For concepts such as Celebrity, the paper relies primarily on specific detectors' hit rates. We suggest the authors verify cross-detector consistency and conduct human blind evaluations to confirm the actual perceptual unlearning. The authors should also explicitly define the unlearning effect, especially when detector judgments conflict with human perception.

3 comments

Dear Reviewer yzEv, Thank you for your detailed review and valuable feedback. We have revised the manuscript and highlighted the changes in blue\color{#1E40AF}{\text{blue}}. Below, we address your comments point by point.


W1. Algorithmic Novelty

We agree that CASE\mathtt{CASE} builds on existing closed-form editing techniques. Our main contribution lies in how concept subspaces are constructed for unlearning, rather than in introducing a new projection operator. Existing closed-form methods primarily rely on fixed text embeddings, which do not capture how concepts are expressed across latent states, timesteps, and layers. CASE\mathtt{CASE} instead constructs editing subspaces from cross-attention activations collected during denoising.

Our analysis provides evidence for this choice. Figure 2 shows approximately 5×5\times higher recall on held-out prompts using activation-derived bases, while Figure 4 shows a better suppression–retention trade-off as the forget and retain anchor sets vary. These results demonstrate how the choice of concept representation affects both the editing directions and the resulting unlearning performance.

W2. Comparison with Recent Methods

We thank the reviewer for this comment. While UCE\mathtt{UCE} and ESD\mathtt{ESD} are earlier representative baselines, our evaluation also includes recent methods such as CURE\mathtt{CURE} (NeurIPS 2025), AdaVD\mathtt{AdaVD} (CVPR 2025), and SAFREE\mathtt{SAFREE} (ICLR 2025). These methods cover different approaches to concept unlearning, including closed-form weight editing and inference-time interventions.

As shown in Table 1, CASE\mathtt{CASE} achieves the highest H-Mean across all four concept categories among the evaluated methods, considering target suppression, retention, adversarial robustness, and generation quality. We further compare CASE\mathtt{CASE} with EraseAnything\mathtt{EraseAnything} on FLUX.1-dev, where CASE\mathtt{CASE} achieves lower target detection and RAB attack success rates across all three evaluated IP concepts (please see our response to Reviewer tYem, W2).

Q1. Joint Key/Value Editing

We thank the reviewer for this question. We apply the editing operator to both Key and Value projections, which affect attention allocation and the information aggregated through attention, respectively. To examine their individual contributions, we conducted the following ablation.

Table R4. Key/Value editing ablation on SD1.5, averaged over three celebrity concepts. Target proportion and retention are reported in %.

EditingTarget ↓Retention ↑
SD57.8751.21
K-only57.2352.07
V-only8.2751.54
K+V (CASE\mathtt{CASE})8.0752.34

The results show that Value editing accounts for most of the observed target suppression, whereas K-only editing has little effect. Joint K/V editing achieves similar suppression and retention to V-only editing under the evaluated setting. These results provide an empirical characterization of the respective contributions of Key and Value editing. We have added the Key/Value editing ablation analysis to Appendix D of the revised manuscript.

Q2. Evaluation Reliability and Unlearning Definition

We thank the reviewer for raising this concern. We define concept unlearning operationally as suppressing target-concept generation while preserving non-target generation capabilities. Detector-based metrics serve as proxies for this objective rather than definitive evidence of perceptual unlearning.

To assess cross-detector consistency, we evaluated the same generated images using two different detectors for each category. For artistic styles and IP characters, we used InternVL2.5-8B-MPO (original detector)[1] and Qwen2.5-VL-7B-Instruct (additional detector)[2]. For celebrities, we used GIPHY Celebrity Detector[3] and ArcFace[4], respectively.

Table R5. Cross-detector evaluation across three concept categories. A/B denotes the original/additional detector. Target detection rates are reported in %, and Cohen's κ\kappa measures agreement on individual images.

CategoryDetectors (A / B)SD A / B ↓CASE A / B ↓SD κ ↑CASE κ ↑
StyleInternVL2.5 / Qwen2.5-VL67.00 / 62.5026.00 / 26.500.2900.281
IPInternVL2.5 / Qwen2.5-VL86.67 / 84.678.33 / 7.170.3080.406
CelebrityGIPHY / ArcFace61.33 / 56.007.67 / 11.170.7050.484

Both detectors show substantial reductions in target detection rates across all three categories, indicating that the observed suppression is not specific to the original evaluation detector. We have added the cross-detector evaluation results to Appendix C of the revised manuscript.

While human blind evaluation was not conducted, we acknowledge that detector agreement does not guarantee perceptual unlearning. If a target remains recognizable to humans despite a negative detector prediction, we regard it as residual target generation rather than successful perceptual suppression.

Dear Reviewer,

As the rebuttal discussion period is coming to an end, we would greatly appreciate it if you could review our responses and additional experiments. We would be happy to address any remaining questions or concerns.

Thank you very much for your time and consideration!

Reviewer_5GaeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on concept unlearning for text-to-image diffusion models. The authors propose to use cross-attention activation during denoising and extract layer-specific forget and retain subspaces using SVD. This is different from existing fixed text embedding-based method. Experiments on different aspects demonstrate the effectiveness of the proposed method, including concept suppression, retention, adversarial robustness, multi-concept unlearning, and transfer to larger diffusion architectures.

Critical strengths
  • The paper is well-written and well-motivated. The key idea of using activations collected during denoising is sound. The probing experiments and controlled comparisons with text-derived bases provide evidence that activation-space representations improve concept coverage.

  • Empirical results is strong and experimental analysis is comprehensive. The proposed method achieves the highest H-Mean across all four evaluated concept categories. The additional experiments on adversarial attacks, anchor selection, multi-concept unlearning, SDXL, and FLUX strengthen the empirical evaluation. The closed-form weight update also avoids gradient-based fine-tuning and adds no inference-time computation.

Critical weaknesses
  • Some parts need further justification. For example, the connection between the activation-space operator and weight updates is not clear. The operator is derived from post-attention activations but applied to both key and value projections in Eq. (3). Since modifying keys also changes attention weights, the preservation property may not hold for the edited attention module. An ablation separating key and value edits would help justify this design.
Questions for authors
  • Can the authors report the subspace construction time and memory cost relative to other closed-form methods, including the cost of collecting denoising trajectories?

  • How much of the improvement comes from activation-derived bases vs. layer-specific editing? Can the authors provide a comparison using a shared activation-derived basis across layers to isolate these factors?

3 comments

Dear Reviewer 5Gae, Thank you for your positive assessment and thoughtful suggestions. We have revised the manuscript and highlighted the changes in blue\color{#1E40AF}{\text{blue}}. Below, we address your comments point by point.


W1. Joint Key/Value Editing and Retention

We thank the reviewer for raising this point. We originally adopted joint K/V editing to modify both attention allocation through Keys and the information aggregated through Values. However, we agree that the retain-subspace preservation property of the editing operator does not guarantee exact preservation of the full attention output. To better understand the contributions of Key and Value editing, we conducted the following ablation.

Table R2. Key/Value editing ablation on SD1.5, averaged over three celebrity concepts. Target proportion and retention are reported in %.

EditingTarget ↓Retention ↑
SD57.8751.21
K-only57.2352.07
V-only8.2751.54
K+V (CASE\mathtt{CASE})8.0752.34

The results show that Value editing accounts for most of the target suppression, while K-only editing has little effect. Joint K/V editing provides a modest additional improvement in both suppression and retention. These findings suggest that Value editing plays the primary role in the observed unlearning performance, with a smaller contribution from Key editing. We have added the Key/Value editing ablation analysis to Appendix D of the revised manuscript.

Q1. Computational Cost

To address this question, we measure the end-to-end editing time, including activation collection, subspace construction, and weight modification, along with peak GPU memory and inference latency. We evaluate three celebrity concepts with three repetitions each (nine paired runs) on an NVIDIA RTX 3090, rotating method order and using a fresh process for each run.

Table R3. Computational cost comparison. Results are reported as mean ±\pm standard deviation across nine runs.

MethodEdit time (s)Peak memory (GiB)Inference (s/image)
SD\mathtt{SD}–-2.39 ±\pm 0.05
UCE\mathtt{UCE}0.82 ±\pm 0.124.18 ±\pm <0.012.38 ±\pm 0.06
CURE\mathtt{CURE}0.82 ±\pm 0.164.08 ±\pm 0.012.39 ±\pm 0.05
CASE\mathtt{CASE}495.63 ±\pm 11.644.35 ±\pm <0.012.41 ±\pm 0.04

CASE\mathtt{CASE} incurs a substantially higher editing cost than UCE and CURE due to its activation-based subspace construction. However, this overhead is confined to offline editing, with no additional computation during inference. Consequently, CASE\mathtt{CASE} maintains inference latency comparable to the original SD model, while requiring only modest additional GPU memory during editing. We have added the computational cost analysis to Appendix B of the revised manuscript.

Q2. Shared vs. Layer-specific Activation Bases

We would like to clarify that CASE\mathtt{CASE} constructs activation bases separately for each cross-attention layer. Unlike text embeddings, which share a common representation space, cross-attention activations depend on the queries and projection weights of each layer (Eq. 1). Since their feature dimensions vary across layers, a single shared activation basis cannot be directly applied to all layers.

Dear Reviewer,

As the rebuttal discussion period is coming to an end, we would greatly appreciate it if you could review our responses and additional experiments. We would be happy to address any remaining questions or concerns.

Thank you very much for your time and consideration!

Thank you to the authors for their rebuttal, which has improved the paper and largely addressed my concerns. I will take these into account during the subsequent discussion period.

Reviewer_tYemOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

the paper presents a novel closer-form concept unlearning method for text-to-image models. In comparison to the previous closed-form solutions that consider only text-space basis while the proposed method uses activation-space basis. To build the activation subspace authors define the forget set and the retain set for the target concept, gather latents for each sample from both sets and later use the SVD-based weight projection similar to CURE method, which did it for the text-space.

The evaluation is done on the majority of the unlearning benchmarks: HUB, Style, IP, Celebrity, and NSFW.

Critical strengths

A simple, training-free method with consistent overall results across categories, scale and attack types.

Critical weaknesses

The main evaluation is done on SD1.4 while the newer models are only lightly examined. There is no robustness evaluation, no FID, and no comparison with recent flow matching (FLUX) -oriented methods such as EraseAnything. The paper also does not explain how CASE is applied to FLUX's joint attention blocks. Given the limited novelty (CURE's operator with a different basis), the paper contribution would benefit from the extensive evaluation of never models and methods made for them. if FLUX.1 experiments are too computationally heavy for the rebuttal period, FLUX.2-klein-base could be used instead.

Questions for authors
  1. Why is only limited FLUX evaluation included?
3 comments

Dear Reviewer tYem, Thank you for your careful review and constructive feedback. We have revised the manuscript and highlighted the changes in blue\color{#1E40AF}{\text{blue}}. Below, we address your comments point by point.


W1. CASE Implementation on FLUX

We clarify how CASE is applied to FLUX.1-dev's joint attention architecture. In Stable Diffusion, image queries attend only to text keys and values:

hSD=softmax⁡(QimgKtxt⊤d)Vtxt.h_{\mathrm{SD}} = \operatorname{softmax}\left(\frac{Q_{\mathrm{img}}K_{\mathrm{txt}}^\top}{\sqrt{d}}\right)V_{\mathrm{txt}}.

In contrast, FLUX allows image queries to attend to both text and image tokens:

hFLUX=softmax⁡(Qimg[Ktxt;Kimg]⊤d)[Vtxt\Vimg]=Aimg,txtVtxt+Aimg,imgVimg.h_{\mathrm{FLUX}} = \operatorname{softmax}\left(\frac{Q_{\mathrm{img}}[K_{\mathrm{txt}};K_{\mathrm{img}}]^\top}{\sqrt{d}}\right)\begin{bmatrix}V_{\mathrm{txt}}\\\V_{\mathrm{img}}\end{bmatrix} = A_{\mathrm{img,txt}}V_{\mathrm{txt}} + A_{\mathrm{img,img}}V_{\mathrm{img}}.

Following the original CASE\mathtt{CASE} formulation, we use Aimg,txtVtxtA_{\mathrm{img,txt}}V_{\mathrm{txt}} to construct the activation basis for each block. This captures the contribution of text values to image-query activations while retaining their dependence on image features. It also corresponds to the text K/V projections that CASE\mathtt{CASE} edits. We then construct the editing operators using the same SVD-based procedure as in Stable Diffusion.

For weight editing, FLUX has two types of attention blocks:

  • Double-stream blocks: Text and image tokens have separate K/V projections, so we directly edit the text K/V weights.
  • Single-stream blocks: Text and image tokens share the same K/V projections. We therefore use a custom attention processor that applies the edited K/V weights to text tokens while keeping the original weights for image tokens.

We have added the FLUX implementation details to Appendix A.2 of the revised manuscript.

W2. Additional Evaluation on FLUX.1-dev

To address the limited evaluation on newer architectures, we conducted additional experiments on FLUX.1-dev, including a comparison with EraseAnything\mathtt{EraseAnything}, a recent method for flow-based models. We evaluate target suppression, within-category retention, adversarial robustness under Ring-A-Bell (RAB), and FID-5k across three IP concepts. We used the official implementation of EraseAnything\mathtt{EraseAnything} [1] and followed the training configuration reported in the paper.

Table R1. Concept-wise evaluation on FLUX.1-dev. Target proportion, retention, and attack success rates are reported in %. H-Mean summarizes performance across the four metrics.

ConceptMethodTarget proportion ↓Retention ↑Attack ↓Quality ↓H-Mean ↑
Buzz LightyearFLUX.1-dev86.8084.4948.5027.810.272
EraseAnything\mathtt{EraseAnything}2.6083.792.2030.280.514
CASE\mathtt{CASE}1.2073.120.5028.320.533
Mickey MouseFLUX.1-dev93.6083.7385.8027.810.143
EraseAnything\mathtt{EraseAnything}87.1081.4775.9028.410.232
CASE\mathtt{CASE}2.2071.410.9029.450.513
PikachuFLUX.1-dev95.0083.5892.7027.810.103
EraseAnything\mathtt{EraseAnything}94.7083.3092.4030.460.106
CASE\mathtt{CASE}11.3073.599.0027.610.529
AverageFLUX.1-dev91.8083.9375.6727.810.186
EraseAnything\mathtt{EraseAnything}61.4782.8556.8329.720.380
CASE\mathtt{CASE}4.9072.713.4728.460.525

As shown in Table R1, CASE\mathtt{CASE} achieves the highest H-Mean across all three concepts, with substantially lower target proportion and attack success rates. In contrast, EraseAnything\mathtt{EraseAnything} effectively suppresses Buzz Lightyear but shows limited effectiveness on Mickey Mouse and Pikachu. These results demonstrate the effectiveness of CASE\mathtt{CASE} on FLUX.1-dev compared with a method specifically designed for flow-based models. We have added this comparison to Section 4.3 of the revised manuscript

[1] https://github.com/tomguluson92/EraseAnything

W3. Novelty Relative to CURE

We agree that CASE\mathtt{CASE} shares the closed-form spectral editing framework of CURE\mathtt{CURE}. Our main contribution lies in how target concepts are identified and represented for unlearning. Existing closed-form methods primarily rely on fixed text embeddings, which may not fully capture how concepts are expressed across latent states, timesteps, and layers. CASE\mathtt{CASE} instead constructs concept subspaces from cross-attention activations collected during denoising.

Our analysis provides evidence for this choice. Figure 2 shows approximately 5×5\times higher recall on held-out prompts using activation-derived bases, while Figure 4 shows a better suppression–retention trade-off as the forget and retain anchor sets vary. Together, these results show that activation-based concept representations improve both coverage of unseen prompts and the effectiveness of closed-form unlearning.

Dear Reviewer,

As the rebuttal discussion period is coming to an end, we would greatly appreciate it if you could review our responses and additional experiments. We would be happy to address any remaining questions or concerns.

Thank you very much for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.