- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes CASE, a training-free, closed-form weight editing method for concept unlearning in diffusion models. By utilizing SVD on cross-attention activations to construct subspaces, it derives a retain-constrained linear editing operator merged into Key/Value weights for targeted suppression. Evaluated on SD 1.5 across Style, IP, Celebrity, and NSFW tasks with comprehensive metrics and ablations, the method demonstrates the empirical effort.
Critical strengths
-
An efficient and lightweight method for concept unlearning. CASE achieves concept unlearning by collecting internal activations, constructing subspaces, and performing closed-form weight editing. It eliminates the need for gradient-based fine-tuning and avoids introducing additional inference-time modules, thereby significantly reducing computational costs. This design offers a practical and effective paradigm for concept editing in diffusion models.
-
Comprehensive evaluation protocol and metrics. The paper systematically evaluates target concept suppression, adjacent concept retention, robustness to adversarial prompts, and overall generation quality. This multi-dimensional analysis, supported by rigorous metrics, goes beyond simple qualitative sample showcases, thereby clearly delineating the strengths and limitations of the proposed method.
Critical weaknesses
-
Limited algorithmic novelty. The proposed method essentially modifies only the cross-attention key/value weights via a closed-form projection. While the use of activation-space subspaces is empirically motivated, the algorithmic contribution is largely a straightforward adaptation of existing closed-form editing ideas. The paper lacks deeper novelty and in-depth exploration regarding the choice of editing angle.
-
Moreover, several compared baselines (e.g., UCE, ESD) were proposed relatively early, and the paper does not adequately compare against more recent state-of-the-art methods, which weakens the justification of the claimed advances.
Questions for authors
see the weakness and the following suggestions: 1: This paper uses the same projection operator to simultaneously modify the Key and Value weights in cross-attention. How do the authors demonstrate that this joint editing effectively suppresses the target concept while preserving non-target concepts? ablation studies is promised to be conducted by editing only Key, only Value.
2: For concepts such as Celebrity, the paper relies primarily on specific detectors' hit rates. We suggest the authors verify cross-detector consistency and conduct human blind evaluations to confirm the actual perceptual unlearning. The authors should also explicitly define the unlearning effect, especially when detector judgments conflict with human perception.
3 comments
Q1. Joint Key/Value Editing
We thank the reviewer for this question. We apply the editing operator to both Key and Value projections, which affect attention allocation and the information aggregated through attention, respectively. To examine their individual contributions, we conducted the following ablation.
Table R4. Key/Value editing ablation on SD1.5, averaged over three celebrity concepts. Target proportion and retention are reported in %.
| Editing | Target ↓ | Retention ↑ |
|---|---|---|
| SD | 57.87 | 51.21 |
| K-only | 57.23 | 52.07 |
| V-only | 8.27 | 51.54 |
| K+V () | 8.07 | 52.34 |
The results show that Value editing accounts for most of the observed target suppression, whereas K-only editing has little effect. Joint K/V editing achieves similar suppression and retention to V-only editing under the evaluated setting. These results provide an empirical characterization of the respective contributions of Key and Value editing. We have added the Key/Value editing ablation analysis to Appendix D of the revised manuscript.
Q2. Evaluation Reliability and Unlearning Definition
We thank the reviewer for raising this concern. We define concept unlearning operationally as suppressing target-concept generation while preserving non-target generation capabilities. Detector-based metrics serve as proxies for this objective rather than definitive evidence of perceptual unlearning.
To assess cross-detector consistency, we evaluated the same generated images using two different detectors for each category. For artistic styles and IP characters, we used InternVL2.5-8B-MPO (original detector)[1] and Qwen2.5-VL-7B-Instruct (additional detector)[2]. For celebrities, we used GIPHY Celebrity Detector[3] and ArcFace[4], respectively.
Table R5. Cross-detector evaluation across three concept categories. A/B denotes the original/additional detector. Target detection rates are reported in %, and Cohen's measures agreement on individual images.
| Category | Detectors (A / B) | SD A / B ↓ | CASE A / B ↓ | SD κ ↑ | CASE κ ↑ |
|---|---|---|---|---|---|
| Style | InternVL2.5 / Qwen2.5-VL | 67.00 / 62.50 | 26.00 / 26.50 | 0.290 | 0.281 |
| IP | InternVL2.5 / Qwen2.5-VL | 86.67 / 84.67 | 8.33 / 7.17 | 0.308 | 0.406 |
| Celebrity | GIPHY / ArcFace | 61.33 / 56.00 | 7.67 / 11.17 | 0.705 | 0.484 |
Both detectors show substantial reductions in target detection rates across all three categories, indicating that the observed suppression is not specific to the original evaluation detector. We have added the cross-detector evaluation results to Appendix C of the revised manuscript.
While human blind evaluation was not conducted, we acknowledge that detector agreement does not guarantee perceptual unlearning. If a target remains recognizable to humans despite a negative detector prediction, we regard it as residual target generation rather than successful perceptual suppression.
- [1] https://huggingface.co/OpenGVLab/InternVL2_5-8B-MPO
- [2] https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct
- [3] https://github.com/Giphy/celeb-detection-oss
- [4] ArcFace: Additive Angular Margin Loss for Deep Face Recognition (CVPR 2019)
Dear Reviewer,
As the rebuttal discussion period is coming to an end, we would greatly appreciate it if you could review our responses and additional experiments. We would be happy to address any remaining questions or concerns.
Thank you very much for your time and consideration!
Dear Reviewer yzEv, Thank you for your detailed review and valuable feedback. We have revised the manuscript and highlighted the changes in blue. Below, we address your comments point by point.
W1. Algorithmic Novelty
We agree that CASE builds on existing closed-form editing techniques. Our main contribution lies in how concept subspaces are constructed for unlearning, rather than in introducing a new projection operator. Existing closed-form methods primarily rely on fixed text embeddings, which do not capture how concepts are expressed across latent states, timesteps, and layers. CASE instead constructs editing subspaces from cross-attention activations collected during denoising.
Our analysis provides evidence for this choice. Figure 2 shows approximately 5× higher recall on held-out prompts using activation-derived bases, while Figure 4 shows a better suppression–retention trade-off as the forget and retain anchor sets vary. These results demonstrate how the choice of concept representation affects both the editing directions and the resulting unlearning performance.
W2. Comparison with Recent Methods
We thank the reviewer for this comment. While UCE and ESD are earlier representative baselines, our evaluation also includes recent methods such as CURE (NeurIPS 2025), AdaVD (CVPR 2025), and SAFREE (ICLR 2025). These methods cover different approaches to concept unlearning, including closed-form weight editing and inference-time interventions.
As shown in Table 1, CASE achieves the highest H-Mean across all four concept categories among the evaluated methods, considering target suppression, retention, adversarial robustness, and generation quality. We further compare CASE with EraseAnything on FLUX.1-dev, where CASE achieves lower target detection and RAB attack success rates across all three evaluated IP concepts (please see our response to Reviewer tYem, W2).