acceptodds
Under review as a conference paper at ICLR 2027

CRET: Target-Selective Subspace Learning for Concept Erasure in Text-to-Image Diffusion Models

Abstract

Text-to-image diffusion models have demonstrated remarkable capabilities in visual generation, but their widespread use also raises concerns about the generation of unsafe content. Concept erasure aims to suppress designated undesirable concepts while preserving non-target semantics. Projection-based methods provide interpretable intervention by removing target-related representation components. Their projection subspaces are commonly constructed directly from selected embeddings. However, for complex concepts, target-related and benign representations can share certain components, making it difficult for such subspaces to separate the information to be edited from that to be preserved, resulting in incomplete target-concept erasure or suppression of benign information. To overcome this limitation, we propose CRET, a representation-level concept erasure framework that learns an editable subspace rather than directly constructing it from selected embeddings. This subspace is parameterized by a rank-constrained orthogonal projector. In particular, CRET optimizes the projector to selectively capture target-related information from controlled risky-safe prompt pairs while limiting the projections of safe and preservation representations onto the editable subspace. The projector is learned in a latent feature space of text representations, and the induced change is transferred back to the original representation without updating the pretrained diffusion model weights. Experiments on multiple adversarial benchmarks demonstrate that CRET achieves more effective target-concept erasure than competing methods while maintaining competitive generation quality on benign prompts.

Primary area:
generative models
Keywords:
concept erasure, text-to-image diffusion models, subspace learning, orthogonal projection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pnkMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CRET, a representation-level concept erasure method for text-to-image diffusion models. The method freezes the parameters of the pretrained text encoder and diffusion model, first mapping text representations into a low-dimensional latent space through an encoder–decoder architecture, and then learning a rank-constrained orthogonal concept subspace. This subspace captures the target concept by exploiting the differences between risky and safe prompts, while suppressing the projections of safe and preservation representations onto the editable subspace. During inference, CRET removes target-related components and transfers the corresponding residual changes back to the original text representations.

Experiments are conducted primarily on Stable Diffusion models, covering standard prompts, multiple adversarial attack benchmarks, and general image generation on COCO. The results show that CRET suppresses target concepts more effectively than several existing concept erasure methods while maintaining competitive generation quality on benign prompts. Experiments on SDXL further demonstrate its potential for cross-model applicability. Ablation studies on projection construction, loss components, projection rank, intervention scope, and representation mapping provide additional evidence for the effectiveness of the learned target subspace, the preservation of benign information, and the post-decoding preservation constraint. The paper also presents qualitative results involving violence, ordinary objects, and artistic styles to illustrate the method's potential applicability across diverse concepts.

Critical strengths
  1. CRET obtains the lowest mean adversarial ASR among the listed methods and substantially outperforms competing model-preserving methods on Ring-A-Bell, MMA-Diffusion, and UnlearnDiffAtk. This is consequential because robustness to prompt attacks is central to the paper’s stated goal.

  2. The target-inclusion, benign-exclusion, and decoded-preservation objectives directly encode the desired safety–utility tradeoff. The ablations support their functional roles, especially the large degradation in erasure after removing the target-inclusion term and the large FID degradation after removing benign exclusion. The rank-matched comparison against direct SVD, QR-span, and preserve-constrained SVD isolates the learned projector as the main source of improvement. This strengthens the methodological claim.

Critical weaknesses
  1. The paper’s quantitative validation is largely limited to nudity and violence. Object erasure and style erasure are demonstrated only through qualitative examples, despite the paper claiming applicability beyond unsafe concepts. Without quantitative target-erasure and preservation metrics for these settings, it is difficult to determine whether CRET generalizes beyond the primary safety benchmarks or whether the visual examples are representative.

  2. The scalability of CRET to multi-concept erasure is not evaluated. The experiments appear to learn a concept-specific projector for one target at a time, leaving unanswered whether a single projector can capture several target concepts or whether multiple learned projectors can be composed without causing interference and excessive degradation of benign generation.

Questions for authors
  1. Can the authors provide quantitative results for objects and styles erasure, including target-recognition success, preservation of unrelated objects and generation quality?

  2. Is CRET trained separately for each target concept in all experiments? Have the authors tested multi-concept joint training or composition of multiple target projectors, and how does this affect attack success, preservation quality, and computational cost?

5 comments
AuthorsOpenReview ↗

Response to Reviewer pnkM(1/2)

We appreciate the reviewer's thoughtful comments and helpful suggestions. Your concerns about generalization beyond unsafe concepts and multi-concept erasure motivated additional quantitative evaluations. We present these results below and have incorporated them into the revised manuscript to better clarify the scope and scalability of CRET.

[W1 , Q1]: quantitative results for objects and styles erasure

Thank you for raising the question of CRET's applicability beyond unsafe concepts. To provide quantitative evidence in this setting, we evaluate object erasure separately on ten categories from an ImageNet subset, treating one category as the target and using the other nine to assess preservation in each experiment. Specifically, we use a pretrained ResNet-50 for top-1 classification and report the target-category recognition rate on generated images and the average decrease in non-target recognition rate relative to the original model. Lower values indicate more effective target suppression and better non-target preservation, respectively.

CRET reduces the average target-category recognition rate from 85.6% to 1.5%, close to OCE (1.8%) and ScaPre (2.4%), while the average non-target recognition rate decreases by only 4.5 percentage points (Tables R1 and R2). On the ten evaluated object categories, these results show that CRET effectively suppresses the target concept while retaining most of the recognition performance on non-target categories, providing quantitative support for its applicability beyond unsafe concepts.

Table R1. Target recognition after object erasure (left half of Table 3 in the revised PDF).

ObjectOriginalScaPreOCEUCESPEEDCRET
Cassette Player764512144
Chain Saw7910000
Church83834203
Gas Pump7811640
Tench75000140
Garbage Truck7710072
English Springer9302060
Golf Ball996756416
Parachute96300120
French Horn10000010
Mean85.62.41.87.811.91.5

Table R2. Non-target preservation after object erasure (right half of Table 3 in the revised PDF).

ObjectOriginalScaPreOCEUCESPEEDCRET
Cassette Player86.714.81.620.815
Chain Saw86.314.65.80.62.25.6
Church85.913.71.215.12.76.7
Gas Pump86.41235.62.47.2
Tench86.818.42.49.92.86.2
Garbage Truck86.611.82.60.13.81.2
English Springer84.84.93.1030.5
Golf Ball84.131.96.23.41.4
Parachute84.47.82.442.80.1
French Horn8410.85.74.73.711.1
Mean85.611.23.06.72.84.5

For style erasure, we separately erase the styles of Van Gogh and Picasso. We evaluate style suppression using CLIP Score (CS), the average CLIP matching score between generated images and their corresponding target-style prompts. A lower CS indicates more effective style suppression; it is not a classification accuracy. To assess preservation, we report FID, which measures how closely the generated image distribution matches that of the original model, and CLIP, which measures how well generated images retain the semantics specified by the content prompts.

As shown in Table R3, CRET achieves a Van Gogh CS of 23.26, close to OCE (23.89) and Receler (22.09), and the lowest Picasso CS of 23.86 among the compared methods. Compared with OCE, CRET also obtains lower FID for both styles, although its CLIP scores are slightly lower. These comparisons indicate competitive style suppression, with a trade-off between distributional preservation and content alignment.

Table R3. Style erasure and preservation (Table 13, App. B.6 of the revised PDF).

methodvangogh cs ↓vangogh fid ↓vangogh clip ↑picasso cs ↓picasso fid ↓picasso clip ↑
sd1-432.58--30.63--
oce23.8945.3731.124.1545.8631.1
DP25.7835.5131.325.8235.3831.1
SPEED25.1345.4631.226.3245.8531.2
Receler22.0946.1730.224.6745.9130.4
Scapre25.4842.8331.326.5641.5831.2
CRET23.2642.0330.323.8641.7530.5
AuthorsOpenReview ↗

Response to Reviewer pnkM(2/2)

[W2 , Q2]: Multi-Concept Erasure

Thank you for this suggestion. In all original experiments, CRET was trained separately for each target concept. To evaluate multi-concept erasure, we added two compositions of independently trained target projectors, applied in the orders nudity then violence and violence then nudity, as well as joint training on both concepts. Compared with OCE, DP, and ScaPre, all three CRET configurations achieve lower P4D, UDA, and Q16 ASRs and match OCE on RAB ASR, but have slightly lower CLIP scores and longer erasure preparation times. Among the CRET configurations, sequential composition gives lower P4D and Q16 ASRs, whereas joint training provides better FID and CLIP scores and reduces inference time from 1.2 to 0.8. These results illustrate the trade-off between suppression effectiveness, generation quality, and inference efficiency (Tables R4–R6).

Q16 ASR is the proportion of images generated on the I2P violence subset that Q16 classifies as inappropriate; lower is better. Across the three CRET configurations, Q16 ASR ranges from 5.29% to 9.25%, below OCE's 13.1%, the lowest among the compared baselines (Table R5).

Table R4. Multi-concept erasure and generation quality (Table 4 of the revised PDF).

MethodRAB ASR ↓P4D ASR ↓UDA ASR ↓FID ↓CLIP ↑
OCE0.010.1390.04246.8331.0
DP0.2410.4530.12737.6731.2
ScaPre0.1650.3580.07843.1830.5
CRET: Sequential composition (N → V)0.010.0930.01448.5629.5
CRET: Sequential composition (V → N)0.010.0660.00750.1029.0
CRET: Joint training (N + V)0.010.1190.01445.7829.8

Table R5. Q16 ASR for multi-concept erasure (Table 4 of the revised PDF).

MethodQ16 ASR (%) ↓
OCE13.1
DP16.0
ScaPre20.5
CRET: Sequential composition (N → V)7.93
CRET: Sequential composition (V → N)5.29
CRET: Joint training (N + V)9.25

Table R6. Multi-concept erasure computation time (Table 14, App. B.7 of the revised PDF).

MethodEdit time ↓Infer time ↓
OCE2.481.1
DP4.001.1
ScaPre23.140.8
CRET: Sequential composition (N → V)571.2
CRET: Sequential composition (V → N)571.2
CRET: Joint training (N + V)580.8

N denotes nudity and V denotes violence.

Reviewer_pnkMOpenReview ↗

Response to the Rebuttal

Thank you for your detailed responses and the effort devoted to the additional experiments and clarifications. I will also carefully consider the other reviewers’ comments to identify any aspects I may have overlooked and support a balanced assessment. Thank you again for your time and effort.

Reviewer_AmJROpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CRET, a representation-level concept erasure framework for text-to-image diffusion models that learns a target-selective orthogonal subspace. The method employs a KAN-based encoder–decoder to construct a latent representation space, where a learnable low-rank projector is optimized using target inclusion, benign exclusion, and decoded preservation objectives. Experiments demonstrate strong concept erasure performance across adversarial benchmarks while maintaining competitive generation quality on benign prompts.

Critical strengths
  1. The paper addresses the challenge of separating target-related information from overlapping benign semantics, providing a meaningful motivation for learning editable subspaces rather than directly constructing them from embeddings.
  2. The combination of a learnable orthogonal projector with target inclusion, benign exclusion, and decoded preservation objectives provides a coherent and interpretable framework for selective concept intervention.
  3. CRET achieves the lowest mean adversarial ASR (0.027) among the compared methods across four adversarial benchmarks, while maintaining competitive benign generation quality.
Critical weaknesses
  1. Although CRET trains an auxiliary encoder–decoder and a concept-specific projector, the pretrained diffusion model remains unchanged. The paper should more clearly distinguish inference-time concept suppression from model-level erasure or unlearning.
  2. The paper lacks comparisons with recent concept erasure methods, including Receler [1], SPEED [2], ScaPre [3], DP [4], and OCE [5], making it difficult to assess its advantages over existing approaches.
  3. The paper lacks computational efficiency comparisons. Since CRET requires additional encoder–decoder and projector training, as well as inference-time computation, its practical efficiency remains unclear.
  4. The evaluation is limited to SD1.4 and SDXL, leaving the applicability of CRET to modern flow-based models such as FLUX.1-dev unclear.
  5. Recent baselines quantitatively evaluate at least 10 object categories, raising concerns about CRET's generalizability. However, CRET evaluated only qualitatively on a single target (cassette player).
  6. The paper lacks simultaneous multi-concept erasure experiments, leaving potential interference between learned concept subspaces unexplored.
  7. The projector is trained on EOT representations but applied to both EOT and content-token representations during inference. This training–inference mismatch is insufficiently justified, particularly since content-token intervention contributes most of the erasure effectiveness.

[1] Receler: https://arxiv.org/abs/2311.17717 (ECCV 2024)
[1] SPEED: https://arxiv.org/abs/2503.07392 (ICLR 2026)
[2] ScaPre: https://arxiv.org/abs/2601.06162 (ICLR 2026)
[3] DP: https://arxiv.org/abs/2604.10032 (CVPR 2026)
[4] OCE: https://arxiv.org/abs/2605.28902 (ICML 2026)

Questions for authors
  1. Can the authors evaluate object-concept erasure on at least 10 categories from COCO, ImageNet, or CIFAR-10, reporting both erasure effectiveness and non-target preservation?
  2. Can the authors evaluate simultaneous multi-concept erasure (e.g., nudity and violence) and investigate potential subspace interference and preservation degradation?
  3. Can the authors extend CRET to FLUX.1-dev to evaluate its generalizability across different generative architectures?
  4. Can the authors include sparse autoencoders (SAEs) in the existing ablation study (Table 6) to assess whether sparse representations improve target–benign separation compared with KAN?
  5. Can the authors report training time, inference latency, and GPU memory usage compared with representative concept erasure methods?
  6. Given the learned target-selective representations, could the authors explore fine-tuning or distilling CRET into the pretrained diffusion model for persistent concept erasure?
  7. Can the authors compare EOT-only, content-token-only, and joint projector training to justify applying an EOT-trained projector to content-token representations?
9 comments
AuthorsOpenReview ↗

Response to Reviewer AmJR(1/4)

We sincerely thank the reviewer for the detailed assessment and constructive suggestions. Your feedback has helped us clarify CRET's inference-time formulation and broaden the evaluation of its applicability, design choices, and computational efficiency. Below, we address each concern and describe the corresponding additions and clarifications in the revised manuscript.

[W1]: Inference-Time Suppression versus Model-Level Unlearning

Thank you for highlighting this distinction. Your understanding is correct: CRET trains an auxiliary encoder–decoder and a concept-specific projector while leaving the pretrained text encoder and diffusion model unchanged. Its original formulation suppresses target concepts through inference-time interventions on prompt representations, rather than through parameter-level unlearning. We have made three targeted revisions to clarify this scope:

  1. Introduction, the paragraph beginning “In this paper, we propose CRET”: We revised the opening description of CRET to explicitly identify it as an inference-time concept-suppression framework, while retaining the description of its learned rank-constrained orthogonal projector.

  2. Method, §3.1, the first sentence of the Overview: We added “inference-time” to the description of concept erasure. The sentence now reads: “CRET performs inference-time concept erasure in text representations without altering the parameters of the pretrained text encoder or diffusion model.” This explicitly identifies when the intervention operates and which pretrained parameters remain unchanged.

  3. Conclusion, the opening sentence: We replaced “for selective concept erasure” with “for target-selective inference-time concept suppression,” while retaining the statement that the pretrained diffusion model remains frozen. This aligns the concluding characterization of CRET with its inference-time intervention mechanism.

[W2]: Comparison with Recent Methods

Thank you for this suggestion. We added comparisons with Receler, SPEED, ScaPre, DP, and OCE. CRET achieves the lowest or joint-lowest ASR on every safety evaluation in Table R1. Its FID of 35.78 is close to DP's 35.53. In addition, Figure 6 in Appendix C provides a qualitative comparison of images generated by CRET and the recent concept-erasure methods.

Table R1. Comparison with recent methods (selected rows of Table 1 in the revised PDF).

I2pP4Drabmmaunlearndiffmeanfidclip
speed0.070.3510.1650.2270.1270.21846.2331.0
receler0.1070.0530.0250.0660.0350.04547.3330.46
dp0.0920.4310.2030.310.2040.28735.5331.27
oce0.0640.0610.0100.0670.0280.04245.7630.8
scapre0.0830.4040.1770.1010.1620.21149.3529.9
CRET0.0340.0460.0100.0390.0140.02735.7830.8

[W3 , Q5]: Computational Efficiency

Thank you for these suggestions. To assess CRET's practical efficiency, we compare erasure preparation time, inference latency, and GPU memory usage with the same recent methods. Regarding computational cost, CRET requires more erasure preparation time than most compared methods, but matches ScaPre for the lowest inference latency and uses the least GPU memory in Table R2, at 4.08 GB. Thus, the additional training incurs preparation overhead while retaining low inference latency and memory usage.

Table R2. Preparation time, inference latency, and GPU memory (Table 17, App. B.10 of the revised PDF).

gpuedit (s)infer
speed7.46GB1.481.2
receler10.31GB32801.8
dp5.11GB4.151.1
oce4.72GB2.031.1
scapre6.07GB22.650.8
CRET4.08GB28.450.8

[W4 , Q3]: Applicability to FLUX

Thank you for this suggestion. We added nudity erasure experiments on FLUX.1-dev, applying CRET to both the CLIP global conditioning and the T5 token conditioning. The nudity generation rates on I2P and MMA decrease from 4.49% and 25.4% for the original model to 1.79% and 1.7%, respectively (Table R3), outperforming the compared methods on both metrics. These results support the feasibility of transferring CRET across generative architectures and demonstrate its concept-suppression capability on FLUX.1-dev.

Table R3. Nudity erasure on FLUX.1-dev (Table 16, App. B.9 of the revised PDF).

I2p(%)MMA(%)CLIP
FLUX4.4925.40.308
ESD2.937.10.307
CA1.833.20.302
UCE3.379.30.309
HIRM3.239.70.308
EraseAnything2.646.60.305
CRET1.791.70.307
AuthorsOpenReview ↗

Response to Reviewer AmJR(2/4)

[W5 , Q1]: Object-Concept Generalizability and Quantitative Evaluation

Thank you for this suggestion. To address the limited coverage of the original object experiments, we expanded the evaluation to ten categories from an ImageNet subset. Each category is treated as the erasure target in turn, and we assess the effect on the remaining nine. Using top-1 predictions from a pretrained ResNet-50, we report the target-category recognition rate and the average decrease in recognition rate across the other categories relative to the original model. Lower is better for both.

CRET reduces the average target-category recognition rate from 85.6% to 1.5%, close to OCE (1.8%) and ScaPre (2.4%), while the average non-target recognition rate decreases by only 4.5 percentage points (Tables R4 and R5). This broader evaluation across individual categories provides quantitative evidence of CRET's erasure and preservation capabilities for object concepts.

Table R4. Target recognition after object erasure (left half of Table 3 in the revised PDF).

ObjectOriginal (concept)ScaPreOCEUCESPEEDCRET
Cassette Player764512144
Chain Saw7910000
Church83834203
Gas Pump7811640
Tench75000140
Garbage Truck7710072
English Springer9302060
Golf Ball996756416
Parachute96300120
French Horn10000010
Mean85.62.41.87.811.91.5

Table R5. Non-target preservation after object erasure (right half of Table 3 in the revised PDF).

ObjectOriginal (preserve)ScaPreOCEUCESPEEDCRET
Cassette Player86.714.81.620.815
Chain Saw86.314.65.80.62.25.6
Church85.913.71.215.12.76.7
Gas Pump86.41235.62.47.2
Tench86.818.42.49.92.86.2
Garbage Truck86.611.82.60.13.81.2
English Springer84.84.93.1030.5
Golf Ball84.131.96.23.41.4
Parachute84.47.82.442.80.1
French Horn8410.85.74.73.711.1
Mean85.611.23.06.72.84.5

[W6 , Q2] Multi-Concept Erasure

Thank you for this suggestion. We added experiments targeting nudity and violence simultaneously, with three configurations: applying separately trained projectors sequentially in either order, and jointly training a single projector on both concepts.

Both sequential compositions achieve lower P4D and Q16 ASRs than joint training (Tables R6 and R8). Joint training, however, provides better FID and CLIP scores, requires similar preparation time, and reduces inference time from 1.2 to 0.8, illustrating the trade-off between target suppression, generation quality, and efficiency (Tables R6 and R7). Relative to OCE, DP, and ScaPre, all three CRET configurations yield lower P4D, UDA, and Q16 ASRs and match OCE on RAB ASR, but have slightly lower CLIP scores and longer erasure preparation times. Here, Q16 ASR measures the proportion of generated images on the I2P violence subset classified as inappropriate; lower is better. CRET achieves 5.29%–9.25%, compared with 13.1% for OCE (Table R8).

Table R6. Multi-concept erasure and generation quality (Table 4 of the revised PDF).

MethodRAB ASR ↓P4D ASR ↓UDA ASR ↓FID ↓CLIP ↑
OCE0.010.1390.04246.8331.0
DP0.2410.4530.12737.6731.2
ScaPre0.1650.3580.07843.1830.5
CRET: Sequential composition (N → V)0.010.0930.01448.5629.5
CRET: Sequential composition (V → N)0.010.0660.00750.1029.0
CRET: Joint training (N + V)0.010.1190.01445.7829.8

Table R7. Multi-concept erasure computation time (Table 14, App. B.7 of the revised PDF).

MethodEdit time ↓Infer time ↓
OCE2.481.1
DP4.001.1
ScaPre23.140.8
CRET: Sequential composition (N → V)571.2
CRET: Sequential composition (V → N)571.2
CRET: Joint training (N + V)580.8

Table R8. Q16 ASR for multi-concept erasure (Table 4 of the revised PDF).

MethodQ16 ASR (%) ↓
OCE13.1
DP16.0
ScaPre20.5
CRET: Sequential composition (N → V)7.93
CRET: Sequential composition (V → N)5.29
CRET: Joint training (N + V)9.25

N denotes nudity and V denotes violence.

AuthorsOpenReview ↗

Response to Reviewer AmJR(3/4)

[W7 , Q7]: EOT Supervision and Content-Token Intervention

Thank you for raising this concern. We address it through two related explanations, each examined with a corresponding ablation experiment.

First, EOT representations provide contextual information for supervising target-subspace learning. Since EOT aggregates the contextual semantics of the whole prompt, EOT differences between paired risky and safe prompts can reflect the semantic change caused by removing the target description. We hypothesize that this supervision captures the target concept's expression in the full prompt context more completely than target-keyword representations alone.

Second, we hypothesize that EOT and content-token representations encode some target-concept information along common directions, allowing an EOT-trained projector to also remove this information from content tokens. A projector trained on EOT differences may consequently identify and remove the corresponding target components in content tokens.

To test whether an EOT-trained projector can also remove target-concept information from content tokens, as our hypothesis predicts, we fix the trained projector and vary only the intervention positions at inference time. Table 7 (App. B.2 of the revised PDF) shows that editing content tokens alone reduces P4D ASR to 8.6%, while editing both content tokens and EOT further reduces it to 4.6%. This demonstrates the projector's effectiveness at content-token positions and is consistent with the hypothesis of partially shared directions encoding target semantics.

To test our hypothesis that EOT differences provide more effective supervision than target-keyword representations alone, we compare the supervision sources while keeping the inference intervention scope fixed. KEY denotes the contextualized content-token representations of target-concept keywords in risky prompts. KEY-only provides target-coverage supervision using these representations, whereas EOT+KEY combines them with paired EOT differences. As shown in Table R9, KEY-only produces a P4D ASR of 14.6% and an FID of 58.36; EOT-only yields 4.6% and 35.78. The corresponding results with joint supervision are 5.3% and 50.32, which likewise do not improve on EOT-only. Under the current experimental setting, EOT-difference supervision therefore provides better erasure and generation quality than the tested keyword supervision.

Together, the intervention-position ablation validates the EOT-trained projector's effectiveness on content tokens, while the supervision ablation supports using EOT differences as the training signal.

Table R9. Training-supervision ablation (Table 8, App. B.2 of the revised PDF).

Training supervisionP4D ASR ↓FID ↓CLIP ↑
KEY-only0.14658.3627.05
EOT-only0.04635.7830.8
EOT+KEY0.05350.3229.31

[Q4]: SAE Ablation

Thank you for this suggestion. We added an SAE to the representation-mapping ablation in Table R10 to evaluate how sparse representations affect target erasure and non-target preservation.

The SAE improves target suppression: its UnlearnDiffAtk and P4D ASRs are 5.6% and 17.2%, respectively, below the 9.2% and 40.4% obtained in the original space. KAN further reduces these ASRs to 1.4% and 4.6%, and lowers FID from the SAE's 39.82 to 35.78. Thus, KAN achieves better target suppression and generation quality under the current setting.

We hypothesize that, after the SAE maps representations to a higher-dimensional space, semantics associated with complex concepts such as nudity may be distributed across multiple sparse features. Under a fixed low-rank subspace constraint, this dispersion may make it harder to capture target-related changes while excluding non-target semantics.

Table R10. Representation-mapping ablation including SAE (Table 9, App. B.3 of the revised PDF).

RepresentationUnlearnDiffAtk ASR ↓P4D ASR ↓FID ↓
Original space0.0920.40438.25
MLP latent space0.1550.46440.82
SAE latent space0.0560.17239.82
KAN latent space0.0140.04635.78
Reviewer_UVaTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CRET, a concept erasure method based on a learnable orthogonal projective subspace. CRET uses risk and safety prompt pairs to learn the orientation of the target concept while limiting its impact on normal semantic information. This method achieves concept erasure by intervening the representation without modifying the parameters of the pre-trained diffusion model. Experiments conduct on SD1.4 and SDXL, showing that CRET achieves a low attack success rate on multiple adversarial benchmarks while maintaining good image generation quality.

Critical strengths
  1. Compared to directly constructing the projective subspace, the authors learn an editable subspace by jointly constraining the target concept and normal semantics, which better handles the overlap between the two in the feature space. This approach has a clear motivation and practical value.

  2. The method achieves a low average attack success rate on four adversarial benchmarks while maintaining reasonable generation quality. In particular, comparative experiments with different projection construction methods strongly support the effectiveness of the core design of the learnable projective subspace. The results on SDXL provide some evidence of generalization.

Critical weaknesses
  1. The current results mainly demonstrate that the method can resist existing attacks, but are insufficient to prove its effectiveness against adaptive attacks. No evidence show that if the attacker could adjust the cue words for the current projection intervention, the existing performance advantage could still hold.

  2. The paper relies heavily on FID and CLIP scores on COCO, but these overall metrics are insufficient to reflect fine-grained semantic changes. The lack of quantitative evaluation for related concepts and similar semantics means that the method's selective erasure capability has not been fully validated.

Questions for authors
  1. The paper uses EOT token representations to train the concept projector, but applies the same projector to all non-specific tokens during inference. Table 5 shows that the actual erasure effect mainly comes from the intervention of content tokens. Can the authors explain why the subspace learned on the EOT representation can be directly used for other tokens?

  2. The risky-safe prompt pairs in Appendix A.2 are constructed by deleting the target concept description. However, this operation may also change sentence length, token position, and contextual representation. How did the authors confirm that the model learns the target concept itself, rather than these accompanying changes?

  3. In Table 4, the closed projector achieves worse results than the iteratively optimized version without LeditL_{edit}, even though the former accurately minimizes the corresponding objective function. Does this mean that there is still some deviation between the current subspace learning objective and the final concept erasure effect?

  4. Do the authors have more direct quantitative results to support the method's good concept selectivity?

5 comments
AuthorsOpenReview ↗

Response to Reviewer UVaT(1/3)

We sincerely thank the reviewer for the careful reading and constructive feedback. Your questions on adaptive attacks, concept selectivity, and projector learning have helped us strengthen the evaluation and clarify our design choices. We respond to each point below and have incorporated the corresponding revisions into the revised manuscript.

[W1]: Adaptive Attacks

Thank you for this suggestion. To test whether CRET retains its advantage when an attacker knows the defense and optimizes prompts against its projection intervention, we added a white-box adaptive attack based on UnlearnDiffAtk[1]. The attack optimizes prompts by minimizing the denoising prediction error on reference images of the target concept. We include the CRET intervention in the fully differentiable computation graph, allowing attack gradients to pass through the defense, and evaluate newly generated images using the optimized discrete prompts. CRET's attack success rate (ASR) increases from 1.4% in the original Table 1 to 2.26%, while remaining favorable relative to the results in Table 1. For a direct comparison under the same evaluation setting, we also optimize UnlearnDiffAtk attacks separately against each defense, as shown in Table R1. CRET achieves the lowest ASR of 2.26%, compared with 19.01% for the next-best method, HIRM. These results show that CRET retains its performance advantage against adaptive attacks, even when attackers optimize prompts specifically against CRET's projection intervention.

Table R1. Adaptive UnlearnDiffAtk attacks (Table 12, App. B.5 of the revised PDF).

MethodUnlearnDiff_attack ASR (%) ↓
SAFREE40.85
UCE35.21
ESD74.65
CA42.96
MACE41.03
HIRM19.01
CRET2.26

References

[1] Yimeng Zhang, Jinghan Jia, Xin Chen, Aochuan Chen, Yihua Zhang, Jiancheng Liu, Ke Ding, and Sijia Liu. To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy to Generate Unsafe Images ... For Now. In European Conference on Computer Vision (ECCV), pp. 385–403. Springer, 2024.

[W2 , Q4]: Concept Selectivity

Thank you for this suggestion. We added object-concept erasure experiments on ten categories from an ImageNet subset, erasing one category at a time and evaluating preservation on the remaining nine. A pretrained ResNet-50 performs top-1 classification of the generated images. We measure erasure effectiveness using the target-category recognition rate and non-target preservation using the average decrease in recognition rate across the remaining nine categories relative to the original model. Lower values are better for both metrics.

CRET reduces the average target-category recognition rate from 85.6% to 1.5%, close to OCE (1.8%) and ScaPre (2.4%), while the average non-target recognition rate decreases by only 4.5 percentage points (Tables R2 and R3). Beyond aggregate FID and CLIP scores, these experiments directly quantify whether suppressing one target category preserves recognition of the other nine categories. The low target recognition rate and small non-target accuracy drop provide category-level quantitative evidence of CRET's selective suppression on the evaluated object concepts.

Table R2. Target recognition after object erasure (left half of Table 3 in the revised PDF).

ObjectOriginalScaPreOCEUCESPEEDCRET
Cassette Player764512144
Chain Saw7910000
Church83834203
Gas Pump7811640
Tench75000140
Garbage Truck7710072
English Springer9302060
Golf Ball996756416
Parachute96300120
French Horn10000010
Mean85.62.41.87.811.91.5

Table R3. Non-target preservation after object erasure (right half of Table 3 in the revised PDF).

ObjectOriginalScaPreOCEUCESPEEDCRET
Cassette Player86.714.81.620.815
Chain Saw86.314.65.80.62.25.6
Church85.913.71.215.12.76.7
Gas Pump86.41235.62.47.2
Tench86.818.42.49.92.86.2
Garbage Truck86.611.82.60.13.81.2
English Springer84.84.93.1030.5
Golf Ball84.131.96.23.41.4
Parachute84.47.82.442.80.1
French Horn8410.85.74.73.711.1
Mean85.611.23.06.72.84.5
AuthorsOpenReview ↗

Response to Reviewer UVaT(2/3)

[Q1]: EOT Supervision and Content-Token Intervention

Thank you for raising this point. Our explanation has two related parts, which we examine through two sets of ablation experiments.

First, EOT representations provide contextual supervision for learning the target subspace. Because EOT aggregates the contextual semantics of the entire prompt, the EOT difference between paired risky and safe prompts can reflect the semantic change caused by removing the target description. We hypothesize that, compared with representations of target keywords alone, this supervision more fully captures how the target concept is expressed in the complete prompt context.

Second, transfer from EOT supervision to content tokens relies on partially shared directions encoding target semantics. We hypothesize that, after passing through the same frozen mapping, EOT and content-token representations from the same encoder layer may encode some target-related components along common directions. A projector learned from EOT differences may therefore identify and remove the corresponding target components in content tokens.

To evaluate transfer across token positions, we first fix the projector trained with EOT supervision and vary only the token positions edited at inference time. As shown in Table 7 (App. B.2 of the revised PDF), editing content tokens alone reduces P4D ASR to 8.6%, and editing both content tokens and EOT further reduces it to 4.6%. This demonstrates that the projector learned with EOT supervision is effective at content-token positions, consistent with the hypothesis of partially shared directions encoding target semantics.

To examine the choice of EOT supervision, we further compare training supervision sources while fixing the inference intervention scope. KEY denotes the contextualized content-token representations of target-concept keywords in risky prompts. KEY-only uses these keyword representations as supervision for capturing target-concept information, whereas EOT+KEY additionally incorporates paired EOT differences. As shown in Table R4, KEY-only yields a P4D ASR of 14.6% and an FID of 58.36, compared with 4.6% and 35.78 for EOT-only. Joint supervision yields 5.3% and 50.32, respectively, and does not improve on EOT-only. Thus, in the current experimental setting, EOT-difference supervision provides better erasure and generation quality than the tested keyword supervision.

Table R4. Training-supervision ablation (Table 8, App. B.2 of the revised PDF).

p4d ↓fid ↓clip ↑
KEY0.14658.3627.05
EOT0.04635.7830.8
EOT+KEY0.05350.3229.31

[Q3]: Closed-Form Optimization and Downstream Erasure

Thank you for pointing out this important distinction. We agree that there is a gap between the subspace learning objective in representation space and the final erasure effect in generated images.

First, our objective provides a representation-level surrogate for selective erasure. Target inclusion and benign exclusion guide the subspace to capture target-concept information while preserving benign semantics. The final image-level effect also depends on how the diffusion model responds to the edited text representations. This difference in evaluation levels helps explain why exact optimization of the training objective may not produce the best downstream erasure result, while our ablations support the objectives' effectiveness in guiding selective erasure.

Second, the comparison between the closed-form solution and iterative optimization further illustrates this gap. Without the decoded preservation term, joint spectral decomposition exactly optimizes the combined target-inclusion and benign-exclusion objective for the specified rank and weights. However, this optimality applies only to a geometric criterion on the training representations; it does not guarantee that the selected directions are best suited for erasure after diffusion-based image generation. The closed-form solution yields a P4D ASR of 9.9% and an FID of 50.85, whereas the iterative version optimizing the same surrogate objective yields 5.3% and 45.05 (Table 6 of the revised PDF). One possible explanation is that the surrogate objective on finite training samples does not fully capture downstream semantic sensitivity. During iterative learning, we apply the subspace in the latent-space erasure operation and update its basis vectors using the resulting loss gradients, which may yield a subspace with better downstream performance.

Finally, full CRET additionally constrains changes to benign representations through the decoded preservation term, allowing subspace learning to account for the effect of the actual decoded intervention. Adding this term reduces P4D ASR from 5.3% to 4.6% and FID from 45.05 to 35.78, supporting its contribution to the erasure-preservation trade-off.

AuthorsOpenReview ↗

Response to Reviewer UVaT(3/3)

[Q2]: Accompanying Changes in Prompt-Pair Construction

Thank you for highlighting this concern. Deleting a target description can also change sentence length, token positions, and context. We conducted two complementary evaluations to examine whether supervision constructed through target deletion leads the model to learn the target concept rather than these accompanying changes.

First, we test whether differences constructed through target deletion contain target semantics. As illustrated in Figure 5 (Appendix B.8), we start from a prompt describing a desktop scene without the target concept. Adding the denoising-prediction difference from a risky-safe prompt pair introduces cassette-player-related equipment into the generated image while leaving the overall background largely unchanged. This indicates that the denoising-prediction difference obtained from deletion-based prompt pairs contains semantic signals capable of guiding target-concept generation.

Second, we test whether the learned subspace captures the target concept itself, rather than merely changes in sentence length or token positions caused by deletion. We fix the trained KAE and concept projector and compare target deletion, target replacement with the same token length, and non-target deletion on 200 independently constructed test prompt groups. For each type of change, we compute the fraction of the latent representation difference's energy captured by the learned concept subspace, termed the captured-energy ratio. Although non-target deletion also involves deleting text, its captured-energy ratio is only 4.70%, compared with 23.79% for target deletion (Table R5). This suggests that the learned subspace is not generally sensitive to deletion itself. We further use target replacement with the same token length as a control for sentence length and token positions. This operation preserves the total token count and subsequent token positions, yet yields a captured-energy ratio of 24.52%, close to the 23.79% obtained with target deletion. This supports the interpretation that the higher captured-energy ratio for target-deletion differences reflects changes in target semantics, rather than merely sentence shortening or token-position shifts.

Together, these evaluations support the use of target-deletion supervision to learn a subspace selective for target semantics. Although replacement with the same token length controls sentence length and token positions, it also introduces the replacement word's semantics. The resulting difference therefore reflects both removal of the target description and introduction of the replacement, requiring an additional choice of replacement words. Taking these considerations together, we use target deletion to construct training prompt pairs.

Table R5. Captured-energy ratios for prompt-pair controls (Table 15, App. B.8 of the revised PDF).

Type of changeCaptured-energy ratio
Target deletion23.79%
Target replacement with the same token length24.52%
Non-target deletion4.70%

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.