acceptodds
Under review as a conference paper at ICLR 2027

Contrastive Representation Learning for Robust Concept Erasure in Diffusion Models

11 views

Abstract

Concept erasure edits pretrained text-to-image diffusion models to remove certain unwanted or unsafe concepts while preserving other capabilities. These erased concepts often stay suppressed under ordinary prompts, however they can be recoverable under adversarial inputs. We introduce CIRCE (Contrastive Image- Representation Concept Erasure), which reshapes the diffusion model’s internal representations through contrastive learning. CIRCE fine-tunes diffusion models to retain benign concepts in the representation space, as well as unlearn the harmful concepts by using them as a negative example. We evaluate CIRCE across objects, artistic styles, and nudity on Stable Diffusion and Qwen-Image-2.1. On UNLEARN-CANVAS, CIRCE achieves the highest mean across six erasure and retention metrics in our comparison. For nudity erasure, CIRCE remains robust across fixed jailbreak prompts, adaptive prompt search, and continuous embedding attacks, while pre- serving strong image quality on benign prompts. A human study with 24 annotators further finds the best observed preservation on general prompts for both backbones. Preservation on benign prompts close to the erased concept remains competitive with existing methods. Overall, our results show that directly reshaping internal representations provides a promising direction for robust concept erasure.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
concept erasure, diffusion models, representation engineering, adversarial robustness, machine unlearning, AI safety

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rcTfOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CIRCE for concept erasure in diffusion models through contrastive representation constraints. The method freezes the base model and text encoder and trains LoRA adapters in the denoiser, combining target-representation clustering, rerouting, separation between target and benign centroids, and benign-representation preservation. Some configurations additionally penalize spatial variance in the predicted clean latent. Experiments evaluate object and artistic-style erasure on Stable Diffusion and nudity erasure on Stable Diffusion and Qwen-Image-2.1. The paper reports recovery rates under multiple text and continuous text-embedding attacks and evaluates preservation using general prompts, benign concepts close to the erased target, and human assessment. Overall, the results indicate a favorable robustness–preservation trade-off in the tested settings.

Critical strengths
  1. The paper evaluates both erasure robustness and preservation of benign generation, including benign prompts close to the target concept, rather than relying only on low target-detection rates. The reported trade-off has value under the attacks and prompt conditions tested.
  2. The experiments cover multiple concept types and different denoising architectures, with baseline retraining, loss ablations, adaptive attacks, and human evaluation. This provides broad empirical evidence of the method's effectiveness.
Critical weaknesses

W1: The advantage of directly supervising internal representations is not sufficiently validated. The paper presents direct supervision of internal representations as an important distinction from output replacement targets. However, parameter updates driven by output targets can also change internal representations. The experiments do not sufficiently separate factors such as the location of supervision and the parameter-update scheme. Therefore, directly imposing representation constraints alone does not establish deeper or more robust concept erasure.

W2: Insufficient mechanistic analysis of representation changes and performance trade-offs. Changing internal representations is central to the paper's motivation, but the analysis primarily examines how individual loss terms affect final metrics. It does not sufficiently verify the intended representation changes or their connection to erasure robustness and benign preservation. The ablations mainly reveal empirical performance trade-offs, with limited explanation of why these trade-offs arise or when they apply, leaving the methodological insights insufficiently developed.

W3: Robustness to visual-inversion recovery has not been validated. The recovery-attack evaluation primarily covers text and continuous text-embedding pathways, without testing visual-inversion methods such as TINA and TINA+, which use reference images and initial latents for recovery. Given that directly changing the denoiser's internal representations is central to the method's motivation, it is important to test whether this intervention also blocks recovery pathways distinct from textual conditioning. The current evidence therefore does not establish whether the method is similarly robust to visual-inversion recovery.

W4: The relationship between SalUn's search-winning configuration and the configuration used in the full evaluation is unclear. The configuration search in Appendix A.1 selects SalUn c5, but the full benchmark comparison reports c4 without clearly explaining this choice. Please explain why the full evaluation of c5 is not reported and provide its results or explain why it was not evaluated, so that it is clear whether the comparison adequately reflects the tuned baseline's performance.

Related work:

  • TINA: Text-Free Inversion Attack for Unlearned Text-to-Image Diffusion Models. CVPR 2026.
  • TINA+: Probing Residual Visual Knowledge in Unlearned Diffusion Models via Diffusion-Consistent Text-Free Inversion. arXiv 2026.
Questions for authors
  1. Can the additional benefit of direct representation supervision over output-target supervision be established while controlling factors such as the updated model components, trainable capacity, training data, and budget? What evidence attributes the gains to the supervision scheme itself?
  2. Do target representations exhibit the intended clustering, rerouting, and separation from benign representations after training? How do these changes relate to attack recovery and benign preservation, particularly the benefits and costs of individual loss terms across tasks?
  3. When access to reference images and initial latents is allowed, does CIRCE remain robust to visual-inversion recovery methods such as TINA/TINA+? If this access setting is outside the intended scope, please clarify the corresponding limits of the conclusions.
  4. Why does the full benchmark report SalUn c4 when c5 wins the configuration search? Can the full c5 results be provided, or can the specific reason for not evaluating or reporting them be explained?

Related work:

  • TINA: Text-Free Inversion Attack for Unlearned Text-to-Image Diffusion Models. CVPR 2026.
  • TINA+: Probing Residual Visual Knowledge in Unlearned Diffusion Models via Diffusion-Consistent Text-Free Inversion. arXiv 2026.
8 comments

We thank the reviewer for their comments. We are encouraged that they acknowledged our broad evaluation, including different adaptive attacks, loss ablations and the human evaluation. Throughout this response, we link figures in our anonymous GitHub repository that illustrate each finding. We invite the reviewer to look at them while they read our response. We will upload the updated paper PDF in the coming days.

W1 / Q1. Representation vs output supervision, controlled

Parameter updates driven by output targets can also change internal representations. The experiments do not sufficiently separate factors such as the location of supervision and the parameter-update scheme.

Can the additional benefit of direct representation supervision over output-target supervision be established while controlling factors such as the updated model components, trainable capacity, training data, and budget?

On SD-1.4, erasing in representations rather than at the output gives about 3× lower attack success at the same NearConcept failure and higher MOS. On Qwen-Image-2.1, clustering, a representation-level term, also lowers attack success at matched image quality.

In our rebuttal, to separate where the supervision acts from everything else, we trained CIRCE with its losses moved to the output. In the output-level version, the same erasure losses act on the denoiser's prediction (averaged over image positions) instead of its internal activations, and the benign anchor keeps the prediction for benign prompts close to the original model's. We trained all four combinations of erasure location and anchor location. All of them update the same layers with the same LoRA rank, training data, timestep range and number of steps, so they differ only in where the losses act.

Table 1: Location of the losses on SD-1.4. Each cell gives PAIR (NearConcept failure, MOS), mean over runs.

anchor on representationsanchor on output
erasure on representations0.075 (0.26, 4.29), CIRCE, 4 runsmodel collapsed (MOS 1.0)
erasure on output0.247 (0.24, 4.04), 4 runs0.227 (0.25, 3.91), 4 runs

At the same NearConcept failure and higher MOS, erasing in representations lowers PAIR from 0.23–0.25 to 0.075. Every representation-level run has lower PAIR than every output-level run, a difference of 0.16 (exact permutation test over the 4 and 8 runs, p = 0.004, the smallest p possible with this many runs) (figure).

On Qwen-Image-2.1 (10 seeds per cell), we compare at matched quality, because output-level erasure lowers PAIR partly by degrading the model. Moving only the erasure losses does not change PAIR, but the two other representation-level terms do: moving the benign anchor to representations lowers PAIR by 0.08 at matched MOS (p = 0.04), and clustering lowers it by 0.22 (p < 0.001, figure).

A standard output-level method confirms this: ESD with an output anchor, tuned over 47 runs, has 0.17 higher PAIR than CIRCE at matched NearConcept failure (p = 0.0004, figure). Across all 274 Qwen-Image-2.1 models we trained (83 CIRCE runs, 116 baseline runs and 75 controls), every output-level run with MOS above 4 is beaten by a CIRCE run with both lower PAIR and higher MOS (figure).

Representation-level erasure is more robust because it moves the concept's internal representation, which output-level training barely does: a probe trained on the original model still finds the concept in 96% of target inputs after ESD with an output anchor, and in 94% after output-level CIRCE, against 71% after CIRCE. An attacker therefore has less of the concept left to recover. This matches what circuit breakers found for LLMs, where rerouting representations gave more robustness than training on outputs alone (Zou et al., 2024). We will add this comparison to Sec. 4 and state that supervising representations moves the robustness–quality frontier.

W2 / Q2. Intended representation changes and their link to outcomes

It does not sufficiently verify the intended representation changes or their connection to erasure robustness and benign preservation. The ablations mainly reveal empirical performance trade-offs, with limited explanation of why these trade-offs arise or when they apply

Do target representations exhibit the intended clustering, rerouting, and separation from benign representations after training? How do these changes relate to attack recovery and benign preservation, particularly the benefits and costs of individual loss terms across tasks?

Yes, each term moves the representations as intended. The trade-off arises because benign prompts close to the concept share much of its representation, so the terms that push the concept away also move them: the more these benign representations drift, the lower the attack success and the higher the NearConcept failure. We measured each term on Qwen-Image-2.1.

Table 2: Representation measurements on Qwen-Image-2.1

termmeasurementCIRCEcomparison
reroutingsimilarity to the original model's activations, target inputs0.540.84–0.86 output-level erasure
separationcosine similarity between target and benign centroids (lower is more separated)0.74 ± 0.02 (n = 10)0.88–0.90 output-level erasure, 0.99 ESD + output anchor, 0.99 original model
clusteringspread of target activations0.230.30 without clustering

CIRCE reroutes target representations far more than output-level erasure, separates target and benign centroids, and tightens the target cluster compared with training without clustering.

Within the 16 runs of CIRCE's family (full recipe and ablations), the drift of benign representations on NearConcept prompts predicts PAIR (Spearman ρ = −0.87) and NearConcept failure (ρ = +0.91). This also suggests when the trade-off applies: a term costs preservation where benign prompts lie close to the target in representation space, and costs little where they do not. The edit re-encodes the concept rather than removing it: a classifier retrained on each model still decodes the concept, with balanced accuracy 0.949 for CIRCE and 0.966 for the base model (figure).

Across tasks, clustering lowers attack success at matched image quality (W1), and its preservation cost depends on the task. On SD it costs nothing on NearConcept, with NearConcept failure between 0.21 and 0.29 for weights 0–1.0 (figure). On Qwen, within CIRCE's runs, tighter clustering goes with more NearConcept damage (ρ = −0.54, p = 0.035). On UnlearnCanvas, retention falls as the clustering weight rises (figure). Separation protects retention, and removing it lowers in-domain accuracy by 6.5 points on styles and 2.7 on objects on average (Table 11). In short, clustering buys robustness, at a preservation cost when nearby benign content moves with the concept, and separation limits that cost. We will include these measurements, the correlations and the effect of each term in the appendix.

W3 / Q3. Visual inversion (TINA / TINA+)

When access to reference images and initial latents is allowed, does CIRCE remain robust to visual-inversion recovery methods such as TINA/TINA+? If this access setting is outside the intended scope, please clarify the corresponding limits of the conclusions.

We thank the reviewer for this question, which led to a finding we found interesting.

TINA+ recovers the target from CIRCE in 57% of cases and from the undefended model in 64% (strict NudeNet), but it also "recovers" nudity from models whose denoiser was trained only on digits or fruit. We trained diffusion models in SD-1.4's latent space only on MNIST digits or only on photos of fruit. These models never generate nudity on their own (NudeNet flags 0/200 of the MNIST model's samples and 2/200 of the fruit model's). Yet TINA+ recovers nudity on 29% (MNIST) and 46% (fruit) of its 118 targets. A white-box attack with the budget of our other white-box attacks recovers nothing from either model (0/80). TINA+ also recovers nudity from every erasure method we tested, between 42% and 67% of targets for all 12 SD-1.4 baselines, so CIRCE (57%) is comparable to existing methods (for example ESD 53%, TRCE 57%, SafeGen 61% and STEREO 67%). The original TINA even succeeds on 57% of targets under strict NudeNet (81% under the lenient detector) with a UNet whose weights are all zero, and the TINA+ authors report that TINA succeeds on a randomly initialised UNet (their Table VII).

Given a reference image and its starting noise, these attacks can thus reproduce the image even from a denoiser that was never trained on the concept (figure). They might therefore measure how well an image can be inverted more than what the model still knows about the concept. We will add these results to the paper and state visual inversion as a limit of our conclusions.

W4 / Q4. SalUn c4 vs c5

The configuration search in Appendix A.1 selects SalUn c5, but the full benchmark comparison reports c4 without clearly explaining this choice.

Why does the full benchmark report SalUn c4 when c5 wins the configuration search? Can the full c5 results be provided, or can the specific reason for not evaluating or reporting them be explained?

The paper reported SalUn (unlearning restricted to salient weights) with c4 because the full c5 evaluation, which requires training and evaluating 70 models (50 styles and 20 objects), had not finished at submission time. We have now run it, and c4 has the higher mean of the six metrics on the full benchmark. The two configurations differ only in α (1.0 for c5, 0.2 for c4), and we will report both rows in the paper (figure).

Table 3: SalUn on the full UnlearnCanvas benchmark (UA: erasure, IRA / CRA: in- and cross-domain retention)

style UA / IRA / CRAobject UA / IRA / CRAmean of six metrics
SalUn c4 (paper)74.8 / 98.1 / 66.782.4 / 99.1 / 84.484.25
SalUn c545.6 / 99.2 / 81.363.9 / 99.3 / 95.980.87

c5 erases much less (style UA 45.6 vs 74.8) and led the search only on its 4-concept, 200-image subset.

We thank the reviewer again for their thoughtful review, which helped make our paper even better. If our responses address the reviewer's concerns, we would appreciate it if they would reconsider their score. We also invite the reviewer to respond, so that we can continue the exchange and further improve the quality of our paper.

Reviewer_upYcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CIRCE, a concept erasure method designed to improve the robustness of diffusion models against adversarial attacks. CIRCE employs contrastive representation learning to reshape target-concept representations through clustering, rerouting, and centroid separation, while preserving benign generation by anchoring representations to those of a frozen base model. The method is evaluated on object and artistic-style erasure using SD 1.5, and on nudity erasure using SD 1.4 and Qwen-Image-2.1. The results demonstrate competitive adversarial robustness and generation quality compared with the evaluated baselines.

Critical strengths

The paper provides a systematic robustness evaluation across different levels of attacker access. It considers fixed and transferred prompts (I2P, Ring-A-Bell, MMA), adaptive black-box attacks (PAIR, prompt refinement, DiffZOO, SneakyPrompt), white-box continuous embedding optimization (CCE-deep), and concept recovery after benign fine-tuning. This evaluation goes beyond standard prompt-based erasure benchmarks and examines robustness under increasingly capable adversarial settings.

The experiments examine the erasure–preservation trade-off across multiple concept categories and model architectures. The evaluation covers 50 artistic styles and 20 object classes on UnlearnCanvas, as well as nudity erasure on both Stable Diffusion and Qwen-Image-2.1. The paper further assesses preservation of concept-adjacent benign content using NearConcept and OVERT, complemented by loss ablations and a human evaluation involving 24 annotators.

Critical weaknesses
  1. Limited comparison with recent representation-based methods. CIRCE builds on existing representation-engineering ideas by applying them to the diffusion denoiser, and the paper discusses its methodological differences from related work. However, the lack of direct comparisons with HiRM (ICLR 2026) and CURE (NeurIPS 2025) makes it difficult to assess the relative benefits of denoiser-level intervention and additional training. In particular, a comparison of erasure–preservation performance, adversarial robustness, and training cost would help establish CIRCE's advantages over these approaches. Furthermore, although TRCE is evaluated in Appendix A.6, it is not included in the main adversarial robustness comparison.

  2. Insufficient evidence for the stability of the main robustness–preservation results across training seeds. Table 12 in Appendix E shows substantial variation across training seeds for the same full objective on Qwen-Image-2.1. The NudeNet-guided PAIR ASR ranges from 8.3% to 43.3%, while the NearConcept failure rate ranges from 12.2% to 41.5%. Although the authors report this variability, it remains unclear how representative the main H1 checkpoint is and whether its favorable robustness–preservation trade-off relative to the baselines persists across training runs. Clarification of the H1 checkpoint selection criteria and its relationship to the repeated runs in Table 12 would help address this concern.

Questions for authors

Q1. Comparison with recent methods: Could the authors provide a comparison with representative recent methods, such as HiRM (ICLR 2026), CURE (NeurIPS 2025), or TRCE (ICCV 2025), under comparable Stable Diffusion settings? Such a comparison would help clarify CIRCE's relative advantages in adversarial robustness, benign preservation, and computational cost, while accounting for differences in attacker access.

Q2. Stability across training seeds: Could the authors evaluate the existing repeated checkpoints using the same judge, prompts, and attack budget as the main results to assess whether CIRCE's advantages persist across training seeds?

Q3. Generality of robustness: Could the authors report adaptive recovery results for representative object or artistic-style targets? Additional adaptive recovery results after benign fine-tuning on Qwen-Image-2.1 would also help establish the scope of the robustness findings.

5 comments

We thank the reviewer for their comments. We are encouraged by their positive feedback about the breadth of our evaluation, in particular the adaptive attacks, the multiple concepts studied and the human evaluation. Throughout this response, we link figures in our anonymous GitHub repository that illustrate each finding. We invite the reviewer to look at them while they read our response. We will upload the updated paper PDF in the coming days.

W1 / Q1. Comparison with HiRM, CURE and TRCE

The lack of direct comparisons with HiRM (ICLR 2026) and CURE (NeurIPS 2025) makes it difficult to assess the relative benefits of denoiser-level intervention and additional training. [...] Furthermore, although TRCE is evaluated in Appendix A.6, it is not included in the main adversarial robustness comparison.

In particular, a comparison of erasure–preservation performance, adversarial robustness, and training cost would help establish CIRCE's advantages over these approaches.

Could the authors provide a comparison with representative recent methods, such as HiRM (ICLR 2026), CURE (NeurIPS 2025), or TRCE (ICCV 2025), under comparable Stable Diffusion settings?

For the rebuttal, we added HiRM (redirects concept representations in the text encoder) and TRCE (cross-attention edit, then early-step fine-tuning) to the main SD-1.4 comparison. We ran HiRM with the authors' official code and configuration, and it reproduces an MMA-Diffusion attack success very close to the one reported in the HiRM paper (2.7% vs. 3.3–8.0%). For TRCE we used the authors' released checkpoint. We evaluated all models with the same attacks, judge and prompts, and the repository plots attack success against preservation for each attack (NearConcept, MOS).

Table 1: SD-1.4 nudity erasure (PAIR averaged over 3 evaluations, CIRCE mean ± sd over 5 runs)

modelPAIRNearConcept failureMOSCLIP fidelitytraining cost
CIRCE0.070 ± 0.0220.26 ± 0.034.28 ± 0.160.93 ± 0.01~35–45 min
TRCE0.0220.343.450.82~5 h
STEREO0.0170.393.400.82–
HiRM0.1390.633.250.8379 s

TRCE and STEREO have lower PAIR than CIRCE, but higher NearConcept failure, lower MOS and lower fidelity. They are also less robust to the paper's white-box attack, which recovers the concept from TRCE in 83% and from STEREO in 30% of images, against 5% for CIRCE. HiRM has higher PAIR and NearConcept failure than CIRCE and lower MOS, but it has the advantage of training much faster (training times measured on the same GPU type).

We leave CURE out of the baselines because we could not reproduce its reported nudity results. CURE has not released its unlearning code. Its official repository says the code "will be released after publication", and it provides only three edited checkpoints for other concepts (cat, Angelina Jolie and Kelly McKernan) but no nudity weights. We re-implemented CURE from these checkpoints, and our construction matches them closely. We then built 12 nudity models in a hyperparameter sweep, but none reached the paper's reported nudity results. With CURE's own evaluation protocol, our best attack success with Ring-A-Bell prompts is 0.215, against 0.013 in the CURE paper. While searching, we found an independent paper that runs CURE as a baseline (PURE) and that also reports much weaker nudity erasure than the CURE paper, with Ring-A-Bell attack success 0.59 against 0.73 for the undefended model. Since a re-implementation that does not match the paper could misrepresent CURE, we prefer not to report it as a baseline. We will add HiRM and TRCE to Table 3 with CLIP fidelity and training cost.

W2 / Q2. Stability across training seeds and H1 selection

[...] it remains unclear how representative the main H1 checkpoint is and whether its favorable robustness–preservation trade-off relative to the baselines persists across training runs. Clarification of the H1 checkpoint selection criteria and its relationship to the repeated runs in Table 12 would help address this concern.

Could the authors evaluate the existing repeated checkpoints using the same judge, prompts, and attack budget as the main results to assess whether CIRCE's advantages persist across training seeds?

CIRCE's advantage persists across seeds and hyperparameters: for good benign preservation (MOS above 3.8), its Pareto frontier lies below the baselines' Pareto frontier across all runs (Shown on this figure), and H1 is typical of its recipe in attack success. For the rebuttal, we ran 14 additional CIRCE runs on Qwen-Image-2.1, 7 with the reported recipe and 7 with the default recipe of Table 12. We re-ran the attacks on all runs, including the Table 12 seeds, with the same judge, prompts and attack budget as the main table. We agree with the reviewer that seed means are more informative than one checkpoint, so the revised PDF will report seed means for every method.

We also found that concept erasure methods in general vary across seeds, and RR varies more than CIRCE (Table 2). Against NearConcept failure, CIRCE's Pareto frontier also lies below the baselines' for NearConcept failure between 5% and 75% (figure).

H1 comes from a good seed, but it is not an outlier: other settings reach similar preservation with lower attack success (for example NearConcept failure 0.13 at PAIR 0.13 and MOS 4.75), and H1 is not on the PAIR frontier against either NearConcept failure or MOS, where other CIRCE runs do better (figure).

We also traced where the seed differences come from. In most seeds, the benign-anchor loss briefly spikes late in training, after the erasure losses have converged, and the number of spikes predicts NearConcept failure across seeds (Spearman ρ = 0.77). H1's seed trains without these spikes, and runs without spikes are better on average (NearConcept failure 0.18 and MOS 4.58, against 0.43 and 4.01 for runs with spikes, p = 0.006). Other concept erasure methods are also sensitive to the seed, as the spread of the baselines in Table 2 below shows.

Table 2: Seed means on Qwen-Image-2.1 (mean ± sd over training runs)

methodrunsPAIRrefinementNearConcept failureMOS
CIRCE, reported recipe8 seeds0.17 ± 0.090.07 ± 0.060.36 ± 0.194.32 ± 0.36
CIRCE, default recipe (Table 12 seeds + 7 new)10 seeds0.16 ± 0.120.07 ± 0.090.39 ± 0.174.01 ± 0.54
RR (circuit breakers)10 seeds0.48 ± 0.200.31 ± 0.200.18 ± 0.154.26 ± 0.42
ESD-η=24 seeds0.09 ± 0.05–0.64 ± 0.072.36
EraseAnything4 seeds0.41 ± 0.06–0.38 ± 0.053.30
Concept Ablation4 seeds0.53 ± 0.07–0.44 ± 0.051.61
STEREO (adversarial training)4 seeds0.90 ± 0.030.72 ± 0.020.03 ± 0.024.64 ± 0.08
undefended model10.92–0.0204.96

Averaged over seeds, CIRCE has the lowest PAIR among methods with MOS ≥ 4, and its 8 seeds have lower PAIR than RR's 10 seeds (Mann–Whitney p = 0.002). The only method with lower PAIR, ESD-η=2, reaches it at MOS 2.36. STEREO preserves NearConcept prompts best (failure 0.03), but PAIR recovers the concept from it in 90% of attempts, close to the undefended model. The ESD-η=2, EraseAnything and Concept Ablation rows are configurations we retrained over seeds for the rebuttal, so they differ from the single checkpoints in the paper's main table.

The Table 12 runs are three seeds of the default recipe. The paper evaluated them with a cheaper ablation protocol in which the NudeNet detector guides and scores the attack, which is why their numbers differ from the main table. The default-recipe row above re-runs the main-table attacks on them.

We selected H1 as a run with prompt-refinement success of at most 10% and NearConcept failure at most 10 points above the base model. It is typical in attack success: its PAIR is 0.22 over three evaluations (0.18 in the paper's single evaluation), close to the median of our CIRCE runs (0.23). CIRCE's advantage does not depend on this choice: across all runs, its Pareto frontier lies below the baselines' for MOS above 3.8 (figure). The revised paper will report seed means in Table 2 and Sec. 6, with H1 shown as one selected run.

Q3. Adaptive recovery on styles, objects and after benign fine-tuning

Could the authors report adaptive recovery results for representative object or artistic-style targets? Additional adaptive recovery results after benign fine-tuning on Qwen-Image-2.1 would also help establish the scope of the robustness findings.

The adaptive attack does not recover styles from CIRCE, and the results on objects are mixed. We ran the adaptive attack (PAIR-style prompt search) on three representative UnlearnCanvas styles (Cubism, Ukiyoe and Van Gogh) and on five objects with released CIRCE checkpoints.

Table 3: UnlearnCanvas styles, runs recovering the target, out of 90

undefendedCIRCE
900

Table 4: UnlearnCanvas objects, CIRCE runs recovering the target, out of 30

BearsSeaArchitecturesTowersHuman
018252929

The undefended model is recovered in 23–30 of 30 attack runs on these objects. A higher clustering weight lowers recovery on some objects (Sea from 18 to 3 of 30, Towers from 29 to 18), while Architectures and Human stay recoverable at every weight we tried.

Benign fine-tuning on Qwen also recovers the concept from CIRCE as well as from the baselines. We increased the number of fine-tuning steps until the defenses broke: after 800 updates, PAIR reaches 0.63–0.72 on CIRCE runs, 0.75–0.83 on ESD with a benign output anchor and 0.92 on the undefended model. In the LLM literature, this is in line with TamperBench (Hossain et al., 2026), which finds that alignment-stage defenses for LLMs, including circuit breakers, largely fail to withstand fine-tuning attacks.

We will rewrite the abstract and conclusion around what our results support most. Black-box adaptive attacks such as PAIR recover the erased concept from most existing baselines, while CIRCE has the lowest attack success among the methods that keep image quality high (MOS ≥ 4), on both SD-1.4 and Qwen-Image-2.1, and on Qwen-Image-2.1 its Pareto frontier lies below the baselines' across all tested hyperparameters. Under the paper's white-box attack, CIRCE is also more robust than TRCE and STEREO. Image inversion, adaptive white-box attacks and tamper resistance remain limitations, for CIRCE and for every baseline we tested.

We thank the reviewer again for their positive assessment, which helped make our paper even better. If our responses address the reviewer's concerns, we would appreciate it if they would reconsider their score. We also invite the reviewer to respond, so that we can continue the exchange and further improve the quality of our paper.

Reviewer_knAROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CIRCE, a representation-level approach to concept erasure in text-to-image diffusion models. It trains denoiser LoRA adapters using losses that cluster target representations, move them away from their frozen-model counterparts, separate target and benign centroids, and anchor benign representations to the original model. The evaluation covers object and style erasure on UnlearnCanvas and nudity erasure on Stable Diffusion 1.4 and Qwen-Image-2.1. CIRCE achieves the highest reported six-metric mean in the UnlearnCanvas comparison and a promising robustness–preservation tradeoff under several recovery attacks. Automatic evaluations and a 24-annotator study support preservation on general prompts, while results on concept-adjacent benign prompts are more mixed.

Critical strengths
  1. An empirical balance between recovery resistance and benign generation. The strongest result is the combination of robustness and preservation rather than superiority on every individual metric. On Qwen, CIRCE obtains superior PAIR success, while achieving substantially higher general-image MOS. On SD, CIRCE has zero observed CCE-deep successes across 80 trials and higher MOS than the compared erased models. These results make representation-level denoiser editing a worthwhile direction.

  2. Evaluation explicitly examines collateral damage and multiple recovery settings. The paper goes beyond fixed jailbreak prompts by including feedback-driven search, continuous conditioning optimization, and benign fine-tuning. It also measures preservation on both general prompts and concept-adjacent benign prompts. The human study provides complementary evidence and appropriately reports weaker preservation results near the erased concept. This broader evaluation makes the practical tradeoffs more visible than an erasure-only benchmark would.

Critical weaknesses
  1. The distinctive contribution of representation clustering is not sufficiently isolated. Representation rerouting with benign preservation has precedent in circuit-breaker methods, which the manuscript acknowledges. The consequential question is what CIRCE’s particular geometry adds when applied to diffusion denoisers. The ablations provide partial but mixed evidence. Removing clustering improves in-domain retention across all nine evaluated UnlearnCanvas targets; on the three style targets, it also maintains or improves erasure accuracy (Table 11). On Qwen, clustering reduces PAIR success in both paired seeds, but the improvement varies substantially: 38.3% to 13.3% in one seed and 46.7% to 43.3% in another, using the ablation’s NudeNet-guided protocol (Table 12).

  2. Variability and model selection leave uncertainty about the reliability of the headline advantage. The repeated Qwen full-objective runs show substantial variation: NearConcept failure ranges from 12.2% to 41.5%, and NudeNet-guided PAIR success ranges from 8.3% to 43.3% (Table 12). These attack results use a different judge from the main table and should not be directly compared with the headline 18.3%. Nevertheless, the variation makes it important to establish whether the selected H1 checkpoint represents typical performance.

  3. Robustness assessment. Have you evaluated embedding attacks designed to bypass the edited representation geometry? For the Qwen continuous-inversion experiments, please specify the optimization budgets, restarts, and configuration selection, and clarify whether successful and unsuccessful outputs were checked with an independent judge.

Questions for authors

Please refer to the strengths and weaknesses above.

9 comments

We thank the reviewer for their comments. We are particularly encouraged by their positive feedback about the breadth of our evaluation and human study. Throughout this response, we link figures in our anonymous GitHub repository that illustrate each finding. We invite the reviewer to look at them while they read our response. We will upload the updated paper PDF in the coming days.

W1. What clustering adds over circuit breakers

The distinctive contribution of representation clustering is not sufficiently isolated. Representation rerouting with benign preservation has precedent in circuit-breaker methods [...]. The consequential question is what CIRCE's particular geometry adds when applied to diffusion denoisers. The ablations provide partial but mixed evidence.

In CIRCE's own setup, CIRCE has lower attack success than circuit-breaker rerouting (RR), and clustering lowers attack success at matched image quality. To isolate what CIRCE's geometry adds, we trained RR with CIRCE's data, LoRA and training budget (10 seeds), and CIRCE without clustering (10 seeds), and attacked both with PAIR and prompt refinement. We then compared each control with CIRCE's runs at matched preservation (NearConcept failure and MOS), since a method can always lower attack success by damaging the model (figure).

Table 1: Attack success of CIRCE minus the control, at matched preservation (Qwen-Image-2.1)

controlattackat matched NearConceptat matched MOS
RR (10 seeds)PAIR−0.18 (p < 0.001)−0.28 (p < 0.001)
RR (10 seeds)prompt refinement−0.13 (p = 0.026)−0.20 (p < 0.001)
no clustering (10 seeds)PAIR−0.15 (p = 0.010)−0.22 (p < 0.001)
no clustering (10 seeds)prompt refinement−0.11 (p = 0.044)−0.13 (p = 0.001)

CIRCE has lower attack success than both controls in every cell. To compare at the same preservation, we only keep runs whose preservation lies in the range that both groups cover, fit attack success against preservation for each group, and report the average gap between the two fits. The p-values come from shuffling the group labels 20,000 times, with each training run counted as one sample.

We also checked that RR is not simply under-tuned. We tried five RR loss weights and, without looking at attack results, kept the one with the best MOS among those whose NearConcept failure is at most CIRCE's (0.36). Over 3 seeds, this tuned RR still has PAIR 0.36 ± 0.09, against 0.17 ± 0.09 for CIRCE's 8 seeds (Mann–Whitney p = 0.032).

On Qwen, clustering reduces PAIR success in both paired seeds, but the improvement varies substantially: 38.3% to 13.3% in one seed and 46.7% to 43.3% in another [...]

The evidence in the paper was mixed because Table 12 had only 2 seeds per arm. The difference between seeds was as large as the effect of clustering, and the arms were not compared at matched preservation. With 10 seeds per arm and comparisons at matched preservation, removing clustering raises attack success on Qwen in all four comparisons above (figure). On SD-1.4, attack success (the mean of PAIR and prompt refinement) goes down as the clustering weight rises (Spearman ρ = −0.67, p = 0.012, 14 runs with weights 0–1.0), while NearConcept failure and MOS stay unchanged (figure). This shows the importance of the clustering loss.

Removing clustering improves in-domain retention across all nine evaluated UnlearnCanvas targets [...] on the three style targets, it also maintains or improves erasure accuracy (Table 11).

On UnlearnCanvas, the clustering weight trades retention against robustness, and the best weight depends on the target. For the rebuttal, we attacked the Table 11 models with the adaptive prompt-search attack (30 attack runs per target) and swept the clustering weight. Removing clustering raises recovery on the six Table 11 objects (Butterfly, Flowers, Frogs, Human, Sandwiches and Sea) from 43 to 74 of 180 attack runs.

In an exploratory sweep (one seed per target), a lower weight than the paper's helps some targets: at weight 0.1, Frogs keeps more in-domain accuracy than without clustering (95.7 vs 94.9) and is recovered in 0 of 30 runs instead of 12 (figure). A higher weight helps others: at weight 2.0, recovery drops from 18 to 3 of 30 runs on Sea and from 29 to 18 on Towers, at similar in-domain accuracy.

The style results in Table 11 are less informative, because two of the three styles are rarely recovered even from the undefended model (4/30 and 15/30). We will include the RR control and the clustering-weight sweeps in our paper.

W2. Is H1 representative?

The repeated Qwen full-objective runs show substantial variation [...]. These attack results use a different judge from the main table [...]. Nevertheless, the variation makes it important to establish whether the selected H1 checkpoint represents typical performance.

H1 is typical of its recipe in attack success, and CIRCE's advantage holds across seeds. For the rebuttal, we ran 14 additional CIRCE runs on Qwen-Image-2.1, 7 with the reported recipe and 7 with the default recipe of Table 12. Table 12 used a cheaper ablation protocol, in which the NudeNet detector guides and scores the attack, so we re-ran the attacks on all runs with the main-table judge, prompts and attack budget. The variation does not come from the judge: under the main-table judge, the three Table 12 seeds have PAIR 0.17, 0.47 and 0.17.

We agree with the reviewer that seed means are more informative than one checkpoint, so in our revised pdf we will report seed means for every method. We also found that concept erasure methods in general vary across seeds, particularly RR varies more than CIRCE (Table 2). Across all runs, CIRCE's frontier lies below the baselines' frontier for MOS above about 3.8 (PAIR vs image quality, every run, PAIR vs NearConcept failure).

H1 comes from a good seed, but it is not an outlier, as other settings reach similar preservation with lower attack success (for example NearConcept failure 0.13 at PAIR 0.13 and MOS 4.75), and H1 is also not on the PAIR frontier against either NearConcept failure or MOS, where other CIRCE runs do better (figure). We will report seed means as the headline.

In addition we traced where the seed differences come from. In most seeds, the benign-anchor loss briefly spikes late in training, after the erasure losses have converged, and the number of spikes can predict NearConcept failure across seeds (Spearman ρ = 0.77). H1's seed trains without these spikes, and runs without spikes are better on average (NearConcept failure 0.18 and MOS 4.58, against 0.43 and 4.01 for runs with spikes, p = 0.006). We will add this note to the paper.

Table 2: Seed means on Qwen-Image-2.1 (mean ± sd over training runs)

methodrunsPAIRrefinementNearConcept failureMOS
CIRCE, reported recipe8 seeds0.17 ± 0.090.07 ± 0.060.36 ± 0.194.32 ± 0.36
CIRCE, default recipe (Table 12 seeds + 7 new)10 seeds0.16 ± 0.120.07 ± 0.090.39 ± 0.174.01 ± 0.54
RR (circuit breakers)10 seeds0.48 ± 0.200.31 ± 0.200.18 ± 0.154.26 ± 0.42
ESD-η=24 seeds0.09 ± 0.05–0.64 ± 0.072.36
EraseAnything4 seeds0.41 ± 0.06–0.38 ± 0.053.30
Concept Ablation4 seeds0.53 ± 0.07–0.44 ± 0.051.61
STEREO (adversarial training)4 seeds0.90 ± 0.030.72 ± 0.020.03 ± 0.024.64 ± 0.08
undefended model10.92–0.0204.96

Averaged over seeds, CIRCE has the lowest PAIR among methods that don't lose too much benign capability (MOS ≥ 4), and its 8 seeds have lower PAIR than RR's 10 seeds (Mann–Whitney p = 0.002). The only method with lower PAIR, ESD-η=2, reaches it at MOS 2.36, which costs too much general image quality to be useful in practice. STEREO preserves NearConcept prompts best (failure 0.03), but PAIR recovers the concept from it in 90% of attempts, close to the undefended model. The ESD-η=2, EraseAnything and Concept Ablation rows are configurations we retrained over seeds for the rebuttal, so they differ from the single checkpoints in the paper's main table.

The advantage does not depend on picking a good run. H1's PAIR (0.22 over three evaluations, 0.18 in the paper's single evaluation) is close to the median of all 83 CIRCE runs we trained (0.23). Among runs with MOS ≥ 4, the median PAIR is 0.27 for CIRCE and 0.68 for the baselines that erase, and about a quarter of CIRCE runs (19 of 83) reach PAIR ≤ 0.30 with NearConcept failure ≤ 0.20. Only 2 of the 94 baseline runs achieve the same. This does not depend on CIRCE's hyperparameters: across all tested settings, 64 of the 66 CIRCE runs with MOS ≥ 4 have lower PAIR than the median baseline run. Every run is one dot in the frontier figures (PAIR vs MOS, PAIR vs NearConcept failure), and for a good MOS above about 3.8, every run on the PAIR vs MOS frontier is a CIRCE run (baselines reach lower PAIR only at MOS of about 2 or below).

W3. White-box robustness and independent judges

Have you evaluated embedding attacks designed to bypass the edited representation geometry?

Under the paper's white-box attack, CIRCE is the most robust model in this comparison. Only after a deliberate search for the strongest adaptive attack against CIRCE's geometry does it reach the level of STEREO under its own strongest attack, while CIRCE keeps much higher image quality on benign prompts (MOS 4.23 against 3.40). The paper's white-box attack (Sec. 6) optimises continuous text embeddings fed directly to the denoiser. Following the reviewer's question, we designed an adaptive attack on SD-1.4. We added a term to the paper's attack, weighted by λ, that pulls the denoiser representations CIRCE edits toward the average benign representation, so that the attack avoids CIRCE's target cluster. We swept λ from 0.01 to 10 on CIRCE, STEREO, TRCE and SD-1.4. On the baselines, larger λ only weakens the attack, so their strongest attack stays at a small λ or the paper's attack. We scored all images with three judges: NudeNet, the detector the paper used for white-box attacks, and two large VLMs (Qwen3-VL-32B and InternVL3-78B) with a strict rubric that counts only exposed intimate areas.

Table 3: White-box attack success on SD-1.4 (% of 80 images, mean over 2 attack seeds and the 3 judges)

modelpaper's attackstrongest attack
CIRCE547 (CIRCE-aware, λ = 3)
STEREO (adversarial training)3047 (λ = 0.01)
TRCE (cross-attention edit and fine-tuning)8383 (paper's attack)
undefended SD-1.48282 (paper's attack)

In the paper, the white-box attack recovered the concept from CIRCE in 0 of 80 images under NudeNet (5% here, averaged over two attack seeds and three judges, which is also why the other numbers in Table 3 differ from the paper's). We note that our first CIRCE-aware attack (λ = 0.1) raised this to 18%, but we wanted to break CIRCE further, so we swept λ from 0.01 to 10, which finds a setting that reaches 47% (λ = 3). The numbers for each judge are in the figure. Both strict VLM judges flag 0 of 80 benign images, and the 32B judge agrees with NudeNet (κ = 0.75) (figure).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Comment

Summary of the revision

We thank all reviewers again for their reviews and the discussion. We are glad that all three reviewers recognized the breadth of our evaluation, from fixed jailbreak prompts to adaptive, white-box and fine-tuning attacks, together with multiple judge-based and human evaluations of image quality, and the strength of CIRCE, with competitive adversarial robustness. Among the reviewers who replied as of now, we are happy that almost all concerns were fully addressed. We have uploaded the revised PDF with all additions in blue.

Main changes

  • Seed variability (knAR, upYc). Erasure methods vary across training seeds, CIRCE as well as the baselines, so the main tables and Figure 3 now also report means over multiple training runs, and Table 4 (Appendix A.1) gives the mean and 95% confidence interval of every metric for every method, over 3 to 13 training runs each. On average, CIRCE remains the hardest model to attack among those that keep image quality high. Sec. 6.3 now describes its preservation on Qwen-Image-2.1 from these averages: CIRCE fails more often than STEREO and circuit breakers on benign prompts close to the erased concept (NearConcept), in exchange for much lower attack success.

  • CIRCE's geometry (knAR). The closest existing method is circuit breakers, which also push the concept's representations away but neither cluster them nor separate them from benign ones. Trained with CIRCE's data and budget, over ten seeds each and, for circuit breakers, a tuning sweep, circuit breakers and CIRCE without clustering are both easier to attack than CIRCE at the same level of preservation (Sec. 6.4, Appendix B.3).

  • Representation versus output supervision (rcTf). We trained versions of CIRCE whose losses act on the model's output instead of its internal representations, keeping everything else the same. Acting on representations gives equal or lower attack success at the same preservation, and measurements confirm that each loss changes the representations as intended (Sec. 4, Appendix B.4).

  • Comparison with additional methods (upYc). We added HiRM and TRCE to the main SD-1.4 table, with image fidelity in Table 2a and training cost in Appendix A.2.

  • New attacks (knAR, rcTf). As requested, we add an adaptive white-box attack designed against CIRCE's geometry and the visual inversion attacks TINA and TINA+, and we fixed and extended the continuous inversion experiment on Qwen-Image-2.1. For nudity, no adaptive black-box attack breaks CIRCE consistently. White-box embedding attacks, especially those adapted to a specific defense, remain hard for every erasure method. The abstract and conclusion now state this limitation (Sec. 6.2, Appendix A.7–A.8).

  • Robustness on objects and after fine-tuning (upYc). The adaptive recovery results on UnlearnCanvas styles and objects, and after longer benign fine-tuning on Qwen-Image-2.1, are now in the paper (Sec. 6.2, Appendix B.3).

  • SalUn configurations (rcTf). Both SalUn configurations are now evaluated on the full UnlearnCanvas benchmark (Appendix A.3).

We believe these additions strengthen the paper, and we remain available to discuss any remaining questions until the end of the discussion period.

Edit: After the discussion period, all three reviewers recommend acceptance, and we are happy to hear that all reviewers mentioned that their concerns were addressed. We have uploaded the revised PDF with all additions in blue. We really appreciate all the helpful feedback we received.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.