- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper investigates the robustness–retention tradeoff in concept erasure of diffusion models from an optimization trajectory perspective, proposing RRCE with separate retain and defence phases
Critical strengths
- The experimental results are favourable against the tested baselines on a number of concepts
Critical weaknesses
- The narrow experimental scope limits the generality of the results. In particular, evaluations are conducted on only small sized models. Larger models such as FLUX was not included. There is only one concept per category. The scope could have been extended to further include some common attack methods.
- The pipeline itself is similar to STEREO [1]. Although cited in the paper, it is not properly credited when laying out the method e.g. in Sec 3
[1] Srivatsan, Koushik, et al. "Stereo: A two-stage framework for adversarially robust concept erasing from text-to-image diffusion models." CVPR, 2025.
Questions for authors
Can the author please provide additional statistical analysis to the experimental results?
3 comments
Responses to Reviewer 3HU3 Comments [2/3]
W1 + Q1 (part 2)
Response: Thank you very much for this useful suggestion.
Five-seed statistics. We repeat training-based methods on SD1.4 Nudity, Dog, and Van Gogh with five random training seeds and the same evaluation protocol. The tables report mean ± standard deviation. UCE and RECE are closed-form edits without training, so their fixed results and the unedited model reference are shown with zero standard deviation.
Nudity (five training seeds)
| Method | Erased ↓ | UD ↓ | CCE ↓ | Retain ↑ | CLIP ↑ | FID ↓ |
|---|---|---|---|---|---|---|
| SD v1.4 (reference) | 28.86 ± 0.00 | 27.37 ± 0.00 | 50.53 ± 0.00 | 97.80 ± 0.00 | 31.36 ± 0.00 | 14.15 ± 0.00 |
| ESD | 15.50 ± 2.69 | 8.84 ± 1.77 | 50.74 ± 1.56 | 89.96 ± 2.36 | 30.81 ± 0.05 | 14.50 ± 0.09 |
| UCE | 5.45 ± 0.00 | 5.26 ± 0.00 | 21.05 ± 0.00 | 93.20 ± 0.00 | 31.08 ± 0.00 | 14.38 ± 0.00 |
| RECE | 1.14 ± 0.00 | 1.05 ± 0.00 | 5.26 ± 0.00 | 93.00 ± 0.00 | 31.06 ± 0.00 | 14.69 ± 0.00 |
| MACE | 2.27 ± 1.96 | 1.05 ± 0.75 | 10.95 ± 1.60 | 49.48 ± 3.89 | 29.37 ± 0.07 | 12.26 ± 0.09 |
| AdvUnlearn | 0.73 ± 1.06 | 0.84 ± 0.88 | 6.53 ± 1.16 | 91.76 ± 4.01 | 29.77 ± 0.07 | 15.38 ± 0.05 |
| STEREO | 1.09 ± 1.13 | 0.42 ± 0.58 | 2.53 ± 1.60 | 71.12 ± 3.77 | 29.79 ± 0.06 | 17.96 ± 0.09 |
| AGE | 1.14 ± 1.17 | 1.69 ± 0.94 | 2.32 ± 1.37 | 77.60 ± 2.05 | 29.96 ± 0.03 | 11.22 ± 0.10 |
| ReCARE | 1.32 ± 1.30 | 1.26 ± 1.37 | 3.58 ± 1.60 | 82.68 ± 2.85 | 30.22 ± 0.04 | 14.83 ± 0.09 |
| AEGIS | 13.41 ± 2.78 | 5.90 ± 1.60 | 45.27 ± 1.49 | 78.16 ± 2.38 | 30.13 ± 0.07 | 11.83 ± 0.12 |
| RRCE | 1.73 ± 1.44 | 0.63 ± 0.94 | 1.26 ± 0.88 | 94.28 ± 4.33 | 30.66 ± 0.03 | 14.68 ± 0.06 |
Dog (five training seeds)
| Method | Erased ↓ | UD ↓ | CCE ↓ | Retain ↑ | CLIP ↑ | FID ↓ |
|---|---|---|---|---|---|---|
| SD v1.4 (reference) | 100.00 ± 0.00 | 94.00 ± 0.00 | 99.80 ± 0.00 | 89.20 ± 0.00 | 31.36 ± 0.00 | 14.15 ± 0.00 |
| ESD | 90.00 ± 2.26 | 82.40 ± 1.67 | 98.00 ± 1.41 | 84.96 ± 3.60 | 30.98 ± 0.06 | 14.78 ± 0.08 |
| UCE | 2.60 ± 0.00 | 22.00 ± 0.00 | 100.00 ± 0.00 | 85.80 ± 0.00 | 31.06 ± 0.00 | 14.96 ± 0.00 |
| RECE | 0.80 ± 0.00 | 14.00 ± 0.00 | 99.80 ± 0.00 | 86.40 ± 0.00 | 31.01 ± 0.00 | 15.41 ± 0.00 |
| MACE | 7.60 ± 2.40 | 57.20 ± 2.28 | 98.40 ± 1.67 | 87.72 ± 2.84 | 31.13 ± 0.06 | 13.70 ± 0.05 |
| AdvUnlearn | 29.56 ± 1.89 | 4.80 ± 3.35 | 98.40 ± 1.67 | 86.00 ± 2.81 | 28.01 ± 0.07 | 15.06 ± 0.06 |
| STEREO | 1.44 ± 1.24 | 2.00 ± 2.00 | 32.20 ± 1.67 | 9.16 ± 3.47 | 28.47 ± 0.04 | 26.60 ± 0.08 |
| AGE | 14.08 ± 1.45 | 20.00 ± 3.16 | 72.00 ± 3.35 | 53.84 ± 1.59 | 30.08 ± 0.06 | 12.46 ± 0.07 |
| ReCARE | 5.56 ± 1.73 | 12.80 ± 1.79 | 18.00 ± 3.03 | 55.12 ± 3.48 | 30.07 ± 0.04 | 14.99 ± 0.08 |
| AEGIS | 2.68 ± 2.14 | 17.60 ± 3.58 | 77.40 ± 3.16 | 20.84 ± 2.42 | 28.91 ± 0.07 | 15.45 ± 0.09 |
| RRCE | 2.56 ± 1.29 | 1.60 ± 1.67 | 10.40 ± 1.79 | 85.64 ± 2.73 | 30.33 ± 0.01 | 13.89 ± 0.06 |
Van Gogh (five training seeds)
| Method | Erased ↓ | UD ↓ | CCE ↓ | Retain ↑ | CLIP ↑ | FID ↓ |
|---|---|---|---|---|---|---|
| SD v1.4 (reference) | 73.20 ± 0.00 | 82.00 ± 0.00 | 33.80 ± 0.00 | 99.00 ± 0.00 | 31.36 ± 0.00 | 14.15 ± 0.00 |
| ESD | 4.00 ± 2.57 | 10.40 ± 2.61 | 54.60 ± 2.97 | 95.00 ± 3.30 | 31.10 ± 0.02 | 13.93 ± 0.05 |
| UCE | 14.00 ± 0.00 | 24.00 ± 0.00 | 43.80 ± 0.00 | 97.00 ± 0.00 | 31.17 ± 0.00 | 13.95 ± 0.00 |
| RECE | 2.00 ± 0.00 | 14.00 ± 0.00 | 49.40 ± 0.00 | 95.40 ± 0.00 | 31.13 ± 0.00 | 13.76 ± 0.00 |
| MACE | 4.20 ± 2.54 | 11.60 ± 1.67 | 27.60 ± 2.00 | 95.36 ± 1.69 | 31.10 ± 0.05 | 14.08 ± 0.10 |
| AdvUnlearn | 2.44 ± 1.71 | 2.80 ± 1.79 | 41.00 ± 1.79 | 14.64 ± 3.50 | 29.38 ± 0.05 | 16.41 ± 0.04 |
| STEREO | 0.96 ± 0.77 | 4.80 ± 3.35 | 1.60 ± 1.67 | 62.28 ± 2.71 | 30.89 ± 0.04 | 15.57 ± 0.05 |
| AGE | 1.20 ± 1.38 | 4.80 ± 2.28 | 4.80 ± 4.00 | 74.96 ± 2.50 | 30.39 ± 0.03 | 11.06 ± 0.07 |
| ReCARE | 1.44 ± 1.51 | 2.40 ± 2.61 | 26.60 ± 2.19 | 79.68 ± 3.75 | 30.85 ± 0.07 | 19.44 ± 0.07 |
| AEGIS | 1.88 ± 1.51 | 4.80 ± 1.79 | 15.60 ± 2.97 | 66.64 ± 3.38 | 29.89 ± 0.07 | 15.10 ± 0.09 |
| RRCE | 1.00 ± 1.21 | 1.20 ± 1.79 | 0.80 ± 1.10 | 93.76 ± 2.85 | 31.10 ± 0.06 | 14.42 ± 0.11 |
The five-seed results show the lowest mean CCE ASR for RRCE on all three tasks, alongside higher mean retention than STEREO and ReCARE.
Responses to Reviewer 3HU3 Comments [3/3]
W1 + Q1 (part 3)
Response: Thank you very much for this useful suggestion.
Additional attack methods and increased attack effort. We added Ring-A-Bell and P4D evaluations on SD1.4 for Nudity, Dog, and Van Gogh, comparing ten erasure methods. The table reports attack success rates (ASR, %, lower is better).
| Method | Nudity: Ring-A-Bell ↓ | Nudity: P4D ↓ | Dog: Ring-A-Bell ↓ | Dog: P4D ↓ | Van Gogh: Ring-A-Bell ↓ | Van Gogh: P4D ↓ |
|---|---|---|---|---|---|---|
| ESD | 50.53 | 69.01 | 28.0 | 38.0 | 22.0 | 30.0 |
| UCE | 29.47 | 76.06 | 44.0 | 70.0 | 40.0 | 66.0 |
| RECE | 10.53 | 66.20 | 20.0 | 54.0 | 10.0 | 48.0 |
| MACE | 6.32 | 75.35 | 22.0 | 48.0 | 8.0 | 42.0 |
| AdvUnlearn | 2.11 | 23.94 | 10.0 | 12.0 | 6.0 | 10.0 |
| STEREO | 1.05 | 24.65 | 4.0 | 0.0 | 4.0 | 0.0 |
| AGE | 25.26 | 89.45 | 16.0 | 24.0 | 18.0 | 24.0 |
| ReCARE | 2.11 | 22.37 | 6.0 | 4.0 | 6.0 | 2.0 |
| AEGIS | 3.52 | 21.12 | 6.0 | 8.0 | 6.0 | 4.0 |
| RRCE | 1.05 | 19.71 | 4.0 | 0.0 | 2.0 | 0.0 |
RRCE achieves the lowest or tied-lowest ASR across both additional attacks and all three concepts. Compared with STEREO, it improves Nudity under P4D and Van Gogh under Ring-A-Bell, and ties in the remaining settings.
We also strengthen the existing UnlearnDiffAtk (UD) and Circumventing Concept Erasure (CCE) evaluations by doubling the optimization budgets and adding three random restarts.
| Concept | Method | CCE: 3000×1 / 6000×1 / 3000×3 | UD: 40×1 / 80×1 / 40×3 |
|---|---|---|---|
| Nudity | RRCE | 1.05 / 2.11 / 1.05 | 0.00 / 0.00 / 0.00 |
| Nudity | STEREO | 2.11 / 3.16 / 2.11 | 0.00 / 0.00 / 0.00 |
| Dog | RRCE | 9.20 / 10.80 / 9.40 | 2.00 / 2.00 / 2.00 |
| Dog | STEREO | 30.60 / 32.60 / 31.00 | 0.00 / 0.00 / 2.00 |
| Van Gogh | RRCE | 0.00 / 0.00 / 0.00 | 0.00 / 0.00 / 0.00 |
| Van Gogh | STEREO | 0.00 / 0.00 / 0.00 | 4.00 / 4.00 / 4.00 |
ASR (%). Settings are iterations × restarts. For three restarts, the highest-scoring candidate on independent validation data is selected per model for CCE and per prompt for UD, then evaluated on held-out test samples.
RRCE's CCE ASR remains lower on Nudity and Dog and tied on Van Gogh, while Dog UD favors STEREO with one restart and ties with three restarts.
W2: RRCE's contribution and its relationship to STEREO
Response: Thank you very much for this insightful comment. RRCE's central contribution is the analysis of cross-time erase–retain interactions and the resulting optimizer design: separate first- and second-moment states for each objective, with adaptive momentum based on adjacent-gradient agreement.
RRCE builds on STEREO's textual-inversion recovery discovery and anchor-guided compositional erasure. We will clarify this relationship in Sec. 3.1. The main differences in the recovery and defense procedure are:
| Method | Recovery discovery | Defense and final model |
|---|---|---|
| STEREO | Stage 1 alternates erasure and textual inversion in a min-max search to collect adversarial tokens. | Stage 2 starts from the pretrained model and performs final compositional erasure using the collected tokens and anchor concepts. |
| RRCE | Each recovery round searches for a token against the current defended model. | The next defense block continues updating that model through anchor-guided erasure and teacher-matching retention, coordinated by objective-specific optimizer states and adaptive momentum. |
The following Nudity ablations test these choices within RRCE. The state comparison fixes the loss schedule and momentum stabilizer, the momentum comparison fixes alternating updates and separate states, and the schedule comparison matches recovery rounds and the total defense budget.
| Controlled change within RRCE | CCE ASR (%) ↓ | Retain (%) ↑ | Source in the paper |
|---|---|---|---|
| Shared → separate first and second moments | 4.21 → 1.05 | 90.80 → 93.20 | Appendix D.3, Table A5 |
| Fixed momentum 0.5 → adaptive momentum | 2.10 → 1.05 | 88.20 → 93.20 | Main-text Table 4 |
| Two-stage → online recovery and defense | 4.21 → 1.05 | 92.60 → 93.20 | Appendix D.1 |
Each change lowers CCE ASR and increases Retain in its matched comparison.
Responses to Reviewer 3HU3 Comments [1/3]
W1 + Q1 (part 1)
Response: Thank you very much for these useful suggestions. We added Harm, Car, and Monet on SD1.4 and evaluated Nudity, Dog, and Van Gogh on FLUX.1-dev.
1. Additional concepts on SD1.4. The evaluation now covers six concepts, with two targets in each of the safety, object, and artistic-style categories. The new experiments use the paper's six-metric protocol. Erased, UD, CCE, and Retain are percentages.
Harm
Car
Monet
RRCE achieves the lowest or tied-lowest CCE ASR while retaining unrelated concepts better than STEREO across all three added SD1.4 targets.
2. FLUX.1-dev. We evaluate three targets using the official FLUX implementations of ESD and UCE as baselines.
Nudity (FLUX.1-dev)
Dog (FLUX.1-dev)
Van Gogh (FLUX.1-dev)
RRCE has lower Erased, UD, and CCE ASR and higher Retain than both baselines on all three targets.