acceptodds
Under review as a conference paper at ICLR 2027

When Compatible Gradients Conflict over Time: Trajectory-Aware Diffusion Model Unlearning

Abstract

Concept erasure in text-to-image diffusion models must prevent adaptive attacks from recovering an erased concept while preserving neighboring semantics and generation quality. Existing methods largely address this robustness–retention trade-off through loss balancing or single-step gradient surgery, implicitly treating erase–retain interference as a local optimization phenomenon. In this paper, we show that this view is incomplete. Across unsafe-content, object, and artistic-style erasure, erase and retain gradients are nearly orthogonal or weakly aligned at individual steps, yet their accumulated directions become increasingly antagonistic over training. We attribute this reversal to two forms of optimizer-history dependence. Specifically, shared first- and second-moment optimizer states mix unmatched erase and retain gradients across optimization steps, creating trajectory conflict despite same-step compatibility. Meanwhile, fixed optimizer memory cannot accommodate directional changes within each objective trajectory. Based on this diagnosis, we propose a trajectory-aware Robust-Retain Concept Erasure framework (RRCE) that alternates recovery discovery with erase–retain defense. The recovery phase uses textual inversion to discover target representations that can still recover the target concept from the current model. The defense phase suppresses these representations through anchor-guided erasure and preserves non-target behavior through teacher-matching retention. To address the two trajectory-level failures, RRCE maintains objective-specific optimizer states and adapts momentum according to adjacent-gradient similarity within each trajectory. Experiments demonstrate state-of-the-art robustness–retention performance against adaptive recovery attacks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Diffusion Models, Machine Unlearning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3HU3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates the robustness–retention tradeoff in concept erasure of diffusion models from an optimization trajectory perspective, proposing RRCE with separate retain and defence phases

Critical strengths
  1. The experimental results are favourable against the tested baselines on a number of concepts
Critical weaknesses
  1. The narrow experimental scope limits the generality of the results. In particular, evaluations are conducted on only small sized models. Larger models such as FLUX was not included. There is only one concept per category. The scope could have been extended to further include some common attack methods.
  2. The pipeline itself is similar to STEREO [1]. Although cited in the paper, it is not properly credited when laying out the method e.g. in Sec 3

[1] Srivatsan, Koushik, et al. "Stereo: A two-stage framework for adversarially robust concept erasing from text-to-image diffusion models." CVPR, 2025.

Questions for authors

Can the author please provide additional statistical analysis to the experimental results?

3 comments
AuthorsOpenReview ↗

Responses to Reviewer 3HU3 Comments [1/3]

W1 + Q1 (part 1)

Response: Thank you very much for these useful suggestions. We added Harm, Car, and Monet on SD1.4 and evaluated Nudity, Dog, and Van Gogh on FLUX.1-dev.

1. Additional concepts on SD1.4. The evaluation now covers six concepts, with two targets in each of the safety, object, and artistic-style categories. The new experiments use the paper's six-metric protocol. Erased, UD, CCE, and Retain are percentages.

Harm

MethodErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
SD v1.4 (reference)37.7059.0078.6097.6031.3614.15
ESD13.0426.0081.8095.4030.7814.46
UCE15.7822.0068.6096.4031.0414.52
RECE8.9312.0054.4096.0031.0214.71
MACE4.508.0046.8092.2029.2712.19
STEREO1.692.0024.6078.6029.8117.98
ReCARE2.672.0037.8090.4030.1714.91
RRCE1.762.0018.6096.0030.6614.67

Car

MethodErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
SD v1.4 (reference)92.4092.0089.00100.0031.3614.15
ESD71.0080.0085.4094.2031.0514.96
UCE21.4032.0082.4099.0030.9614.73
RECE1.2012.0086.2097.6031.0915.51
MACE8.6046.0086.8098.2031.1813.37
STEREO0.402.008.8041.8029.4518.56
ReCARE3.804.0012.4082.4029.9414.74
RRCE0.002.004.2098.4030.6414.23

Monet

MethodErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
SD v1.4 (reference)78.8084.0042.6098.4031.3614.15
ESD13.4010.0044.2092.0031.1013.89
UCE17.0026.0063.8096.4031.1614.01
RECE6.2012.0055.8095.0031.1313.64
MACE9.0012.0060.2094.4031.1413.96
STEREO0.002.000.0065.2030.9115.00
ReCARE1.002.002.6080.0030.8419.89
RRCE0.000.000.0096.4031.1214.38

RRCE achieves the lowest or tied-lowest CCE ASR while retaining unrelated concepts better than STEREO across all three added SD1.4 targets.

2. FLUX.1-dev. We evaluate three targets using the official FLUX implementations of ESD and UCE as baselines.

Nudity (FLUX.1-dev)

ModelErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
FLUX.1-dev (reference)45.2647.3668.42100.0030.8720.41
ESD3.6013.6827.3695.2028.4423.08
UCE1.2010.5215.7894.0024.5630.71
RRCE0.000.001.2097.4030.5623.59

Dog (FLUX.1-dev)

ModelErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
FLUX.1-dev (reference)100.0092.80100.00100.0030.8720.41
ESD12.8062.40100.0090.6028.1224.32
UCE10.4032.2096.0092.8023.7832.13
RRCE3.6012.0023.4094.2030.4224.68

Van Gogh (FLUX.1-dev)

ModelErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
FLUX.1-dev (reference)12.0014.0020.4066.4030.8720.41
ESD4.406.2019.4062.0029.6722.84
UCE2.003.6016.8063.6028.7426.22
RRCE0.001.003.0064.4030.7222.78

RRCE has lower Erased, UD, and CCE ASR and higher Retain than both baselines on all three targets.

AuthorsOpenReview ↗

Responses to Reviewer 3HU3 Comments [2/3]

W1 + Q1 (part 2)

Response: Thank you very much for this useful suggestion.

Five-seed statistics. We repeat training-based methods on SD1.4 Nudity, Dog, and Van Gogh with five random training seeds and the same evaluation protocol. The tables report mean ± standard deviation. UCE and RECE are closed-form edits without training, so their fixed results and the unedited model reference are shown with zero standard deviation.

Nudity (five training seeds)

MethodErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
SD v1.4 (reference)28.86 ± 0.0027.37 ± 0.0050.53 ± 0.0097.80 ± 0.0031.36 ± 0.0014.15 ± 0.00
ESD15.50 ± 2.698.84 ± 1.7750.74 ± 1.5689.96 ± 2.3630.81 ± 0.0514.50 ± 0.09
UCE5.45 ± 0.005.26 ± 0.0021.05 ± 0.0093.20 ± 0.0031.08 ± 0.0014.38 ± 0.00
RECE1.14 ± 0.001.05 ± 0.005.26 ± 0.0093.00 ± 0.0031.06 ± 0.0014.69 ± 0.00
MACE2.27 ± 1.961.05 ± 0.7510.95 ± 1.6049.48 ± 3.8929.37 ± 0.0712.26 ± 0.09
AdvUnlearn0.73 ± 1.060.84 ± 0.886.53 ± 1.1691.76 ± 4.0129.77 ± 0.0715.38 ± 0.05
STEREO1.09 ± 1.130.42 ± 0.582.53 ± 1.6071.12 ± 3.7729.79 ± 0.0617.96 ± 0.09
AGE1.14 ± 1.171.69 ± 0.942.32 ± 1.3777.60 ± 2.0529.96 ± 0.0311.22 ± 0.10
ReCARE1.32 ± 1.301.26 ± 1.373.58 ± 1.6082.68 ± 2.8530.22 ± 0.0414.83 ± 0.09
AEGIS13.41 ± 2.785.90 ± 1.6045.27 ± 1.4978.16 ± 2.3830.13 ± 0.0711.83 ± 0.12
RRCE1.73 ± 1.440.63 ± 0.941.26 ± 0.8894.28 ± 4.3330.66 ± 0.0314.68 ± 0.06

Dog (five training seeds)

MethodErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
SD v1.4 (reference)100.00 ± 0.0094.00 ± 0.0099.80 ± 0.0089.20 ± 0.0031.36 ± 0.0014.15 ± 0.00
ESD90.00 ± 2.2682.40 ± 1.6798.00 ± 1.4184.96 ± 3.6030.98 ± 0.0614.78 ± 0.08
UCE2.60 ± 0.0022.00 ± 0.00100.00 ± 0.0085.80 ± 0.0031.06 ± 0.0014.96 ± 0.00
RECE0.80 ± 0.0014.00 ± 0.0099.80 ± 0.0086.40 ± 0.0031.01 ± 0.0015.41 ± 0.00
MACE7.60 ± 2.4057.20 ± 2.2898.40 ± 1.6787.72 ± 2.8431.13 ± 0.0613.70 ± 0.05
AdvUnlearn29.56 ± 1.894.80 ± 3.3598.40 ± 1.6786.00 ± 2.8128.01 ± 0.0715.06 ± 0.06
STEREO1.44 ± 1.242.00 ± 2.0032.20 ± 1.679.16 ± 3.4728.47 ± 0.0426.60 ± 0.08
AGE14.08 ± 1.4520.00 ± 3.1672.00 ± 3.3553.84 ± 1.5930.08 ± 0.0612.46 ± 0.07
ReCARE5.56 ± 1.7312.80 ± 1.7918.00 ± 3.0355.12 ± 3.4830.07 ± 0.0414.99 ± 0.08
AEGIS2.68 ± 2.1417.60 ± 3.5877.40 ± 3.1620.84 ± 2.4228.91 ± 0.0715.45 ± 0.09
RRCE2.56 ± 1.291.60 ± 1.6710.40 ± 1.7985.64 ± 2.7330.33 ± 0.0113.89 ± 0.06

Van Gogh (five training seeds)

MethodErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
SD v1.4 (reference)73.20 ± 0.0082.00 ± 0.0033.80 ± 0.0099.00 ± 0.0031.36 ± 0.0014.15 ± 0.00
ESD4.00 ± 2.5710.40 ± 2.6154.60 ± 2.9795.00 ± 3.3031.10 ± 0.0213.93 ± 0.05
UCE14.00 ± 0.0024.00 ± 0.0043.80 ± 0.0097.00 ± 0.0031.17 ± 0.0013.95 ± 0.00
RECE2.00 ± 0.0014.00 ± 0.0049.40 ± 0.0095.40 ± 0.0031.13 ± 0.0013.76 ± 0.00
MACE4.20 ± 2.5411.60 ± 1.6727.60 ± 2.0095.36 ± 1.6931.10 ± 0.0514.08 ± 0.10
AdvUnlearn2.44 ± 1.712.80 ± 1.7941.00 ± 1.7914.64 ± 3.5029.38 ± 0.0516.41 ± 0.04
STEREO0.96 ± 0.774.80 ± 3.351.60 ± 1.6762.28 ± 2.7130.89 ± 0.0415.57 ± 0.05
AGE1.20 ± 1.384.80 ± 2.284.80 ± 4.0074.96 ± 2.5030.39 ± 0.0311.06 ± 0.07
ReCARE1.44 ± 1.512.40 ± 2.6126.60 ± 2.1979.68 ± 3.7530.85 ± 0.0719.44 ± 0.07
AEGIS1.88 ± 1.514.80 ± 1.7915.60 ± 2.9766.64 ± 3.3829.89 ± 0.0715.10 ± 0.09
RRCE1.00 ± 1.211.20 ± 1.790.80 ± 1.1093.76 ± 2.8531.10 ± 0.0614.42 ± 0.11

The five-seed results show the lowest mean CCE ASR for RRCE on all three tasks, alongside higher mean retention than STEREO and ReCARE.

AuthorsOpenReview ↗

Responses to Reviewer 3HU3 Comments [3/3]

W1 + Q1 (part 3)

Response: Thank you very much for this useful suggestion.

Additional attack methods and increased attack effort. We added Ring-A-Bell and P4D evaluations on SD1.4 for Nudity, Dog, and Van Gogh, comparing ten erasure methods. The table reports attack success rates (ASR, %, lower is better).

MethodNudity: Ring-A-Bell ↓Nudity: P4D ↓Dog: Ring-A-Bell ↓Dog: P4D ↓Van Gogh: Ring-A-Bell ↓Van Gogh: P4D ↓
ESD50.5369.0128.038.022.030.0
UCE29.4776.0644.070.040.066.0
RECE10.5366.2020.054.010.048.0
MACE6.3275.3522.048.08.042.0
AdvUnlearn2.1123.9410.012.06.010.0
STEREO1.0524.654.00.04.00.0
AGE25.2689.4516.024.018.024.0
ReCARE2.1122.376.04.06.02.0
AEGIS3.5221.126.08.06.04.0
RRCE1.0519.714.00.02.00.0

RRCE achieves the lowest or tied-lowest ASR across both additional attacks and all three concepts. Compared with STEREO, it improves Nudity under P4D and Van Gogh under Ring-A-Bell, and ties in the remaining settings.

We also strengthen the existing UnlearnDiffAtk (UD) and Circumventing Concept Erasure (CCE) evaluations by doubling the optimization budgets and adding three random restarts.

ConceptMethodCCE: 3000×1 / 6000×1 / 3000×3UD: 40×1 / 80×1 / 40×3
NudityRRCE1.05 / 2.11 / 1.050.00 / 0.00 / 0.00
NuditySTEREO2.11 / 3.16 / 2.110.00 / 0.00 / 0.00
DogRRCE9.20 / 10.80 / 9.402.00 / 2.00 / 2.00
DogSTEREO30.60 / 32.60 / 31.000.00 / 0.00 / 2.00
Van GoghRRCE0.00 / 0.00 / 0.000.00 / 0.00 / 0.00
Van GoghSTEREO0.00 / 0.00 / 0.004.00 / 4.00 / 4.00

ASR (%). Settings are iterations × restarts. For three restarts, the highest-scoring candidate on independent validation data is selected per model for CCE and per prompt for UD, then evaluated on held-out test samples.

RRCE's CCE ASR remains lower on Nudity and Dog and tied on Van Gogh, while Dog UD favors STEREO with one restart and ties with three restarts.

W2: RRCE's contribution and its relationship to STEREO

Response: Thank you very much for this insightful comment. RRCE's central contribution is the analysis of cross-time erase–retain interactions and the resulting optimizer design: separate first- and second-moment states for each objective, with adaptive momentum based on adjacent-gradient agreement.

RRCE builds on STEREO's textual-inversion recovery discovery and anchor-guided compositional erasure. We will clarify this relationship in Sec. 3.1. The main differences in the recovery and defense procedure are:

MethodRecovery discoveryDefense and final model
STEREOStage 1 alternates erasure and textual inversion in a min-max search to collect adversarial tokens.Stage 2 starts from the pretrained model and performs final compositional erasure using the collected tokens and anchor concepts.
RRCEEach recovery round searches for a token against the current defended model.The next defense block continues updating that model through anchor-guided erasure and teacher-matching retention, coordinated by objective-specific optimizer states and adaptive momentum.

The following Nudity ablations test these choices within RRCE. The state comparison fixes the loss schedule and momentum stabilizer, the momentum comparison fixes alternating updates and separate states, and the schedule comparison matches recovery rounds and the total defense budget.

Controlled change within RRCECCE ASR (%) ↓Retain (%) ↑Source in the paper
Shared → separate first and second moments4.21 → 1.0590.80 → 93.20Appendix D.3, Table A5
Fixed momentum 0.5 → adaptive momentum2.10 → 1.0588.20 → 93.20Main-text Table 4
Two-stage → online recovery and defense4.21 → 1.0592.60 → 93.20Appendix D.1

Each change lowers CCE ASR and increases Retain in its matched comparison.

Reviewer_bu5COpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the optimization conflict between erasure and retention in diffusion model concept erasure. The authors find that the two sets of gradients are nearly orthogonal for a single step, but they become opposed once they accumulate over many steps. Building on this, the authors propose RRCE. The method alternates between an adversarial recovery search and erasure defense. Text inversion finds representations that can still recover the target concept, and training then combines anchor guided erasure with teacher matched retention. In the optimization, erasure and retention keep separate momentum and second order states, and the momentum is adjusted according to the similarity of adjacent gradients. The experiments cover unsafe content, objects, and artistic styles. They show that the method strikes a good balance between resisting recovery attacks and preserving nearby semantics.

Critical strengths
  • Analyzing the conflict between erasure and retention through optimization history is an illuminating angle.
  • The design of separate states and adaptive momentum is clear, and the ablation of the main components is fairly thorough.
  • On the Dog task, the CCE attack success rate is 9.2% while retention reaches 87%, which is a good trade-off.
Critical weaknesses
  • The gradient turning has not ruled out the influence of random sampling, and the attribution to curvature is not yet sufficient.
  • The experiments are limited to SD1.4 and three targets, and they lack error ranges over multiple training seeds.
  • The retention evaluation centers on the nearby concepts used in training. The behavior on concepts not seen in training is still unclear.
Questions for authors
  1. After fixing the samples, the noise, and the diffusion timestep, do the gradients still turn substantially?
  2. Does the same cumulative conflict appear when training starts from the original model?
  3. Is the cumulative conflict in Figure 1 equally visible within the effective memory window of Adam?
  4. Does shuffling the order of the adaptive momentum coefficients still beat a fixed 0.5?
  5. How much do the main results vary across several training seeds?
  6. With more attack steps and random restarts, does the advantage hold?
  7. How well are nearby concepts that never appear in training preserved?
  8. How much extra GPU memory do the dual optimizer and the gradient history cost?
5 comments
AuthorsOpenReview ↗

Responses to Reviewer bu5C Comments [1/4]

W1 + Q1: Gradient variation under fixed inputs

Response: Thank you very much for this insightful comment. Starting from pretrained SD1.4, we fixed the samples, noise, and diffusion timestep and measured adjacent-gradient cosine similarity along a 200-step trajectory, using the same 8,192 parameter coordinates throughout.

ObjectiveMean cosineMinimum cosineMaximum angle (approx.)
Erasure E0.8930.84033°
Retention R0.8880.71145°

The probe shows moderate directional changes while adjacent gradients remain substantially aligned.

This residual turning reflects changes in the conditional gradient field as the model parameters evolve. Our curvature analysis describes this parameter-induced variation. During stochastic training, changes in the sampled inputs also contribute to the observed gradient variation. We will clarify this joint interpretation in the manuscript.

RRCE's adaptive momentum coefficient responds to the directional consistency of these observed gradients within each objective. When adjacent gradients align, a larger coefficient retains more history, supporting temporal averaging of sampling fluctuations along a stable direction. When alignment falls, it lowers the coefficient, shortening the effective memory and increasing the current gradient's weight so that outdated directions have less influence on the update.

The main-text Nudity ablation supports this adaptive choice: with alternating updates and separate optimizer states held fixed, replacing fixed β1=0.5\beta_1=0.5 with adaptive momentum reduces CCE ASR from 2.10% to 1.05% and raises Retain from 88.20% to 93.20%.

Fixing the samples, noise, and diffusion timestep is an extreme diagnostic condition for isolating parameter-induced variation, whereas standard diffusion training uses random sampling to cover data and noise conditions.

Q2: Cumulative conflict from the original model

Response: Thank you very much for this useful suggestion. Starting from the original SD1.4 model, we ran 200 joint AdamW steps using the original Dog text condition without recovery tokens, and computed the following statistics on 8,192 recorded parameter coordinates.

Cumulative-gradient statisticResult
Evaluated points with negative cosine193/199
Minimum cumulative cosine−0.5963
Cumulative cosine at step 200−0.0540
Sum of same-time inner products+3.46 × 10⁻⁷
Sum of cross-time inner products−1.11 × 10⁻⁶

The cumulative cosine is negative at 193 of the 199 points where it is defined. The negative cross-time sum outweighs the positive same-time sum, showing that cumulative conflict also appears when training starts from the unmodified model.

Q3: Conflict within Adam's effective memory window

Response: Thank you very much for this insightful comment. In the Fig. 1 Dog-erasure diagnostic with SD1.4 and full U-Net training, we ran 200 steps with fixed β1=0.9\beta_1=0.9 and compared the cosine between exponentially weighted erase and retain gradients over three history ranges. The negative-correlation rate is the fraction of points with defined cosine at which it is negative.

Accumulation rangeMean cosineNegative-correlation rate
Latest 10 steps+0.0276618.3%
Latest 30 steps−0.0277568.6%
Full EMA history−0.0315171.2%

The latest 30 steps contain approximately 95.8% of the EMA weight. Their mean cosine of −0.02775 and negative-correlation rate of 68.6% are close to the full-history values of −0.03151 and 71.2%, placing the observed negative interaction within Adam's effective memory window.

Q4: Shuffling the adaptive momentum coefficients

Response: Thank you very much for this useful suggestion. For Dog erasure, we compare the original adaptive coefficients, a shuffled ordering of the same values, and fixed 0.5, keeping separate optimizer states for erasure and retention.

Training variantErased ↓CCE ASR ↓Retain ↑
Adaptive coefficients, separate states0.40%9.20%87.00%
Shuffled adaptive coefficients, separate states1.00%36.00%85.00%
Fixed 0.5, separate states0.40%13.80%81.50%

Shuffling increases CCE ASR from 13.80% to 36.00% relative to fixed 0.5, although Retain increases from 81.50% to 85.00%. The original adaptive ordering achieves the lowest CCE ASR and highest Retain among these controls. Because shuffling preserves the values and changes only their positions, this comparison supports the importance of when the coefficients are applied.

AuthorsOpenReview ↗

Responses to Reviewer bu5C Comments [2/4]

W2 + Q5 (part 1)

Response: Thank you very much for this useful suggestion. We expand the evaluation to FLUX.1-dev and additional SD1.4 concepts.

Model and concept coverage. We evaluate Nudity, Dog, and Van Gogh on FLUX.1-dev using the official FLUX implementations of ESD and UCE, the two original baselines with such implementations.

Nudity (FLUX.1-dev)

ModelErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
FLUX.1-dev (reference)45.2647.3668.42100.0030.8720.41
ESD3.6013.6827.3695.2028.4423.08
UCE1.2010.5215.7894.0024.5630.71
RRCE0.000.001.2097.4030.5623.59

Dog (FLUX.1-dev)

ModelErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
FLUX.1-dev (reference)100.0092.80100.00100.0030.8720.41
ESD12.8062.40100.0090.6028.1224.32
UCE10.4032.2096.0092.8023.7832.13
RRCE3.6012.0023.4094.2030.4224.68

Van Gogh (FLUX.1-dev)

ModelErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
FLUX.1-dev (reference)12.0014.0020.4066.4030.8720.41
ESD4.406.2019.4062.0029.6722.84
UCE2.003.6016.8063.6028.7426.22
RRCE0.001.003.0064.4030.7222.78

RRCE achieves lower Erased, UD, and CCE ASR and higher Retain than ESD and UCE on all three FLUX tasks.

We also add Harm, Car, and Monet on SD1.4, bringing the evaluation to six concepts, with the same six-metric protocol.

Harm

MethodErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
SD v1.4 (reference)37.7059.0078.6097.6031.3614.15
ESD13.0426.0081.8095.4030.7814.46
UCE15.7822.0068.6096.4031.0414.52
RECE8.9312.0054.4096.0031.0214.71
MACE4.508.0046.8092.2029.2712.19
STEREO1.692.0024.6078.6029.8117.98
ReCARE2.672.0037.8090.4030.1714.91
RRCE1.762.0018.6096.0030.6614.67

Car

MethodErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
SD v1.4 (reference)92.4092.0089.00100.0031.3614.15
ESD71.0080.0085.4094.2031.0514.96
UCE21.4032.0082.4099.0030.9614.73
RECE1.2012.0086.2097.6031.0915.51
MACE8.6046.0086.8098.2031.1813.37
STEREO0.402.008.8041.8029.4518.56
ReCARE3.804.0012.4082.4029.9414.74
RRCE0.002.004.2098.4030.6414.23

Monet

MethodErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
SD v1.4 (reference)78.8084.0042.6098.4031.3614.15
ESD13.4010.0044.2092.0031.1013.89
UCE17.0026.0063.8096.4031.1614.01
RECE6.2012.0055.8095.0031.1313.64
MACE9.0012.0060.2094.4031.1413.96
STEREO0.002.000.0065.2030.9115.00
ReCARE1.002.002.6080.0030.8419.89
RRCE0.000.000.0096.4031.1214.38

Across the added SD1.4 concepts, RRCE achieves the lowest or tied-lowest CCE ASR. It improves retention over STEREO on all three targets, including Monet, where the two methods tie on CCE ASR.

AuthorsOpenReview ↗

Responses to Reviewer bu5C Comments [3/4]

W2 + Q5 (part 2)

Response: Thank you very much for this useful suggestion.

Training variability. We repeat the training-based methods with five random training seeds on SD1.4 Nudity, Dog, and Van Gogh, keeping the evaluation protocol fixed. The tables report mean ± standard deviation. UCE and RECE use closed-form edits without training, so training seeds do not apply. Their fixed results and the unedited SD1.4 reference are displayed with zero standard deviation.

Nudity (five training seeds)

MethodErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
SD v1.4 (reference)28.86 ± 0.0027.37 ± 0.0050.53 ± 0.0097.80 ± 0.0031.36 ± 0.0014.15 ± 0.00
ESD15.50 ± 2.698.84 ± 1.7750.74 ± 1.5689.96 ± 2.3630.81 ± 0.0514.50 ± 0.09
UCE5.45 ± 0.005.26 ± 0.0021.05 ± 0.0093.20 ± 0.0031.08 ± 0.0014.38 ± 0.00
RECE1.14 ± 0.001.05 ± 0.005.26 ± 0.0093.00 ± 0.0031.06 ± 0.0014.69 ± 0.00
MACE2.27 ± 1.961.05 ± 0.7510.95 ± 1.6049.48 ± 3.8929.37 ± 0.0712.26 ± 0.09
AdvUnlearn0.73 ± 1.060.84 ± 0.886.53 ± 1.1691.76 ± 4.0129.77 ± 0.0715.38 ± 0.05
STEREO1.09 ± 1.130.42 ± 0.582.53 ± 1.6071.12 ± 3.7729.79 ± 0.0617.96 ± 0.09
AGE1.14 ± 1.171.69 ± 0.942.32 ± 1.3777.60 ± 2.0529.96 ± 0.0311.22 ± 0.10
ReCARE1.32 ± 1.301.26 ± 1.373.58 ± 1.6082.68 ± 2.8530.22 ± 0.0414.83 ± 0.09
AEGIS13.41 ± 2.785.90 ± 1.6045.27 ± 1.4978.16 ± 2.3830.13 ± 0.0711.83 ± 0.12
RRCE1.73 ± 1.440.63 ± 0.941.26 ± 0.8894.28 ± 4.3330.66 ± 0.0314.68 ± 0.06

Dog (five training seeds)

MethodErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
SD v1.4 (reference)100.00 ± 0.0094.00 ± 0.0099.80 ± 0.0089.20 ± 0.0031.36 ± 0.0014.15 ± 0.00
ESD90.00 ± 2.2682.40 ± 1.6798.00 ± 1.4184.96 ± 3.6030.98 ± 0.0614.78 ± 0.08
UCE2.60 ± 0.0022.00 ± 0.00100.00 ± 0.0085.80 ± 0.0031.06 ± 0.0014.96 ± 0.00
RECE0.80 ± 0.0014.00 ± 0.0099.80 ± 0.0086.40 ± 0.0031.01 ± 0.0015.41 ± 0.00
MACE7.60 ± 2.4057.20 ± 2.2898.40 ± 1.6787.72 ± 2.8431.13 ± 0.0613.70 ± 0.05
AdvUnlearn29.56 ± 1.894.80 ± 3.3598.40 ± 1.6786.00 ± 2.8128.01 ± 0.0715.06 ± 0.06
STEREO1.44 ± 1.242.00 ± 2.0032.20 ± 1.679.16 ± 3.4728.47 ± 0.0426.60 ± 0.08
AGE14.08 ± 1.4520.00 ± 3.1672.00 ± 3.3553.84 ± 1.5930.08 ± 0.0612.46 ± 0.07
ReCARE5.56 ± 1.7312.80 ± 1.7918.00 ± 3.0355.12 ± 3.4830.07 ± 0.0414.99 ± 0.08
AEGIS2.68 ± 2.1417.60 ± 3.5877.40 ± 3.1620.84 ± 2.4228.91 ± 0.0715.45 ± 0.09
RRCE2.56 ± 1.291.60 ± 1.6710.40 ± 1.7985.64 ± 2.7330.33 ± 0.0113.89 ± 0.06

Van Gogh (five training seeds)

MethodErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
SD v1.4 (reference)73.20 ± 0.0082.00 ± 0.0033.80 ± 0.0099.00 ± 0.0031.36 ± 0.0014.15 ± 0.00
ESD4.00 ± 2.5710.40 ± 2.6154.60 ± 2.9795.00 ± 3.3031.10 ± 0.0213.93 ± 0.05
UCE14.00 ± 0.0024.00 ± 0.0043.80 ± 0.0097.00 ± 0.0031.17 ± 0.0013.95 ± 0.00
RECE2.00 ± 0.0014.00 ± 0.0049.40 ± 0.0095.40 ± 0.0031.13 ± 0.0013.76 ± 0.00
MACE4.20 ± 2.5411.60 ± 1.6727.60 ± 2.0095.36 ± 1.6931.10 ± 0.0514.08 ± 0.10
AdvUnlearn2.44 ± 1.712.80 ± 1.7941.00 ± 1.7914.64 ± 3.5029.38 ± 0.0516.41 ± 0.04
STEREO0.96 ± 0.774.80 ± 3.351.60 ± 1.6762.28 ± 2.7130.89 ± 0.0415.57 ± 0.05
AGE1.20 ± 1.384.80 ± 2.284.80 ± 4.0074.96 ± 2.5030.39 ± 0.0311.06 ± 0.07
ReCARE1.44 ± 1.512.40 ± 2.6126.60 ± 2.1979.68 ± 3.7530.85 ± 0.0719.44 ± 0.07
AEGIS1.88 ± 1.514.80 ± 1.7915.60 ± 2.9766.64 ± 3.3829.89 ± 0.0715.10 ± 0.09
RRCE1.00 ± 1.211.20 ± 1.790.80 ± 1.1093.76 ± 2.8531.10 ± 0.0614.42 ± 0.11

Across the five-seed comparisons, RRCE achieves the lowest mean CCE ASR on all three tasks while maintaining higher mean retention than STEREO and ReCARE.

Reviewer_hPBAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the trade-off between robustness and non-target semantic preservation in concept erasure for diffusion models. The authors observe that the erasure and preservation gradients are usually approximately orthogonal at a single step, but that their directions accumulated over time can become conflicting; consecutive gradients for the same objective can likewise show substantial directional variation (Fig. 1). On this basis they propose RRCE: textual inversion is used to iteratively discover the conditions under which the target concept is recoverable, after which anchor-guided erasure and teacher-matching preservation are performed; during optimization, separate AdamW states are maintained for the two objectives, and the first-order momentum coefficient is adjusted according to the cosine similarity between consecutive gradients of the same objective (Sec. 3, Eqs. 2–12).

The experiments build on SD v1.4 and cover three erasure tasks—nudity, dog, and Van Gogh—using attack success rates under direct prompts, UD, and CCE, together with neighboring-concept preservation rate, CLIP, and FID. The main tables and appendix provide a range of baselines, component ablations, and analyses of optimizer states (Tables 1–4, A3–A7).

Critical strengths
  1. A distinctive cross-time optimization perspective is offered. Fig. 1 contrasts same-step and cumulative gradient similarities, and Eqs. (8)–(10) analyze the historical mixing induced by shared first-order states, pushing the discussion beyond step-wise gradient projection.

  2. The optimizer design is explicit and the implementation is relatively straightforward. Separate erase/retain states are maintained, and β1 is adjusted according to the similarity between consecutive gradients; the appendix further discusses bias correction for the time-varying momentum (Sec. 3.2; Appendix E).

  3. The results and ablations are reasonably thorough. On the dog task, RRCE attains a CCE ASR of 9.20% with a Retain of 87.00% (Table 2); on the Van Gogh task, the direct, UD, and CCE ASRs are all 0.00%, with a Retain of 96.40% (Table 3). The authors also report that not all CLIP/FID metrics are the best, thereby avoiding a single-metric narrative.

Critical weaknesses
  1. The mechanistic explanation and causal isolation could still be strengthened. The cross-time negative correlation in Fig. 1 is a property of the gradient sequence; even with separate optimizers, the cross-time interaction between the gradients of the original losses need not disappear. We suggest reporting more clearly, under an identical gradient sequence, number of updates, and update order, the actual parameter update directions, the cumulative conflict, and the resulting performance differences produced by shared versus separate states, and clarifying which training configuration the diagnostic statistics are drawn from. The paper already includes Table 4, A5, and a theoretical analysis in the appendix, so this is a suggestion for strengthening the causal evidence rather than a claim that the authors omitted an optimizer ablation.

  2. The experimental scope offers limited support for broad generalization claims. The main experiments cover only SD v1.4 and three target concepts (Sec. 4.1), and applicability to other model versions, a wider range of target concepts, or different attack budgets remains unclear. The authors do evaluate two adaptive attacks, UD and CCE, so it would be inaccurate to describe the work as not testing adaptive attacks.

  3. The treatment of the performance trade-off and of uncertainty could be improved. The main tables largely report a single set of metrics, with no variance or confidence intervals across independent training runs. Some margins are small, for example the nudity CCE ASR of 1.05% versus 2.11% (Table 1); reporting run-to-run stability, sensitivity to the detector threshold, and a more complete Pareto comparison would make it easier to judge whether the advantage is robust.

Questions for authors

Q1. Which training configuration, which concept, and which parameter layers were the gradient trajectories in Fig. 1 taken from? If separate states are used or the update order is changed, how do the cumulative cosine similarity and the actual update direction change?

Q2. Regarding the negative cross-time cross term in Eq. (9), can the authors further distinguish the gradient geometry of the objectives themselves from the additional effect induced by the shared optimizer state? Can an experiment with a controlled gradient sequence quantify the two?

Q3. Can results be provided on multiple random training seeds, on other erasure concepts, or on other diffusion models, to demonstrate the stability and the scope of applicability of the improvements in Tables 1–3?

Q4. Is the performance ranking sensitive to the strength of the UD/CCE attacks, the number of attack optimization iterations, and the detector threshold used to compute ASR? The current settings are already reported in Sec. 4.1; the question here concerns sensitivity rather than missing parameters.

Q5. How much training memory and total time overhead do the two sets of AdamW states, the textual inversion, and the multi-round defense each incur? The appendix already compares training time; can the cost of the optimizer states and of the recovery discovery be broken down further?

4 comments
AuthorsOpenReview ↗

Responses to Reviewer hPBA Comments [1/4]

W1 + Q1 + Q2: Mechanistic explanation and causal isolation

Response: Thank you very much for this insightful comment. We would like to address your concern from three perspectives. First, we clarify the diagnostic settings and compare shared and separate optimizer states under both update orders. Second, we replay identical gradient sequences to isolate the effect of state sharing. Finally, we use a matched training ablation to evaluate the benefits of state separation.

1. Diagnostic settings and update order. Figure 1 studies dog erasure on SD 1.4 with full U-Net training: panels (a,b) use 500 joint AdamW steps from a checkpoint after two erasure rounds, with the original concept and two recovered tokens, and measure cumulative gradients over all trainable parameters. Panel (c) uses 120 joint steps from the same checkpoint and measures adjacent gradients on a fixed coordinate subset. Panel (d) starts from the original SD1.4 model and uses three 500-step defense blocks, with shared AdamW driving joint-objective training and read-only objective-specific moment recurrences processing the same gradients on a fixed coordinate subset.

For the update-order comparison, each configuration starts from the Fig. 1(a,b) checkpoint with β1=0.9\beta_1=0.9 and β2=0.999\beta_2=0.999. Each alternating run contains 200 erasure updates and 200 retention updates, and the cosines cover all trainable parameters along that run's trajectory. We vary state sharing and update order under these settings:

Alternating training configurationCumulative raw-gradient E/R cosineCumulative parameter-update E/R cosine
Shared states, E→R−0.3476+0.9939
Shared states, R→E−0.3525+0.9965
Separate states, E→R−0.3833−0.1039
Separate states, R→E−0.3825−0.1015

We also conduct evaluations on joint optimization with 200 joint updates:

Reference configurationCumulative raw-gradient E/R cosineCumulative attributed-update E/R cosine
Shared states, joint optimization−0.3587−0.3318

For alternating runs, updates are the measured erasure-step and retention-step parameter displacements. For the joint reference, the shared update is attributed to its E/R first-moment components under the common preconditioner, with weight decay and numerical residuals divided equally.

In both update orders, shared-state cumulative updates are nearly parallel, whereas separate-state updates have negative cosine. The raw gradients remain negatively correlated in all four runs.

2. Replay with identical input gradients. We replay a separately recorded RRCE trajectory for SD1.4 Dog erasure with full U-Net training, fixing the gradients, recorded momentum coefficients, update count, and E→R order. Both state configurations process three blocks of 500 erasure/retention update pairs, totaling 3,000 updates, on the same 1,796 recorded parameter coordinates:

Defense blockRaw-gradient cosine, both statesShared-state update cosineSeparate-state update cosine
1−0.9622+0.6620−0.3149
2−0.9816+0.7110−0.2196
3−0.9675+0.7041−0.2515

The raw-gradient geometry is identical, but changing the state configuration changes the cumulative update directions in every block. Thus, the gradient term ⟨giE,gjR⟩\langle g_i^E,g_j^R\rangle and the optimizer-induced update geometry are distinct, and separating states does not remove negative correlation in the supplied gradients.

3. Performance under a matched training ablation. The Nudity comparison fixes the loss schedule, defense budget, and stabilized momentum rule:

Optimizer statesUD ASR ↓CCE ASR ↓Retain ↑
Shared first and second moments1.05%4.21%90.80%
Separate first moments only0.00%2.11%86.80%
Separate second moments only2.11%2.11%88.00%
Separate first and second moments, RRCE0.00%1.05%93.20%

With shared optimizer states, erasure and retention use a common gradient history. Their past gradients are combined in the first moment, while their squared gradients are combined in the second moment to determine the scaling of each parameter update. High alignment between these mixed updates can coexist with cancellation between the erase and retain contributions inside the shared first moment. This mixing can make temporal smoothing and preconditioning less suited to each objective's own gradient sequence.

RRCE separates both histories so that each objective receives temporal smoothing and preconditioning based on its own gradients. The aim is to accommodate each objective's sampling variability and gradient geometry while both continue to update the same model. Under this interpretation, the less aligned updates reflect distinct directions for erasure and retention. The matched ablation supports the benefit of this separation: separating both moments reduces CCE ASR from 4.21% to 1.05% and raises Retain from 90.80% to 93.20%, improvements of 3.16 and 2.40 percentage points.

AuthorsOpenReview ↗

Responses to Reviewer hPBA Comments [2/4]

W2 + Q3: Model, concept, and attack-budget coverage

Response: We sincerely appreciate this useful comment. We expand our experiments in three ways: i) We use FLUX.1-dev as an additional backbone. ii) We add three more erasure targets for SD 1.4. iii) We test doubled UD and CCE budgets.

More Models. We evaluate Nudity, Dog, and Van Gogh against ESD and UCE on FLUX.1-dev.

Nudity (FLUX.1-dev)

ModelErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
FLUX.1-dev (reference)45.2647.3668.42100.0030.8720.41
ESD3.6013.6827.3695.2028.4423.08
UCE1.2010.5215.7894.0024.5630.71
RRCE0.000.001.2097.4030.5623.59

Dog (FLUX.1-dev)

ModelErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
FLUX.1-dev (reference)100.0092.80100.00100.0030.8720.41
ESD12.8062.40100.0090.6028.1224.32
UCE10.4032.2096.0092.8023.7832.13
RRCE3.6012.0023.4094.2030.4224.68

Van Gogh (FLUX.1-dev)

ModelErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
FLUX.1-dev (reference)12.0014.0020.4066.4030.8720.41
ESD4.406.2019.4062.0029.6722.84
UCE2.003.6016.8063.6028.7426.22
RRCE0.001.003.0064.4030.7222.78

On these three FLUX tasks, RRCE has lower Erased, UD, and CCE ASR and higher Retain than both ESD and UCE.

Additional SD1.4 concepts. We add Harm (safety), Car (object), and Monet (style).

Harm

MethodErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
SD v1.4 (reference)37.7059.0078.6097.6031.3614.15
ESD13.0426.0081.8095.4030.7814.46
UCE15.7822.0068.6096.4031.0414.52
RECE8.9312.0054.4096.0031.0214.71
MACE4.508.0046.8092.2029.2712.19
STEREO1.692.0024.6078.6029.8117.98
ReCARE2.672.0037.8090.4030.1714.91
RRCE1.762.0018.6096.0030.6614.67

Car

MethodErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
SD v1.4 (reference)92.4092.0089.00100.0031.3614.15
ESD71.0080.0085.4094.2031.0514.96
UCE21.4032.0082.4099.0030.9614.73
RECE1.2012.0086.2097.6031.0915.51
MACE8.6046.0086.8098.2031.1813.37
STEREO0.402.008.8041.8029.4518.56
ReCARE3.804.0012.4082.4029.9414.74
RRCE0.002.004.2098.4030.6414.23

Monet

MethodErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
SD v1.4 (reference)78.8084.0042.6098.4031.3614.15
ESD13.4010.0044.2092.0031.1013.89
UCE17.0026.0063.8096.4031.1614.01
RECE6.2012.0055.8095.0031.1313.64
MACE9.0012.0060.2094.4031.1413.96
STEREO0.002.000.0065.2030.9115.00
ReCARE1.002.002.6080.0030.8419.89
RRCE0.000.000.0096.4031.1214.38

RRCE has the lowest or tied-lowest CCE ASR on these three additional targets. On Monet, RRCE and STEREO both have 0.00% CCE ASR, while Retain is 96.40% and 65.20%, respectively.

Attack budgets. On the original three SD1.4 targets, we increase CCE from 3,000 to 6,000 iterations and UD from 40 to 80. We report the ASR (%) in the following table:

ConceptMethodCCE: 3000×1 / 6000×1UD: 40×1 / 80×1
NudityRRCE1.05 / 2.110.00 / 0.00
NuditySTEREO2.11 / 3.160.00 / 0.00
DogRRCE9.20 / 10.802.00 / 2.00
DogSTEREO30.60 / 32.600.00 / 0.00
Van GoghRRCE0.00 / 0.000.00 / 0.00
Van GoghSTEREO0.00 / 0.004.00 / 4.00

Under both budgets, RRCE has lower CCE ASR on Nudity and Dog and ties on Van Gogh. For UD, Nudity remains tied, RRCE remains lower on Van Gogh, and STEREO remains lower on Dog.

AuthorsOpenReview ↗

Responses to Reviewer hPBA Comments [3/4]

W3 + Q3 + Q4: Training variability, performance trade-offs, and evaluation sensitivity

Response: Thank you very much for this useful suggestion. We report five-seed statistics, the joint CCE–Retain comparison, and sensitivity to detection criteria.

1. Five-seed training. The following tables give mean ± standard deviation. UCE and RECE use closed-form edits without training, so their results are fixed with zero deviation.

Nudity (five training seeds)

MethodErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
SD v1.4 (reference)28.86 ± 0.0027.37 ± 0.0050.53 ± 0.0097.80 ± 0.0031.36 ± 0.0014.15 ± 0.00
ESD15.50 ± 2.698.84 ± 1.7750.74 ± 1.5689.96 ± 2.3630.81 ± 0.0514.50 ± 0.09
UCE5.45 ± 0.005.26 ± 0.0021.05 ± 0.0093.20 ± 0.0031.08 ± 0.0014.38 ± 0.00
RECE1.14 ± 0.001.05 ± 0.005.26 ± 0.0093.00 ± 0.0031.06 ± 0.0014.69 ± 0.00
MACE2.27 ± 1.961.05 ± 0.7510.95 ± 1.6049.48 ± 3.8929.37 ± 0.0712.26 ± 0.09
AdvUnlearn0.73 ± 1.060.84 ± 0.886.53 ± 1.1691.76 ± 4.0129.77 ± 0.0715.38 ± 0.05
STEREO1.09 ± 1.130.42 ± 0.582.53 ± 1.6071.12 ± 3.7729.79 ± 0.0617.96 ± 0.09
AGE1.14 ± 1.171.69 ± 0.942.32 ± 1.3777.60 ± 2.0529.96 ± 0.0311.22 ± 0.10
ReCARE1.32 ± 1.301.26 ± 1.373.58 ± 1.6082.68 ± 2.8530.22 ± 0.0414.83 ± 0.09
AEGIS13.41 ± 2.785.90 ± 1.6045.27 ± 1.4978.16 ± 2.3830.13 ± 0.0711.83 ± 0.12
RRCE1.73 ± 1.440.63 ± 0.941.26 ± 0.8894.28 ± 4.3330.66 ± 0.0314.68 ± 0.06

Dog (five training seeds)

MethodErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
SD v1.4 (reference)100.00 ± 0.0094.00 ± 0.0099.80 ± 0.0089.20 ± 0.0031.36 ± 0.0014.15 ± 0.00
ESD90.00 ± 2.2682.40 ± 1.6798.00 ± 1.4184.96 ± 3.6030.98 ± 0.0614.78 ± 0.08
UCE2.60 ± 0.0022.00 ± 0.00100.00 ± 0.0085.80 ± 0.0031.06 ± 0.0014.96 ± 0.00
RECE0.80 ± 0.0014.00 ± 0.0099.80 ± 0.0086.40 ± 0.0031.01 ± 0.0015.41 ± 0.00
MACE7.60 ± 2.4057.20 ± 2.2898.40 ± 1.6787.72 ± 2.8431.13 ± 0.0613.70 ± 0.05
AdvUnlearn29.56 ± 1.894.80 ± 3.3598.40 ± 1.6786.00 ± 2.8128.01 ± 0.0715.06 ± 0.06
STEREO1.44 ± 1.242.00 ± 2.0032.20 ± 1.679.16 ± 3.4728.47 ± 0.0426.60 ± 0.08
AGE14.08 ± 1.4520.00 ± 3.1672.00 ± 3.3553.84 ± 1.5930.08 ± 0.0612.46 ± 0.07
ReCARE5.56 ± 1.7312.80 ± 1.7918.00 ± 3.0355.12 ± 3.4830.07 ± 0.0414.99 ± 0.08
AEGIS2.68 ± 2.1417.60 ± 3.5877.40 ± 3.1620.84 ± 2.4228.91 ± 0.0715.45 ± 0.09
RRCE2.56 ± 1.291.60 ± 1.6710.40 ± 1.7985.64 ± 2.7330.33 ± 0.0113.89 ± 0.06

Van Gogh (five training seeds)

MethodErased ↓UD ↓CCE ↓Retain ↑CLIP ↑FID ↓
SD v1.4 (reference)73.20 ± 0.0082.00 ± 0.0033.80 ± 0.0099.00 ± 0.0031.36 ± 0.0014.15 ± 0.00
ESD4.00 ± 2.5710.40 ± 2.6154.60 ± 2.9795.00 ± 3.3031.10 ± 0.0213.93 ± 0.05
UCE14.00 ± 0.0024.00 ± 0.0043.80 ± 0.0097.00 ± 0.0031.17 ± 0.0013.95 ± 0.00
RECE2.00 ± 0.0014.00 ± 0.0049.40 ± 0.0095.40 ± 0.0031.13 ± 0.0013.76 ± 0.00
MACE4.20 ± 2.5411.60 ± 1.6727.60 ± 2.0095.36 ± 1.6931.10 ± 0.0514.08 ± 0.10
AdvUnlearn2.44 ± 1.712.80 ± 1.7941.00 ± 1.7914.64 ± 3.5029.38 ± 0.0516.41 ± 0.04
STEREO0.96 ± 0.774.80 ± 3.351.60 ± 1.6762.28 ± 2.7130.89 ± 0.0415.57 ± 0.05
AGE1.20 ± 1.384.80 ± 2.284.80 ± 4.0074.96 ± 2.5030.39 ± 0.0311.06 ± 0.07
ReCARE1.44 ± 1.512.40 ± 2.6126.60 ± 2.1979.68 ± 3.7530.85 ± 0.0719.44 ± 0.07
AEGIS1.88 ± 1.514.80 ± 1.7915.60 ± 2.9766.64 ± 3.3829.89 ± 0.0715.10 ± 0.09
RRCE1.00 ± 1.211.20 ± 1.790.80 ± 1.1093.76 ± 2.8531.10 ± 0.0614.42 ± 0.11

RRCE has the lowest mean CCE ASR on all three tasks.

2. Joint CCE–Retain comparison. Using the five-seed means for training-based methods and the fixed results for closed-form methods, RRCE is nondominated in the CCE–Retain comparison among the compared erasure methods on all three tasks.

RRCE achieves the lowest mean CCE ASR while maintaining strong retention. Across all three tasks, RRCE also achieves higher mean Retain, higher mean CLIP, and lower mean FID than STEREO, supporting its joint robustness–preservation advantage across repeated training runs.

3. Detection criteria. We vary the Dog and Nudity thresholds and the Van Gogh top-k criterion:

Concept and evaluation criteriaRRCE CCE ASR (%)STEREO CCE ASR (%)
Dog: thresholds 0.4 / 0.5 / 0.610.80 / 9.20 / 8.8032.40 / 30.60 / 28.80
Nudity: thresholds 0.5 / 0.75 / 0.92.11 / 1.05 / 0.004.21 / 2.11 / 0.00
Van Gogh: top-1 / top-3 / top-50.00 / 0.00 / 4.400.00 / 0.00 / 4.60

RRCE's CCE ASR remains lower than or equal to STEREO's across these tested criteria.

AuthorsOpenReview ↗

Responses to Reviewer hPBA Comments [4/4]

Q5: Training time and memory costs

Response: Thank you very much for this meaningful comment. We measure Van Gogh erasure with cross-attention-only U-Net training, using the same task and trainable parameters:

Measured time and memory

ConfigurationTotal training timeTotal optimizer-update timePeak allocated memoryPeak reserved memory
Full RRCE25.68 min35.57 s13.15 GiB13.30 GiB
Shared states, fixed momentum29.72 min8.08 s12.48 GiB12.63 GiB
Separate states, fixed momentum25.71 min7.58 s12.80 GiB12.97 GiB

Relative to shared states with fixed momentum, full RRCE's optimizer-update time increases by 27.49 seconds and peak allocated memory increases by 0.67 GiB. Optimizer-update time sums all optimizer steps and is already included in the total training time.

Optimizer storage. For PP trainable parameters, the additional first- and second-moment buffers contain 2P2P FP32 scalars and require 8P8P bytes. The previous-gradient buffers for the two objectives require another 8P8P bytes, giving 16P16P bytes of additional persistent storage, which scales linearly with the number of trainable parameters. In this configuration, each buffer group occupies approximately 0.33 GiB, totaling 0.66 GiB. With fixed momentum, separating states raises peak allocated memory by 0.32 GiB.

Recovery and defense costs. These phase times are measured in the same configuration and normalized to the 25.68-minute total:

ComponentTimeShare
Three defense blocks14.34 min55.83%
Two textual-inversion searches11.03 min42.96%
Loading, initialization, saving, and other overhead0.31 min1.21%
Total25.68 min100%

The two defense blocks after the initial block take 9.62 minutes. During the two recovery searches, diffusion-model weights are fixed while only token embeddings are optimized.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.