acceptodds
Under review as a conference paper at ICLR 2027

Not Every Time and Frequency Need to Be Forgotten in Diffusion Unlearning

Abstract

Data unlearning aims to remove the influence of specific training samples from a trained model. In fine-tuning methods, data unlearning relies primarily on loss maximization over forget samples, which often leads to quality degradation or incomplete forgetting. Existing methods perform unlearning uniformly across diffusion stages, ignoring diffusion dynamics from noise to data. Our systematic study of diffusion phases shows that forgetting in diffusion models is uneven across time and frequency, with theoretical justification of distributive distortion and forgetting-utility trade-off. By selectively forgetting time and frequency in diffusion models, we achieve both higher unlearning success rates and improved generation quality across diverse settings, including both conditional and unconditional scenarios. We also introduce an improved SSCD metric that measures dissimilarity using a normalized perturbation distance. Together, we provide practical insights for understanding and improving data unlearning in diffusion models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, Data Unlearning, Diffusion Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5KegOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies data unlearning in diffusion models and argues that forgetting should not be applied uniformly across all diffusion timesteps and frequency components. The authors analyze the phase-dependent behavior of diffusion models and propose a time-selective unlearning strategy that concentrates updates on the timesteps most relevant to memorization, together with a frequency-selective strategy that suppresses unnecessary high-frequency modifications during unlearning. These selection mechanisms are designed as general plug-in components that can be combined with existing unlearning objectives. The paper also introduces a normalized variant of SSCD to reduce the confounding effect of severe image degradation when evaluating forgetting. Experiments across multiple settings show that the proposed approach generally improves the trade-off between forgetting effectiveness and generation quality.

Critical strengths
  1. Diffusion-specific insight: The paper shows that effective forgetting in diffusion models is concentrated in particular timestep and frequency regions, offering a clear perspective on where unlearning should be applied.

  2. Simple and general design: The proposed selective mechanisms are lightweight, compatible with existing unlearning objectives, and improve the trade-off between forgetting effectiveness and generation quality.

Critical weaknesses
  1. Incomplete experimental evaluation: The paper does not include comparisons with more recent data-unlearning methods such as "Unlearning in Diffusion Models: A Unified Framework with KL Divergence and Likelihood Constraints (ICML 2026)" and "ReTrack: Data Unlearning in Diffusion Models through Redirecting the Denoising Trajectory (AISTATS 2026)", whose experimental settings are closely related to this work. In addition, although the proposed P(t)P(t) and βi,t(r)\beta_{i,t}(r) introduce the key hyperparameters kk and ss, no ablation or sensitivity analysis is provided for them.

  2. Concerns about SSCDnorm\mathrm{SSCD}^{\mathrm{norm}}: As one of the paper's main contributions, the motivation and formulation of SSCDnorm\mathrm{SSCD}^{\mathrm{norm}} appear questionable. The intended normalization focuses on the direction of the unlearning-induced change while discarding its magnitude, although the magnitude of the change is also important for determining whether unlearning is actually effective. Moreover, the squared norm in the denominator of Eq. (13) means that the resulting perturbation does not have a fixed ℓ2\ell_2 radius, which appears inconsistent with the paper's description of the operation as a projection onto an ℓ2\ell_2 ball.

Questions for authors
  1. In Eq. (13), the denominator contains the squared ℓ2\ell_2 norm, which means that the resulting perturbation does not have a fixed ℓ2\ell_2 radius of ρ\rho. Is this a typo, or is this the intended formulation? If intended, could the authors clarify how it is consistent with the description of projecting the perturbation onto an ℓ2\ell_2 ball?

  2. There appear to be several possible typos or formatting issues, including the quotation marks in the caption of Table 2, the missing equation reference in the third paragraph of Appendix B, and the duplicated Sections C.5 and C.6. Could the authors confirm whether these are typos?

  3. In Table 3, the filter band ablation is evaluated using SSCD\mathrm{SSCD}, whereas the time window ablation is evaluated using SSCDnorm\mathrm{SSCD}^{\mathrm{norm}}. What is the reason for using different forgetting metrics for these two ablations? Would the conclusions remain consistent if both were evaluated using the same metrics?

  4. Could the authors provide visualizations of the normalized samples used to compute SSCDnorm\mathrm{SSCD}^{\mathrm{norm}}, preferably for different values of ρ\rho? I am concerned that sufficiently large extrapolation along the reconstruction-error direction may produce severely degraded or perceptually meaningless images, which would make the interpretation of the normalized metric less clear.

4 comments

We appreciate the reviewer's comments about the additional recent baselines and rationale behind the proposed method. Here, we provide more experimental results and explanations.

We welcome any further questions from the reviewer.


[W1-1] Recent baselines

We agree that investigating further recent papers in data unlearning is useful. Thus, among the two papers that the reviewer suggested, we adopt the proposed selective unlearning in ReTrack [1], which fits well with our experimental setups. Based on the ReTrack implementation, we apply our selective unlearning. Note that the formulation of ReTrack applies unlearning to the k-nearest samples, which requires longer training while providing stable unlearning. Thus, we test whether selective unlearning can accelerate convergence, as shown in Figure 8 in the original submission. We have added the ReTrack comparison to Section 4.2.

For ReTrack+Our method, we slightly modify the selective unlearning for the initial 5 steps and apply full-step unlearning in a step-wise way to match the final target. The results below show that selective unlearning can accelerate convergence and effectively improve unlearning performance, especially in the early training stages.

Full-prompt evaluation

MetricMethod12345Final
ASR (%) ↑ReTrack4.210.433.358.372.995.8
+Ours12.550.072.975.079.2100.0
SSCD ↓ReTrack0.6950.5720.4780.3980.3370.111
+Ours0.5450.4180.3470.3140.3040.110
CLIP-IQA ↑ReTrack0.7830.7570.7230.7160.7060.698
+Ours0.7870.7690.7420.7430.7450.719

Partial-prompt evaluation

MetricMethod12345Final
ASR (%) ↑ReTrack6.218.841.768.879.297.9
+Ours47.979.291.789.691.7100.0
SSCD ↓ReTrack0.5620.4620.3770.3050.2480.088
+Ours0.3360.2430.2160.2160.2130.090
CLIP-IQA ↑ReTrack0.7840.7600.7350.7390.7390.745
+Ours0.7920.7720.7570.7570.7440.730

[1] ReTrack: Data Unlearning in Diffusion Models through Redirecting the Denoising Trajectory (AISTATS 2026)


[W1-2] hyperparameter ablations

For kk and ss, we state that For the proposed method, we fix the time suppression intensity at k=0k=0 in Equation (7) and the high-frequency suppression weight at s=0s=0 in Fourier filtering to simplify the design. in the Appendix. However, we agree that it is hard to see. Thus, we moved it to the experimental setup.

To push further, in addition to Table 3 (refer to Q3 below), we agree that the sensitivity analysis for frequency and time can support the strength of the proposed method. Since our primary method selectively controls time and frequency while retaining all other experimental settings from SISS [2], we conducted independent ablation studies varying only time steps and frequency cutoffs. The results are as follows:

[Time step only]

Time WindowSSCDnorm (t=250)\mathrm{SSCD}^{\mathrm{norm}} \ (t = 250) ↓Aesthetic (t=250t = 250) ↑SSCDnorm (t=500)\mathrm{SSCD}^{\mathrm{norm}} \ (t = 500) ↓Aesthetic (t=500t = 500) ↑
Full [0, 1000] (Base)0.43004.09400.50104.8450
[0, 500]0.46265.52770.47595.8395
[250, 750]0.40935.60110.43355.9527
[500, 1000]0.48095.82240.44365.8874

[Freq only]

Frequency CutoffSSCDnorm (t=250)\mathrm{SSCD}^{\mathrm{norm}} \ (t = 250) ↓Aesthetic (t=250t = 250) ↑SSCDnorm (t=500)\mathrm{SSCD}^{\mathrm{norm}} \ (t = 500) ↓Aesthetic (t=500t = 500) ↑
0 (Base)0.43004.09400.50104.8450
0.100.42095.50930.43885.5616
0.150.41645.70390.43115.7962

The results show that both time and frequency selection independently improve forgetting/convergence while protecting quality. However, the effect of using each method individually is weaker than that of using both methods together, as shown in Table 2. We have included these independent ablations in Appendix C.3.

[2] Data Unlearning in Diffusion Models (SISS), ICLR 2025.

Additionally, we summarize the experiments in the paper, including the Appendix, to make the scope of the submitted evaluation easier to assess as follows:

DimensionEvaluation
Model BackbonesU-Net DDPM, SD v1.4, LDM (Tab. 9), SD v3.0 (DiT / Flow-matching; Tab. 5)
Optimization ObjectivesGA (NegGrad), SISS, EraseDiff, naive deletion, DPO, KTO (Tabs. 2, 9)
Evaluation MetricsFID, SSCD, CLIP-IQA, unlearning success rate, SSCD_norm, Aesthetic score (Tabs. 2, 5, 9, 13, 14; Fig. 8)
Additional EvaluationMIA (SecMI/PIAN) (Tab. 4), LLM-based OOD detection (Tab. 10), NN preservation (Tab. 11; neighbor sets in Fig. 10)
Task ScopeData-level unlearning, Concept unlearning (Nudity) (Fig. 9)

The bolded items indicate additional evaluations beyond those reported in SISS [2] or ReTrack [1].

[W2-1, Q1] Eq. (13): squared norm or fixed-radius projection?

Sorry for the confusion. It is a typo. We replace the squared L2 norm with the L2 norm, as follows:

SSCDnorm=SSCD(x0,x0+ρx^0(xt,θ)−x0∥∣x^0(xt,θ)−x0∥∣2+ε)\mathrm{SSCD}^{\mathrm{norm}} = \mathrm{SSCD}(\mathbf{x}_0, \mathbf{x}_0 + \rho \frac{\hat{\mathbf{x}}_0(\mathbf{x}_t,\theta) - \mathbf{x}_0}{\||\hat{\mathbf{x}}_0(\mathbf{x}_t,\theta) - \mathbf{x}_0\||_2 + \varepsilon})

We have corrected the equation. The evaluation was already calculated using this equation, so the reported numerical results remain unchanged.

See further rationale behind normalization in [W2-2] below.


[W2-2] Motivation and formulation of normalized SSCD

The goal of normalized SSCD is to control the utility-privacy trade-off. We believe that machine unlearning is an important topic, but the evaluation still requires some bound on image perturbation. For example, in privacy domains, we can easily find privacy budgets in similar literature, such as the attack radius in adversarial attacks or the (ϵ,δ)(\epsilon,\delta) budget in differential privacy. These bounds are designed to evaluate performance under the same constraints.

This analysis cannot be directly applied to generative modeling, but we at least need to match the image level within some threshold, in our case ρ\rho. This helps to evaluate SSCD under similar image perturbations. This is directly connected to the reviewer's concern about image visualization in [Q4] below. When we only care about small SSCD values (or, equivalently, enlarge ρ\rho in SSCD-norm), the resulting images are noisy or grey.

Therefore, we fix the radius to handle the privacy-quality trade-off. We have clarified the rationale in Section 4.1, immediately after the equation, i.e., normalized SSCD compares reconstruction-error directions at the same perturbation scale.


[Q2] Typos and formatting issues

Thank you for pointing out the typos and formatting errors. We fixed all the minor problems that the reviewer pointed out. Going further, we also checked other quotation marks, such as those in Figures 20 to 24, and fixed the quotation marks accordingly. Sorry for the rush during the submission phase. Thus, we have carefully checked all the details again following the reviewer's suggestion. Thank you for improving the overall quality of the paper.


[Q3] Different metrics for the two ablations

We conducted both experiments separately and initially reported each score. However, due to the 9-page constraint, we concatenated the two tables, and the evaluation varied. We clearly state that we do not cherry-pick the evaluation. Below, we provide the results with the same evaluation under SSCD-norm.

Filter bandSSCD-norm↓ (t=250t=250)Aes.↑ (t=250t=250)SSCD-norm↓ (t=500t=500)Aes.↑ (t=500t=500)Time windowSSCD-norm↓ (t=250t=250)Aes.↑ (t=250t=250)SSCD-norm↓ (t=500t=500)Aes.↑ (t=500t=500)
Base0.5444.0940.5024.845Base (Full)0.4304.0940.5014.845
Low-pass0.3325.5530.3875.904[250, 750]0.3495.5200.3996.095
Mid-pass0.3774.3740.4175.043[250, 500]0.4155.7900.4395.951
High-pass0.6254.1430.5974.670[333, 666]0.3995.7060.4205.696

Thank you for suggesting the unification of the evaluation metric for the results. We have updated Table 3 in the revised manuscript.

[Q4] Normalized-sample visualizations across ρ\rho

We select ρ=100\rho=100, which is the empirical sweet spot (optimal balance point) of the privacy-utility trade-off. When ρ>200\rho>200, the quality itself is too bad for human eyes, and ρ<20\rho<20 is too similar to the original images.

MethodSSCD_50 ↓ (t=250t=250)SSCD_50 ↓ (t=500t=500)SSCD_100 ↓ (t=250t=250)SSCD_100 ↓ (t=500t=500)SSCD_150 ↓ (t=250t=250)SSCD_150 ↓ (t=500t=500)SSCD_200 ↓ (t=250t=250)SSCD_200 ↓ (t=500t=500)
SISS0.7860.7780.4300.5010.3950.3460.2950.259
+Ours0.7420.8100.3490.3990.2510.2230.2030.179
KTO0.7270.8080.3660.4420.2830.2770.2180.193
+Ours0.7020.7930.3400.3770.2750.2390.2140.201

In the revised version, we have added normalized-sample visualizations for ρ∈{50,100,150,200}\rho \in \{50,100,150,200\} in Figure 25 of Appendix D, allowing readers to directly inspect the effect of increasing the perturbation radius.

Thank you for the detailed response, which has addressed my concerns. I have raised my score to 3.

Reviewer_gWR9OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies data unlearning in diffusion models. The authors argue that unlearning should not treat all timesteps and frequencies equally; they provide extensive experiments to show how different timestep ranges affect unlearning and how to avoid degrading unlearned samples' quality with low-pass filters. Therefore, the authors propose a time- and frequency-selective unlearning method, which can be easily integrated with existing unlearning methods. Besides, the authors propose an improved metric (normalized SSCD). Experiments on CelebA-HQ, compared with GA, SISS, DPO, and KTO, show the effectiveness of the proposed method.

Critical strengths
  • The analysis of the timesteps affected by unlearning is extensive (also using previous observations to support), providing evidence for the claim that unlearning is uneven across time.
  • The proposed method is simple and can be easily integrated with existing unlearning methods.
Critical weaknesses
  • My main concern is why we need to make sure of the quality of the unlearned samples. The authors mentioned that "ensuring the quality of unlearned samples is crucial not only for aesthetics but also for preventing privacy attacks"; then why not consider generating random faces for unlearned IDs? Also, since the authors mentioned privacy attacks, would this case lead to more threats to retain IDs?
  • The experiments are on CelebA-HQ, where one ID has about 5 images, which might make forgetting easy. It would be better to consider more diverse datasets where the retain set has more images belong to the forgetton ID.
Questions for authors
  • Figure 6 provides "Over-forgotten samples differ in high frequencies, while retain samples remain unchanged", how about well forgotten samples?
  • Table 11 provides "Reconstruction on the five nearest neighbors of each forgotten image"; do these five nearest neighbors belong to the same person as the forgotten image?
  • In Table 9, DPO+ours (from t=250) and KTO+base (from t=250) are the same as DPO base (from t=500), pls check the numbers in the table to make sure they are correct.
4 comments

We appreciate the reviewer's comments about the reliability of the experimental setups and the additional evaluation. Here, we provide more experimental results and explanations, which we will include in the revised version shortly.


[W1] Why preserve forgotten-sample quality?

The goal of sample-level unlearning is to remove the influence of a requested training record while preserving utility on the retain distribution, ideally approximating a model retrained without that record. This is a well-established setting motivated in part by GDPR’s right to erasure of personal data. If a record is redundant or non-memorized, removing it may correctly produce little observable change; our experiments focus on memorized records because their individual influence is measurable and creates the clearest privacy or copyright risk. Representative examples include the exact reconstruction shown in Figure 20 or "Ann Graham Lotz" in Figures 21 and 22 of our submission.

We further present three experiments in our paper that emphasize the importance of high-quality unlearned samples: (i) nearby sample quality, (ii) clues for identifying unlearned samples, and (iii) MIA indistinguishability. All these experiments indicate that poor quality at certain data points can provide clues for identifying unlearned points and also distort nearby images.

  1. As shown in Table 11, the reconstructed images of the five nearest neighbors of each forgotten image retain SSCD scores close to those of the pretrained model, while our method improves their aesthetic quality over SISS at both evaluation timesteps. These results show that selective unlearning mitigates quality degradation in nearby samples.
ModelSSCD ↓ (t=250t=250)Aesthetic ↑ (t=250t=250)SSCD ↓ (t=500t=500)Aesthetic ↑ (t=500t=500)
Orig.0.88176.38940.74056.7483
SISS0.88126.24410.73946.5261
+ Ours0.87976.44670.74396.5873
  1. In Table 10, a higher score indicates that the generated image is perceived as realistic (in-distribution), making it harder to detect that unlearning has occurred. This could provide a clue for identifying unlearned samples, for example, in relearning attacks [1]. Our method receives higher realism scores, indicating that the unlearned outputs are perceived as more realistic and in-distribution.
LLM EvaluatorSISSSISS + Ours
ChatGPT-50.220.67
Claude Sonnet 4.50.130.83
Gemini 2.5 Flash0.000.50
  1. As shown in Table 4, we can also see the importance of quality through membership inference attacks (MIAs). Under SecMI, our method reduces both measures that separate the unlearned and hold-out distributions. Under PIAN, it reduces the standardized distance for SISS and modestly reduces AUC separation for KTO, but the unlearned and hold-out distributions remain clearly distinguishable.
AttackMethodAUCsep ↓ (Base)AUCsep ↓ (+Ours)Δd ↓ (Base)Δd ↓ (+Ours)
SecMIPretrained0.58—0.03—
SecMISISS1.000.794.041.07
SecMIKTO0.760.681.600.34
PIANPretrained0.68—0.54—
PIANSISS1.001.008.594.12
PIANKTO0.990.940.870.86

All of the above experiments show the importance of quality even after unlearning. We will clarify this point in the rebuttal and clearly distinguish our setting from "Output Likelihood Suppression" or "Concept Unlearning," which deletes all the information about a mode in the output distribution and generally does not care about deleted quality [2].

[1] Unlearning's Blind Spots: Over-Unlearning and Prototypical Relearning, ICML 2026.

[2] Position: The Term “Machine Unlearning” Is Overused in LLMs, ICML 2026.

[W2] More images per forgotten identity

We apologize for the confusing experimental setup. Following SISS [1], each selected face defines a separate single-record unlearning task, and the model is trained in a continual manner (Section 4.1; Appendix C.6). Indices 10000-10005 were used only to maintain consistent visualizations in the paper.

To test sensitivity to the selected records, we additionally performed five independently sampled evaluations, selecting six random faces in each run (5 models X 6 images = 30 deletions). Applying our selective scheme improves the mean result, not confined to the six visualized records, as shown in Table 15:

MethodSSCDnorm\mathrm{SSCD}^{\mathrm{norm}} ↓ (t=250t=250)Aesthetic ↑ (t=250t=250)SSCDnorm\mathrm{SSCD}^{\mathrm{norm}} ↓ (t=500t=500)Aesthetic ↑ (t=500t=500)
SISS0.427±0.0990.427\pm0.0994.729±0.6604.729\pm0.6600.399±0.0440.399\pm0.0445.255±0.6675.255\pm0.667
SISS + Ours0.380±0.0950.380\pm0.0955.373±0.3365.373\pm0.3360.377±0.0730.377\pm0.0735.680±0.3105.680\pm0.310

As reported in Table 14, we observed statistical validity by fixing the target records and repeating training five times. The standard deviations are small.

MethodSSCDnorm\mathrm{SSCD}^{\mathrm{norm}} ↓ (t=250t=250)Aesthetic ↑ (t=250t=250)SSCDnorm\mathrm{SSCD}^{\mathrm{norm}} ↓ (t=500t=500)Aesthetic ↑ (t=500t=500)
SISS0.430±0.0030.430\pm0.0034.094±0.0544.094\pm0.0540.501±0.0040.501\pm0.0044.845±0.0514.845\pm0.051
SISS + Ours0.349±0.0020.349\pm0.0025.520±0.0635.520\pm0.0630.399±0.0030.399\pm0.0036.095±0.0496.095\pm0.049

Furthermore, we also tested our method in a broader setting to validate the results on more images in the conditional setting, i.e., T2I unlearning for 200 prompts. The main conditional experiment follows the 45 manually selected prompts used by SISS and trains a separate model for each prompt. To evaluate a substantially larger forget set, we additionally tested 200 memorized prompts by Wen et al. [2] and trained one model to unlearn all 200 prompts using mini-batches. This is a distinct multi-record scale test rather than a direct aggregation of the original per-prompt experiment.

ModelMethodSSCD ↓Aesthetic ↑
SD v1.4Pretrained reference--6.0893
SISS0.76406.1187
SISS + Ours0.46506.1760
SD v3Pretrained reference--6.4183
SISS0.67466.3665
SISS + Ours0.66356.3890

On SD v1.4, selective unlearning reduces SSCD by 39.1% relative to SISS while slightly improving aesthetic quality. On SD v3, the SSCD gain is more modest (1.6%), but the selective version still improves both reported metrics relative to SISS.

We also include ablation studies of time, frequency, and the perturbation radius ρ\rho in the responses to Reviewer 5Keg.

[1] Data Unlearning in Diffusion Models, ICLR 2025.

[2] Detecting, Explaining, and Mitigating Memorization in Diffusion Models, ICLR 2024.


[Q1] Frequency changes in well-forgotten samples

As shown in the right panel of Figure 13 in Appendix C.6, samples from the forget set that preserve aesthetic quality exhibit smaller differences in high-frequency components before and after unlearning. Figure 14 additionally shows that the PSDs of retain samples remain largely unchanged. We agree that investigating the difference between well-forgotten and over-forgotten samples is interesting. We believe that distortion of high-frequency components, which correspond to image details, can contribute substantially to this difference. We have added this explanation to Section 3.2 of the revised manuscript.


[Q2] Identity of the five nearest neighbors

We already visualized the nearby images using DINOv3 embeddings in Figure 10.

As CelebA-HQ consists of independent images of celebrities, the nearest neighbors do not need to share the identity of the unlearned person. This is another important reason why we care about the quality after unlearning, since the model output can affect nearby images (not of the same person), which can reduce the utility of unlearned models.

Thank you for pointing our the visualization and we will clarify this point.


[Q3] Repeated values in Table 9

As the values are measured under the same experimental settings using the same seed images, the measures can be identical even though the unlearning method varies. However, as mentioned in the experimental settings, we ran the experiment five times and observed low standard deviations. You can find evidence of statistical reliability in Tables 14 and 15.

Thank the authors for the response, which addressed some of my concerns. However, it does not directly explore methods other than generating random faces, nor does it assess the privacy risks associated with retaining images belonging to the same identity. And it does not fully address the issues of retaining identity information for a large number of images. For now I keep my score.

Thank you for pointing out the unresolved points from the initial rebuttal. We hope these clarifications help the reviewer reconsider the assessment. We welcome any further questions or suggestions.


Q1. Why not consider generating random faces for unlearned IDs?

We agree that random replacement is a relevant alternative for data unlearning. Given the limited rebuttal period, the table below summarizes published evidence from ReTrack [1], suggested by Reviewer 5Keg. In short, ReTrack maps the unlearning data to the k-nearest points of the retain dataset. Its neighbor-count ablation expands the target set from 1 to 1,000, using more retained neighbors. We believe that using more neighbors broadens the set of replacement targets, providing an alternative related to the reviewer’s suggestion of random replacement.

[Effect of the number of redirection neighbors on MNIST T-Shirt. Table 4 of ReTrack]

neighbor number (k)Forget-set generation ↓NLL ↑FID ↓
10.00% ± 0.00%8.29 ± 0.042.22 ± 1.02
100.00% ± 0.00%8.28 ± 0.032.09 ± 1.04
1000.00% ± 0.00%8.28 ± 0.042.04 ± 1.02
10000.00% ± 0.00%8.28 ± 0.042.01 ± 1.05

Note that EraseDiff, another related work in the paper, guides predictions on the forget data toward random noise, not random images. ReTrack preserves generation quality better than EraseDiff, which supports the suggestion of the reviewer.

Our contribution, however, is a selective mechanism designed to be agnostic to the unlearning method and its choice of replacement target. For ReTrack+Ours, we slightly modify the selective unlearning for the initial 5 steps and apply full-step unlearning in a step-wise way to match the final target. The results below show that selective unlearning can accelerate convergence and effectively improve unlearning performance, especially in the early training stages.

[Full-prompt evaluation on ReTrack and ReTrack+Ours (Selective)]

MetricMethod12345Final
ASR (%) ↑ReTrack4.210.433.358.372.995.8
+Ours12.550.072.975.079.2100.0
SSCD ↓ReTrack0.6950.5720.4780.3980.3370.111
+Ours0.5450.4180.3470.3140.3040.110
CLIP-IQA ↑ReTrack0.7830.7570.7230.7160.7060.698
+Ours0.7870.7690.7420.7430.7450.719

Thus, we believe that the reviewer's question about the choice of unlearning method is useful, especially regarding random-image replacement, while our method still has strengths across different base unlearning methods.


Q2. Privacy risks associated with retaining images belonging to the same identity

[Comparison of unlearning settings and approaches]

MethodUnlearning settingTarget and approach
ReTrack [1]Sample-levelRedirect toward retained neighbors
SISS [2]Sample-levelRetain loss + forget loss
ESD [3]Concept-levelSuppress specified concepts through text-guided model fine-tuning
GUIDE [4]Identity-levelErase a target identity while preserving other identities and generation quality

As the reviewer mentioned, sample-wise deletion cannot guarantee full identity removal across the dataset. However, full identity removal concerns the design of the underlying unlearning algorithm, as illustrated by GUIDE [4], which explicitly targets identity erasure in 3D generative models. Within the scope of data unlearning, we clarify again that we have strong experimental support from other measures and tasks, including data unlearning with ReTrack [1], SISS [2], concept unlearning, and other backbone models. We summarize the experiments in the paper, including the Appendix, to make the scope of the submitted evaluation easier to assess as:

[Experimental Scope]

DimensionEvaluation
Model BackbonesU-Net DDPM, SD v1.4, LDM (Tab. 9), SD v3.0 (DiT / Flow-matching; Tab. 5)
Optimization ObjectivesGA (NegGrad), SISS, EraseDiff, naive deletion, DPO, KTO (Tabs. 2, 9)
Evaluation MetricsFID, SSCD, CLIP-IQA, unlearning success rate, SSCD_norm, Aesthetic score (Tabs. 2, 5, 9, 13, 14; Fig. 8)
Additional EvaluationMIA (SecMI/PIAN) (Tab. 4), LLM-based OOD detection (Tab. 10), NN preservation (Tab. 11; neighbor sets in Fig. 10)
Task ScopeData-level unlearning, Concept unlearning (Nudity) (Fig. 9)

The bolded items indicate additional evaluations beyond those reported in SISS [2] or ReTrack [1].


[1] ReTrack: Data Unlearning in Diffusion Models through Redirecting the Denoising Trajectory (AISTATS 2026).

[2] Data Unlearning in Diffusion Models (ICLR 2025).

[3] Erasing Concepts from Diffusion Models (ICCV 2023).

[4] Generative Unlearning for Any Identity (CVPR 2024).

Reviewer_rdfcOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

the author propose a novel SSCD score that computes the computes the normalized distance between the original and generated image using SSCD score (Pizzi et al., 2022). The evaluation is mainly done on the unconditional and conditional unlearning tasks.

Critical strengths

None

Critical weaknesses

the validation of the proposed metric is limited: no correlation with the human eval, no analysis how the proposed metric improves the existing unlearning benchmarks like HUB, Style, IP, Celebrity, and NSFW.

Questions for authors
  1. Would normalized SSCD help to detect the concept erasure that preserves the rest of the context? For example using EraseAnything (https://arxiv.org/abs/2603.27599) or a similar method
2 comments

We appreciate the reviewer's comments. We would like to highlight the following contributions, also noted by other reviewers, and our rebuttal updates for the reviewer's consideration. We would appreciate it if the reviewer could also take these aspects into account when assessing the paper.

(1) Our Contributions

  • Diffusion-specific insight: Our analysis provides evidence that forgetting is uneven across time and frequency [5Keg, gWR9].
  • Simple and general design: Our selective mechanisms are lightweight and easily integrated with existing unlearning objectives [5Keg, gWR9].
  • Forgetting–quality trade-off: Our method improves forgetting effectiveness while preserving generation quality [5Keg], also evaluating a normalized variant of SSCD to compare unlearning at a fixed perturbation scale.

(2) During rebuttal, we mainly cover

  • Recent baselines and ablations: ReTrack comparisons and separate time/frequency experiments (5Keg [W1-1, W1-2]).
  • Metric clarification: The normalization formula and consistent evaluation across ablations (5Keg [W2, Q1, Q3]).
  • Quality and privacy: Nearest-neighbor preservation, anomaly detection, MIA, and larger-scale experiments (gWR9 [W1, W2]).

We welcome any further questions or suggestions.


[W1] Validation of normalized SSCD

We appreciate the reviewer's comments about our normalized metric. The goal of normalized SSCD is to control the utility-privacy trade-off. We believe that machine unlearning is an important topic, but the evaluation still requires some bound on image perturbation. For example, in privacy domains, we can easily find privacy budgets in similar literature, such as the attack radius in adversarial attacks or the (ϵ,δ)(\epsilon,\delta) budget in differential privacy. These bounds are designed to evaluate performance under the same constraints.

This analysis cannot be directly applied to generative modelling, but we at least need to match the image level within some threshold, in our case ρ\rho. This helps to evaluate SSCD under similar image perturbations. When we only care about small SSCD values (or, equivalently, enlarge ρ\rho in SSCD-norm), the resulting images are noisy or grey.

Therefore, we fix the radius to handle the privacy-quality trade-off. We further explain the rationale behind the normalized scores in detail and state that the normalized score is not an advanced score for existing models, but rather measures unlearning performance in a controlled experimental setup.

Thus, we propose SSCD_norm to investigate the trade-off from a normalized perspective. We also include experimental results with other measures and tasks, including data unlearning in the new ReTrack [1], SISS [2], and other types of concept unlearning and other backbone models. We summarize the experiments in the paper, including the Appendix, to make the scope of the submitted evaluation easier to assess as follows:

DimensionEvaluation
Model BackbonesU-Net DDPM, SD v1.4, LDM (Tab. 9), SD v3.0 (DiT / Flow-matching; Tab. 5)
Optimization ObjectivesGA (NegGrad), SISS, EraseDiff, naive deletion, DPO, KTO (Tabs. 2, 9)
Evaluation MetricsFID, SSCD, CLIP-IQA, unlearning success rate, SSCD_norm, Aesthetic score (Tabs. 2, 5, 9, 13, 14; Fig. 8)
Additional EvaluationMIA (SecMI/PIAN) (Tab. 4), LLM-based OOD detection (Tab. 10), NN preservation (Tab. 11; neighbor sets in Fig. 10)
Task ScopeData-level unlearning, Concept unlearning (Nudity) (Fig. 9)

The bolded items indicate additional evaluations beyond those reported in SISS [2] or ReTrack [1].

[1] ReTrack: Data Unlearning in Diffusion Models through Redirecting the Denoising Trajectory (AISTATS 2026)

[2] Data Unlearning in Diffusion Models (SISS), ICLR 2025.


As a side note, asked by reviewer 5Keg, we fixed a typo in the SSCD_norm to be consistent with the perturbation having a fixed radius, as follows:

SSCDnorm=SSCD(x0,x0+ρx^0(xt,θ)−x0∥∣x^0(xt,θ)−x0∥∣2+ε)\mathrm{SSCD}^{\mathrm{norm}} = \mathrm{SSCD}(\mathbf{x}_0, \mathbf{x}_0 + \rho \frac{\hat{\mathbf{x}}_0(\mathbf{x}_t,\theta) - \mathbf{x}_0}{\||\hat{\mathbf{x}}_0(\mathbf{x}_t,\theta) - \mathbf{x}_0\||_2 + \varepsilon})

The evaluation was already calculated using this equation, so the table does not change at all.

[Q1] Concept erasure with context preservation

Please refer to [W1] above for the general reason for SSCD_norm and its rationale. For the concept erasure, in our submitted paper, we showed how our method works in the setting of concept erasure. In particular, Figure 9 in Section 4.2 provides the same time selection for nudity concept unlearning using Ring-a-Bell prompts and NudeNet evaluation. Across five unlearning steps, the selective version yields a consistently lower nudity detection rate while maintaining higher CLIP-IQA than SISS (Figure 9). At the final step, our method shows less than 0.6 on the nudity rate while keeping the quality (CLIP-IQA).

Training StepNudity Rate (SISS) ↓Nudity Rate (Ours) ↓CLIP-IQA (SISS) ↑CLIP-IQA (Ours) ↑
10.850.800.610.65
20.830.760.610.68
30.820.690.620.68
40.810.660.610.67
50.770.590.610.69

Since we mainly focus on data unlearning, the result supports the broader interpretation that phase-aware updates can benefit concept-level as well as data-level unlearning. The concept-unlearning experiment is included in the main discussion, while data-level unlearning remains the paper's primary scope.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.