- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies data unlearning in diffusion models and argues that forgetting should not be applied uniformly across all diffusion timesteps and frequency components. The authors analyze the phase-dependent behavior of diffusion models and propose a time-selective unlearning strategy that concentrates updates on the timesteps most relevant to memorization, together with a frequency-selective strategy that suppresses unnecessary high-frequency modifications during unlearning. These selection mechanisms are designed as general plug-in components that can be combined with existing unlearning objectives. The paper also introduces a normalized variant of SSCD to reduce the confounding effect of severe image degradation when evaluating forgetting. Experiments across multiple settings show that the proposed approach generally improves the trade-off between forgetting effectiveness and generation quality.
Critical strengths
-
Diffusion-specific insight: The paper shows that effective forgetting in diffusion models is concentrated in particular timestep and frequency regions, offering a clear perspective on where unlearning should be applied.
-
Simple and general design: The proposed selective mechanisms are lightweight, compatible with existing unlearning objectives, and improve the trade-off between forgetting effectiveness and generation quality.
Critical weaknesses
-
Incomplete experimental evaluation: The paper does not include comparisons with more recent data-unlearning methods such as "Unlearning in Diffusion Models: A Unified Framework with KL Divergence and Likelihood Constraints (ICML 2026)" and "ReTrack: Data Unlearning in Diffusion Models through Redirecting the Denoising Trajectory (AISTATS 2026)", whose experimental settings are closely related to this work. In addition, although the proposed and introduce the key hyperparameters and , no ablation or sensitivity analysis is provided for them.
-
Concerns about : As one of the paper's main contributions, the motivation and formulation of appear questionable. The intended normalization focuses on the direction of the unlearning-induced change while discarding its magnitude, although the magnitude of the change is also important for determining whether unlearning is actually effective. Moreover, the squared norm in the denominator of Eq. (13) means that the resulting perturbation does not have a fixed radius, which appears inconsistent with the paper's description of the operation as a projection onto an ball.
Questions for authors
-
In Eq. (13), the denominator contains the squared norm, which means that the resulting perturbation does not have a fixed radius of . Is this a typo, or is this the intended formulation? If intended, could the authors clarify how it is consistent with the description of projecting the perturbation onto an ball?
-
There appear to be several possible typos or formatting issues, including the quotation marks in the caption of Table 2, the missing equation reference in the third paragraph of Appendix B, and the duplicated Sections C.5 and C.6. Could the authors confirm whether these are typos?
-
In Table 3, the filter band ablation is evaluated using , whereas the time window ablation is evaluated using . What is the reason for using different forgetting metrics for these two ablations? Would the conclusions remain consistent if both were evaluated using the same metrics?
-
Could the authors provide visualizations of the normalized samples used to compute , preferably for different values of ? I am concerned that sufficiently large extrapolation along the reconstruction-error direction may produce severely degraded or perceptually meaningless images, which would make the interpretation of the normalized metric less clear.
4 comments
Additionally, we summarize the experiments in the paper, including the Appendix, to make the scope of the submitted evaluation easier to assess as follows:
| Dimension | Evaluation |
|---|---|
| Model Backbones | U-Net DDPM, SD v1.4, LDM (Tab. 9), SD v3.0 (DiT / Flow-matching; Tab. 5) |
| Optimization Objectives | GA (NegGrad), SISS, EraseDiff, naive deletion, DPO, KTO (Tabs. 2, 9) |
| Evaluation Metrics | FID, SSCD, CLIP-IQA, unlearning success rate, SSCD_norm, Aesthetic score (Tabs. 2, 5, 9, 13, 14; Fig. 8) |
| Additional Evaluation | MIA (SecMI/PIAN) (Tab. 4), LLM-based OOD detection (Tab. 10), NN preservation (Tab. 11; neighbor sets in Fig. 10) |
| Task Scope | Data-level unlearning, Concept unlearning (Nudity) (Fig. 9) |
The bolded items indicate additional evaluations beyond those reported in SISS [2] or ReTrack [1].
[W2-1, Q1] Eq. (13): squared norm or fixed-radius projection?
Sorry for the confusion. It is a typo. We replace the squared L2 norm with the L2 norm, as follows:
We have corrected the equation. The evaluation was already calculated using this equation, so the reported numerical results remain unchanged.
See further rationale behind normalization in [W2-2] below.
[W2-2] Motivation and formulation of normalized SSCD
The goal of normalized SSCD is to control the utility-privacy trade-off. We believe that machine unlearning is an important topic, but the evaluation still requires some bound on image perturbation. For example, in privacy domains, we can easily find privacy budgets in similar literature, such as the attack radius in adversarial attacks or the budget in differential privacy. These bounds are designed to evaluate performance under the same constraints.
This analysis cannot be directly applied to generative modeling, but we at least need to match the image level within some threshold, in our case . This helps to evaluate SSCD under similar image perturbations. This is directly connected to the reviewer's concern about image visualization in [Q4] below. When we only care about small SSCD values (or, equivalently, enlarge in SSCD-norm), the resulting images are noisy or grey.
Therefore, we fix the radius to handle the privacy-quality trade-off. We have clarified the rationale in Section 4.1, immediately after the equation, i.e., normalized SSCD compares reconstruction-error directions at the same perturbation scale.
[Q2] Typos and formatting issues
Thank you for pointing out the typos and formatting errors. We fixed all the minor problems that the reviewer pointed out. Going further, we also checked other quotation marks, such as those in Figures 20 to 24, and fixed the quotation marks accordingly. Sorry for the rush during the submission phase. Thus, we have carefully checked all the details again following the reviewer's suggestion. Thank you for improving the overall quality of the paper.
[Q3] Different metrics for the two ablations
We conducted both experiments separately and initially reported each score. However, due to the 9-page constraint, we concatenated the two tables, and the evaluation varied. We clearly state that we do not cherry-pick the evaluation. Below, we provide the results with the same evaluation under SSCD-norm.
| Filter band | SSCD-norm↓ () | Aes.↑ () | SSCD-norm↓ () | Aes.↑ () | Time window | SSCD-norm↓ () | Aes.↑ () | SSCD-norm↓ () | Aes.↑ () |
|---|---|---|---|---|---|---|---|---|---|
| Base | 0.544 | 4.094 | 0.502 | 4.845 | Base (Full) | 0.430 | 4.094 | 0.501 | 4.845 |
| Low-pass | 0.332 | 5.553 | 0.387 | 5.904 | [250, 750] | 0.349 | 5.520 | 0.399 | 6.095 |
| Mid-pass | 0.377 | 4.374 | 0.417 | 5.043 | [250, 500] | 0.415 | 5.790 | 0.439 | 5.951 |
| High-pass | 0.625 | 4.143 | 0.597 | 4.670 | [333, 666] | 0.399 | 5.706 | 0.420 | 5.696 |
Thank you for suggesting the unification of the evaluation metric for the results. We have updated Table 3 in the revised manuscript.
[Q4] Normalized-sample visualizations across
We select , which is the empirical sweet spot (optimal balance point) of the privacy-utility trade-off. When , the quality itself is too bad for human eyes, and is too similar to the original images.
| Method | SSCD_50 ↓ () | SSCD_50 ↓ () | SSCD_100 ↓ () | SSCD_100 ↓ () | SSCD_150 ↓ () | SSCD_150 ↓ () | SSCD_200 ↓ () | SSCD_200 ↓ () |
|---|---|---|---|---|---|---|---|---|
| SISS | 0.786 | 0.778 | 0.430 | 0.501 | 0.395 | 0.346 | 0.295 | 0.259 |
| +Ours | 0.742 | 0.810 | 0.349 | 0.399 | 0.251 | 0.223 | 0.203 | 0.179 |
| KTO | 0.727 | 0.808 | 0.366 | 0.442 | 0.283 | 0.277 | 0.218 | 0.193 |
| +Ours | 0.702 | 0.793 | 0.340 | 0.377 | 0.275 | 0.239 | 0.214 | 0.201 |
In the revised version, we have added normalized-sample visualizations for in Figure 25 of Appendix D, allowing readers to directly inspect the effect of increasing the perturbation radius.
Thank you for the detailed response, which has addressed my concerns. I have raised my score to 3.
We appreciate the reviewer's comments about the additional recent baselines and rationale behind the proposed method. Here, we provide more experimental results and explanations.
We welcome any further questions from the reviewer.
[W1-1] Recent baselines
We agree that investigating further recent papers in data unlearning is useful. Thus, among the two papers that the reviewer suggested, we adopt the proposed selective unlearning in ReTrack [1], which fits well with our experimental setups. Based on the ReTrack implementation, we apply our selective unlearning. Note that the formulation of ReTrack applies unlearning to the k-nearest samples, which requires longer training while providing stable unlearning. Thus, we test whether selective unlearning can accelerate convergence, as shown in Figure 8 in the original submission. We have added the ReTrack comparison to Section 4.2.
For ReTrack+Our method, we slightly modify the selective unlearning for the initial 5 steps and apply full-step unlearning in a step-wise way to match the final target. The results below show that selective unlearning can accelerate convergence and effectively improve unlearning performance, especially in the early training stages.
Full-prompt evaluation
Partial-prompt evaluation
[1] ReTrack: Data Unlearning in Diffusion Models through Redirecting the Denoising Trajectory (AISTATS 2026)
[W1-2] hyperparameter ablations
For k and s, we state that For the proposed method, we fix the time suppression intensity at k=0 in Equation (7) and the high-frequency suppression weight at s=0 in Fourier filtering to simplify the design. in the Appendix. However, we agree that it is hard to see. Thus, we moved it to the experimental setup.
To push further, in addition to Table 3 (refer to Q3 below), we agree that the sensitivity analysis for frequency and time can support the strength of the proposed method. Since our primary method selectively controls time and frequency while retaining all other experimental settings from SISS [2], we conducted independent ablation studies varying only time steps and frequency cutoffs. The results are as follows:
[Time step only]
[Freq only]
The results show that both time and frequency selection independently improve forgetting/convergence while protecting quality. However, the effect of using each method individually is weaker than that of using both methods together, as shown in Table 2. We have included these independent ablations in Appendix C.3.
[2] Data Unlearning in Diffusion Models (SISS), ICLR 2025.