acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Sequential Reappearance in Diffusion Data-Point Unlearning

Abstract

Diffusion data-point unlearning is typically evaluated immediately after each deletion, even though subsequent requests may repeatedly update the same model. We identify sequential reappearance, a failure mode in which an instance that is initially judged to be forgotten later returns to the memorized regime without reuse of the deleted data or adversarial fine-tuning. To capture this behavior, we introduce a target-level evaluation protocol that tracks whether each target is forgotten immediately, remains forgotten at the end of the sequence, or reappears during subsequent deletions. We further find that targets that later reappear exhibit greater local recovery accessibility after deletion than targets that remain forgotten. We characterize this vulnerability as recovery accessibility and show that this post-deletion local geometry is associated with subsequent reappearance risk. Motivated by this finding, we propose a recovery-guided unlearning method that iteratively identifies the most recoverable region around each target and extends the deletion objective to that region. Experiments show that our method improves forgetting persistence across long deletion sequences while maintaining competitive generation quality, highlighting the importance of evaluating diffusion unlearning beyond immediate deletion efficacy.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, Diffusion Models, Sequential Unlearning, Data Memorization, Unlearning Persistence

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LuJ4OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the authors propose sequential unlearning at the data-point level for Diffusion Models. The authors argue that existing unlearning methods, when applied to sequential unlearning, suffer from sequential reappearance. Basically, previously unlearned targets reappear when other targets are sequentially unlearned, which is a critical vulnerability. So here they propose LASTING, which extends SISS with PGD over bounded image/latent perturbations to identify low-denoising-loss neighbors for forget-side updates, reducing target reappearance during sequential diffusion unlearning.

Critical strengths
  1. This is a promising direction, as in practical real-world deployment, we need scalable and reliable unlearning methods that can handle sequential requests.
  2. Concept reappearance is a critical vulnerability with unlearning algorithms, and in a sequential setup, this will increase further. In this work, LASTING offers good improvement over other baseline methods with respect to this.
Critical weaknesses
  1. This sequential unlearning setup has been explored at the concept level in both the Diffusion and LLM setups. So this is a meaningful extension at the data-point level, but the paper lacks in the evaluation/metrics section. Catastrophic forgetting (model degradation) is a common problem at the concept level. In the case of CelebA-HQ, since it has access to full retention data, this won’t be very significant. But when it comes to SD models, we don’t have access to retain data, and we mostly use a small synthetic dataset, so retention is affected. So, for the SD experiments, we need more metrics to measure this. Here, the authors have not reported FID comparisons for the SD experiments as well.
  2. LASTING doesn’t perform very well on scaling to SD models, and also when full retention data is not available. With SD models, we can see that the CLIP-IQA metric shows that retention is comparatively poor compared to other methods, so, as I mentioned in the 1st point, the model might be facing catastrophic forgetting.
  3. The sequential unlearning setup for CelebA-HQ assumes that it knows all the future deletion requests, which is not practical (Appendix C.1). In the real-world deployment, we only have the current unlearned model along with the current unlearning request and previous requests (if stored).
  4. Novelty/Contribution - Sequential reappearance is not a significant contribution, as it has already been shown in the literature that fine-tuning the unlearned model on even random data causes reappearance of the unlearned target. Unlearning is essentially fine-tuning, so the reappearance phenomenon is expected. Even sequential unlearning works at the concept level for LLMs and Diffusion Models face the problem of concept reappearance, but they mainly focus on Catastrophic forgetting, as it’s an even bigger problem.
  5. Limited to just the SDv1.5 model only.
Questions for authors
  1. This is a promising direction, but currently it requires substantial changes. See the points mentioned in Weaknesses.
  2. Better evaluation metrics for SD experiments. It is better to focus experiments on SD models, as in practice, you don’t have access to retained data, so we need methods that perform well even without it. Also, unlearned models should have good retention performance, which isn’t really measured here.
  3. [Future direction] I would like to see how the unlearning persists when earlier deleted targets or related variants reappear in later retain batches, measuring recovery speed, reappearance rates, and retained semantic utility.
7 comments
AuthorsOpenReview ↗

Response to Reviewer LuJ4 (1/3)

We thank the reviewer for the constructive feedback. We agree that forgetting persistence should be evaluated alongside generation performance on non-deleted content. Below, we present additional SD generation evaluations, examine sequential unlearning without advance knowledge of future deletion requests, and clarify our contribution relative to prior work.

W1·W2 + Q2. Generation quality and retention in SD

CLIP-IQA in Table 4 measures perceptual image quality without a reference image and was evaluated on outputs from deletion-target prompts. Since these results alone cannot establish general generation ability or text alignment, we additionally compare the Base model with the models at K=10, 20, and 30 using fixed COCO prompts.

We select one caption for each of 10,000 distinct COCO val2014 images using selection seed 2026 and generate 10,000 images per model state. All methods use the same captions, 512×512 resolution, PNDM with 50 steps, and a guidance scale of 7.5.

We evaluate SD v1.4 using FID-COCO, FID-base, CLIPScore, CLIP-IQA, and MUSIQ. FID-COCO measures distributional differences from real COCO images, while FID-base measures deviation from the original model's generation distribution. CLIPScore measures image–text alignment (Hessel et al., 2021), whereas CLIP-IQA (Wang et al., 2023) and MUSIQ (Ke et al., 2021) assess perceptual image quality without a reference image. The quality metrics use fixed COCO prompts and seeds, while Final Reappearance uses the deletion-target evaluation from Table 4. Base is the model before unlearning. Bold indicates the best value among updated methods, excluding Base and including ties.

Below, we show K=30, the final checkpoint of the 30-request sequence. Please refer to Table 20 in the appendix for the full COCO generation-quality results at K=10, 20, and 30.

Table C1. SD v1.4 (K=30)

MetricBaseSISSLASTING (Ours)ReTrackFU
Final Reappearance (%) ↓—22.455.886.9031.25
FID-COCO ↓23.688128.866428.199629.530326.8380
FID-base ↓0.00008.41488.19359.59986.8054
CLIPScore ↑0.81150.81590.81620.80790.8159
CLIP-IQA ↑0.88590.88870.88620.87240.8800
MUSIQ ↑70.439069.864269.939069.355769.0722

LASTING largely preserves perceptual quality and text alignment relative to Base, with higher CLIPScore, nearly unchanged CLIP-IQA, and only a 0.50-point decrease in MUSIQ. It achieves the highest CLIPScore and MUSIQ among the updated methods and lower FID-COCO than SISS and ReTrack, alongside the lowest reappearance rate. These results show that LASTING maintains competitive generation quality on non-deleted content while improving forgetting persistence.

W3. Sequential unlearning without advance knowledge of future deletion requests

The original protocol in Appendix C.1 uses a fixed retain pool that excludes all 300 deletion targets in advance. To test whether LASTING's persistence gains hold without advance knowledge of future deletion requests, we ran sequential unlearning experiments that exclude only previously deleted targets and the current deletion target from the retain pool. Future targets remain eligible for retention until their deletion requests arrive. We compare LASTING with SISS, ReTrack, and FU over 300 deletion requests. FID follows the original CelebA-HQ evaluation protocol.

Below, we show K=300, the final checkpoint. Please refer to Table 22 in the appendix for the K=100 and K=200 results.

Table C2. Sequential unlearning (K=300, seed=42)

MetricLASTING (Ours)SISSReTrackFU
Mean Immediate SSCD ↓0.20730.29330.33740.2869
Mean Final SSCD ↓0.40590.41150.58260.6562
FID ↓21.262219.390254.909229.3144
Final [email protected] ↑95.00%92.67%64.33%45.33%
Final Reappearance ↓4.68%7.02%28.63%54.55%

LASTING achieves the lowest Final Reappearance and highest Final Success among the compared methods. These results show that LASTING's persistence gains hold without advance knowledge of future deletion requests.

AuthorsOpenReview ↗

Response to Reviewer LuJ4 (2/3)

W4 + part of W1. Contribution relative to prior recovery and sequential-unlearning studies

Prior work on recovery after fine-tuning and concept-level sequential unlearning also examines whether forgetting persists through later model updates. Our contribution, however, is to link each target's reappearance risk to its Recovery Accessibility after deletion. We use this analysis to decide which representations to update during unlearning. We track deleted targets through ordinary subsequent deletions without target reuse or a separate recovery intervention. This analysis motivates selecting locally recoverable, low-denoising-loss representations as update targets, with benefits under both SISS and FU.

The results below examine the direction of neighbor selection and its applicability to another objective. Bold indicates the lowest reappearance rate within each base objective.

Table C3. Extended neighbor-selection ablation (K=200, %)

Forget-side update targetSISS Final Success ↑SISS Final Reapp. ↓FU Final Success ↑FU Final Reapp. ↓
Original target94.505.5065.5034.17
Random Neighbor75.5023.7461.5038.19
High-Loss Neighbor54.0037.2045.5051.85
Low-Loss Neighbor (LASTING, Ours)97.502.5095.504.50

Our methodological contribution is to turn the Recovery Accessibility finding into an explicit rule for choosing what to unlearn. LASTING targets the locally recoverable, low-loss representations implicated by our analysis of sequential reappearance. The ablation directly tests this selection rule. Low-loss selection improves persistence under both SISS and FU, while random perturbations worsen it. High-loss selection substantially worsens persistence, raising reappearance rates to 6.8× for SISS and 1.5× for FU relative to original-target unlearning. This connects an explanation of why deleted targets return to a concrete intervention that improves forgetting persistence across different objectives.

Q3. Reintroduction of previously deleted targets or variants

We thank the reviewer for suggesting this promising research direction. Given the time constraints, we conducted a small exploratory experiment on CelebA-HQ, starting from each method's K=200 checkpoint. We randomly selected 10 targets that had SSCD<0.7 both immediately after deletion and at K=200 under both methods. During each of the next 10 deletion stages (201–210), all 10 selected targets were reintroduced, one target in each of 10 distinct optimizer updates. Each target was inserted once per stage, giving 10 re-exposures per target and 100 target insertions in total. For each method, a matched run without reinsertion served as the control.

The rates below refer to the 10 selected targets. Bold indicates the lowest reappearance count within each reinsertion condition, including ties.

Table C4.

MethodReinsertionFinal Reapp. (K=210) ↓Ever Reapp. (K=201–210) ↓Re-exposures until first reappearance
SISSNo0/100/10—
SISSYes4/104/101, 3, 4, 10
LASTINGNo0/100/10—
LASTINGYes2/102/105, 7

All 10 targets receive 10 re-exposures each over stages 201–210. The last column reports the cumulative re-exposure count at the first stage-end detection (SSCD≥0.7) for each reappearing target. A dash indicates no reappearance.

Neither method showed reappearance without reinsertion. With reinsertion, LASTING had fewer reappearing targets (2/10 versus 4/10), and its earliest return occurred after five re-exposures, compared with one for SISS. SISS may retain nearby representations that allow deleted targets to return quickly. LASTING updates these representations during unlearning, which may make rapid recovery less likely. Developing methods that remain robust to repeated re-exposure is a promising direction for future work. We thank the reviewer for this suggestion.

AuthorsOpenReview ↗

Response to Reviewer LuJ4 (3/3)

W5. Additional evaluation on SD v2.1

Thank you for suggesting evaluation on an additional model. We selected SD2.1 to evaluate LASTING beyond SD1.4.

Evaluating instance-level unlearning requires first establishing that the model reproduces the target image before deletion. Otherwise, failure to reproduce it afterward does not demonstrate suppression of previously memorized content. In its SD v1.4 experiments, SISS uses prompts that induce reproduction of specific training images to evaluate instance-level unlearning (Alberti et al., 2025, §5.3). Following this setting, we evaluate whether reproduction of memorized images is suppressed after deletion and whether this suppression persists through subsequent deletion updates.

Because the same prompt can elicit different memorization behavior across models, we used candidate memorization prompts from prior work (Jeon et al., 2025, Appendix D) rather than directly reusing the SD1.4 prompt set. We generated images using these candidates and manually inspected the outputs to identify 30 prompt–target pairs exhibiting memorization. We then used these pairs for sequential unlearning on SD2.1.

We evaluate the final model after 30 deletion requests (K=30) using both the original and modified prompts for each target, yielding 60 prompt–target cases in total. For each case, we generate 16 images and consider the target detected if at least one image has an SSCD similarity to the target reference at or above the threshold.

Somepalli et al. identify candidate copies of training images using SSCD similarity above 0.5 in their Stable Diffusion experiments (Somepalli et al., 2023, §3–4), while Finding DoRI uses scores above 0.7 to detect reproduction of memorized images (Kowalczuk et al., §3.2). To examine sensitivity to the detection threshold, we evaluate τ∈{0.4, 0.5, 0.6, 0.7, 0.8}, covering these prior cutoffs and nearby values. We apply all five thresholds to the same generated images and SSCD scores.

Table C5. Forgetting performance on SD2.1 (K=30, %)

MetricSISSFUReTrackLASTING (Ours)
Final Reappearance (τ=0.4) ↓22.220.0038.895.77
Final Reappearance (τ=0.5) ↓20.690.0022.225.17
Final Reappearance (τ=0.6) ↓18.330.0010.001.67
Final Reappearance (τ=0.7) ↓13.330.001.670.00
Final Reappearance (τ=0.8) ↓6.670.000.000.00
Final Success (τ=0.4) ↑76.67100.0061.6795.00
Final Success (τ=0.5) ↑78.33100.0076.6795.00
Final Success (τ=0.6) ↑81.67100.0090.0098.33
Final Success (τ=0.7) ↑86.67100.0098.33100.00
Final Success (τ=0.8) ↑93.33100.00100.00100.00

LASTING matches or outperforms SISS and ReTrack on both forgetting metrics across all evaluated thresholds. Although FU achieves 100% Final Success at every evaluated threshold, we observed severe quality degradation when inspecting its generated images.

General generation quality. To assess whether improved forgetting persistence preserves general generation ability, we evaluate the same checkpoints on non-deleted COCO prompts and compare them with Base. FID-COCO measures distributional differences from real COCO images, while FID-base measures deviation from Base's generation distribution. CLIPScore measures image–text alignment, and CLIP-IQA and MUSIQ assess perceptual quality without a reference image.

Table C6. General generation quality on COCO for SD2.1 (K=30)

MetricBaseSISSFUReTrackLASTING (Ours)
FID-COCO ↓24.4130.0798.0341.6126.98
FID-base ↓0.0014.8385.9032.079.47
CLIPScore ↑0.81850.81820.68490.78770.8179
CLIP-IQA ↑0.93710.86860.56450.86040.8857
MUSIQ ↑71.1567.2159.7269.3467.75

LASTING largely preserves Base's text alignment and perceptual quality, with nearly unchanged CLIPScore and modest reductions in CLIP-IQA and MUSIQ. Its FID-COCO remains close to Base, and it achieves the lowest FID-COCO and FID-base among the updated methods. FU's strong forgetting performance comes with substantial generation-quality degradation. Its FID-COCO is approximately 4.0× that of Base.

Overall, LASTING achieves stronger forgetting persistence than SISS and ReTrack while maintaining competitive generation quality and text alignment on non-deleted prompts.

We will release the images and the full list of prompts used in this evaluation on GitHub.

Reviewer_MkPwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether the individual data point remain forgotten in an unlearning process as a diffusion model does further unlearning requests on other data points. It shows that previously forgotten images can reappear after later unlearning on other images, and it finds that immediately after unlearning, images that later reappear tend to have nearby variants that the model can denoise with a relatively low loss. To quantify this gap, the paper introduces recovery accessibility, which measures the relative reduction in denoising loss achieved by searching around each target image. Motivated by this observation, they propose LASTING, which directly searches for nearby variants with low denoising loss around the target image and uses that variant in place of the original image as the target of the unlearning process. Experiments on CelebA-HQ and Stable Diffusion show improved forgetting persistence while maintaining competitive generation quality.

Critical strengths
  1. The paper evaluates whether forgetting lasts through later deletion requests. This addresses a practical concern: removing another image should not bring back samples that were already forgotten.

  2. The paper includes a number of experiments across different models and unlearning methods. It tests different numbers of deletion requests and examines the main design choices. The comparison with SISS under a similar compute budget is also helpful.

  3. The observation about nearby variants particularly insightful. Immediately after unlearning, images that later reappear tend to have nearby variants that the model can denoise with a relatively low loss. This is an useful insight. And LASTING is straightforward and can be added to existing unlearning methods without redesigning their objectives. The results with both SISS and FU suggest that its benefit is not limited to a single method.

Critical weaknesses
  1. Visual samples do not clearly support the advantage suggested by the SSCD results In several visual results, SISS makes the orignal image harder to recognize than LASTING while LASTING gets better average SSCD scores.

  2. 0.7 as the SSCD threshold needs stronger justification The paper follows prior work, but it is unclear why this cutoff reliably separates forgotten targets from recovered ones in the current setting. This matters because the reported success and reappearance rates on CelebA-HQ depend directly on this threshold.

Questions for authors
  1. SISS appears to disrupt the target images more strongly, while LASTING has a better performance on SSCD scores. It raises a concern whether the numerical improvement reflects a meaningful improvement in preventing the target image from being reproduced. Can authors explain this gap and possibly make it clear that how many individual examples that has such phenomenon are in the experiments. Because unlearning technique is to make the model no longer able to generate the target image, not to get a good score on metrics.

  2. Can author explain why the threshold 0.7 can be considered as the standard to judge if a target image is successfully unlearned. Also what evidence supports treating an image below 0.7 as successfully forgotten, particularly when some examples still look similar to the original? A comparison across a reasonable range of thresholds would help establish whether the conclusions are robust.

Considering mentioned problems, the main contribution can be limited to the finding that immediately after unlearning, images that later reappear tend to have nearby variants that the model can denoise with a relatively low loss. This is insightful, however, if above 2 questions can not get reasonable explanation, the score will be dropped.

4 comments
AuthorsOpenReview ↗

Response to Reviewer MkPw (1/1)

We thank the reviewer for the constructive feedback. We explain how SSCD relates to visual appearance, assess persistence across multiple thresholds, and manually inspect discrepancies between visual judgments and SSCD scores.

W1 + Q1-a. SSCD and visual appearance

SSCD (Self-Supervised Copy Detection) is an image descriptor trained to match originals with transformed copies (Pizzi et al., 2022). Blur, compression, or visible noise can therefore coexist with a high score when features identifying the original remain. SSCD is widely used to assess training-image reproduction in diffusion models, and we follow this evaluation practice (Somepalli et al., 2023b).

Some of the paper's qualitative examples indeed show stronger distortion or lower SSCD for SISS than LASTING. We added individual SSCD values to Figure 5 so readers can directly compare distortion, noise, and scores. Our central goal is to evaluate how many successfully forgotten targets remain forgotten through subsequent deletion updates, rather than how strongly individual outputs are distorted.

W2 + Q2. Threshold interpretation and robustness

We agree that a single SSCD cutoff may not separate target reproduction equally well for every image. We follow prior work (Somepalli et al., 2023b) in using τ=0.7 as a consistent evaluation criterion and address this limitation by evaluating thresholds from 0.60 to 0.80.

To illustrate the visual meaning of these thresholds, we added Figure 4 to Appendix B.4, showing final reconstructions near SSCD values of 0.60, 0.65, 0.70, 0.75, and 0.80. Each score is within 0.01 of the indicated value, using CelebA-HQ at K=300.

We also added Tables 15 and 16 to Appendix B.4, comparing Final and Ever Reappearance, respectively, across these five thresholds and K∈{50,100,200,300}. Both metrics are evaluated using existing SSCD scores. Ever Reappearance counts any return during subsequent deletions. Below are the Final Reappearance results at K=200 and K=300, averaged over three training seeds.

Table B1. Final Reappearance (K=200, %) ↓

Methodτ=0.60τ=0.65τ=0.70τ=0.75τ=0.80
SISS14.528.224.332.001.33
EraseDiff75.4266.3051.9931.6311.33
ReTrack26.4416.8610.236.353.90
FU22.4917.9212.067.173.33
Prompt-Free66.6255.8044.1730.0013.00
LASTING8.754.703.011.501.34

Table B2. Final Reappearance (K=300, %) ↓

Methodτ=0.60τ=0.65τ=0.70τ=0.75τ=0.80
SISS20.5013.797.934.452.22
EraseDiff75.0168.6953.8931.558.57
ReTrack67.4259.6149.8436.6919.19
FU26.2023.8219.0612.785.22
Prompt-Free73.8063.8952.3336.5617.33
LASTING12.015.593.121.781.22

Across the 20 K–threshold settings, LASTING achieves the lowest or tied-lowest Final Reappearance in 19 and Ever Reappearance in all 20. The only exception is Final Reappearance at K=200 and τ=0.80: 1.33% for SISS versus 1.34% for LASTING, a difference of 0.01 percentage points. These results support that LASTING's gains reflect sustained suppression of target reappearance across thresholds, rather than merely favorable scores within a narrow range around τ=0.7.

Q1-b. Manual inspection of visual–SSCD discrepancies

Moreover, to check for ambiguities in score-based judgments, we manually inspected the 164 targets for which SISS had higher Final SSCD than LASTING at K=300 (training seed 42). Our judgments focused on distinctive facial features identifying the individual, rather than general shape or appearance. SISS appeared to remove the original more effectively in 15 cases (9.1% of this subset). In 2 of these cases, SSCD classified SISS as reappearing and LASTING as forgotten at τ=0.7.

Even treating these two cases as non-reappearances for SISS reduces its Final Reappearance from 29/298 (9.73%) to 27/298 (9.06%), still above LASTING's 7/300 (2.33%). We will release all inspected images and assessment results on GitHub.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer,

Thank you again for your time and thoughtful review of our work. As the discussion period draws to a close, we would like to let you know that we remain available for any further questions about our responses. Please feel free to let us know if any points would benefit from clarification or if additional information would be helpful. We would be happy to follow up.

We sincerely appreciate your time and consideration.

I thank the authors for their detailed response. The rebuttal has adequately addressed my concerns, and I have no further questions. I will maintain my original rating. I encourage the authors to incorporate the clarifications and additional results into the final version.

AuthorsOpenReview ↗

Thank You for Your Positive Follow-up and Constructive Feedback

Thank you very much for your thoughtful follow-up and for acknowledging that our rebuttal has adequately addressed your concerns. We sincerely appreciate your constructive feedback, which helped us strengthen the evaluation and clarify the interpretation of our results.

As you suggested, we will incorporate the additional analyses and clarifications into the revised manuscript. Thank you again for your time and careful consideration.

Reviewer_p2dgOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper talks about an interesting phenomenon they find: the forgotten data may reappear in the later update for the diffusion model, and they connect this with the recovery accessibility. To fix this problem, the authors propose a method called LASTING to search for the nearby representation to increase the forgetting persistence. Their experiments on stable diffusion also demonstrate the effectiveness of the proposed method.

Critical strengths

The sequential reappearance phenomenon is an important but easily overlooked problem in data unlearning. The authors also provide a possible explanation of the phenomenon: influence of local neighboring regions, providing insight for future research on forgetting persistence.

The paper also provides sufficient experiments on Stable diffusion and CelebA, comparison with multiple baselines, and enough ablation experiments to show the effectiveness of their LASTING method especially in longer deletion sequences.

Critical weaknesses

The methods proposed in this paper are relatively limited in novelty. Undoubtedly, the LASTING method can effectively improve the forgetting persistence compared to most other baselines, however, this method is closer to improvement of existing forgetting methods.

LASTING’s cost for improving forgetting persistence can exist. It does not outperform other baseline across all metrics. Directly replacing the forgetting target in the loss function may also introduce trade-offs in other aspects of performance. For example, compared with SISS, LASTING achieves a relatively modest improvement in forgetting persistence but worse FID performance, indicating lower generation quality.

Questions for authors

The authors show a correlation between local recovery accessibility and sequential reappearance, while it is limited to a L∞L_\infty-bounded neighborhood. Could the authors further clarify under what conditions this relationship holds, and whether there exists other factors may also affect forgetting persistence?

Moreover, since LASTING directly replaces the origional target, could the authors further discuss whether this replacement introduces additional costs or trade-offs beyond the forgetting persistence?

4 comments
AuthorsOpenReview ↗

Response to Reviewer p2dg (1/2)

We thank the reviewer for the constructive feedback. We address the contribution, the computational and quality trade-offs, and the conditions under which recovery accessibility is associated with subsequent reappearance.

W1. Methodological novelty

Our central contribution is the target-level analysis of sequential reappearance and the resulting perspective on which representations should be updated during unlearning. We find that a successfully forgotten target can remain vulnerable to subsequent deletion updates when nearby representations retain low denoising loss. Motivated by this observation, LASTING selects a locally recoverable representation as the forget-side update target.

Tables 5 and 7 examine the direction of neighbor selection and its applicability to another unlearning objective.

Table A1. (From Tables 5 and 7)

Base objectiveForget-side update targetFinal Reappearance (K=200, %) ↓
SISSOriginal target5.50
SISSRandom Neighbor23.74
SISSHigh-Loss Neighbor37.20
SISSLow-Loss Neighbor (LASTING)2.50
FUOriginal target34.17
FULow-Loss Neighbor (FU+LASTING)4.50

At K=200, random perturbations and high-loss neighbor selection worsen persistence, whereas low-loss neighbor selection improves it for both SISS and FU. These results support recovery-guided selection of the update target as a contribution that extends beyond improving one particular objective.

W2 + Q2. Computational cost, generation quality, and forgetting persistence

The additional cost of target replacement comes from the inner search for a forget-side neighbor. The number of inner-search steps, J, controls the trade-off between computation and forgetting persistence.

Table A2. (From Tables 9, 10, and 12)

MethodTime per target (s) ↓Final Reappearance (K=300, %) ↓
SISS257.709.73
LASTING, J=4647.264.33
LASTING, J=8960.432.33

Reducing J from 8 to 4 decreases time per target by approximately 32.6%, while retaining a lower reappearance rate than SISS in this setting. The search budget can therefore be adjusted to the desired persistence and computational budget.

Quality on non-deleted images. To examine whether LASTING improves forgetting persistence while maintaining reconstruction quality comparable to SISS, we randomly select 200 CelebA-HQ retain images at each K and reconstruct the same images with Base, SISS, and LASTING after adding noise at t=250. FID and MUSIQ are computed on these reconstructions. The FID reference consists of the 30,000−K images excluding deletion targets. K denotes the cumulative number of deletion requests. Evaluation images differ across K, so Base scores vary.

Table A3.

MethodFID (K=200) ↓MUSIQ (K=200) ↑FID (K=300) ↓MUSIQ (K=300) ↑
Base48.641056.386847.287856.8207
SISS48.828753.792748.258354.4251
LASTING (Ours)48.955654.709048.023753.6027

LASTING achieves retain-reconstruction FID and MUSIQ scores comparable to SISS while improving forgetting persistence.

For SD v1.4, we compare Base, SISS, and LASTING on general generation at K=10, 20, and 30. We select one caption for each of 10,000 distinct COCO val2014 images and generate 10,000 images per model state. All methods use the same captions, 512×512 resolution, PNDM with 50 steps, and a guidance scale of 7.5. Base is a common reference across K. Bold indicates the better result between SISS and LASTING at each K, including ties.

Table A4.

MetricBaseSISS (K=10)LASTING (K=10)SISS (K=20)LASTING (K=20)SISS (K=30)LASTING (K=30)
FID-COCO ↓23.688128.497528.330127.653926.490428.866428.1996
FID-base ↓0.00007.82808.04387.89717.35648.41488.1935
CLIPScore ↑0.81150.81490.81570.81560.82230.81590.8162
CLIP-IQA ↑0.88590.88720.88640.87390.86980.88870.8862
MUSIQ ↑70.439070.115370.211768.453067.423869.864269.9390

LASTING achieves lower FID-COCO and higher CLIPScore than SISS at all three checkpoints, while CLIP-IQA and MUSIQ remain close to SISS. These results show that LASTING improves forgetting persistence while maintaining competitive general-generation quality.

AuthorsOpenReview ↗

Response to Reviewer p2dg (2/2)

Q1-a. Scope of the association between recovery accessibility and reappearance

We vary the local measurement radius to examine the scope of this association. The original radius is 4/255 in the [0,1] pixel scale, corresponding to 8/255 in the normalized [−1,1] image space. We compare 2/255, 4/255, and 8/255, using fixed timesteps and a fixed noise bank with eight projected search steps.

The following results use CelebA-HQ targets that were successfully forgotten immediately after deletion and have a complete follow-up of 30 deletion stages. Each cell reports Spearman ρ / partial Spearman ρ controlling for Immediate SSCD / AUROC.

Table A5.

Method2/2554/255: original radius8/255
EraseDiff0.446 / 0.446 / 0.7070.528 / 0.529 / 0.7400.553 / 0.553 / 0.752
ReTrack0.454 / 0.368 / 0.7740.436 / 0.381 / 0.7500.380 / 0.385 / 0.645
SISS0.450 / 0.218 / 0.6930.452 / 0.234 / 0.7070.452 / 0.243 / 0.709
FU0.210 / 0.209 / 0.6060.358 / 0.349 / 0.6690.252 / 0.232 / 0.628

Across all four methods, Recovery Accessibility remains positively associated with subsequent reappearance at every evaluated radius, including after controlling for Immediate SSCD.

Q1-b. Other factors associated with forgetting persistence

To examine other factors that may be associated with forgetting persistence, we considered Immediate SSCD, Pre-deletion SSCD, and Forgetting Magnitude as additional candidate measures. Pre-deletion SSCD is measured under the model immediately before the target's deletion. Forgetting Magnitude is defined as Pre-deletion SSCD minus Immediate SSCD.

For each measure, we calculate its Spearman correlation with the maximum future SSCD over the next 30 deletion stages, its partial Spearman correlation controlling for Immediate SSCD, and its AUROC for reappearance within that window.

Each cell reports Spearman ρ / partial Spearman ρ / AUROC.

Table A6.

MethodImmediate SSCDPre-deletion SSCDForgetting MagnitudeRecovery Accessibility
EraseDiff0.032 / – / 0.5010.123 / 0.121 / 0.6220.047 / 0.080 / 0.5830.528 / 0.529 / 0.740
ReTrack0.511 / – / 0.8070.073 / 0.009 / 0.507-0.464 / -0.016 / 0.2170.436 / 0.381 / 0.750
SISS0.650 / – / 0.7300.125 / 0.002 / 0.692-0.611 / -0.063 / 0.2980.452 / 0.234 / 0.707
FU0.435 / – / 0.6510.239 / 0.213 / 0.602-0.303 / 0.187 / 0.4010.358 / 0.349 / 0.669

Partial correlation is undefined for Immediate SSCD itself. Negative correlations and AUROC below 0.5 for Forgetting Magnitude indicate the direction in which larger immediate reductions in similarity are associated with lower subsequent reappearance risk.

Immediate SSCD is positively associated with maximum future SSCD for most methods. Immediate SSCD is informative for SISS, ReTrack, and FU, whereas its association is weak for EraseDiff.

The associations of Pre-deletion SSCD and Forgetting Magnitude generally weaken after controlling for Immediate SSCD, suggesting overlap with immediate residual similarity.

However, further reducing Immediate SSCD does not necessarily improve forgetting persistence. To examine this, we increase the SISS training budget to approximately match LASTING. The single-run results from Table 10 use the same deletion order and training seed 42 at K=100, 200, and 300.

Table A7. (From Table 10)

MethodImmediate SSCD (K=100) ↓Final Reapp. (K=100, %) ↓Immediate SSCD (K=200) ↓Final Reapp. (K=200, %) ↓Immediate SSCD (K=300) ↓Final Reapp. (K=300, %) ↓
SISS0.33664.000.30245.500.28079.73
Increased-budget SISS0.116111.000.085129.500.066431.00
LASTING0.28071.000.25772.500.22812.33

Increased-budget SISS achieves the lowest Immediate SSCD but the highest Final Reappearance at every reported K. Thus, stronger immediate forgetting alone does not ensure persistence under subsequent deletion updates, highlighting the importance of considering Recovery Accessibility.

Thank you for the detailed responses and additional analyses.

The new experiments clarify the relationship between Recovery Accessibility and sequential reappearance, and the additional computational and generation-quality evaluations also provide a clearer picture of the trade-offs. I appreciate these clarifications.

However, I still concern about the methodological novelty: LASTING still seems to be an incremental improvement to existing unlearning objectives.

Overall, your rebuttal has strengthened the empirical evidence and helped me better understand the generation-quality evaluations. Nevertheless, my original assessment of the paper's contribution remains unchanged, and I will maintain my rating of 3 (Weak Acceptance).

AuthorsOpenReview ↗

Thank You for Your Time and Consideration

Thank you very much for carefully reviewing our responses and additional analyses. We sincerely appreciate your thoughtful engagement and understand that methodological novelty remains your main concern.

Our contribution is to turn the Recovery Accessibility finding into a concrete rule for selecting the forget-side update target. To further examine this principle, we extended the original SISS neighbor-selection ablation to FU by additionally evaluating random and high-loss neighbors.

The table combines these new FU ablations with previously reported results (Tables 3, 5, and 7) in the fixed-retain CelebA-HQ setting. Bold indicates the best value within each objective.

Table A8. Extended neighbor-selection ablation (K=200, %)

Forget-side update targetSISS Final Success ↑SISS Final Reapp. ↓FU Final Success ↑FU Final Reapp. ↓
Original target94.505.5065.5034.17
Random Neighbor75.5023.7461.5038.19
High-Loss Neighbor54.0037.2045.5051.85
Low-Loss Neighbor (LASTING, Ours)97.502.5095.504.50

Low-loss selection improves persistence under both objectives. High-loss selection raises reappearance rates to 6.8× for SISS and 1.5× for FU relative to original-target unlearning. Random perturbations also worsen persistence. These results strengthen our methodological contribution of turning the Recovery Accessibility finding into a target-selection principle that improves forgetting persistence across different unlearning objectives.

We hope this clarification and the results presented in our responses to the other reviewers help inform your assessment of our contribution and its novelty. These include generation quality on non-deleted prompts and model extension (Reviewer LuJ4), as well as SSCD threshold analyses (Reviewer MkPw). Thank you again for your thoughtful consideration. Please let us know if further clarification would be helpful.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

Comment:

We thank reviewers p2dg, MkPw, and LuJ4 for their insightful and constructive feedback. We appreciate their recognition of the importance of persistent forgetting under sequential deletion requests. Below, we summarize how our responses address the main concerns, including the follow-up analyses. Detailed responses are provided under each review.

  • Contribution and novelty (Reviewers p2dg, LuJ4). We clarify that our contribution is to translate the Recovery Accessibility finding into a concrete rule for selecting the forget-side update target. We search for locally recoverable, low-denoising-loss representations and direct unlearning toward them. We also extend the original neighbor-selection ablation under SISS to FU, supporting the importance of this selection rule across different objectives.

  • Generation quality and model extension (Reviewers p2dg, LuJ4). We evaluate general generation quality and text alignment using 10,000 COCO captions per model version (SD v1.4 and SD v2.1). Our follow-up response reports three-seed SD v1.4 results at K=10, 20, and 30, jointly evaluating forgetting performance and generation quality. For SD2.1, we also evaluate sequential unlearning on 30 memorization prompt–target pairs across multiple SSCD thresholds. Together with CelebA-HQ retain-image reconstruction results, these evaluations assess forgetting persistence alongside generation ability on non-deleted prompts.

  • SSCD evaluation (Reviewer MkPw). We clarify the interpretation of SSCD and the threshold choice following prior work. To address the limitations of a single threshold, we report Final and Ever Reappearance across multiple thresholds and inspect discrepancies between visual judgments and SSCD scores. Individual SSCD values in Figure 5 provide further context for the qualitative examples.

  • Recovery Accessibility (Reviewer p2dg). We examine sensitivity to the measurement radius and compare Recovery Accessibility with Immediate SSCD, Pre-deletion SSCD, and Forgetting Magnitude. The increased-budget SISS comparison further examines whether stronger immediate forgetting alone ensures persistence.

  • Sequential deletion and later re-exposure (Reviewer LuJ4). We report results for sequential deletion without advance knowledge of future requests at K=100, 200, and 300. We also examine reintroduction of previously forgotten targets into retain batches in a ten-target exploratory setting with matched controls.

We sincerely appreciate the reviewers' time and thoughtful consideration. Please let us know if further clarification or supporting information would be helpful.

Best, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.