acceptodds
Under review as a conference paper at ICLR 2027

Data Unlearning via Inverse Distillation

Abstract

Multi-step matching models, including flow and diffusion models, produce high-quality outputs but incur substantial inference costs and may reproduce unwanted components of their training datasets. We introduce Inverse Distillation Unlearning (IDU), a unified framework that simultaneously distills a teacher multi-step matching model into an efficient one-step student generator and suppresses outputs corresponding to a designated training subset. We first formulate distillation as a min-max objective over a data distribution and then represent this distribution as a mixture of the forget-set and the generated distributions. This allows us to compare this mixture with the teacher's training distribution and recover only the retained data at the optimum. Our method requires only a pretrained full-data teacher and data from the forget set, without access to retained training examples, extra feature extractors or classifiers. Extensive experiments on MNIST and CIFAR-10 datasets under flow-matching and score-based diffusion settings demonstrate that IDU substantially reduces the generation frequency of forgotten classes while preserving generation quality on the retained classes. To the best of our knowledge, IDU is the first unified framework for simultaneous unlearning and distillation in unconditional flow-matching and score-based models.

Primary area:
generative models
Keywords:
Machine Unlearning, Diffusion models, Flow Matching, Acceleration of diffusion/flow models, Distillation of diffusion/flow models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_smu1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Inverse Distillation Unlearning (IDU), a framework that combines data unlearning and distillation into a one-step generative model. Starting from a pretrained diffusion or flow-matching teacher, the method trains a one-step generator to forget a selected subset of training data without requiring access to the retained samples. This is achieved by matching a mixture of forgotten and generated samples to the teacher's original data distribution, encouraging the generator to reproduce only the retained data. The authors evaluate IDU on MNIST and CIFAR-10 using flow-matching and score-based diffusion models with good results.

Critical strengths
  • Elegant and intuitive idea: Combining the forget set with generated samples to recover the retained distribution is an interesting approach.

  • No retained data required during unlearning: This is a clear benefit of the method, although there is a concern regarding model selection that needs to be addressed (see weaknesses).

  • Promising experimental results: The method substantially reduces the generation of forgotten classes while largely preserving generation quality. It also works with both flow-matching and score-based diffusion models, showing flexibility across different generative models.

Critical weaknesses
  • Scope and scalability

The experiments are limited to MNIST and CIFAR-10, two simple, low-resolution image datasets. Given the generality claimed by the authors, I expect at least one experiment on a larger, higher-resolution dataset and a more realistic architecture, such as a latent diffusion model or a diffusion transformer (DiT). Testing another modality, such as audio, would also help support the generality of the approach. I would not describe experiments on only two datasets as "extensive." Can the authors provide a new dataset and a new (transformer based) architecture?

  • Theoretical guarantees

Theorem 1 states that the retained distribution is recovered when the mixture coefficient equals the true forget-set proportion (ρ=π\rho=\pi). However, the FM experiments use ρ=0.4\rho=0.4 and ρ=0.6\rho=0.6 when forgetting two classes, despite the true proportion being approx. 0.2. Is there a theoretical interpretation for these choices? It is unclear whether the guarantee still applies when ρ≠π\rho\ne\pi.

Additionally, the theoretical guarantee is derived for the original inverse-distillation objective, but the SiD experiments use a modified generator loss. Could the authors clarify whether the guarantee still holds for this modified objective? If not, the distinction between theoretical and empirical forgetting should be made explicit.

  • Data unlearning

The authors claim to address general data unlearning, but all experiments involve forgetting entire classes. I would like to see experiments on randomly selected subsets of samples across different classes. In particular, can the method forget individual images while preserving visually similar training examples? This is not demonstrated in the current experiments, and the claims should be softened if such experiments cannot be provided.

  • Experimental setting

A main advantage of IDU is that retained data are not required during optimization. However, the authors still use retained-data for hyperparameter and checkpoint selection. I would like to see an experiment where retained data are also unavailable for validation, to show how the method performs in this setting.

Checkpoint selection seems particularly important, since the authors report that forgetting can be unstable during training. Could they provide training curves for FGR and FID, including the later training stages, rather than only reporting selected checkpoints?

  • Baselines

Some comparisons with existing unlearning methods, rely on results reported in previous papers rather than controlled experiments. Differences in architectures, initialization, and evaluation protocols make these comparisons difficult to interpret. I understand that some implementations may not be available, but additional controlled baselines are needed. For example, SISS followed by ordinary distillation would be an interesting comparison to test whether simultaneous unlearning and distillation is actually beneficial.

  • Evaluation metrics

FID is a global metric and may hide degradation in individual retained classes. Table 5 already shows that larger values of ρ\rho can suppress additional MNIST digits (2, 5, 8, and 9). Could the authors report per-class generation frequencies and quality metrics for the retained classes?

Reducing forgotten-class generation frequency (FGR) does not necessarily demonstrate data unlearning. FGR measures output suppression but does not establish that the influence of individual forgotten training samples has been removed. A generator could avoid certain classes while still retaining memorized information about the forgotten samples. I would like to see stronger sample-level forgetting evaluations, such as memorization or nearest-neighbor analyses, or potentially membership inference attacks adapted to one-step generators.

  • Computational efficiency

The method is presented as efficient, but there is no detailed comparison of unlearning time, GPU cost, or training overhead. IDU requires a pretrained teacher, a trainable fake model, and a generator. While one-step inference is clearly faster, it would be useful to understand the overall computational cost and trade-off compared with existing unlearning methods.

  • Additional concerns

The authors suggest that larger architectures might alleviate the observed forgetting instability. Can they provide any experimental evidence supporting this hypothesis? (This might need to be done in future iterations of the manuscript if the time does not allow it now)

IDU relies on a frozen teacher trained on the complete dataset. Could the authors discuss whether information from the forget set might still be transferred to the student, and what implications this has for the unlearning guarantees?

  • Minor comments

Please clarify whether the reported results are averaged over five independent training runs or five evaluations of the same trained model.

In Appendix B.9, the retrained FM baseline fails to generate digit 2, substantially increasing its Retain FID. Could the authors investigate and clarify whether this is an optimization issue and whether additional tuning or independent runs improve the baseline? Since retraining is the main reference for success here, it is important that this baseline is properly optimized.

Questions for authors

Overall, the idea is interesting and promising, but I find the experimental evaluation insufficient to fully support the paper's claims, particularly regarding general data unlearning, scalability, and comparisons with existing methods. Please tackle the experimental requests and explanations given in the weaknesses section.

6 comments
AuthorsOpenReview ↗

Reply Rebuttal Comment by Author

We thank the reviewer for the detailed review and address the concerns below.

W1. Scope and scalability. We evaluated IDU on FFHQ at 64×6464\times64 resolution [1], using the 3,465 images labeled dark in FFHQ-Aging as the forget set [2]. The results below use the classifier and FGR correction described in the General Reply.

MetricVP-EDM teacherIDU (ρ=0.05\rho=0.05)IDU (ρ=0.10\rho=0.10)IDU (ρ=0.15\rho=0.15)
Retain FID ↓—1.691.741.97
Observed FGR (%) ↓4.470.980.410.30
Classifier-adjusted FGR (%) ↓4.260.030.000.00

At ρ=0.05\rho=0.05, IDU achieved an observed FGR of 0.98% and a Retain FID of 1.69. The classifier-adjusted FGR was 0.03%, compared with 4.26% for the teacher. The adjusted estimates at ρ=0.10\rho=0.10 and ρ=0.15\rho=0.15 were clipped to zero because the observed rates fell below the classifier's estimated false-positive rate.

To test a forget set smaller than 1%, we also fine-tuned the FFHQ-64 SiD student on 642 images with an annotated absolute yaw of at least 50∘50^\circ (0.917% of FFHQ). At ρ=0.01\rho=0.01, observed FGR decreased from 0.72% to 0.30%. The corresponding classifier-adjusted estimates were 0.63% and 0.00%, with the latter obtained by clipping. The General Reply describes the calibration protocol.

We also ran a preliminary IDU experiment on CelebA at 64×6464\times64 using a pretrained U-ViT-S/4 pixel-space diffusion transformer [12]. The student had yet to converge by the end of the rebuttal period. Early checkpoints showed a substantial reduction in the generation frequency of the target attribute group.

W2. Theoretical guarantees. In our paper, we give convergence guarantees only for the theoretically correct value ρ=π\rho = \pi. This value gives decent practical metrics and trade-off, especially for small forget datasets (see FFHQ experiments above). We empirically found that our IDU allows us to increase ρ\rho to enhance forgetting. This effect is especially notable for large forget portions and weak FM architecture you are referring. In other words, other ρ\rho values were grid-searched to get the best forget–retain trade-off, beyond the pretty good baseline ρ=π\rho = \pi. We highlighted this distinction in the revision.

In the case of SiD, SiD itself is not theoretically justified to converge to the target distribution, even with pure distillation without unlearning. Thus, we cannot formally guarantee anything for it, but in practice its loss modification achieves superior results, and our IDU is fully compatible with it.

W3. Data unlearning. To address this issue, we run two close-to-practice setups where we delete individual images or much smaller forget datasets from the training data. Note that we erase the entire classes in the original paper only for easier and faster evaluation. We never use explicit class labels in our IDU anyway, since it can handle any subset of the training data as we demonstrate below.

We unlearn small red-cars subclass from the unconditional CIFAR-10 dataset, which constitutes around 2% of the total data.

CIFAR-10 - forget red cars uncond (init FGR - 1.98%, init FID - 1.94)

ρ\rho0.050.01
IDU's Retain FID ↓2.122.10
IDU's Observed FGR (%) ↓0.561.67

For the small forget sets studied here, setting ρ\rho close to the forget-set proportion π\pi provided a practical starting point with small FID and 3x forgetting. Further grid search might reveal even better configurations in terms of the retain-forget trade-off.

AuthorsOpenReview ↗

Reply Rebuttal Comment by Author

W4. Retain data.

When retained data are unavailable for validation, we recommend starting with ρ≈π\rho\approx\pi. A modest increase can strengthen suppression, with a possible cost in Retain FID, as observed on FFHQ-64. For the EDM-based SiD setting, the forgetting rate and Retain FID convergence of our IDU is stable over the training horizon; thus, we recommend taking the final EMA checkpoint from a run with a prespecified training budget. In our flow-matching experiments, late-stage increases in FGR can occur, primarily due to the limited FM model capacity, not our IDU design (see W8 reply).

In our experiments, the correct portion gives decent results, and slightly higher values, used with caution, can further improve forgetting rates. These are basically the results of our SiD CIFAR-10 run with ρ=0.2\rho = 0.2, we provide the metrics convergence curves in Appendix D.2.

When only image classifier is available for validation, we propose Filtered Teacher FID as a proxy for Retain FID. It uses a synthetic reference built from teacher generations classified as belonging to retained classes.

The table compares this synthetic reference with the real retained-data reference on CIFAR-10, with classes 1 and 9 designated for forgetting.

MethodFiltered Teacher FID ↓Retain FID ↓
Retrain teacher1.172.26
Retrain distillation2.252.82
IDU (ρ=0.20\rho=0.20)2.553.15
ReTrack + SiD3.454.00
SISS + SiD4.455.26
MSGM + SiD7.347.80

The two metrics give broadly consistent model rankings in this comparison. This suggests that Filtered Teacher FID, together with FGR, can serve as a practical validation proxy when only image classifier is available. See Appendix D.1 of the revised manuscript for the Filtered Teacher FID evaluation protocol.

W5. Baselines We added SiD distillation of the unlearned multi-step models obtained after EraseDiff, SISS, MSGM, and ReTrack procedures as baselines. Unfortunately, UOT-Unlearn code for unlearning is not available for reproduction and comparison. The results are provided below

CIFAR-10 - forget 1,9 classes uncond (init FGR - 20%)

methodEraseDiff distilledSISS distilledMSGM distilledReTrack distilledOur IDU
Retain FID ↓Diverged5.267.804.003.15
Observed FGR (%) ↓Diverged0.520.471.132.36

MNIST - forget 3,7 classes uncond (init FGR - 20%)

methodEraseDiff dist.SISS dist.MSGM dist.ReTrack dist.Retrained teacher dist.Our IDU
Retain FID ↓Diverged1.472.301.231.201.65
Observed FGR (%) ↓Diverged0.500.760.680.000.95

We found that unlearned models cause extra instability in distillation, as they change generation trajectories so much that they stop matching the trajectories for which SiD is designed (see the general reply for details). In contrast, our IDU ensures stable compatibility with SiD design, achieves superior retain quality and does not require retained data for training with comparable compute (see W7 reply).

AuthorsOpenReview ↗

Reply Rebuttal Comment by Author

W6. Evaluation metrics. We report per-class FID and generation frequencies for unconditional CIFAR-10 generation in the SiD setting. The forget classes are class 1 (automobile) and class 9 (truck).

Per-class FID (lower is better). For each retained class, we used all 5,000 ground-truth CIFAR-10 training images from that class as the reference and collected 10,000 generated images assigned to it by the top-1 prediction of a ResNet-56 classifier.

The original submission reported Retain FID against the full retained set. FID estimates are sensitive to sample size, and each class provides only 5,000 reference images. We therefore use Retain FID computed against the larger, pooled reference set as our primary quality metric and report per-class FID here to show how quality varies across retained classes.

Retained CIFAR-10 classOriginal EDM teacherStandard SiD studentRetrained teacherDistilled retrained teacherIDU
0 - Airplane8.579.938.819.6212.91
2 - Bird10.359.689.979.2411.45
3 - Cat11.0210.9210.8310.6113.15
4 - Deer7.577.527.267.898.07
5 - Dog11.4110.5410.7010.1512.19
6 - Frog8.848.639.048.359.25
7 - Horse7.417.307.177.968.24
8 - Ship6.386.796.376.768.20

The per-class comparison between IDU and the distilled retraining baseline follows the same overall direction as pooled Retain FID, with gaps that vary across retained classes.

Per-class generation frequencies (%). Each frequency is the percentage of all generated images assigned to that class by the classifier. FGR is the sum of the frequencies of classes 1 and 9 before rounding, calculated in one seed.

ρ\rhoClass 0Class 1Class 2Class 3Class 4Class 5Class 6Class 7Class 8Class 9FGR
0.0511.246.9710.209.6810.5210.0411.8911.1011.367.0013.97
0.1011.074.2010.4310.4811.2911.8311.9912.2211.724.778.96
0.2012.961.1010.3811.3212.2412.8711.5113.6912.631.292.39
0.4012.290.6911.8812.1613.3413.1713.1412.4110.320.591.29
0.6011.521.1111.8712.1111.8814.1213.9912.709.910.781.89
0.8012.660.7713.6111.4112.8614.1312.7613.257.980.561.33

Increasing ρ\rho from 0.05 to 0.20 reduced FGR from 13.97% to 2.39%. The lowest observed FGR was 1.29% at ρ=0.40\rho=0.40. Across the tested values from ρ=0.05\rho=0.05 to 0.600.60, each retained class accounted for approximately 10% to 14% of retained-class generations. At ρ=0.80\rho=0.80, the relative frequency of class 8 (ship) fell to 8.1% among retained-class generations. This shift occurred at the largest tested weight, well above the theoretical choice ρ=π≈0.20\rho=\pi\approx0.20, and illustrates a possible retention cost of such a large forget-set weight.

W7. Computational efficiency. We compare three CIFAR-10 pipelines that start from the same pretrained teacher and produce a one-step generator with unlearning. Our primary comparison is with ReTrack + SiD, the sequential baseline closest to IDU in both Retain FID and FGR. The teacher-unlearning checkpoints correspond to approximately 20,000 updates for ReTrack and 12,000 for SISS, both at a global batch size of 512.

In our SiD and IDU runs, each training iteration includes one update of the auxiliary (fake) score network and one update of the generator. The generator first synthesizes a batch of 512 images for the fake score network update and then synthesizes a fresh batch of 512 images for its own update. The training loop increments the image counter by 512 once per iteration. Thus, the logged counts of approximately 50M, 49M, and 31M correspond to the number of images generated for each update phase separately. Across both phases, the generator produces approximately 100M, 98M, and 62M training images, respectively.

PipelineTeacher unlearning (M images, one batch counted per update)SiD or IDU (M generated images per update phase)Total logged training count (M images)Retain FID ↓FGR (%) ↓
IDU—~50~503.152.36
ReTrack + SiD~10.2~49~594.001.13
SISS + SiD~6.1~31~375.260.52

IDU and ReTrack + SiD use comparable training volumes under this counting convention. IDU achieves lower Retain FID, while ReTrack + SiD achieves lower FGR. SISS + SiD reaches its reported checkpoint at a smaller training count and achieves the lowest FGR of the three, with a higher Retain FID. Per-update computation and accelerator time can differ across pipelines.

Reviewer_sVs6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Inverse Distillation Unlearning (IDU), which distills a multi-step generative model into a one-step generator while removing a forget set. The fake model is trained on a mixture of forget and generated data. The generator is penalized for reproducing forget samples. The generator provably recovers the retained distribution. Training needs only the teacher and the forget samples. The paper shows experiments on MNIST and CIFAR-10 forget one or two classes.

Critical strengths

1. Simple and principled recipe

IDU models the fake model’s data as a mixture of forget and generated data. This gives a clear optimality result (Theorem 1) and avoids the conflicting forget/retain losses of many existing unlearning methods. It requires only one additional forward pass through the fake model and introduces essentially one new hyperparameter, ρ.

2. Practical and general

Training uses only the frozen teacher and the forget samples, with no retained data, classifier, or feature extractor. The method works with different matching teachers (flow matching and score-based diffusion) and does not require a specific generator architecture. It produces an efficient one-step generator and can also fine-tune an already distilled one.

Critical weaknesses

1. Lack of comparison with baselines

The main weakness of the paper is that the method is barely compared to prior work. The only comparisons are with UOT-Unlearn (in a setting that is not equivalent) and SFD (single class, IDU is worse on both Retain FID and FGR). I encourage the authors to compare against same-setup baselines like:

  • unlearning the teacher with SISS, EraseDiff (or another method discussed in the related work and Section 5), then distilling it with SiD
  • the reverse order: distilling into a one-step generator first, then unlearning it

I understand that extensive experiments may not be feasible during the rebuttal period and do not consider them mandatory, but they would strengthen the work.

2. Experiments are limited to the class cond. forgetting and unconditional generation

All models in the experiments are unconditional, and the forget sets are always entire classes. The paper never examines what happens when a model is explicitly asked to generate content it was meant to forget. That would be closer to real-world use. I understand that the recipe does not need class or concept labels during training. But how does IDU perform when applied to a conditional teacher? What does the resulting one-step generator produce when prompted with a forgotten class?

3. Generalization to small forget sets

In all experiments, the forget set is between 10 and 20% of the training data. The fake model sees the entire forget distribution. The paper does not study more realistic cases like:

  • the training set is web-scale, so the forget set is a tiny fraction of it
  • the forget set contains only a few samples (images of a specific person)
  • the forget set is a subset of a class (forgetting blue trucks while keeping red trucks) In particular, how should ρ be chosen when π is very small?
Questions for authors
  1. How does IDU perform when applied to a conditional teacher? What does the generator produce when prompted with a forgotten concept?
  2. How does IDU compare to same-setup baselines or to (a) unlearning the teacher then distilling it with SiD, or (b) distilling first and then unlearning the one-step generator?
  3. How does IDU behave when the forget set is a very small fraction of the training data or contains only a few samples?
  4. How should ρ be chosen in practice without access to retained data, given that ρ and checkpoints are currently selected using Retain FID?
3 comments
AuthorsOpenReview ↗

Reply Rebuttal Comment by Author

We thank the reviewer for the feedback and are glad that the idea of unlearning via data mixture in the fake model was found interesting. We address the questions below.

W1 and Q2. Baseline comparison. We added both required baseline types.

(a) First, SiD distillation of the unlearned multi-step models obtained after EraseDiff [8], SISS [9], MSGM [10], and ReTrack [11] procedures. The results are provided below.

CIFAR-10 - forget 1,9 classes uncond (init FGR - 20%)

methodEraseDiff distilledSISS distilledMSGM distilledReTrack distilledOur IDU
Retain FID ↓Diverged5.267.804.003.15
Observed FGR (%) ↓Diverged0.520.471.132.36

MNIST - forget 3,7 classes uncond (init FGR - 20%)

methodEraseDiff dist.SISS dist.MSGM dist.ReTrack dist.Retrained teacher dist.Our IDU
Retain FID ↓Diverged1.472.301.231.201.65
Observed FGR (%) ↓Diverged0.500.760.680.000.95

We found that unlearned models cause extra instability in distillation, as they change generation trajectories so much that they stop matching the trajectories for which SiD is designed (see the general reply for details). In contrast, our IDU ensures stable compatibility with SiD and does not require retained data for training.

(b) Then, we unlearn already distilled generator via our IDU, i.e., a fine-tuning of a pre-trained generator. Unfortunately, UOT-Unlearn code for unlearning is not available for reproduction and comparison. We provide the improved results of the fine-tune experiment from the original Appendix B.7 (Section 4 in revision):

CIFAR-10 - forget 1,9 classes uncond (init FGR - 20%)

methodFT IDUIDU from scratch
Retain FID2.943.15
FGR (%)4.792.36

W2 and Q1. Class forgetting only. Indeed, in the original paper, we ran the unconditional methods mostly on class forgetting tasks. This was done only for easier and faster evaluation, and we never use explicit class labels in our IDU anyway.

Our IDU is not appropriate for full class forgetting in the conditional setup. If we condition the pre-trained generator on a forget label, it will not forget anything, since the mixture of generated and forget data will consist only of forget samples, and the fake model will learn a forget-class distribution identical to the teacher's. Thus, all generator gradient updates in SiD will be zero.

But, our conditioned IDU is perfectly suited to scenarios where one needs to forget some of the outputs generated under the same given prompt. To demonstrate this, we run our IDU to forget a data portion within a given class from the pre-trained generator, e.g., red cars in conditional CIFAR-10 (20% inside the automobile class).

CIFAR-10 - forget red cars cond (init overall FGR - 1.98%, init overall FID - 1.71)

ρ\rho0.20
IDU's overall Retain FID ↓2.17
IDU's overall FGR (%) ↓0.05

IDU effectively identifies the target subclass and erases it while leaving the other classes intact, yielding the best FID and the lowest overall FGR across the experiments.

W3 and Q3. Smaller forget datasets. We tested our IDU on (a) a forget set containing only a few samples and (b) subclass forget scenarios. In the web-scale variant, the model is usually conditioned on a prompt, and the conditioned set is much smaller and closer to the subclass case.

We run IDU to forget red cars in CIFAR-10 in the unconditional fine-tune setup (around 2%):

CIFAR-10 - forget red cars uncond (init FGR - 1.98%, init FID - 1.94)

ρ\rho0.050.01
IDU's Retain FID ↓2.122.10
IDU's Observed FGR (%) ↓0.561.67

For the small forget sets studied here, setting ρ\rho close to the forget-set proportion π\pi provided a practical starting point with small FID and 4x forgetting. Further grid search might reveal even better configurations in terms of the retain-forget trade-off.

To test a forget set smaller than 1%, we also fine-tuned the FFHQ-64 SiD student on 642 images with an annotated absolute yaw of at least 50∘50^\circ (0.917% of FFHQ). At ρ=0.01\rho=0.01, observed FGR decreased from 0.72% to 0.30%. The corresponding classifier-adjusted estimates were 0.63% and 0.00%, with the latter obtained by clipping. The General Reply describes the calibration protocol.

AuthorsOpenReview ↗

Reply Rebuttal Comment by Author

Q4. Choice of ρ\rho. When retained data are unavailable for validation, we recommend starting with ρ\rho close to the estimated forget-set proportion π\pi. A modest increase of ρ\rho can strengthen suppression at a possible cost in Retain FID. In our experiments, the correct portion gives decent results, especially in cases of small forget datasets. For example, on FFHQ-64, increasing ρ\rho from 0.05 to 0.15 lowered the observed FGR from 0.98% to 0.30% and increased Retain FID from 1.69 to 1.97. This illustrates the trade-off when increasing the forget-set weight above π\pi.

For the EDM-based SiD setting, the forgetting rate and retain FID convergence of our IDU is stable over the training horizon; thus, we recommend taking the final EMA checkpoint from a run with a prespecified training budget. In our flow-matching experiments, late-stage increases in FGR can occur, primarily due to the weak FM architecture, not a drawback of our IDU. The TorchCFM U-Nets used in those experiments have 25.8 million parameters on MNIST and 35.7 million on CIFAR-10, compared with approximately 55.7 million per network for the corresponding EDM DDPM++ backbones. Limited model capacity is one plausible contributor.

For the demonstration, we provide training metric curves of our SiD CIFAR-10 run with the theoretical ρ=0.2\rho = 0.2 in Appendix D.2.

We grid-searched over values of ρ\rho using metric evaluations to find the best possible retain--forget balance and study the method's behaviour. When real retained images are unavailable, we propose using Filtered Teacher FID alongside FGR for validation. We describe the synthetic reference and compare it with Retain FID in our response to W4.

I thank the authors for the detailed rebuttal and the additional experiments.

The two-stage baselines address my first concern. They show a trade-off rather than a clear improvement: IDU obtains the best Retain FID but higher FGR than SISS, MSGM, and ReTrack on CIFAR, and it is outperformed on both metrics by SISS and ReTrack on MNIST. I think the paper should present this honestly and emphasize its actual advantages (single stage, no retained data).

The red-car and FFHQ experiments are welcome, but they do not fully cover realistic settings such as text-to-image models. I encourage the authors to test this setting.

I am raising my score.

Reviewer_hmweOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a method called Data Unlearning via Inverse Distillation, called the Inverse Distillation Unlearning (IDU) framework. This simultaneously distils a slow multi-step generative model into a fast single-step gen. while erasing a forget set. IDU only relies on a teacher model and specific samples to be forgotten.

Critical strengths

Strengths:

  1. The proposed objective has no gradient conflicts.
  2. It does not rely on retain data
  3. It accomplishes two things at the same time: speeding up by reducing the number of generation steps and forgetting through unlearning.
Critical weaknesses

Weakness:

  1. The only datasets used are toy datasets (MNIST and CIFAR). Challenging situations arise in high-resolution and large datasets such as CELEBA-HQ and FFHQ.
  2. The paper is motivated by small subset removal, but the experiment focuses more on class unlearning. While class labels are not given, erasing a large portion of the dataset is a different problem than erasing a small subset. Have the authors experimented on a small subset?
  3. While the theoretical value of rho is the proportion of the dataset to forget, for MNIST and CIFAR, the numbers need a higher rho. If the rho is set too high, results become unstable. Could you elaborate on that?
Questions for authors

See above

1 comment
AuthorsOpenReview ↗

Reply Rebuttal Comment by Authors

We thank the reviewer for the careful reading and answer the questions below.

W1. High-resolution experiments. We evaluated IDU on FFHQ at 64×6464\times64 resolution [1]. The forget set comprised 3,465 images labeled dark in the FFHQ-Aging glasses annotations [2], approximately 4.95% of the dataset. We report both observed and classifier-adjusted FGR using the evaluation protocol described in the General Reply.

MetricVP-EDM teacherIDU (ρ=0.05\rho=0.05)IDU (ρ=0.10\rho=0.10)IDU (ρ=0.15\rho=0.15)
Retain FID ↓—1.691.741.97
Observed FGR (%) ↓4.470.980.410.30
Classifier-adjusted FGR (%) ↓4.260.030.000.00

At ρ=0.05\rho=0.05, the observed FGR was 0.98%, compared with 4.47% for the teacher. The corresponding classifier-adjusted estimates were 0.03% and 4.26%. Increasing ρ\rho from 0.05 to 0.15 reduced the observed FGR from 0.98% to 0.30% and increased Retain FID from 1.69 to 1.97. The two adjusted estimates of 0.00% result from clipping below-zero estimates to zero.

W2. Smaller forget sets. We further reduce the forget set size in our experiments by running close-to-practice setup: unlearn a small subclass instead of the whole class from the pre-trained generator, and erase specific images from the model.

We unlearn the red-cars subclass from the unconditional CIFAR-10 dataset, which constitutes around 2% of the total data.

CIFAR-10 - forget red cars uncond (init FGR - 1.98%, init FID - 1.94)

ρ\rho0.050.01
IDU's Retain FID ↓2.122.10
IDU's Observed FGR (%) ↓0.561.67

For the small forget sets studied here, setting ρ\rho close to the forget-set proportion π\pi provided a practical starting point with small FID and 3x forgetting. Further grid search might reveal even better configurations in terms of the retain-forget trade-off.

To test a forget set smaller than 1%, we also fine-tuned the FFHQ-64 SiD student on 642 images with an annotated absolute yaw of at least 50∘50^\circ (0.917% of FFHQ). At ρ=0.01\rho=0.01, observed FGR decreased from 0.72% to 0.30%. The corresponding classifier-adjusted estimates were 0.63% and 0.00%, with the latter obtained by clipping. The General Reply describes the calibration protocol.

W3. ρ\rho effect. Indeed, the theoretically correct portion ρ≈π\rho \approx \pi gives decent metrics and trade-off, especially for small forget datasets (see these exps in the General Reply). However, the theory sometimes might set insufficient weight for the forget loss, resulting in forgetting rates that are a bit far from the best ones (see Table~3 in the submission). In practice, our IDU allows us to increase the forget-loss scaling factor to strengthen forgetting of the target images, at the low cost of the retain quality.

On the other hand, an excessively large forget-loss scaling factor first prevents effective propagation of the generator model gradients. Second, it does not allow us to reproduce the teacher data mixture, as the forget data portion occupies too much probability mass meant for other samples.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General reply

We thank all three reviewers for their careful and constructive feedback. We answer the shared points here. During the rebuttal, all new experiments were run with one-seed evaluation to make it in time. For the next iteration, we will publish full-size results, but for now, we report the results available at the time of writing.

We are providing a revised manuscript. All additions to the text based on our rebuttal are marked in blue.

  • sVs6 (W1, Q2) and smu1 (W5) — Added comparisons with EraseDiff, SISS, MSGM, and ReTrack followed by SiD, along with examples of how teacher outputs change before distillation (Section 4, Table 2, and Appendix E).
  • sVs6 (W1, Q2) — Updated the CIFAR-10 fine-tuning results using the converged SiD checkpoint and moved the comparison into the main text (Section 4, Table 3).
  • hmwe (W1–W3), sVs6 (W2–W3, Q1, Q3–Q4), and smu1 (W1–W3) — Added FFHQ-64 experiments on dark sunglasses and large head rotations, including classifier-adjusted FGR, and unconditional and class-conditional experiments on the CIFAR-10 red-car subclass (Section 4 and Appendix C).
  • sVs6 (Q4) and smu1 (W4) — Introduced Filtered Teacher FID as a validation proxy when retained data are unavailable and compared it with Retain FID (Appendix D.1).
  • sVs6 (Q4) and smu1 (W4, W8) — Added CIFAR-10 SiD training curves and discussed late-stage instability in flow matching (Appendix D.2).
  • smu1 (W9) — Clarified that reported means and standard deviations are computed from five evaluations of the same trained checkpoint using distinct sampling seeds (Section 4 and Tables 1 and 3).

High-resolution experiments were added in Appendix C in the revision.

We have launched additional joint distillation and unlearning runs at 64×6464\times64 for the FFHQ settings described above and a separate CelebA experiment using a pretrained U-ViT-S/4 teacher. These runs had yet to converge at the time of writing. We will report the completed results in the final version.

Large head rotations on FFHQ-64. To test IDU on a forget set smaller than 1% of the training data, we selected all FFHQ images with an absolute head yaw of at least 50∘50^\circ according to the public FFHQ-Aging annotations [2], which were obtained with Face++ [6]. This yielded 642 of the 70,000 images (0.917%). We fine-tuned the public one-step SiD checkpoint (α=1.0\alpha=1.0) [3] at ρ=0.01\rho=0.01.

We measured FGR on generated images using a pretrained SemiUHPE head-pose estimator [7]. We split the labeled real FFHQ-64 images into calibration and holdout halves within each of the forget and retain groups. On the calibration half, we selected the largest predicted-yaw threshold that achieved at least 59% recall. At the resulting threshold of 47.09∘47.09^\circ, holdout recall was 58.57% and the false-positive rate was 0.352%. Because false positives matter at this low prevalence, we applied the correction described under References and details, assuming that the holdout error rates also apply to generated images.

Modelρ\rhoReported FID ↓Observed FGR (%) ↓Classifier-adjusted FGR (%) ↓
Initial one-step SiD student (α=1.0\alpha=1.0)—1.710.720.63
IDU fine-tuning0.011.710.300.00

Fine-tuning reduced observed FGR from 0.72% to 0.30%. The adjusted estimate was clipped to zero because the observed rate fell below the estimated false-positive rate.

We also launched a separate joint distillation and unlearning run on the same subset at ρ=0.01\rho=0.01. Across the checkpoints evaluated so far, observed FGR has remained near 0.30%, and the adjusted estimate has remained at 0.00% after clipping. Retain FID has yet to stabilize in this ongoing run.

AuthorsOpenReview ↗

General reply

Baseline comparison. We applied EraseDiff [8], SISS [9], MSGM [10], and ReTrack [11] to the same pretrained unconditional teacher used by IDU, and then distilled the unlearned models via standard distillation SiD algorithm. We also included the distilled student of the teacher trained from scratch on the retained data. The UOT-Unlearn code is not publicly available.

We found that unlearned teachers are less stable in distillation, unlike our proposed IDU approach. Empirical findings suggest that unlearned teachers result in unpredictable generation trajectories, different from the known paths of the teacher, refitted to the dataset without the forget data. EraseDiff corrupts the forget data and semantically similar pictures to pure noise; MSGM mitigates the effect for unrelated images and corrupts them partially, whereas ReTrack extremely blurs the forget samples and does not affect others. The closest method to retraining is SISS, which basically minimizes the same objective (see the Appendix E in the revision for examples). Since SiD and other distillation methods are built on top of the known retrained teacher trajectories, unlearned teachers' paths do not fit them and may cause divergence: we observed mode collapse and severely noisy distilled generations in heuristic SiD. We report Retain FID and FGR for checkpoints where we managed to find stable hyperparameters.

CIFAR-10 - forget 1,9 classes uncond (init FGR - 20%)

methodEraseDiff distilledSISS distilledMSGM distilledReTrack distilledOur IDU
Retain FID ↓Diverged5.267.804.003.15
Observed FGR (%) ↓Diverged0.520.471.132.36

MNIST - forget 3,7 classes uncond (init FGR - 20%)

methodEraseDiff dist.SISS dist.MSGM dist.ReTrack dist.Retrained teacher dist.Our IDU
Retain FID ↓Diverged1.472.301.231.201.65
Observed FGR (%) ↓Diverged0.500.760.680.000.95

On CIFAR-10, a more visually diverse benchmark of natural images, IDU achieved lower Retain FID (3.15) than all three converged sequential unlearning baselines. Distillation increased Retain FID from 2.61 to 5.26 for SISS and from 3.45 to 7.80 for MSGM. All three converged sequential baselines achieved lower FGR than IDU (0.47–1.13% versus 2.36%). On MNIST, a simpler benchmark of handwritten digits, distilled ReTrack achieved lower Retain FID and FGR than IDU. Distillation after EraseDiff diverged on both datasets.

This comparison was included in Section 4 in the revision.

Smaller forget data sizes. We verified our IDU on the 5%, 2% and 1% forget data portions, and it still effectively reduces generation frequency as for large portions from the original paper.

The experiments on the 5% and 1% portions were conducted on the FFHQ dataset and are reported in the first paragraph. For the 2% portion, we separated all red cars from the CIFAR-10 dataset (around 2%), trained a new classifier for evaluation (macro recall and precision ≈\approx 96%), and ran our IDU in fine-tune mode only on these samples. The results are in the table below.

CIFAR-10 - forget red cars uncond (init FGR - 1.98%, init FID - 1.94)

ρ\rho0.050.01
IDU's Retain FID ↓2.122.10
IDU's Observed FGR (%) ↓0.561.67

With the most straightforward and quick search strategy for the ρ\rho coefficient, we achieved 3x forgetting with a small FID on the retain data. Further grid search might reveal even better configurations in terms of the retain-forget trade-off.

These experiments were described in Appendix C in the revision.

References and details

Correction for classification errors. Let FF denote the observed fraction of generated images classified as having the target attribute, and let PP denote the underlying fraction. If the classifier's recall rr and false-positive rate ff measured on labeled FFHQ-64 images also apply to generated images, then

F=Pr+(1−P)f.F = Pr + (1-P)f.

For the classifier, we measured r=0.83492r=0.83492 and f=0.0095142f=0.0095142 on labeled FFHQ-64 images. We used these rates to estimate the adjusted FGR as P^=clip⁡[0,1]((F−f)/(r−f))\widehat{P}=\operatorname{clip}_{[0,1]}((F-f)/(r-f)) and report the result as a percentage.

AuthorsOpenReview ↗

General reply

[1] Tero Karras, Samuli Laine, and Timo Aila. A Style-Based Generator Architecture for Generative Adversarial Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. FFHQ dataset.

[2] Roy Or-El, Soumyadip Sengupta, Ohad Fried, Eli Shechtman, and Ira Kemelmacher-Shlizerman. Lifespan Age Transformation Synthesis. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 739–755, 2020. FFHQ-Aging annotations.

[3] Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang. Score Identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation. In Proceedings of the 41st International Conference on Machine Learning (ICML), PMLR 235, pp. 62307–62331, 2024. One-step SiD checkpoint for FFHQ-64 (α=1.0\alpha=1.0).

[4] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the Design Space of Diffusion-Based Generative Models. In Advances in Neural Information Processing Systems 35 (NeurIPS), 2022. Unconditional VP-EDM teacher checkpoint for FFHQ-64.

[5] Mantas Birškus. Glasses Detector, v1.0.0, 2024. Sunglasses classifier weights (ShuffleNetV2 ×1.0).

[6] Face++ Cognitive Services. Face Analyze API documentation, head-pose attributes.

[7] Huayi Zhou, Fei Jiang, Jin Yuan, Yong Rui, Hongtao Lu, and Kui Jia. Semi-Supervised Unconstrained Head Pose Estimation in the Wild. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(3), pp. 3082–3099, 2026. Pretrained SemiUHPE weights.

[8] Jing Wu, Trung Le, Munawar Hayat, and Mehrtash Harandi. Erasing Undesirable Influence in Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 28263–28273, 2025.

[9] Silas Alberti, Kenan Hasanaliyev, Manav Shah, and Stefano Ermon. Data Unlearning in Diffusion Models. In International Conference on Learning Representations (ICLR), 2025.

[10] Wan Jiang, He Wang, Xin Zhang, Dan Guo, Zhaoxin Fan, Yunfeng Diao, and Richang Hong. Moderating the Generalization of Score-Based Generative Model. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 360–369, 2025.

[11] Qitan Shi, Cheng Jin, Jiawei Zhang, and Yuantao Gu. ReTrack: Data Unlearning in Diffusion Models Through Redirecting the Denoising Trajectory. In Proceedings of the 29th International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 300, pp. 2818–2826, 2026.

[12] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All Are Worth Words: A ViT Backbone for Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22669–22679, 2023.

[13] Zemin Huang, Weijian Luo, Zhengyang Geng, and Guojun Qi. One-Step Diffusion and Flow Distillation Through Implicit Generator Matching. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(8), pp. 9156–9167, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.