- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Inverse Distillation Unlearning (IDU), a framework that combines data unlearning and distillation into a one-step generative model. Starting from a pretrained diffusion or flow-matching teacher, the method trains a one-step generator to forget a selected subset of training data without requiring access to the retained samples. This is achieved by matching a mixture of forgotten and generated samples to the teacher's original data distribution, encouraging the generator to reproduce only the retained data. The authors evaluate IDU on MNIST and CIFAR-10 using flow-matching and score-based diffusion models with good results.
Critical strengths
-
Elegant and intuitive idea: Combining the forget set with generated samples to recover the retained distribution is an interesting approach.
-
No retained data required during unlearning: This is a clear benefit of the method, although there is a concern regarding model selection that needs to be addressed (see weaknesses).
-
Promising experimental results: The method substantially reduces the generation of forgotten classes while largely preserving generation quality. It also works with both flow-matching and score-based diffusion models, showing flexibility across different generative models.
Critical weaknesses
- Scope and scalability
The experiments are limited to MNIST and CIFAR-10, two simple, low-resolution image datasets. Given the generality claimed by the authors, I expect at least one experiment on a larger, higher-resolution dataset and a more realistic architecture, such as a latent diffusion model or a diffusion transformer (DiT). Testing another modality, such as audio, would also help support the generality of the approach. I would not describe experiments on only two datasets as "extensive." Can the authors provide a new dataset and a new (transformer based) architecture?
- Theoretical guarantees
Theorem 1 states that the retained distribution is recovered when the mixture coefficient equals the true forget-set proportion (). However, the FM experiments use and when forgetting two classes, despite the true proportion being approx. 0.2. Is there a theoretical interpretation for these choices? It is unclear whether the guarantee still applies when .
Additionally, the theoretical guarantee is derived for the original inverse-distillation objective, but the SiD experiments use a modified generator loss. Could the authors clarify whether the guarantee still holds for this modified objective? If not, the distinction between theoretical and empirical forgetting should be made explicit.
- Data unlearning
The authors claim to address general data unlearning, but all experiments involve forgetting entire classes. I would like to see experiments on randomly selected subsets of samples across different classes. In particular, can the method forget individual images while preserving visually similar training examples? This is not demonstrated in the current experiments, and the claims should be softened if such experiments cannot be provided.
- Experimental setting
A main advantage of IDU is that retained data are not required during optimization. However, the authors still use retained-data for hyperparameter and checkpoint selection. I would like to see an experiment where retained data are also unavailable for validation, to show how the method performs in this setting.
Checkpoint selection seems particularly important, since the authors report that forgetting can be unstable during training. Could they provide training curves for FGR and FID, including the later training stages, rather than only reporting selected checkpoints?
- Baselines
Some comparisons with existing unlearning methods, rely on results reported in previous papers rather than controlled experiments. Differences in architectures, initialization, and evaluation protocols make these comparisons difficult to interpret. I understand that some implementations may not be available, but additional controlled baselines are needed. For example, SISS followed by ordinary distillation would be an interesting comparison to test whether simultaneous unlearning and distillation is actually beneficial.
- Evaluation metrics
FID is a global metric and may hide degradation in individual retained classes. Table 5 already shows that larger values of can suppress additional MNIST digits (2, 5, 8, and 9). Could the authors report per-class generation frequencies and quality metrics for the retained classes?
Reducing forgotten-class generation frequency (FGR) does not necessarily demonstrate data unlearning. FGR measures output suppression but does not establish that the influence of individual forgotten training samples has been removed. A generator could avoid certain classes while still retaining memorized information about the forgotten samples. I would like to see stronger sample-level forgetting evaluations, such as memorization or nearest-neighbor analyses, or potentially membership inference attacks adapted to one-step generators.
- Computational efficiency
The method is presented as efficient, but there is no detailed comparison of unlearning time, GPU cost, or training overhead. IDU requires a pretrained teacher, a trainable fake model, and a generator. While one-step inference is clearly faster, it would be useful to understand the overall computational cost and trade-off compared with existing unlearning methods.
- Additional concerns
The authors suggest that larger architectures might alleviate the observed forgetting instability. Can they provide any experimental evidence supporting this hypothesis? (This might need to be done in future iterations of the manuscript if the time does not allow it now)
IDU relies on a frozen teacher trained on the complete dataset. Could the authors discuss whether information from the forget set might still be transferred to the student, and what implications this has for the unlearning guarantees?
- Minor comments
Please clarify whether the reported results are averaged over five independent training runs or five evaluations of the same trained model.
In Appendix B.9, the retrained FM baseline fails to generate digit 2, substantially increasing its Retain FID. Could the authors investigate and clarify whether this is an optimization issue and whether additional tuning or independent runs improve the baseline? Since retraining is the main reference for success here, it is important that this baseline is properly optimized.
Questions for authors
Overall, the idea is interesting and promising, but I find the experimental evaluation insufficient to fully support the paper's claims, particularly regarding general data unlearning, scalability, and comparisons with existing methods. Please tackle the experimental requests and explanations given in the weaknesses section.
6 comments
Reply Rebuttal Comment by Author
W4. Retain data.
When retained data are unavailable for validation, we recommend starting with . A modest increase can strengthen suppression, with a possible cost in Retain FID, as observed on FFHQ-64. For the EDM-based SiD setting, the forgetting rate and Retain FID convergence of our IDU is stable over the training horizon; thus, we recommend taking the final EMA checkpoint from a run with a prespecified training budget. In our flow-matching experiments, late-stage increases in FGR can occur, primarily due to the limited FM model capacity, not our IDU design (see W8 reply).
In our experiments, the correct portion gives decent results, and slightly higher values, used with caution, can further improve forgetting rates. These are basically the results of our SiD CIFAR-10 run with , we provide the metrics convergence curves in Appendix D.2.
When only image classifier is available for validation, we propose Filtered Teacher FID as a proxy for Retain FID. It uses a synthetic reference built from teacher generations classified as belonging to retained classes.
The table compares this synthetic reference with the real retained-data reference on CIFAR-10, with classes 1 and 9 designated for forgetting.
| Method | Filtered Teacher FID ↓ | Retain FID ↓ |
|---|---|---|
| Retrain teacher | 1.17 | 2.26 |
| Retrain distillation | 2.25 | 2.82 |
| IDU () | 2.55 | 3.15 |
| ReTrack + SiD | 3.45 | 4.00 |
| SISS + SiD | 4.45 | 5.26 |
| MSGM + SiD | 7.34 | 7.80 |
The two metrics give broadly consistent model rankings in this comparison. This suggests that Filtered Teacher FID, together with FGR, can serve as a practical validation proxy when only image classifier is available. See Appendix D.1 of the revised manuscript for the Filtered Teacher FID evaluation protocol.
W5. Baselines We added SiD distillation of the unlearned multi-step models obtained after EraseDiff, SISS, MSGM, and ReTrack procedures as baselines. Unfortunately, UOT-Unlearn code for unlearning is not available for reproduction and comparison. The results are provided below
CIFAR-10 - forget 1,9 classes uncond (init FGR - 20%)
| method | EraseDiff distilled | SISS distilled | MSGM distilled | ReTrack distilled | Our IDU |
|---|---|---|---|---|---|
| Retain FID ↓ | Diverged | 5.26 | 7.80 | 4.00 | 3.15 |
| Observed FGR (%) ↓ | Diverged | 0.52 | 0.47 | 1.13 | 2.36 |
MNIST - forget 3,7 classes uncond (init FGR - 20%)
| method | EraseDiff dist. | SISS dist. | MSGM dist. | ReTrack dist. | Retrained teacher dist. | Our IDU |
|---|---|---|---|---|---|---|
| Retain FID ↓ | Diverged | 1.47 | 2.30 | 1.23 | 1.20 | 1.65 |
| Observed FGR (%) ↓ | Diverged | 0.50 | 0.76 | 0.68 | 0.00 | 0.95 |
We found that unlearned models cause extra instability in distillation, as they change generation trajectories so much that they stop matching the trajectories for which SiD is designed (see the general reply for details). In contrast, our IDU ensures stable compatibility with SiD design, achieves superior retain quality and does not require retained data for training with comparable compute (see W7 reply).
Reply Rebuttal Comment by Author
W6. Evaluation metrics. We report per-class FID and generation frequencies for unconditional CIFAR-10 generation in the SiD setting. The forget classes are class 1 (automobile) and class 9 (truck).
Per-class FID (lower is better). For each retained class, we used all 5,000 ground-truth CIFAR-10 training images from that class as the reference and collected 10,000 generated images assigned to it by the top-1 prediction of a ResNet-56 classifier.
The original submission reported Retain FID against the full retained set. FID estimates are sensitive to sample size, and each class provides only 5,000 reference images. We therefore use Retain FID computed against the larger, pooled reference set as our primary quality metric and report per-class FID here to show how quality varies across retained classes.
| Retained CIFAR-10 class | Original EDM teacher | Standard SiD student | Retrained teacher | Distilled retrained teacher | IDU |
|---|---|---|---|---|---|
| 0 - Airplane | 8.57 | 9.93 | 8.81 | 9.62 | 12.91 |
| 2 - Bird | 10.35 | 9.68 | 9.97 | 9.24 | 11.45 |
| 3 - Cat | 11.02 | 10.92 | 10.83 | 10.61 | 13.15 |
| 4 - Deer | 7.57 | 7.52 | 7.26 | 7.89 | 8.07 |
| 5 - Dog | 11.41 | 10.54 | 10.70 | 10.15 | 12.19 |
| 6 - Frog | 8.84 | 8.63 | 9.04 | 8.35 | 9.25 |
| 7 - Horse | 7.41 | 7.30 | 7.17 | 7.96 | 8.24 |
| 8 - Ship | 6.38 | 6.79 | 6.37 | 6.76 | 8.20 |
The per-class comparison between IDU and the distilled retraining baseline follows the same overall direction as pooled Retain FID, with gaps that vary across retained classes.
Per-class generation frequencies (%). Each frequency is the percentage of all generated images assigned to that class by the classifier. FGR is the sum of the frequencies of classes 1 and 9 before rounding, calculated in one seed.
| Class 0 | Class 1 | Class 2 | Class 3 | Class 4 | Class 5 | Class 6 | Class 7 | Class 8 | Class 9 | FGR | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.05 | 11.24 | 6.97 | 10.20 | 9.68 | 10.52 | 10.04 | 11.89 | 11.10 | 11.36 | 7.00 | 13.97 |
| 0.10 | 11.07 | 4.20 | 10.43 | 10.48 | 11.29 | 11.83 | 11.99 | 12.22 | 11.72 | 4.77 | 8.96 |
| 0.20 | 12.96 | 1.10 | 10.38 | 11.32 | 12.24 | 12.87 | 11.51 | 13.69 | 12.63 | 1.29 | 2.39 |
| 0.40 | 12.29 | 0.69 | 11.88 | 12.16 | 13.34 | 13.17 | 13.14 | 12.41 | 10.32 | 0.59 | 1.29 |
| 0.60 | 11.52 | 1.11 | 11.87 | 12.11 | 11.88 | 14.12 | 13.99 | 12.70 | 9.91 | 0.78 | 1.89 |
| 0.80 | 12.66 | 0.77 | 13.61 | 11.41 | 12.86 | 14.13 | 12.76 | 13.25 | 7.98 | 0.56 | 1.33 |
Increasing from 0.05 to 0.20 reduced FGR from 13.97% to 2.39%. The lowest observed FGR was 1.29% at . Across the tested values from to , each retained class accounted for approximately 10% to 14% of retained-class generations. At , the relative frequency of class 8 (ship) fell to 8.1% among retained-class generations. This shift occurred at the largest tested weight, well above the theoretical choice , and illustrates a possible retention cost of such a large forget-set weight.
W7. Computational efficiency. We compare three CIFAR-10 pipelines that start from the same pretrained teacher and produce a one-step generator with unlearning. Our primary comparison is with ReTrack + SiD, the sequential baseline closest to IDU in both Retain FID and FGR. The teacher-unlearning checkpoints correspond to approximately 20,000 updates for ReTrack and 12,000 for SISS, both at a global batch size of 512.
In our SiD and IDU runs, each training iteration includes one update of the auxiliary (fake) score network and one update of the generator. The generator first synthesizes a batch of 512 images for the fake score network update and then synthesizes a fresh batch of 512 images for its own update. The training loop increments the image counter by 512 once per iteration. Thus, the logged counts of approximately 50M, 49M, and 31M correspond to the number of images generated for each update phase separately. Across both phases, the generator produces approximately 100M, 98M, and 62M training images, respectively.
| Pipeline | Teacher unlearning (M images, one batch counted per update) | SiD or IDU (M generated images per update phase) | Total logged training count (M images) | Retain FID ↓ | FGR (%) ↓ |
|---|---|---|---|---|---|
| IDU | — | ~50 | ~50 | 3.15 | 2.36 |
| ReTrack + SiD | ~10.2 | ~49 | ~59 | 4.00 | 1.13 |
| SISS + SiD | ~6.1 | ~31 | ~37 | 5.26 | 0.52 |
IDU and ReTrack + SiD use comparable training volumes under this counting convention. IDU achieves lower Retain FID, while ReTrack + SiD achieves lower FGR. SISS + SiD reaches its reported checkpoint at a smaller training count and achieves the lowest FGR of the three, with a higher Retain FID. Per-update computation and accelerator time can differ across pipelines.
Reply Rebuttal Comment by Author
We thank the reviewer for the detailed review and address the concerns below.
W1. Scope and scalability. We evaluated IDU on FFHQ at 64×64 resolution [1], using the 3,465 images labeled
darkin FFHQ-Aging as the forget set [2]. The results below use the classifier and FGR correction described in the General Reply.At ρ=0.05, IDU achieved an observed FGR of 0.98% and a Retain FID of 1.69. The classifier-adjusted FGR was 0.03%, compared with 4.26% for the teacher. The adjusted estimates at ρ=0.10 and ρ=0.15 were clipped to zero because the observed rates fell below the classifier's estimated false-positive rate.
To test a forget set smaller than 1%, we also fine-tuned the FFHQ-64 SiD student on 642 images with an annotated absolute yaw of at least 50∘ (0.917% of FFHQ). At ρ=0.01, observed FGR decreased from 0.72% to 0.30%. The corresponding classifier-adjusted estimates were 0.63% and 0.00%, with the latter obtained by clipping. The General Reply describes the calibration protocol.
We also ran a preliminary IDU experiment on CelebA at 64×64 using a pretrained U-ViT-S/4 pixel-space diffusion transformer [12]. The student had yet to converge by the end of the rebuttal period. Early checkpoints showed a substantial reduction in the generation frequency of the target attribute group.
W2. Theoretical guarantees. In our paper, we give convergence guarantees only for the theoretically correct value ρ=π. This value gives decent practical metrics and trade-off, especially for small forget datasets (see FFHQ experiments above). We empirically found that our IDU allows us to increase ρ to enhance forgetting. This effect is especially notable for large forget portions and weak FM architecture you are referring. In other words, other ρ values were grid-searched to get the best forget–retain trade-off, beyond the pretty good baseline ρ=π. We highlighted this distinction in the revision.
In the case of SiD, SiD itself is not theoretically justified to converge to the target distribution, even with pure distillation without unlearning. Thus, we cannot formally guarantee anything for it, but in practice its loss modification achieves superior results, and our IDU is fully compatible with it.
W3. Data unlearning. To address this issue, we run two close-to-practice setups where we delete individual images or much smaller forget datasets from the training data. Note that we erase the entire classes in the original paper only for easier and faster evaluation. We never use explicit class labels in our IDU anyway, since it can handle any subset of the training data as we demonstrate below.
We unlearn small red-cars subclass from the unconditional CIFAR-10 dataset, which constitutes around 2% of the total data.
CIFAR-10 - forget red cars uncond (init FGR - 1.98%, init FID - 1.94)
For the small forget sets studied here, setting ρ close to the forget-set proportion π provided a practical starting point with small FID and 3x forgetting. Further grid search might reveal even better configurations in terms of the retain-forget trade-off.