- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies machine unlearning under retain-forget entanglement and challenges the common objective of driving accuracy on the forget set toward zero. The central observation is that a model retrained from scratch without the forget set can nevertheless retain high accuracy on those forgotten examples when similar examples remain in the retain set. Therefore, aggressively reducing forget-set accuracy can “overshoot” the retrained reference rather than approximate it.
The paper formalizes this intuition through a total-variation-based separability bound relating forget accuracy and accuracy on the adjacent retain distribution. It then introduces SWIFT, which replaces the unavailable retrained model with two proxy references. The forget-set loss distribution is matched to that of held-out calibration samples from the same distribution, while retention is constrained by a one-sided Wasserstein loss relative to either a lightweight probe trained on retain data or, in some settings, the original model. The method is evaluated on CIFAR-100, VGGFace2, ToxiGen, and SBIC in class-based and instance-based unlearning settings.
Overall, I find the central critique of zero-forget-accuracy objectives useful and timely, and the instance-based formulation provides a meaningful stress test for entangled unlearning.
Critical strengths
-
The paper identifies an important evaluation issue in approximate unlearning. The distinction between suppressing performance on the forget set and matching the behavior of a retrained model is conceptually important. The empirical examples make the issue concrete: for example, the retrained model remains highly accurate on forgotten instances when related retained samples remain.
-
The instance-based entanglement setting is well motivated. Randomly splitting examples from the same subgroup between forget and adjacent-retain sets creates a more challenging setting than removing an entire subclass. Proposition 1 provides a clean population-level explanation of why selectively suppressing one distribution becomes impossible as the forget and adjacent distributions become indistinguishable.
-
The experiments cover several modalities and datasets. The paper includes image classification, face recognition, toxicity classification, and social-bias/offense classification rather than demonstrating the phenomenon on a single benchmark.
-
The paper evaluates against actual retraining. This is particularly valuable because the paper's primary claim concerns deviation from the retrained reference rather than simply minimizing forget accuracy.
-
The paper is reasonably transparent about some limitations. In particular, it explicitly acknowledges that SWIFT can also damage held-out examples from the forget distribution; on VGGFace2 and CIFAR-100, the test-set forget-distribution accuracy remains noticeably below that of retraining.
Critical weaknesses
-
The novelty relative to recent reference-aligned unlearning methods is unclear. The central premise, rather than maximizing the forget loss, align forgotten samples with the behavior of unseen/reference data has recently appeared in closely related work. In particular, RAUL / Multi-Objective Reference-Aligned Machine Unlearning, published in PMLR in May 2026, replaces unbounded forgetting with alignment toward a reference distribution representing unseen data, including an empirical held-out reference set, and reports closeness to retraining. Proceedings of Machine Learning Research ReGUn similarly argues for aligning forget examples to truly unseen examples via a held-out class-conditioned reference. The manuscript cites these works in the appendix, but does not experimentally compare against RAUL. The distinction claimed by the authors evaluating the calibration reference under the current parameters and introducing an additional retain-side reference is potentially meaningful, but I do not currently see sufficient evidence demonstrating that these changes constitute a substantial advance over the existing reference-alignment approach. This is especially important for RAUL because it is a peer-reviewed 2026 publication rather than merely contemporaneous unpublished work.
-
There are serious reproducibility inconsistencies in the submitted manuscript. The main text states that the VGGFace2 experiment uses InceptionResnetV1, whereas Appendix A.6 states that VGGFace2 uses a ResNet-50, and the following page again discusses the VGGFace2 model as ResNet-50. Furthermore, Table 12 contains an unresolved placeholder, “XXX”, for the CIFAR-100 gradient-ascent learning rate. These issues make it difficult to reproduce or fully evaluate the baseline comparisons and should definitely be resolved in the rebuttal/revision.
-
Statistical uncertainty is not reported. The main tables report individual point estimates without means, standard deviations, or confidence intervals across multiple runs. Given that the claimed advantages sometimes involve relatively small deviations from retraining, it is difficult to determine how much of the difference is robust to initialization, data splitting, or unlearning optimization. At minimum, I would like to see repeated runs for SWIFT, retraining, and the strongest relevant baselines on one representative vision and one text dataset.
Questions for authors
See weakness