- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper address how to evaluate a single unlearned model against the natural randomness of gold-standard retraining model. It proposed that each deleted sample’s true-label log-odds should fall inside a region induced by the distribution of retraining models. And designed a leave-one-retrain-out evaluation (BLORO), which standardizes the unlearned model’s score against K−1 retrained references and calibrates thresholds using held-out retraining models from other unleanring requests. The author also proposed ReCo to control the distance between the unlearning data sample and its target. The experiments on CIFAR10/100 with ResNet-18 show the effectiveness of ReCo.
Critical strengths
- The motivation itself is reasonable. I agree with that point-reference evaluation is fragile, and distributional evaluation does not describe the single model actually deployed in practical.
- The evaluation hygiene is careful. Configurations are frozen before evaluation and CR/ED are never used for tuning.
- The results in Fig. 3(c) and Table 4 show that similar LiRA AUCs can hide very different mixes of under- and over-forgetting. This is a valuable point for the community.
Critical weaknesses
- The novelty of SRC is limited. SRC is essentially “does a per-sample statistic fall within a prediction interval of the retraining distribution.” This is close to existing per-example evaluations[1].
- BLORO depends heavily on K. In Table 12, ReCo’s CR falls from 0.827 at K=3 to 0.599 at K=20, and ED at K=3 is about 10× larger. At K=5, the setting used in the main tables, CR is 0.706, versus 0.599 with the full reference set. CR/ED at K=5 therefore do not reflect the population quantity well. The method ranking looks stable, but absolute values are not comparable across papers that use different K, which weakens BLORO as a standard metric.
- For a fair comparison, the baselines (e.g., FT, GA, SCRUB, SalUn) should be given the same nonmember-derived information, for example per-sample gating or early stopping when samples reach the ReCo targets, to separate the potential value of the target itself from the value of the controller. Table 4 ablates inside ReCo but does not address this.
- ReCo suffers from over-forgetting issue. The proxy-target error, an MAE of about 4.2 log-odds (Table 13), also looks large. How does it compare with the typical retraining standard deviation per sample? Without that comparison, Fig. 3(a)’s claim that targets “typically lie near” retraining behavior is hard to judge. Also, over-forgetting is a privacy exposure signal.
- Computational cost of ReCo are not reported. Repeatedly evaluating the full forget set, plus backtracking line search, may be costly.
- Experimental scope is narrow for both datasets and architectures.
- The controller has many components. A sensitivity analysis is encouraged.
[1] Hayes, Jamie, et al. "Inexact unlearning needs more careful evaluations to avoid a false sense of privacy." 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 2025.
Questions for authors
How CR changes along the continued-training trajectory? Would baselines also converge toward compatibility with longer training?