- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates the reliability of using a single retrained model as the reference for evaluating machine unlearning. Through experiments across multiple datasets, architectures, forgetting settings, and evaluation metrics, the authors identify two sources of instability: retraining variance, where models with nearly identical utility obtain different unlearning scores due to random seeds or training configurations, and metric fragility, where some metrics fluctuate even for a fixed model. In particular, MIA-R and distance-based metrics are shown to be unstable, while MIA-F and accuracy-based metrics are comparatively consistent. The paper therefore recommends treating retraining as a distribution represented by several checkpoints, reporting reference intervals instead of point estimates, and prioritizing robust metrics such as MIA-F. Overall, the work highlights an important weakness in current unlearning evaluation practices and proposes practical guidelines for more reliable benchmarking.
Critical strengths
This paper identifies an important yet overlooked issue in machine unlearning evaluation: existing studies often treat a single retrained model as an absolute reference. The authors demonstrate that retrained checkpoints with nearly identical test accuracy can produce substantially different unlearning scores, potentially undermining the reliability of existing evaluations. Extensive experiments across diverse settings provide solid evidence of this problem and its practical impact. In particular, the detailed analysis of MIA-R’s failure modes offers valuable insights and may encourage future work on developing more reliable evaluation protocols and metrics.
Critical weaknesses
The main weakness of this paper is its relatively limited technical contribution, as it primarily analyzes evaluation instability caused by randomness rather than proposing a new unlearning method or metric. Nevertheless, the identified problem is important and meaningful to the field. In addition, the stability of MIA-F does not necessarily demonstrate its effectiveness: although MIA-F remains close to 0.5 for retrained models, the paper does not show that it can reliably distinguish successful, partial, and failed unlearning. Finally, the recommendation to use three to five retrained models is not supported by a convergence or sample-size analysis. A more detailed study of how many checkpoints are needed to obtain stable reference intervals would strengthen the paper’s practical contribution.
Questions for authors
- Can the authors evaluate whether MIA-F can reliably distinguish successful unlearning, partial unlearning, and complete failure to unlearn, rather than only showing that it is stable on retrained models?
- Can the authors provide a convergence or sample-size analysis to justify why three to five retrained models are sufficient for estimating stable medians, confidence intervals, or interquartile ranges? I would be willing to raise my score if the authors can adequately address these two concerns.