acceptodds
Withdrawn from ICLR 2027

Samplewise Retraining Compatibility: Outcome-Level Evaluation and Guidance for Machine Unlearning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Approximate machine unlearning is commonly evaluated by fidelity to exact retraining. Existing evaluations either compare a realized unlearned model against a single retraining reference or compare repeated unlearning and retraining outcomes distributionally. The former reduces stochastic retraining to a single reference realization, while the latter characterizes repeated outcomes rather than the single model produced for an individual deletion request. This leaves an outcome-level question: how should a realized unlearned model be evaluated against retraining's natural variability? We formalize samplewise retraining compatibility (SRC) to evaluate whether each deleted sample's realized behavior falls within the range naturally produced by stochastic retraining. To operationalize SRC with finite retraining references, we introduce BLORO, which calibrates unlearning deviation against retraining's own variability. To pursue SRC without retraining references, we propose ReCo, which estimates sample-specific proxy targets from difficulty-conditioned nonmember behavior and adaptively controls forgetting toward them. Experiments show that BLORO remains stable with finite retraining references, ReCo achieves strong SRC while remaining competitive on conventional set-level unlearning metrics, and models obtained with ReCo better preserve their post-unlearning behavior on deleted samples during subsequent training on retained data. Together, these results support SRC as a principle for evaluating and guiding approximate unlearning.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, Unlearning Evaluation, Samplewise Retraining Compatibility, Exact Retraining, Stochastic Retraining, Adaptive Unlearning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_owY5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper address how to evaluate a single unlearned model against the natural randomness of gold-standard retraining model. It proposed that each deleted sample’s true-label log-odds should fall inside a region induced by the distribution of retraining models. And designed a leave-one-retrain-out evaluation (BLORO), which standardizes the unlearned model’s score against K−1 retrained references and calibrates thresholds using held-out retraining models from other unleanring requests. The author also proposed ReCo to control the distance between the unlearning data sample and its target. The experiments on CIFAR10/100 with ResNet-18 show the effectiveness of ReCo.

Critical strengths
  1. The motivation itself is reasonable. I agree with that point-reference evaluation is fragile, and distributional evaluation does not describe the single model actually deployed in practical.
  2. The evaluation hygiene is careful. Configurations are frozen before evaluation and CR/ED are never used for tuning.
  3. The results in Fig. 3(c) and Table 4 show that similar LiRA AUCs can hide very different mixes of under- and over-forgetting. This is a valuable point for the community.
Critical weaknesses
  1. The novelty of SRC is limited. SRC is essentially “does a per-sample statistic fall within a prediction interval of the retraining distribution.” This is close to existing per-example evaluations[1].
  2. BLORO depends heavily on K. In Table 12, ReCo’s CR falls from 0.827 at K=3 to 0.599 at K=20, and ED at K=3 is about 10× larger. At K=5, the setting used in the main tables, CR is 0.706, versus 0.599 with the full reference set. CR/ED at K=5 therefore do not reflect the population quantity well. The method ranking looks stable, but absolute values are not comparable across papers that use different K, which weakens BLORO as a standard metric.
  3. For a fair comparison, the baselines (e.g., FT, GA, SCRUB, SalUn) should be given the same nonmember-derived information, for example per-sample gating or early stopping when samples reach the ReCo targets, to separate the potential value of the target itself from the value of the controller. Table 4 ablates inside ReCo but does not address this.
  4. ReCo suffers from over-forgetting issue. The proxy-target error, an MAE of about 4.2 log-odds (Table 13), also looks large. How does it compare with the typical retraining standard deviation per sample? Without that comparison, Fig. 3(a)’s claim that targets “typically lie near” retraining behavior is hard to judge. Also, over-forgetting is a privacy exposure signal.
  5. Computational cost of ReCo are not reported. Repeatedly evaluating the full forget set, plus backtracking line search, may be costly.
  6. Experimental scope is narrow for both datasets and architectures.
  7. The controller has many components. A sensitivity analysis is encouraged.

[1] Hayes, Jamie, et al. "Inexact unlearning needs more careful evaluations to avoid a false sense of privacy." 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 2025.

Questions for authors

How CR changes along the continued-training trajectory? Would baselines also converge toward compatibility with longer training?

Reviewer_4pw1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SRC, a framework for evaluating machine unlearning by checking whether the behavior of individual forgotten samples is within the variability of the performance in exact retraining runs. The paper also proposes BLORO, an evaluation procedure that accounts for this variability using a limited number of retrained models. Finally, it also introduces ReCo, an unlearning method that estimates sample-specific forgetting targets from held-out nonmember data and adaptively adjusts the forgetting process. The results show improved samplewise compatibility while maintaining competitive performance on standard unlearning metrics. The authors also investigate whether these improvements persist after further training on retained data.

Critical strengths
  • The idea that different deleted samples may require different degrees of forgetting is interesting and could be useful for future work.

  • Evaluating unlearning against the natural variability of retraining, rather than a single retrained model, is also an interesting and enw approach.

  • The paper is well written, and the proposed framework is clearly explained.

  • The authors include several relevant unlearning baselines, strong membership inference attacks (LiRA and RMIA), and useful ablation studies.

Critical weaknesses
  • Experimental evaluation

The experiments are limited to CIFAR-10 and CIFAR-100 using ResNet-18. I find this too limited to show that the method generalizes to more complex settings. I would like to see experiments with larger and modern models, at least ViTs (Base) or other transformer-based architectures, and more challenging datasets such as Tiny-ImageNet or higher-resolution datasets.

Additionally, the authors only evaluate forgetting 100 randomly selected samples. I would expect experiments with different forgetting ratios (e.g., 2%, 10%, 50%), as well as full-class forgetting. This is particularly important given that the method relies on assumptions about the relationship between nonmembers and forgotten samples, which may not hold in more challenging settings.

The paper presents SRC and ReCo as general approaches to machine unlearning, but all experiments are limited to image classification. I think this limitation should be clearly stated in the abstract, intro and conclusion. Otherwise, I request experiments on another modality, such as text, to support the claims.

  • Over-forgetting and one-sided optimization

ReCo seems to over-forget quite often, even though one of the motivations behind SRC is to distinguish under-forgetting from over-forgetting. I understand that ReCo uses one-sided control, but why was this design chosen? Could the authors provide an ablation comparing one-sided and two-sided optimization to see whether explicitly enforcing both bounds improves compatibility?

  • BLORO stability

The abstract states that BLORO remains stable with a limited number of retraining references. However, Table 12 shows that both CR and ED change substantially as the number of retraining references increases. Can the authors clarify this and discuss what it means for comparing different unlearning methods?

  • Proxy-target assumption

I understand that ReCo needs proxy targets because exact retraining is too expensive. However, I am not convinced that the underlying assumption holds in general. Just because two images have similar difficulty does not mean they would receive similar predictions after retraining. For example, a forgotten sample might still be classified with 95% confidence after retraining because it resembles other training examples, while another sample with similar difficulty might receive much lower confidence.

I appreciate the comparison with actual retraining results, but I struggle to understand how well this proxy assumption generalizes to more challenging settings, such as class-wise forgetting or highly memorized samples. I would like to see a clearer explanation of why this assumption should hold, together with experiments validating it in more complex settings.

  • Usefulness of the proposed compatibility metrics

I am not convinced that the proposed compatibility metrics provide a meaningful measure of successful unlearning.

The original model, which has not undergone any unlearning, already achieves almost the same CR as ReCo (from Table 9). What does SRC actually add beyond existing privacy metrics? I would like the authors to include the original model in the main compatibility tables and discuss this limitation. At the moment, I am not convinced that CR provides enough additional information to justify its use as an unlearning evaluation metric.

Another concern is that SRC relies only on a single sample-level statistic, the true-label log-odds. Matching this statistic does not necessarily mean that a sample has been successfully forgotten. I think evaluating the full output probability distributions would provide a more complete picture. Additionally, matching the marginal retraining behavior of individual samples is not the same as matching the joint distribution of retraining outcomes. I would like the authors to discuss these limitations.

  • Test accuracy degradation

ReCo generally achieves lower test accuracy than several baselines. Can the authors explain why this happens and whether it could be improved, for example by adding a retained-data loss term?

  • Computational efficiency

The paper is missing a computational efficiency analysis. ReCo requires additional labeled nonmember data, multiple augmented views, repeated evaluations of the complete deletion set, and an adaptive backtracking procedure.

Efficiency is one of the main motivations for approximate unlearning, given that unlearning requests can come often, so I think this is an important missing experiment. The authors should report wall-clock runtime comparisons against the baselines, together with an analysis of the additional memory and computational costs.

  • Missing baselines

FedOSD: The method is evaluated without its native recovery stage. I understand the motivation for comparing unlearning-stage endpoints, but I would also like to see a comparison with the complete FedOSD pipeline.

FaLW: This is mentioned as a closely related sample-aware unlearning approach, but no experimental comparison is provided. I would like to see it included as a baseline.

ReGUn: Reference-Guided Machine Unlearning (Mirlach, J., ICLR Workshop 2026) is another relevant missing baseline and related work. It already proposes guiding forgotten samples toward nonmember behavior using held-out data, which is very similar to the motivation behind ReCo. I would like the authors to discuss these differences and provide a direct comparison to show the additional benefit of ReCo's sample-specific targets and controller.

  • Minor comments

The GDPR reference should be corrected. The bibliography lists "Protection Regulation" as the author of a 2018 Intouch publication. The authors should cite the actual EU regulation instead.

Figure 3 is difficult to read because the text is too small.

Table 2 reports extremely large ED values for EWCSGA. Although the authors explain this, I would like to understand why it happens. Is it a numerical issue, unsuitable hyperparameters, or a genuine failure of the method? This also raises some concerns about the reliability of the implementation for the rest of the metrics.

The experiments use only two independent runs. I would like to see at least three seeds in all results to better assess the robustness of the method.

Questions for authors

I think the ideas and intuition are interesting and novel, but the experimental evaluation is not good enough to show that it all works as the authors claim. Please see the points in the weaknesses and address them or provide an explanation.

Reviewer_4WivOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies an issue with regards to unlearning and retraining models wherein exact matches with retrained models ignore the stochastic variations introduced due to several factors and introduce confidence intervals based on log-odds for multiple retrained models and asserts that an unlearned model falling within the confidence intervals for the unlearned sample (SRC) is a better calibration. It introduces BLORO as a leave-one-retrain-out procedure that measures an unlearned model's deviation against K−1 references and calibrates thresholds using held-out retrained models from deletion requests, measuring Compatibility Rate (CR) and Excess Deviation (ED). When retraining isn't an option, the work proposes RECO that offload the threshold estimation for confidence intervals to non-members culminating in an erasure objective that applies to samples outside the interval. Evaluations over CIFAR-10/100 with ResNet-18, are conducted in standard and federated setups with sample-specific erasure as the objective.

Critical strengths
  1. The work identifies a critical evaluation view that hasn't been addressed thus far. Retraining on its own introduces stochasticity so comparisons of unlearned models must factor that into their evaluations. Figure 3 clarifies this further and comparing against one or few retraining runs could be rebutted with the observations of this work.
  2. Retraining stochasticity motivates new metrics for evaluation which this work proposes and could be useful in general since I think the unlearning research community hasn't settled on them yet and it isn't very easy to measure unlearning.
  3. The work also proposes an informed unlearning method that factors in these variations and measures against the proposed metrics.
Critical weaknesses
  1. The work doesn't mention the original model's (all samples trained, no unlearning or retraining) CR as a baseline. This makes it hard to interpret the numbers in the main table and others. It could be the case that the proposed metrics could inherit some limitations due to the very problem that the authors identify. For example, if the CR of the original model is high on the unlearned samples, some of the deleted samples already belonged in the bandwidth? This also prevents the reader from interpreting the available bandwidth. I am reserved about the utility of the evaluation protocol presented due to this limitation.
  2. The method apparently depends heavily on the number of retrains available and the CR is sensitive to this as shown in Table 12. The authors should explain this phenomenon in context with the other methods mentioned in the table.
  3. I think that the work proposes an evaluation metric that motivates its unlearning method and it is therefore bound to outperform the methods it compares against, which may be unfair. Without a baseline that gets the same 960 labeled non-members and a per-sample matching objective, it isn't possible to tell whether ReCo adds anything.
  4. The authors should report the compute consumed in comparison to the other methods to allow for a performance/compute trade-off evaluation.
Questions for authors
  1. If retrains are available in your evaluation, why not do ReCo with retrain interval also (train on K, evaluate on held-out retrains) as an upper bound? This could isolate proxy-target error from controller error in ReCo's CR.
  2. If retraining randomness is fixed (same init, same data order, only the deleted samples removed), how big is the per-sample retrain variability?
  3. Could you also test class-wise and larger populations of sample-wise deletions? Several other works that are referenced do so. These change the model significantly.
  4. Could you include the wall-clock / FLOP cost of ReCo vs each baseline, including the per-step forget-set evaluation and backtracking? How many steps does ReCo take before it stops on average?
  5. Could the authors add the original model's CR and ED metrics to the main tables?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.