acceptodds
Withdrawn from ICLR 2027

Delete Once Means Once: Request Idempotence in Approximate Machine Unlearning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Machine unlearning has emerged as a promising way to remove the influence of data from a trained model. However, practical deletion services are typically required to be idempotent: deleting the same data twice should leave the system unchanged. We show that this natural property is not guaranteed by stateless approximate unlearning algorithms. We define a stochasticity-aware Duplicate Amplification Ratio (DAR) and find that a second identical NegGrad+ request produces 4.60x the predictive drift of ordinary single-request stochasticity in our core CIFAR-100/ResNet-18 setting. This amplification is robust to training seeds, distinct from extra compute, and tied to a fixed-point mismatch between deletion semantics and the forgetting surrogate. The severity is heterogeneous—five vision settings show DAR from 1.49 to 6.40, while a recommender boundary case shows weaker utility impact—but repeated replay can eventually drive the model into a catastrophic high-confidence degenerate regime. We argue that request idempotence should be enforced at the service boundary and provide DELETE GUARD, a lightweight stateful wrapper that enforces the correctness contract without modifying the underlying optimizer.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, Idempotence, Approximate Unlearning, Deep Learning, Data Deletion, CIFAR-100, NegGrad+

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_aRFQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies request idempotence in approximate machine unlearning: repeating a completed deletion request should not trigger further model changes when the deletion target is unchanged. The authors introduce the duplicate amplification ratio (DAR), measure drift under repeated unlearning, and use optimization controls to examine its cause. They also present DeleteGuard, a stateful wrapper that prevents repeated execution of requests with the same key. The contribution is mainly an empirical analysis of request handling, not a new unlearning algorithm.

Critical strengths
  • It makes a useful distinction between deletion-service correctness and the behavior of an unlearning optimizer.

  • The stochastic baselines, retain-only continuation, and replication across three training seeds strengthen the evidence for the reported NegGrad+ behavior.

Critical weaknesses
  • The technical contribution is limited. It mainly analyzes a narrow failure case and introduces no new unlearning algorithm.

Section 3.3 explains that invoking an optimizer twice applies more updates, and that the forgetting gradient need not vanish at a retrained model. This follows directly from the stated objective. DeleteGuard addresses the issue through request tracking, not a change to the unlearning procedure. A characterization of which objectives or stopping conditions avoid this failure would add more depth beyond documenting its severity.

  • Successful completion of the first (approximate) deletion is not clearly defined/established.

The contract applies after a successful request, but this work doesn't give an operational criterion for success. Table 5 reports 98.90% forget-training accuracy and 0.086 JS distance to retraining after the first request. These values alone do not establish success or failure; a retrained model may also predict the requested examples correctly. It should distinguish successful execution of an update job from successful removal of the requested influence. The exact-retraining stress test is useful, but it doesn't show that the approximate endpoints used to compute DAR satisfy the first-deletion condition.

  • The applicability to large language models (LLMs) and vision LLMs remains unclear.

The service contract is model-independent. The practical difficulty is defining the request and deciding when it has been satisfied. The experiments use explicit training-record deletions in image classification and recommendation. For (v)LLMs, this paper doesn't show how fact- or concept-level requests would map to forget sets, how equivalent requests would be recognized, or how completion would be assessed. The authors acknowledge the absence of language-model experiments. This limits the demonstrated scope instead of showing that idempotence itself is unsuitable for large models.

  • The strongest findings rely heavily on NegGrad+ and a fixed optimization setting.

The image breadth experiments test NegGrad+, while the core configuration fixes the forgetting weight, learning rate, and invocation length. No sensitivity study is reported for these choices. The retain-only control is helpful, but it doesn't show whether the magnitude of the effect persists under other reasonable unlearning budgets. The long-horizon collapse also arises from a single trajectory, as the authors acknowledge.

  • The statistical procedures are not described in enough detail to assess the reported uncertainty.

The main analysis uses five unlearning trajectories. Pairwise distances reuse the same models, and each second-request endpoint depends on its first-request endpoint. These observations are not independent. Appendix D names a hierarchical bootstrap and a run-label permutation test, but doesn't explain the resampling hierarchy or how the paired trajectories are handled. Those details are needed to assess the confidence interval and p-value. The information here is insufficient to determine whether the procedures account for the dependence correctly.

Minor issues, e.g.:

  • Define DARk_k explicitly for k>2k>2, since Table 2 reports DAR3_3 and DAR4_4, whereas Section 3.2 defines only DAR2_2.

-Table 3 describes Δ\Delta test accuracy as a "drop" but reports negative values. Define it as accuracy at k=2k=2 minus accuracy at k=1k=1, or report positive drop magnitudes.

Questions for authors
  1. What defines a successful first deletion, and what evidence shows that the approximate k=1k=1 endpoints meet this condition?

  2. How much drift occurs when a compute-matched retain-only update starts from the same RETRAIN-A checkpoint used in the exact-target stress test?

Reviewer_rrqJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper makes the observation that gradient ascent based methods fail to be stable under repeated unlearning requests. They showed this for several image models and datasets, and varying the size of the set being removed (and the number of duplicate requests). In contrast, other methods like fine-tuning did not suffer as significantly. The authors propose a database system to keep track of what data has already been removed to handle future duplicate requests.

Critical strengths
  1. Several metrics and seeds were used. This makes me confident the shown empirical are mostly sounds, up to questions about how hyperparameters were chosen.
  2. The specific evaluation of idempotency is novel to the best of my knowledge. I think this can make the paper a valuable evaluation paper, however it currently focused on methods already known to fail unlearning (even before measuring idempotency); see below.
Critical weaknesses
  1. The empirical observations are largely focused on gradient ascent based approaches (NegGrad), which already are known to fail unlearning [1]. More generally, any method which only uses the model weights and forget set is unable to unlearn [2]. Current consensus (to the best of my knowledge) is that fine-tuning is a guaranteed unlearning algorithm with errors controllable by the number of training steps and adapting the number of finetuning steps to the datapoint [3] (in fact, note the best performing methods in [1] are variants of fine-tuning). The paper’s results on finetuning seem to show it is also mostly idempotent, making the evaluations novelty/relevance less clear.
  2. Furthermore, Idempotency seems to add to the literature on adversarial deletion requests breaking unlearning methods. In fact, it is already known supposedly exact unlearning methods can fail under adaptive requests [4]; this is not mentioned in the paper, and I think limits the novelty of studying duplicate requests on their own.

[1] Hayes, Jamie, et al. "Inexact unlearning needs more careful evaluations to avoid a false sense of privacy." 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). IEEE, 2025.

[2] Thudi, Anvith, et al. "On the necessity of auditable algorithmic definitions for machine unlearning." 31st USENIX security symposium (USENIX Security 22). 2022.

[3] Sepahvand, Nazanin Mohammadi, et al. "Leveraging per-instance privacy for machine unlearning." International Conference on Machine Learning (2025).

[4] Gupta, Varun, et al. "Adaptive machine unlearning." Advances in Neural Information Processing Systems 34 (2021)

Questions for authors

I think the paper as is does not significantly add to existing understanding of what unlearning methods work. However, I believe the main novelty is adding another evaluation to test existing methods and add to our understanding of when they work (or not). My questions are mainly suggestions towards reframing the paper for this. I do not expect all of them to be answered, but even answering most of these could significantly improve the impact of the paper and I would consider raising my score.

  1. Could the authors rephrase the introduction to focus on the limitation of fine-tuning to meet idempotency? The experimental results show it is significantly better than NegGrad, but can you focus on how these numbers could still be improved?
  2. For the evaluation of fine-tuning, can you isolate different data deletion sets? It is known its effectiveness under a fixed steps budget varies with the datapoints (see reference [3] from earlier); are some less idempotent than others?
  3. Similarly, can you ablate the number of training steps? Fine-tuning has a bias term which depends on how converged the training algorithm gets, and the drift you observe could be explained by that.
  4. There are also other ways in which fine-tuning is known to become more effective at unlearning. One is by adding l1 regularization (see implementation in [1] of SPARSITY) or by adding some small noise during training and fine-tuning (see [3] and the noise levels they used without hurting test accuracy). Does doing this already make idempotency less of a concern?

Below are questions regarding the current write-up. These are mostly minor, given I again think the paper needs more significant restructuring.

  1. In Section 2, the “Continual and repeated unlearning” paragraph, you are missing a reference to “Adaptive Machine Unlearning” [4]; that paper discuss how adversarial requests could break supposedly exact unlearning methods. I think you can reframe idempotency as an example of this that is easy to implement and evaluate.
  2. I saw a fixed \beta = 0.95 was used for NegGrad. How was this chosen? Also how were the other hyperparameters chosen?
  3. Line 215 says “three unlearning realizations” were used; what are these exactly and how were they sampled? I assume forget sets sampled from the training set with replacement?
  4. In Section 5.2 line 264 states “A reviewer could reasonably argue that a…”. Keep in mind this paper should ultimately be read by non-reviewers, so consider rephrasing to “One could” or some other general language; also you can just state “We have not controlled for whether this effect is due to the unlearning objective used or the optimization algorithm.”
  5. In Figure 2, for “Retain-only continuation” is that retraining or finetuning? Also, consider rephrasing the caption to make this clearer.
Reviewer_x9HAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies request idempotence in approximate machine unlearning, focusing on whether processing the same deletion request twice causes unnecessary model changes. It introduces the Duplicate Amplification Ratio (DAR) to measure replay-induced drift relative to ordinary optimization randomness. Experiments on image classification and recommendation tasks show that repeated NegGrad+ updates can cause substantial predictive drift and utility degradation. The paper further analyzes this behavior through optimization controls and gradient measurements, and presents DELETE GUARD, a stateful wrapper for preventing duplicate unlearning operations.

Critical strengths
  1. The paper identifies a relevant issue that is largely overlooked in one-shot machine unlearning evaluations. Treating duplicate requests as a zero-target-change case provides a clear way to examine whether unlearning operations behave consistently after a deletion has been processed.
  2. The experimental analysis includes useful controls beyond simply repeating an optimization procedure. In particular, the compute-matched retain-only experiment, exact-retraining stress test, and cross-training-seed results provide convincing evidence that the observed NegGrad+ drift is associated with the forgetting objective rather than just additional optimization.
Critical weaknesses
  1. The main technical contribution is limited. As shown in Section 3.3, replaying a finite-step unlearning procedure simply performs additional optimization, so model drift is expected unless the first invocation reaches a fixed point. The experiments quantify this behavior for NegGrad+, but offer limited new theoretical or algorithmic insight. Moreover, DELETE GUARD relies on standard request-key deduplication and does not fully address semantically identical deletion requests submitted under different keys. It is therefore unclear what the paper contributes beyond applying a well-established systems principle to approximate unlearning.

  2. The experimental evidence does not establish a sufficiently broad or consequential failure mode. The strongest results are concentrated on NegGrad+, while Table 2 reports DAR2 values of only 1.15 and 1.06 for fine-tuning and SalUn, respectively. The MovieLens experiment shows no ranking-utility degradation, and the catastrophic collapse in Figure 4 is based on a single trajectory. In addition, the DAR estimator is not exactly calibrated to one under perfect idempotence with finite paired runs, which complicates comparisons across experimental settings. Overall, the evidence supports a specific optimizer sensitivity, but the broader significance of the finding remains insufficiently established.

Questions for authors
  1. The formal contract in Section 3 requires repeated deletion of the same data to be idempotent. However, DELETE GUARD appears to identify duplicates through request keys. Would resubmitting the same deletion set under a different key trigger another unlearning operation? If so, how does the implementation satisfy the stated contract?

  2. In Eqs. (6)–(8), Vdup includes matched first- and second-request pairs, while Vstoch excludes same-run pairs. Under perfect idempotence, this makes the empirical DAR depend on the number of runs. Could the authors clarify this normalization and its implications for comparing different settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.