- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies request idempotence in approximate machine unlearning: repeating a completed deletion request should not trigger further model changes when the deletion target is unchanged. The authors introduce the duplicate amplification ratio (DAR), measure drift under repeated unlearning, and use optimization controls to examine its cause. They also present DeleteGuard, a stateful wrapper that prevents repeated execution of requests with the same key. The contribution is mainly an empirical analysis of request handling, not a new unlearning algorithm.
Critical strengths
-
It makes a useful distinction between deletion-service correctness and the behavior of an unlearning optimizer.
-
The stochastic baselines, retain-only continuation, and replication across three training seeds strengthen the evidence for the reported NegGrad+ behavior.
Critical weaknesses
- The technical contribution is limited. It mainly analyzes a narrow failure case and introduces no new unlearning algorithm.
Section 3.3 explains that invoking an optimizer twice applies more updates, and that the forgetting gradient need not vanish at a retrained model. This follows directly from the stated objective. DeleteGuard addresses the issue through request tracking, not a change to the unlearning procedure. A characterization of which objectives or stopping conditions avoid this failure would add more depth beyond documenting its severity.
- Successful completion of the first (approximate) deletion is not clearly defined/established.
The contract applies after a successful request, but this work doesn't give an operational criterion for success. Table 5 reports 98.90% forget-training accuracy and 0.086 JS distance to retraining after the first request. These values alone do not establish success or failure; a retrained model may also predict the requested examples correctly. It should distinguish successful execution of an update job from successful removal of the requested influence. The exact-retraining stress test is useful, but it doesn't show that the approximate endpoints used to compute DAR satisfy the first-deletion condition.
- The applicability to large language models (LLMs) and vision LLMs remains unclear.
The service contract is model-independent. The practical difficulty is defining the request and deciding when it has been satisfied. The experiments use explicit training-record deletions in image classification and recommendation. For (v)LLMs, this paper doesn't show how fact- or concept-level requests would map to forget sets, how equivalent requests would be recognized, or how completion would be assessed. The authors acknowledge the absence of language-model experiments. This limits the demonstrated scope instead of showing that idempotence itself is unsuitable for large models.
- The strongest findings rely heavily on NegGrad+ and a fixed optimization setting.
The image breadth experiments test NegGrad+, while the core configuration fixes the forgetting weight, learning rate, and invocation length. No sensitivity study is reported for these choices. The retain-only control is helpful, but it doesn't show whether the magnitude of the effect persists under other reasonable unlearning budgets. The long-horizon collapse also arises from a single trajectory, as the authors acknowledge.
- The statistical procedures are not described in enough detail to assess the reported uncertainty.
The main analysis uses five unlearning trajectories. Pairwise distances reuse the same models, and each second-request endpoint depends on its first-request endpoint. These observations are not independent. Appendix D names a hierarchical bootstrap and a run-label permutation test, but doesn't explain the resampling hierarchy or how the paired trajectories are handled. Those details are needed to assess the confidence interval and p-value. The information here is insufficient to determine whether the procedures account for the dependence correctly.
Minor issues, e.g.:
- Define DAR explicitly for , since Table 2 reports DAR and DAR, whereas Section 3.2 defines only DAR.
-Table 3 describes test accuracy as a "drop" but reports negative values. Define it as accuracy at minus accuracy at , or report positive drop magnitudes.
Questions for authors
-
What defines a successful first deletion, and what evidence shows that the approximate endpoints meet this condition?
-
How much drift occurs when a compute-matched retain-only update starts from the same RETRAIN-A checkpoint used in the exact-target stress test?