- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies relearning resistance in machine unlearning when forgotten data are relearned using Adam rather than standard gradient descent. It derives an upper bound on Relearning Convergence Delay (RCD) under a frozen Adam preconditioner and uses this result to motivate Gradient-Entropy Maximization (GEM), a regularizer designed to spread the forget-set gradient across model parameters and make relearning more difficult. Experiments on CIFAR-100 classification, Stable Diffusion concept removal, and TOFU language-model unlearning show that GEM can improve RCD for several methods, although the gains vary across baselines, tasks, and relearning learning rates.
Critical strengths
-
This paper studies a meaningful and relatively underexplored question: whether unlearned knowledge remains difficult to recover when relearning is performed with Adam.
-
GEM can be easily combined with existing unlearning methods, and the experiments cover image classification, image generation, and language-model unlearning with several representative baselines.
Critical weaknesses
- The claimed relearning resistance is too closely tied to Adam. The evaluation assumes that relearning follows an Adam optimization trajectory, but an attacker can choose SGD, AdamW, related benign data, or a different recovery procedure. Prior work has shown that a small amount of related but non-identical data can reactivate supposedly forgotten knowledge [a]. It is therefore unclear whether GEM makes the target knowledge genuinely harder to recover or only makes a particular Adam trajectory converge more slowly.
The theoretical analysis further narrows this setting by assuming , , and a preconditioner frozen at the beginning of relearning. It also requires a locally positive-definite Hessian, a monotonic condition-number assumption, and a step size determined by the largest Hessian eigenvalue at each iteration. These assumptions remove the momentum and evolving second-moment estimate used by practical Adam and AdamW, leaving a substantial gap between the theory and the optimizers used in the experiments.
- The gradient-entropy score and RCD do not provide a certificate that the target knowledge has been reliably removed. Theorems 3.1 and 3.4 only establish upper bounds on RCD and the relevant preconditioning term. Increasing such an upper bound does not imply that the actual RCD must increase. This gap is also visible in Appendix D.4, where the entropy score and measured RCD move in the same direction in only 5 of 10 comparisons for the 30% forgetting settings, with similarly partial agreement under Random 50%.
More fundamentally, a large RCD may result from poor conditioning or generally slower optimization rather than the absence of the target information from the model. Prior work has recovered apparently forgotten knowledge through benign relearning [a], broader behavioral and privacy tests [b], and probes of internal representations [c]. The paper therefore needs independent recovery attacks and residual-knowledge evaluations before interpreting a larger Adam RCD as stronger unlearning; otherwise, the term “certificate” overstates the guarantee.
[a] Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning. ICLR 2025.
[b] MUSE: Machine Unlearning Six-Way Evaluation for Language Models. ICLR 2025.
[c] An Illusion of Unlearning? Assessing Machine Unlearning Through Internal Representations. AISTATS 2026.
Questions for authors
-
Does GEM improve relearning resistance when the attacker uses SGD with momentum, AdamW, or a different recovery procedure?
-
Since Theorems 3.1 and 3.4 provide only upper bounds, what formal argument connects maximizing the gradient-entropy score to an increase in the actual RCD? How should the proposed “certificate” be interpreted given the partial disagreement between the score and RCD in Appendix D.4?
-
Can the analysis be extended to practical Adam with nonzero momentum and an evolving second-moment estimate, or can the authors provide evidence that the frozen-preconditioner approximation accurately predicts the behavior observed in the experiments?