acceptodds
Under review as a conference paper at ICLR 2027

Caffeine: Machine Unlearning via Curvature-Aware Gradient Ascent

Abstract

Deleting data from a trained model without retraining requires an update that cancels the deleted data's contribution. Influence functions specify this update via the inverse Hessian, but evaluating it requires iterative passes over the training data, which is often no longer available when a deletion request arrives. Existing approximations read the retained data, restrict the change to the final layer, or drop curvature rescaling altogether. We present Caffeine, a source-free unlearning algorithm that operates as a curvature-aware gradient ascent. Since the update moves along the gradient, it needs the curvature strictly along that one direction—a single scalar that two forget-set gradients evaluated a small step apart determine without any exact second-order computation or retained data. Caffeine measures this scalar at each saved training checkpoint and aggregates the resulting curvature-scaled steps into a single update to every layer of the deployed model. Across class and random-subset deletion on five image and text datasets, Caffeine is the only approximate method that both forgets and preserves accuracy without retained data, running up to faster than retraining on image tasks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
machine unlearning, source-free unlearning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2wM6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Caffeine, an approximate machine unlearning method that operates without access to the retained data. At each saved training checkpoint, it uses two forget-set gradient evaluations to estimate curvature along the gradient direction. It scales the gradient by the inverse of this estimate, then combines the checkpoint updates using a learning-rate-weighted signed RMS rule to update the final model. The authors evaluate class deletion on three image datasets and two text datasets, and random-subset deletion on two image datasets. The reported results show a better trade-off between forget accuracy and utility than the evaluated approximate baselines, together with substantial runtime savings over retraining on the image tasks.

Critical strengths
  1. The method addresses a meaningful constraint with a relatively simple procedure. Preserving utility without access to retained data is a practical challenge. Caffeine estimates its update scale from forget-set gradients, avoids second-order automatic differentiation, and can modify the full network. These properties make it a potentially useful approach when training checkpoints are available.
  2. The reported efficiency–utility trade-off is promising. For example, on TinyImageNet class deletion, Caffeine achieves 46.37% test accuracy compared with 47.00% for retraining, while reducing runtime from 4,103 to 170 seconds. If stronger evaluations confirm its forgetting effectiveness, this efficiency improvement would be a meaningful contribution.
Critical weaknesses
  1. The evaluation does not sufficiently distinguish unlearning from changes in prediction behavior. Class deletion is assessed primarily through near-zero forget accuracy. Suppressing the deleted class’s output can achieve this without removing the relevant training information from the representation. The paper criticizes final-layer methods for leaving such information intact, but does not provide a corresponding representation or relearning evaluation to establish its own advantage. The random-subset results provide limited additional evidence. In Tables 6 and 7, the original model already has an MIA AUC of approximately 0.52, suggesting that the chosen attack has little ability to distinguish members before unlearning. Consequently, an AUC near 0.5 after applying Caffeine is weak evidence of successful removal. This limitation affects the paper’s central claim, rather than merely the breadth of its evaluation.
  2. The signed RMS aggregation is insufficiently justified, and the ablations do not establish its contribution. Equation (11) takes the sign from the weighted sum of historical updates and the magnitude from their RMS. For a coordinate with two equally weighted updates of 11 and −0.99-0.99, the resulting update is approximately 0.9950.995, despite the near cancellation of the two directions. Thus, the rule does not inherently produce small updates in oscillatory regions or establish the claimed trust-region behavior. The paper does not directly compare this aggregation with a weighted mean, nor does it include a well-tuned gradient-ascent baseline under the same data-access constraint and computational budget. It is therefore difficult to determine whether the gains come from curvature estimation, historical checkpoints, or simply a different update scale. These controls are particularly important given the severe degradation of several evaluated baselines.
Questions for authors
  1. Clarification of the experimental protocol: Does retraining start from an initialization that has not been trained on the deleted data? The statement that all methods, including retraining, start from the same original models and checkpoints is ambiguous. Also, which dataset and splits are used for the five-class “IMDB” experiment? Reference [35] points to the Stanford Sentiment Treebank.
  2. Evidence for the aggregation rule: What precisely does Remark 3 establish about signed RMS aggregation? How does that result constrain the update applied to the final model, beyond a bound relative to historical update magnitudes? Do you have comparisons against weighted-mean aggregation and fixed-step gradient ascent using the same checkpoints and comparable computational budgets? These comparisons would materially affect my assessment of the method’s contribution.
  3. Evidence for forgetting: Do you have results from a membership-inference attack with demonstrated discriminatory power on the original model, or representation/relearning evaluations against the retrained model? Please also report variability across runs for the AUC results and reconcile the discrepancies between Section 4.3 and Table 6. This would help determine how much evidence the reported attack provides. These questions concern evidence needed to support the current claims, rather than requests to extend the work to new applications. The supplied PDF does not include the cited supplementary material; if it already addresses these points, please identify the relevant results.
3 comments

Dear Reviewer,

The authors have kindly provided detailed responses to your comments. We would appreciate it if you could take a moment to review their rebuttal and share any further feedback or updates to your evaluation.

Thank you for your time and thoughtful consideration.

I appreciate the authors' efforts in addressing my concerns. The additional representation-level evaluation, frozen linear probe, and SST-5 MIA experiment provide stronger evidence of the forgetting effectiveness. The clarification of the retraining protocol and the additional aggregation ablations are also helpful.

However, some concerns remain. In particular, the curvature estimation is still not directly validated, and the poor performance of some baselines (e.g., SFMU and HFOL) remains insufficiently explained. These issues leave some uncertainty about the technical validity and reliability of the experimental comparisons.

Overall, the rebuttal has improved the paper considerably, but I believe some important concerns remain unresolved. Therefore, I maintain my original rating.

We thank the reviewer for acknowledging that the rebuttal has improved the paper considerably. The review listed two weaknesses and three questions. As we read the reviewer's comment, these are now addressed: evidence of forgetting (W1, Q3) by the frozen-backbone probe, the SST-5 MIA and the standard deviations over three training seeds; the aggregation rule (W2, Q2) by Proposition 3 and by the weighted-mean and fixed-step comparisons, which the review said "would materially affect my assessment"; and the protocol (Q1).

Neither remaining point was among the review's questions. We answer both below and respectfully ask the reviewer to reconsider the rating in light of them. All appendix references are to the Supplementary Material PDF (pages 14-24); the probe table cited is now Table 13.

1. Direct validation of the curvature estimate. This is in the paper (Appendix F.4, Table 12, page 24 of the supplementary PDF, added in the latest revision). Caffeine uses only the scalar Kt=gf⊤Hfgf/∥gf∥22K_t = g_f^\top H_f g_f / \lVert g_f \rVert_2^2. We compare its two-gradient estimate K^t\hat K_t with the exact KtK_t at every checkpoint. The exact value is computed with Hessian-vector products by double backpropagation; it is the same number the materialized Hessian would give.

Checkpoint (epoch)Exact KtK_tK^t\hat K_t (ours)Relative error
10189.31183.293.18%
20232.93218.975.99%
30242.05231.194.49%
4088.2988.300.01%
5047.3851.067.77%

The error is below 8% at every checkpoint (smaller CNN of Table 11, CIFAR-10, class 0). Replacing K^t\hat K_t by the exact KtK_t inside Caffeine changes test, forget and retain accuracy by at most 0.15 points (Table 11).

2. Baseline performance. The review mentioned the degradation of several baselines as the reason for requesting controls. Those controls are now in Table 10 (page 22). They use no third-party code, and the same model and checkpoints as Caffeine:

VariantTestForgetRetain
Retrain85.480.0089.98
Fixed-step ascent, α=2×10−3\alpha = 2\times10^{-3}75.4016.9887.29
Fixed-step ascent, α=3×10−3\alpha = 3\times10^{-3}68.632.8483.02
1/Kt1/K_t + weighted mean82.2210.4588.20
Caffeine81.130.0587.86

Our conclusions therefore do not rest on SFMU or HFOL: every result is measured against retraining on the same model, and the controls above show the same picture. SFMU, GradTao and SCRUB run from the authors' released code with their recommended hyperparameters (Appendix D, Table 8, page 20). HFOL's released code targets the small models of its paper, and we adapted it to ResNet-18 without changing the algorithm. We did not tune any baseline further and say so.

Why these two fail. Neither method was designed for class deletion: both papers evaluate sample-level or random-subset deletion.

SFMU. In the authors' own setting the released code reproduces the results reported in their paper (CIFAR-10, linear classifier on ResNet-18 features, 10% random-subset deletion: they report 70% test, 71% remaining and 68% forget accuracy; we obtain for SFMU, 79.3% test, 83.20% forget, and 83.14% retain accuracy, so its behavior here is a property of the method under class deletion, not of our setup. SFMU modifies only the final layer. Its backbone is identical to the original model's, so the deleted data remains in the features: a new linear head recovers the deleted class from SFMU's features at 88.62%, exactly as from the original model's (Table 13, page 24). A final-layer correction of the size SFMU applies does not remove a whole class.

HFOL. Its paper evaluates a small CNN and a linear model; ResNet-18, with 11.7M parameters, is outside that range. HFOL serves a deletion by adding the stored vector of every deleted sample to the model, followed by calibrated noise (Appendix D). Each vector is an approximation, and the errors and the noise accumulate with the number of samples deleted at once. With 5,000 samples on CIFAR-10 the accumulated update collapses the model, under class deletion (10.33% test accuracy) and under random-subset deletion (10.30%); on TinyImageNet it leaves the model unchanged (61.94% forget accuracy against 62.43%). It runs out of memory for two or more classes and cannot run on DistilBERT, whose attention kernel does not support the double backward pass it needs.

We thank the reviewer again for the constructive feedback.

Reviewer_MgArOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose a approximate machine unlearning method named Caffeine where the access to the retained data is not required. For each saved training checkpoint, the proposed Caffeine first computes a scale curvature along the gradient of the forget data and then apply it to scale the gradient-ascent update. Finally, such update are aggregated into a single full-model parameter update using a learning-rate-weighted signed root-mean-square scheme. Experiments are conducted on both the image datasets and text datasets with two unlearning scenarios. And the reported results demonstrated a better trade-off betwwen the forgetting performance and post-unlearning utility.

Critical strengths

(1) This paper aims to solve the machine unlearning task in the scenario where the retain data can not be accessed. It used the forget data, saved training checkpoints and deployed model.

(2) The proposed Caffeine estimated the curvature using only two first-order gradient evaluations per checkpoint, avoiding explicit Hessian computation or inversion.

(3) Experiments are extensive across several data forgetting settings, and the reported results demonstrate a favorable balance between data forgetting and post-unlearning utility performance, together with substantial speedups over retraining.

Critical weaknesses

(1) Although the proposed Caffeine does not require retained training data, it relies on multiple checkpoints saved during the original training process. In the real implementation, such task setting has a big limitation when only the deployed model is available. So this attribute will limit the practical usage of the proposed method.

(2) In equation 11, the checkpoints updates are combines using a signed RMS. However, when these updates from different checkpoints point in opposite directions, their sum may nearly cancel while the RMS magnitude remains large. The authors should give more explanation regarding this potential problem.

(3) From the reported experiment results, the original models already have membership inference AUC values close to 0.5, which means that the attack has limited ability to distinguish training samples from unseen samples even before unlearning. However, the MIA after forgetting is still near 0.5, which reflect the randomly prediction.

(4) The experiment setting of Table 4 is unclear and the authors should clarify the dataset, providing the exact experimental settings

Questions for authors

(1) What theoretical evidence supports describing the signed RMS update as a safe trust region, especially when updates from different checkpoints cancel each other?

(2) How does the requirement to save multiple training checkpoints affect Caffeine's practical applicability and storage cost?

1 comment

Dear Reviewer,

The authors have kindly provided detailed responses to your comments. We would appreciate it if you could take a moment to review their rebuttal and share any further feedback or updates to your evaluation.

Thank you for your time and thoughtful consideration.

Reviewer_b8qdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the optimization for solving machine unlearning problems, and proposes a gradient-based unlearning algorithm, Caffeine.

The key assumption in Caffeine is that the curvature of the loss on the forget set can provide a useful estimate of how much the model should be updated to remove the influence of the forgotten data from a trained model.

To run Caffeine, it requires number of CC checkpoints θt\theta_t at every training step tt (assuming a target model θ∗\theta^* was trained with CC steps on the complete dataset D=Df+DrD = D_f + D_r), and access only to the forget data DfD_f, no retain data DrD_r is needed.

The method approximates the curvature by using gradient of θt\theta_t over DfD_f, eventually, across tt it summarize a fine-grained update to the target model θ∗\theta^*.

Through experiments, the paper shows that comparing to other source-free unlearning algorithms, the proposed method is achieving high accuracies (Test, Forget, Retain, and MIA), while requiring less computational time and memory.

Critical strengths
  • The proposed unlearning algorithm is source-free that requires no retain data.

    • Reviewer agrees with such a setup that training data can be inaccessible at the time of fine-tuning the pre-trained models.
  • The proposed unlearning algorithm provides a trade-off between computational time-efficiency, unlearning accuracy, and the storage of checkpoints across different training stages.

    • Reviewer also agrees that assuming the access to the checkpoints across different training stages is reasonable, as people usually save checkpoints as training goes on.
  • Experiments show evidences that the proposed unlearning algorithm demonstrates superior performance and efficiency than other source-free unlearning algorithms.

Critical weaknesses
  • Lack of important technical details. Specifically, the paper never defines the main objective L(⋅)L(\cdot) or provides details of it. Is LL the classifier training objective (e.g., entropy-loss)?

  • Overly simple experimental setup. The unlearning experiments shown in this paper is overly toy that it primarily focuses on unlearning image or text classifiers trained on small label space (e.g., 5, 10, 17, 200 classes), where a re-training is feasible.

    • Such a setup may not be practical and meaningful that for example, Table 1 focuses on CIFAR-10 and experiment unlearned up to 3 classes of it (30% of your label space) -- where do image of those unlearned classes go to in this case?
  • Lack of direct gradient and Hessian validation or analysis. Since paper is proposing an Hessian approximation with gradients, having experiments that directly show estimation error can make the paper more convincing.

    • Given the reported use of "NVIDIA A100 80 GB GPUs", such targeted analyses appear worth investigating, although the feasibility of explicitly materializing the full Hessian depends on the number of trainable parameters.
  • Experimental correctness and transparency. Some comparing methods demonstrate significant failure in the paper's reported results, which makes reviewer concern about their experimental correctness and transparency.

    • Specifically, two unlearning algorithms SFMU [1] and HFOL [2] show no effect at all for "1 Class" unlearning on CIFAR-10 in Table 1, they are even shown higher forget accuracies after unlearning.

References:

[1] Towards source-free machine unlearning, CVPR 2025

[2] Hessian-free online certified unlearning, ICLR 2025

Questions for authors

See questions in the Critical Weaknesses.

2 comments

Dear Reviewer,

The authors have kindly provided detailed responses to your comments. We would appreciate it if you could take a moment to review their rebuttal and share any further feedback or updates to your evaluation.

Thank you for your time and thoughtful consideration.

Reviewer_b8qdOpenReview ↗

Response to Rebuttal

Thanks authors for their substantial effort in rebuttal.

Two of my initial concerns regarding the 1) unlearning objective and 2) experimental correctness, have been clarified. And 1) unlearning objective has been revised in the submitted manuscript.

Remaining concerns:

  • Overly simple experiment setup: While I agreed that the dataset this paper used is a consistent setup to existing work, this concern remains. Potential way to convincingly justify the proposed method effectiveness, can be demonstrating unlearning performance on generative models, whose objective is also more complex beyond the cross-entropy formation.
  • Direct gradient and Hessian validation or analysis: authors have tried to provide an unlearning performance evaluation by swapping hessians that are estimated in different ways. It would be great to provide direct error analysis over heissian.

In summary, my concerns are partially resolved, and I would like to finalize my score as 3 - Weak accept. My justification is that this paper proposes a simple yet effective optimization method for solving unlearning problems, supported by extensive experiments on classification tasks. But in the meantime it has limited evidence that the proposed method can generalize to tasks beyond classification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.