- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies certified continual learning–unlearning for regularization-based continual learning. It derives an algorithm-dependent excess-risk bound, develops a natural-forgetting procedure and a Hessian-based correction procedure, and analyzes the effects of regularization and deletion order. Experiments with a frozen ResNet-18 feature extractor and a trainable head on task-partitioned CIFAR-100 compare approximation error, accuracy, runtime, and storage.
Critical strengths
- The paper addresses the important tension between maintaining previously learned tasks and removing the influence of deleted tasks, rather than treating unlearning as an isolated post-training operation.
- The unified treatment of task-additive regularizers and the comparison between natural forgetting and second-order corrections provide a useful conceptual account of how regularization and deletion order interact.
- The evaluation considers several regularizers, early versus late deletions, and multiple requests, and reports both predictive quality and resource usage.
Critical weaknesses
- Limited effectiveness of natural forgetting for targeted unlearning. The natural forgetting method in Eq. (9) performs no explicit correction to remove the requested task's influence. Instead, it relies on subsequent continual-learning updates and calibrated Gaussian noise. Its practical effectiveness remains unclear, particularly for recently learned tasks or deletion requests arriving at the final learning stage, where few or no subsequent updates are available. The noise required to achieve certified unlearning may also significantly affect retained-task utility.
- Insufficient experimental validation. The experiments mainly focus on CIFAR-100 with a frozen feature extractor and a trainable classification head. More diverse datasets, model architectures, and continual-learning settings are needed. More importantly, the paper lacks direct empirical evaluations of unlearning effectiveness, even basic membership inference attack (MIA) experiments. Existing works with theoretical unlearning guarantees still conduct MIA evaluations to validate forgetting effectiveness. More recent methods, such as Unlearning Evaluation through Subset Statistical Independence (ICLR 2026), also provide alternative ways to assess unlearning. The current evaluation, which primarily relies on parameter approximation errors and predictive accuracy, is insufficient to fully assess practical forgetting effectiveness.
Questions for authors
- How effective is natural forgetting for targeted task-level unlearning, particularly when a deletion request concerns a recently learned task or arrives at the final learning stage? Since the method performs no explicit correction, how is the required Gaussian noise calibrated, and what is its impact on retained-task utility?
- Can the authors provide more comprehensive evaluations across different continual-learning benchmarks, model architectures, and deletion schedules? In particular, can they include MIA-based evaluations or other suitable empirical unlearning metrics, alongside retained-task accuracy, to provide more direct evidence of forgetting effectiveness?
6 comments
Response to Reviwer 2U54 (2/3)
New Unlearning sequence evaluation. We evaluated unlearning performance under various request sequences in Appendix G.1 of the original manuscript. As suggested, we have moved these results to the main text, as shown in Figure 2 on page 10 of the revised manuscript.
On CIFAR-100, we evaluate HU and NF under three request sequences over a 20-task stream, all of which eventually delete tasks . Here, is forward-synchronous and satisfies Eq. (16), whereas the other two sequences are asynchronous. HU achieves a lower final approximation error than NF across all three sequences, demonstrating its effectiveness and robustness under complex multi-request schedules. The forward-synchronous sequence achieves the lowest final approximation error, consistent with our theoretical analysis in Theorems 5.2 and E.1.
We have also added corresponding experiments using the CNN on CIFAR-10, reported in Figure 8 of Appendix H.1 on page 56 in the revised manuscript. These experiments consider a 10-task stream with three schedules:
Q_1 = (4:\{2\};\,7:\{5,6\};\,9:\{8\}), Q_2 = (4:\{2\};\,7:\{5\};\,9:\{6,8\}), Q_3 = (7:\{5,6\};\,9:\{2,8\}). \end{aligned}$$ Here, $Q_1$ is forward-synchronous, while $Q_2$ and $Q_3$ are asynchronous. EWC on CIFAR-10 exhibits the same schedule ordering as EWC on CIFAR-100. At $t=9$, when all schedules have deleted $\{2,5,6,8\}$, HU achieves approximation errors of $0.31779$, $0.32247$, and $0.33198$ under $Q_1$, $Q_2$, and $Q_3$, respectively. After learning the final retained task at $t=10$, these errors decrease to $0.31048$, $0.31359$, and $0.31613$, preserving the ordering. Thus, the forward-synchronous schedule also achieves the lowest final error in this setting. **New evaluation of the unlearning–accuracy tradeoff on CIFAR-10.** Table 1(b) on page 9 of the revised manuscript presents our newly added CIFAR-10 evaluations, comparing NF and HU across EWC, MAS, Diag-Hessian, scalar $\ell_2$, and accumulated $\ell_2$ regularization. We use five tasks and issue a single deletion request at $t=5$, separately deleting Task 2 or Task 4. When predictive accuracy is prioritized, we should choose MAS for both deletion settings. Its parameter-dependent penalties constrain important directions while allowing adaptation elsewhere, consistent with Section 3. When approximation quality is prioritized, we should choose HU with Diag-Hessian for Task 2 and HU with accumulated $\ell_2$ for Task 4. This dependence on the deletion regime is consistent with Sections 4–5, where stronger effective regularization can attenuate the deleted task’s influence. All reported Pareto-optimal configurations use HU, indicating that explicit correction is beneficial in the evaluated five-task setting. **New evaluation on unlearning performance along CLU trajectory for CIFAR-10.** We added the experiment result of unlearning approximation error along CLU trajectory for CIFAR-10 in the new FIGURE 10 in page 57 of the revised manuscript. For CIFAR-10, we use a 10-task stream with requests deleting $\\{2\\}$, $\\{5\\}$, and $\\{6,8\\}$ at $t=4,7,9$, respectively. As shown in Figure 10 of the revised manuscript, both Hessian unlearning and forgetting-enhanced Hessian achieve lower final errors than natural forgetting under all three regularizers, with Hessian unlearning attaining the lowest error in each case. These results demonstrate the benefit of explicit Hessian correction for this multi-request setting.Response to Reviwer 2U54 (3/3)
Comparison to non-certified methods We consider non-certified unlearning baselines, including GA, NegGrad+, and an adaptation of the UniCLUN unlearning objective, within our continual learning setting (Chatterjee et al., 2024; Kurmanji et al., 2024; Huang et al., 2025). Under the evaluated configurations, optimizing the objectives of GA and NegGrad+ can cause parameter explosion and numerical divergence, preventing a finite approximation-error comparison with retraining. Although the UniCLUN adaptation remains numerically stable, continued optimization increases its approximation error to approximately 10 while reducing its test accuracy to 8.62% on CIFAR-100. Calibrating additional noise to this empirical error would further perturb an already degraded model, limiting its usefulness as a competitive baseline for our method.
These baselines also rely on more permissive data-access assumptions. GA requires access to the forget data, while NegGrad+ and the UniCLUN adaptation additionally use retained examples through a replay buffer. In contrast, our regularization-based framework assumes that raw data from previously trained tasks, including the task to be deleted, are unavailable when an unlearning request arrives. Consequently, these adapted baselines are not directly comparable under our data-access constraints and, in the evaluated configurations, perform poorly even when granted additional data access.
Certified unlearning framework requires a provable indistinguishability guarantee between the parameter distributions of the unlearned and retrained models. Empirical evaluations, including MIA-based evaluations, provide weaker evidence and do not establish this guarantee. Consequently, strong empirical performance by a non-certified method does not necessarily translate into satisfactory performance when the method is incorporated into a certified unlearning framework. Consequently, the evaluated non-certified baselines generally fail to provide effective unlearning under our certified-unlearning criteria (Mu & Klabjan, 2026).
We have added this discussion in Appendix H.4.
Regarding certified unlearning benchmarks, to the best of our knowledge, this is the first work to study certified unlearning across this family of CL methods. Consequently, no existing certified unlearning method provides a directly comparable baseline in this setting, and we include natural forgetting as an approximate unlearning benchmark, as it serves as a CL counterpart to gradient-based unlearning in offline learning.
We thank you again for your careful review and hope that our responses have addressed your concerns. We would be happy to answer any further questions. If you find our responses satisfactory, we would appreciate your consideration of a higher score.
Response to Reviwer 2U54 (1/3)
Thank you for your valuable suggestions and comments. We address your questions and provide clarifications point by point below.
1. Response on W1 and Q1.
First, we clarify that our main contribution is not natural forgetting but Hessian-based unlearning in Section 5. We use natural forgetting as a simple benchmark for comparison with Hessian-based unlearning and to enhance it, yielding forgetting-enhanced Hessian unlearning in Appendix F.
Natural forgetting indeed has limitations for targeted unlearning and can perform poorly when deleting recent tasks. However, it requires no additional request-time model update, providing a particularly simple unlearning option. Also, since forgetting is an inherent property of continual learning, it is natural and well motivated to investigate how this effect can be exploited to remove deleted-task influence. Moreover, natural forgetting mirrors gradient-based methods in offline unlearning that continue training on retained tasks, supporting the motivation for studying natural forgetting in CL (Koloskova et al., 2025).
Also, we further introduce natural forgetting as a complementary unlearning option, broadening the framework’s applicability across different computational budgets and deletion scenarios. Our theoretical analysis in Theorem 4.1 explicitly characterizes its unlearning performance and provides a basis for choosing between natural forgetting and Hessian-based unlearning by comparing their theoretical guarantees in Theorems 4.1 and 5.2 and their operational costs. Motivated by the simplicity of natural forgetting, we also propose a forgetting-enhanced Hessian unlearning method in Appendix F for choice to balance operational cost and utility.
Our experiments also explicitly illustrate the limitations and practical effectiveness of natural forgetting. Table 1 shows that natural forgetting performs well when deleting early tasks, identifying a regime in which it can be useful. In other regimes, we primarily use Hessian-based unlearning or forgetting-enhanced Hessian unlearning. In Table 2 of the revised manuscript, we also show its benefit in terms of low storage cost. Figures 3–6 on page 10 of the revised manuscript also show that natural forgetting reduces membership leakage while retaining accuracy, although it performs worse than Hessian-based unlearning.
To calibrate noise for natural forgetting, Theorem 4.1 explicitly establishes an upper bound on the approximation error ∣wt−S1:t−wt−S≤t∣ between the natural-forgetting model wt−S1:t and the retrained model. The noise standard deviation is proportional to this proven upper bound, following the standard certified-unlearning procedure introduced in Section 3.2.
2. Response on W2 and Q2.
Following your suggestion, we have significantly expanded our experiments to include additional benchmarks, datasets, and model architectures, as well as MIA-based evaluations. Specifically, we add experiments on CIFAR-10 using a convolutional neural network trained from scratch, comprising three convolutional blocks with 64, 128, and 256 output channels, respectively, and 2,473,610 trainable parameters in total. CIFAR-10 is a commonly used benchmark in continual learning experiments (Xie et al., 2025, Structural-Entropy-Based Sample Selection for Efficient and Effective Learning, ICLR).
New MIA evaluation. we have added MIA-based evaluations in the new Figures 3–6. Specifically, we evaluate membership leakage using a standard modified-entropy MIA and LiRA (Carlini et al., 2022), a state-of-the-art attack that uses shadow models to calibrate a likelihood-ratio test for membership inference. We also tested the evaluation methods proposed in Unlearning Evaluation through Subset Statistical Independence (ICLR 2026), but they did not yield informative results in our setting and are therefore not included. We compare four model states: the checkpoint immediately after learning the task to be deleted, natural forgetting (NF), Hessian unlearning (HU), and complete retraining with the deleted task excluded. Figure 3 shows that HU substantially reduces membership leakage for the deleted task, achieving modified-entropy AUCs of 0.510 and 0.498 on CIFAR-10 and CIFAR-100, respectively, close to retraining (0.509 and 0.501). Figures 4 and 5 show consistent LiRA results: AUC decreases from 0.791 and 0.690 immediately after learning to 0.511 and 0.494 after HU, versus 0.536 and 0.563 for NF. With smaller empirical approximation errors, HU requires less perturbation and achieves higher test accuracies than NF (56.53%/77.00% versus 49.56%/76.06%; Figure 6), approaching retraining performance on CIFAR-100. Although NF provides weaker protection than HU, it still reduces AUC relative to the just-learned checkpoint, demonstrating its effectiveness. Overall, HU provides stronger protection against the evaluated MIAs while preserving higher test accuracy.