- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies machine unlearning in replay-based continual learning, aiming to approximate retraining without the forget set while preserving performance on retained data. It identifies two challenges to applying standard influence functions: the non-stationarity of sequentially trained models and unreliable curvature estimation from limited, task-imbalanced data. The proposed CLIF framework addresses these challenges through three phases: a residual-gradient correction of the influence target, a task-balanced diagonal preconditioner based on within-task gradient variances, and sequential projections that select forget-related update directions and remove components aligned with retained-buffer gradients. The paper provides conditional approximation bounds and evaluates CLIF on CIFAR-10, CIFAR-100, and ImageNet-100 under class-wise and random-sample unlearning. The reported results show improved agreement with retraining and better retention than the compared baselines.
Critical strengths
The paper newly adapt IF unlearning to the rehearsal-based CL.
Critical weaknesses
My main concern is that the paper has very limited methodological novelty, which I think is overstated. The proposed method looks largely like an application of IF-based unlearning to rehearsal-based CL. The authors identify challenges in applying standard IF to CL and organize the response into three phases, but several of the changes seem rather contrived: they distinguish CLIF from standard IF without clearly showing why these particular changes are necessary or how they fundamentally address the sequential nature of CL. The theoretical analysis also does little to establish their significance. My concerns are as follows.
-
In Phase I, the IF unlearning involving is simply an application of second-order approximation. Writing the target as a subtraction involving and adds no new information or computational step. Algorithm 1 computes directly. I therefore find it difficult to regard this algebraic reformulation as a substantial algorithmic contribution.
-
Phase II replaces the Hessian with a damped, task-averaged diagonal statistic based on gradient variance. Phase III projects the update using forget and retain gradient subspaces. These two components appear to be adaptations of existing ideas rather than fundamentally new designs. The latter idea appears equally applicable to standard machine unlearning outside CL. Some of the ablation results show only small performance changes and do not establish why these particular designs are necessary.
-
The theoretical analysis does not characterize the effects of these heuristic choices. This is especially important because Phases II and III determine the actual update. The guarantee in Theorem 4.1 is stated in terms of . However, the size of its first two terms is exactly what we need to know: how much error does the CFI substitution introduce, and how much error does the projection introduce? The paper provides no meaningful bound on either term based on the properties of the proposed designs. The proof mainly applies a Taylor expansion and standard smoothness and error-bound arguments after substituting and . Likewise, the projection analysis bounds its effect using the distance between the projected and unprojected updates without showing that this distance is small. As a result, these theorems seem rather naive relative to the claimed methodological contributions. They restate the errors introduced by the heuristic components instead of explaining or controlling them.
-
``The Forgetting-Retention Dilemma: Certified Unlearning Theory in Continual Learning (ICML 2026)" already studies Hessian-based, second-order unlearning in CL, but the paper does not discuss it. The earlier paper addresses a more complex setting where learning and unlearning requests are interleaved, and the historical datasets requested for deletion need not be available again at unlearning time. CLIF mainly considers one request after CL training and assumes that the forget set is fully available. The advantage of CLIF over the existing work should be clearly discussed.
Questions for authors
See critical weakness.
5 comments
Response to Reviewer sxLj - W2
W2.
Phase II replaces the Hessian with a damped, task-averaged diagonal statistic based on gradient variance. Phase III projects the update using forget and retain gradient subspaces. These two components appear to be adaptations of existing ideas rather than fundamentally new designs. The latter idea appears equally applicable to standard machine unlearning outside CL. Some of the ablation results show only small performance changes and do not establish why these particular designs are necessary.
Response:
The contribution lies in how these operations are constructed and combined to address the limited, task-imbalanced information available after replay-based CL.
Phase II addresses a distinct estimation problem. CFI is not merely a damped diagonal Hessian replacement. It applies two adaptations motivated by CL: (i) within-task gradient centring, which removes the squared empirical-mean contribution under non-stationarity; and (ii) equal task weighting, which avoids allowing the much larger current-task dataset to dominate the scaling statistic. Damping ensures numerical invertibility, while these two operations determine the undamped preconditioning structure.
Phase III addresses historical-task interference. The retain subspace is constructed specifically from the surviving replay buffer, which is the available representation of past tasks. The forget subspace is constructed from the requested deletion samples. While the underlying projection operations can also be applied outside CL, their role here is to balance targeted deletion against interference with historical tasks whose original training data are no longer accessible.
The ablations show more than marginal differences. Table 6 reports substantial changes in several settings:
- Without CFI, CIFAR-100 forget-class test accuracy increases from 0.00% to 15.00%, showing weaker forgetting despite a relatively small change in combined accuracy.
- Without variance-form centring, combined test accuracy decreases from 49.96% to 42.86% on CIFAR-10 and from 29.63% to 20.31% on ImageNet-100.
- Without target projection / retain-overlap removal, it decreases from 49.96% to 38.24% / 39.71% on CIFAR-10 and from 29.63% to 17.48% / 20.25% on ImageNet-100.
These results support CLIF as a practical combination of complementary mechanisms developed for the constraints of replay-based CL, rather than three unrelated modifications to a conventional IF implementation. We have further clarified these design choices in Sections 4.2–4.3, with additional controlled evaluations in Appendix E.12.
Response to Reviewer sxLj - W3
W3.
The theoretical analysis does not characterize ... explaining or controlling them.
Response:
We thank the reviewer for highlighting this. We further refine the analysis to relate these errors to the proposed designs and provide empirical measurements of the resulting residuals.
(1) Scope of Theorem 4.1. Theorem 4.1 identifies two challenges specific to sequential CL: , which captures the discrepancy between accessible and full-history retain gradients, and , which allows the sequentially retrained model to remain non-stationary with respect to the full-history objective. We provide the following refinement to further characterise how CFI and projection affect the final update.
(2) Corollary 4.2: characterising component errors and the actual CLIF update.
Let , , , and . Define and , where is the smallest principal angle between the estimated forget and retain subspaces.
CFI approximation error. Since and , we obtain . Thus, the relevant error is the mismatch between CFI and the retain Hessian along the actual update direction.
Projection error. Using , we obtain . This explicitly separates the effects of forget-subspace truncation and retain-overlap removal.
A tighter bound for the final update. Theorem 4.1 bounds these two residual vectors separately, even though their contributions can cancel. We therefore define . The actual projected step satisfies , and consequently, . Theorem 4.1 therefore remains valid with the tighter residual . This refinement evaluates the residual of the update CLIF actually applies.
(3) Empirical validation of the residual bounds. We additionally measure , , , and the Taylor remainder on the three benchmarks (Table R10). While the intermediate errors can be relatively large, their combined residual is much smaller: is 0.34, 2.11, and 1.60, compared with , 14.07, and 22.70 on CIFAR-100, CIFAR-10, and ImageNet-100, respectively.
These results demonstrate that separately bounding the CFI and projection residuals can overestimate the residual of the final update, while the refinement provides a tighter, empirically informative characterisation.
The refined Corollary 4.2 and its proof are incorporated into Section 4.4 and Appendix C.2, with empirical residual analysis in Appendix E.13.
Table R10: Empirical diagnostics of the CFI and projection residuals.
| Dataset | upper bound | Taylor remainder | |||
|---|---|---|---|---|---|
| CIFAR-100 | |||||
| CIFAR-10 | |||||
| ImageNet-100 |
Response to Reviewer sxLj - W1
We thank the reviewer for the thoughtful assessment of the methodological contributions, theoretical analysis, and related work. We have further clarified CLIF's distinctive contributions and strengthened the revised manuscript with refined theoretical analysis, additional experimental evidence, and a direct comparison with closely related work.
W1.
Response:
Thank you. Machine unlearning for continual learning (CL) models, which is distinct from sequential unlearning on static models, is an under-explored yet critical problem for privacy compliance and post-deployment data correction in lifelong systems. Much like prior studies in influence functions and empirical Fisher approximations (e.g., Bae et al. (2022); Kunstner et al. (2019)), we view the clean algebraic form of Proposition 4.1 as a strength: a simple, exact decomposition that diagnoses why classical IFs catastrophically collapse in CL and resolves that failure with clear empirical gains. Thus, the contribution of Phase I is not the identity gtarget=gR, but the identification and correction of a key source of failure in conventional IF under sequential CL.
(1) Identifying the missing contribution. Conventional deletion IF derives its update under the stationarity assumption gA=0, leaving only the forget-gradient term. However, sequential CL does not generally satisfy this assumption. Proposition 4.1 derives the local correction without assuming stationarity: gtarget=residual-gradient correction∣R∣∣R∣+∣S∣gA−classical IF term∣R∣∣S∣gS=gR. Although the final expression equals gR, the decomposition explicitly identifies the residual-gradient contribution neglected by conventional IF and recovers the classical deletion target when gA=0.
(2) Distinction from conventional IF. Classical IF approximates the effect of removing samples from a stationary training solution, whereas Phase I derives a local correction for a sequentially trained, non-stationary model. Direct optimisation on R is mathematically equivalent to this correction, but does not by itself resolve unlearning in CL: the accessible retain set contains only the current-task data and a small historical buffer, so unconstrained updates may interfere with previously learned knowledge. This motivates additional CL-specific extensions on preconditioning and subspace control in Phases II-III.
(3) Empirical importance. Table 6 directly evaluates the missing residual-gradient contribution by replacing Phase I's target with the classical IF target while keeping the other components unchanged. The combined test accuracy Accat drops from 38.67% to 6.70% on CIFAR-100, and from 49.96% and 29.63% to 0% on CIFAR-10 and ImageNet-100, respectively. This demonstrates that the correction is consequential under CL non-stationarity, despite its simple algebraic form.
Thus, Phase I retains the local second-order interpretation of IF and recovers conventional deletion IF in the stationary case, while adapting its update target to the non-stationary CL setting. We have further clarified this distinction in Section 4.1