acceptodds
Under review as a conference paper at ICLR 2027

CLIF: When Continual Learning Meets Influence-based Forgetting

Abstract

Machine unlearning in continual learning (CL) addresses the growing need to remove parts of a CL model's accumulated knowledge in response to privacy regulations, biases, or noise. Influence functions (IFs) offer an effective, retraining-free route to unlearning in the standard single-task setting, but we show that simply applying IFs to CL models leads to common failure. We identify two core challenges (Cs) as the root causes: (C1) Non-stationarity. Sequential training drives the final CL model far from the stationary point of the joint full-history loss that IFs assume. This leaves a residual gradient that standard IFs ignore, biasing the unlearning target. (C2) Curvature unreliability. IFs rely on the inverse of Hessian to convert removal gradients into parameter updates. But in CL, Hessian is estimated from small, distributionally skewed available data, yielding ill-conditioned inversions that corrupt parameter updates. To overcome both challenges, we propose CLIF (Continual-Learning with Influence-based Forgetting), a novel framework—the first, to our knowledge, to rehabilitate IF-based unlearning for replay-based CL—via three complementary phases. Phase I addresses C1 by introducing a bias-corrected influence target that incorporates residual gradients overlooked by standard IFs. Phase II addresses C2 via a devised Continual Fisher Information preconditioner whose variance-form Fisher provides a full-rank, stably invertible alternative to standard Hessian inverse. Phase III further proposes a forget-subspace projection that confines updates to forget-relevant directions while removing retain-aligned components to reduce performance degradation. Extensive experiments across two unlearning scenarios show that CLIF matches the gold-standard retraining behaviour more closely than standard IFs and state-of-the-art baselines, delivering effective unlearning while retaining strong performance on remaining tasks.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual learning, machine unlearning, influence function

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sxLjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies machine unlearning in replay-based continual learning, aiming to approximate retraining without the forget set while preserving performance on retained data. It identifies two challenges to applying standard influence functions: the non-stationarity of sequentially trained models and unreliable curvature estimation from limited, task-imbalanced data. The proposed CLIF framework addresses these challenges through three phases: a residual-gradient correction of the influence target, a task-balanced diagonal preconditioner based on within-task gradient variances, and sequential projections that select forget-related update directions and remove components aligned with retained-buffer gradients. The paper provides conditional approximation bounds and evaluates CLIF on CIFAR-10, CIFAR-100, and ImageNet-100 under class-wise and random-sample unlearning. The reported results show improved agreement with retraining and better retention than the compared baselines.

Critical strengths

The paper newly adapt IF unlearning to the rehearsal-based CL.

Critical weaknesses

My main concern is that the paper has very limited methodological novelty, which I think is overstated. The proposed method looks largely like an application of IF-based unlearning to rehearsal-based CL. The authors identify challenges in applying standard IF to CL and organize the response into three phases, but several of the changes seem rather contrived: they distinguish CLIF from standard IF without clearly showing why these particular changes are necessary or how they fundamentally address the sequential nature of CL. The theoretical analysis also does little to establish their significance. My concerns are as follows.

  1. In Phase I, the IF unlearning involving gRg_R is simply an application of second-order approximation. Writing the target as a subtraction involving gAg_A and gSg_S adds no new information or computational step. Algorithm 1 computes gRg_R directly. I therefore find it difficult to regard this algebraic reformulation as a substantial algorithmic contribution.

  2. Phase II replaces the Hessian with a damped, task-averaged diagonal statistic based on gradient variance. Phase III projects the update using forget and retain gradient subspaces. These two components appear to be adaptations of existing ideas rather than fundamentally new designs. The latter idea appears equally applicable to standard machine unlearning outside CL. Some of the ablation results show only small performance changes and do not establish why these particular designs are necessary.

  3. The theoretical analysis does not characterize the effects of these heuristic choices. This is especially important because Phases II and III determine the actual update. The guarantee in Theorem 4.1 is stated in terms of Γ\Gamma. However, the size of its first two terms is exactly what we need to know: how much error does the CFI substitution introduce, and how much error does the projection introduce? The paper provides no meaningful bound on either term based on the properties of the proposed designs. The proof mainly applies a Taylor expansion and standard smoothness and error-bound arguments after substituting δI\delta_I and δproj\delta_{\mathrm{proj}}. Likewise, the projection analysis bounds its effect using the distance between the projected and unprojected updates without showing that this distance is small. As a result, these theorems seem rather naive relative to the claimed methodological contributions. They restate the errors introduced by the heuristic components instead of explaining or controlling them.

  4. ``The Forgetting-Retention Dilemma: Certified Unlearning Theory in Continual Learning (ICML 2026)" already studies Hessian-based, second-order unlearning in CL, but the paper does not discuss it. The earlier paper addresses a more complex setting where learning and unlearning requests are interleaved, and the historical datasets requested for deletion need not be available again at unlearning time. CLIF mainly considers one request after CL training and assumes that the forget set is fully available. The advantage of CLIF over the existing work should be clearly discussed.

Questions for authors

See critical weakness.

5 comments
AuthorsOpenReview ↗

Response to Reviewer sxLj - W1

We thank the reviewer for the thoughtful assessment of the methodological contributions, theoretical analysis, and related work. We have further clarified CLIF's distinctive contributions and strengthened the revised manuscript with refined theoretical analysis, additional experimental evidence, and a direct comparison with closely related work.

W1.

In Phase I, the IF unlearning involving $g_R$ is simply an application of second-order approximation. Writing the target as a subtraction involving $g_A$ and $g_S$ adds no new information or computational step. Algorithm 1 computes $g_R$ directly. I therefore find it difficult to regard this algebraic reformulation as a substantial algorithmic contribution.

Response:

Thank you. Machine unlearning for continual learning (CL) models, which is distinct from sequential unlearning on static models, is an under-explored yet critical problem for privacy compliance and post-deployment data correction in lifelong systems. Much like prior studies in influence functions and empirical Fisher approximations (e.g., Bae et al. (2022); Kunstner et al. (2019)), we view the clean algebraic form of Proposition 4.1 as a strength: a simple, exact decomposition that diagnoses why classical IFs catastrophically collapse in CL and resolves that failure with clear empirical gains. Thus, the contribution of Phase I is not the identity gtarget=gR\mathbf{g}_{\mathrm{target}}=\mathbf{g}_\mathcal{R}, but the identification and correction of a key source of failure in conventional IF under sequential CL.

(1) Identifying the missing contribution. Conventional deletion IF derives its update under the stationarity assumption gA=0\mathbf{g}_\mathcal{A}=0, leaving only the forget-gradient term. However, sequential CL does not generally satisfy this assumption. Proposition 4.1 derives the local correction without assuming stationarity: gtarget=∣R∣+∣S∣∣R∣gA⏟residual-gradient correction−∣S∣∣R∣gS⏟classical IF term=gR.\mathbf{g}_{\mathrm{target}}=\underbrace{\frac{|\mathcal{R}|+|\mathcal{S}|}{|\mathcal{R}|}\mathbf{g}_{\mathcal{A}}}_{\text{residual-gradient correction}}-\underbrace{\frac{|\mathcal{S}|}{|\mathcal{R}|}\mathbf{g}_{\mathcal{S}}}_{\text{classical IF term}}=\mathbf{g}_{\mathcal{R}}. Although the final expression equals gR\mathbf{g}_\mathcal{R}, the decomposition explicitly identifies the residual-gradient contribution neglected by conventional IF and recovers the classical deletion target when gA=0\mathbf{g}_\mathcal{A}=0.

(2) Distinction from conventional IF. Classical IF approximates the effect of removing samples from a stationary training solution, whereas Phase I derives a local correction for a sequentially trained, non-stationary model. Direct optimisation on R\mathcal{R} is mathematically equivalent to this correction, but does not by itself resolve unlearning in CL: the accessible retain set contains only the current-task data and a small historical buffer, so unconstrained updates may interfere with previously learned knowledge. This motivates additional CL-specific extensions on preconditioning and subspace control in Phases II-III.

(3) Empirical importance. Table 6 directly evaluates the missing residual-gradient contribution by replacing Phase I's target with the classical IF target while keeping the other components unchanged. The combined test accuracy Accat\mathrm{Acc_{at}} drops from 38.67% to 6.70% on CIFAR-100, and from 49.96% and 29.63% to 0% on CIFAR-10 and ImageNet-100, respectively. This demonstrates that the correction is consequential under CL non-stationarity, despite its simple algebraic form.

Thus, Phase I retains the local second-order interpretation of IF and recovers conventional deletion IF in the stationary case, while adapting its update target to the non-stationary CL setting. We have further clarified this distinction in Section 4.1

AuthorsOpenReview ↗

Response to Reviewer sxLj - W2

W2.

Phase II replaces the Hessian with a damped, task-averaged diagonal statistic based on gradient variance. Phase III projects the update using forget and retain gradient subspaces. These two components appear to be adaptations of existing ideas rather than fundamentally new designs. The latter idea appears equally applicable to standard machine unlearning outside CL. Some of the ablation results show only small performance changes and do not establish why these particular designs are necessary.

Response:

The contribution lies in how these operations are constructed and combined to address the limited, task-imbalanced information available after replay-based CL.

Phase II addresses a distinct estimation problem. CFI is not merely a damped diagonal Hessian replacement. It applies two adaptations motivated by CL: (i) within-task gradient centring, which removes the squared empirical-mean contribution under non-stationarity; and (ii) equal task weighting, which avoids allowing the much larger current-task dataset to dominate the scaling statistic. Damping ensures numerical invertibility, while these two operations determine the undamped preconditioning structure.

Phase III addresses historical-task interference. The retain subspace is constructed specifically from the surviving replay buffer, which is the available representation of past tasks. The forget subspace is constructed from the requested deletion samples. While the underlying projection operations can also be applied outside CL, their role here is to balance targeted deletion against interference with historical tasks whose original training data are no longer accessible.

The ablations show more than marginal differences. Table 6 reports substantial changes in several settings:

  • Without CFI, CIFAR-100 forget-class test accuracy increases from 0.00% to 15.00%, showing weaker forgetting despite a relatively small change in combined accuracy.
  • Without variance-form centring, combined test accuracy decreases from 49.96% to 42.86% on CIFAR-10 and from 29.63% to 20.31% on ImageNet-100.
  • Without target projection / retain-overlap removal, it decreases from 49.96% to 38.24% / 39.71% on CIFAR-10 and from 29.63% to 17.48% / 20.25% on ImageNet-100.

These results support CLIF as a practical combination of complementary mechanisms developed for the constraints of replay-based CL, rather than three unrelated modifications to a conventional IF implementation. We have further clarified these design choices in Sections 4.2–4.3, with additional controlled evaluations in Appendix E.12.

AuthorsOpenReview ↗

Response to Reviewer sxLj - W3

W3.

The theoretical analysis does not characterize ... explaining or controlling them.

Response:

We thank the reviewer for highlighting this. We further refine the analysis to relate these errors to the proposed designs and provide empirical measurements of the resulting residuals.

(1) Scope of Theorem 4.1. Theorem 4.1 identifies two challenges specific to sequential CL: ϵB\epsilon_B, which captures the discrepancy between accessible and full-history retain gradients, and ρCL\rho_{\rm CL}, which allows the sequentially retrained model to remain non-stationary with respect to the full-history objective. We provide the following refinement to further characterise how CFI and projection affect the final update.

(2) Corollary 4.2: characterising component errors and the actual CLIF update.

Let H=HR\mathbf{H}=\mathbf{H}_{\mathcal{R}}, M=F^CFI+λI\mathbf{M}=\widehat{\mathbf{F}}_{\mathrm{CFI}}+\lambda\mathbf{I}, δI=M−1gR\boldsymbol{\delta}_I=\mathbf{M}^{-1}\mathbf{g}_{\mathcal{R}}, and δproj=(I−PB′)PSδI\boldsymbol{\delta}_{\mathrm{proj}}=(\mathbf{I}-\mathbf{P}_{\mathcal{B}'})\mathbf{P}_{\mathcal{S}}\boldsymbol{\delta}_I. Define c=∥PSδI∥2∥δI∥2c=\frac{\Vert\mathbf{P}_{\mathcal{S}}\boldsymbol{\delta}_I\Vert_2}{\Vert\boldsymbol{\delta}_I\Vert_2} and ω=∥PB′PSδI∥2∥PSδI∥2≤cos⁡θ1\omega=\frac{\Vert\mathbf{P}_{\mathcal{B}'}\mathbf{P}_{\mathcal{S}}\boldsymbol{\delta}_I\Vert_2}{\Vert\mathbf{P}_{\mathcal{S}}\boldsymbol{\delta}_I\Vert_2}\leq\cos\theta_1, where θ1\theta_1 is the smallest principal angle between the estimated forget and retain subspaces.

CFI approximation error. Since gR=MδI\mathbf{g}_{\mathcal{R}}=\mathbf{M}\boldsymbol{\delta}_I and M⪰λI\mathbf{M}\succeq\lambda\mathbf{I}, we obtain rF=∥(H−M)δI∥2≤∥H−M∥2λ∥gR∥2r_F=\Vert(\mathbf{H}-\mathbf{M})\boldsymbol{\delta}_I\Vert_2\leq\frac{\Vert\mathbf{H}-\mathbf{M}\Vert_2}{\lambda}\Vert\mathbf{g}_{\mathcal{R}}\Vert_2. Thus, the relevant error is the mismatch between CFI and the retain Hessian along the actual update direction.

Projection error. Using δI−δproj=(I−PS)δI+PB′PSδI\boldsymbol{\delta}_I-\boldsymbol{\delta}_{\mathrm{proj}}=(\mathbf{I}-\mathbf{P}_{\mathcal{S}})\boldsymbol{\delta}_I+\mathbf{P}_{\mathcal{B}'}\mathbf{P}_{\mathcal{S}}\boldsymbol{\delta}_I, we obtain rP≤∥H∥2λ(1−c2+cω)∥gR∥2r_P\leq\frac{\|\mathbf{H}\|_2}{\lambda}\big(\sqrt{1-c^2}+c\omega\big)\Vert\mathbf{g}_{\mathcal{R}}\Vert_2. This explicitly separates the effects of forget-subspace truncation and retain-overlap removal.

A tighter bound for the final update. Theorem 4.1 bounds these two residual vectors separately, even though their contributions can cancel. We therefore define rFP=∥gR−Hδproj∥2≤rF+rPr_{FP}=\Vert\mathbf{g}_{\mathcal{R}}-\mathbf{H}\boldsymbol{\delta}_{\mathrm{proj}}\Vert_2\leq r_F+r_P. The actual projected step satisfies ∥δproj∥2=c1−ω2∥δI∥2≤c1−ω2λ∥gR∥2\Vert\boldsymbol{\delta}_{\mathrm{proj}}\Vert_2=c\sqrt{1-\omega^2}\Vert\boldsymbol{\delta}_I\Vert_2\leq\frac{c\sqrt{1-\omega^2}}{\lambda}\Vert\mathbf{g}_{\mathcal{R}}\Vert_2, and consequently, rFP≤(1+c1−ω2∥H∥2λ)∥gR∥2r_{FP}\leq\left(1+\frac{c\sqrt{1-\omega^2}\Vert\mathbf{H}\Vert_2}{\lambda}\right)\Vert\mathbf{g}_{\mathcal{R}}\Vert_2. Theorem 4.1 therefore remains valid with the tighter residual Γproj=rFP+LH2∥δproj∥22≤Γ\Gamma_{\mathrm{proj}}=r_{FP}+\frac{L_H}{2}\Vert\boldsymbol{\delta}_{\mathrm{proj}}\Vert_2^2\leq\Gamma. This refinement evaluates the residual of the update CLIF actually applies.

(3) Empirical validation of the residual bounds. We additionally measure rFr_F, rPr_P, rFPr_{FP}, and the Taylor remainder on the three benchmarks (Table R10). While the intermediate errors can be relatively large, their combined residual is much smaller: rFPr_{FP} is 0.34, 2.11, and 1.60, compared with rF+rP=5.19r_F+r_P=5.19, 14.07, and 22.70 on CIFAR-100, CIFAR-10, and ImageNet-100, respectively.

These results demonstrate that separately bounding the CFI and projection residuals can overestimate the residual of the final update, while the refinement provides a tighter, empirically informative characterisation.

The refined Corollary 4.2 and its proof are incorporated into Section 4.4 and Appendix C.2, with empirical residual analysis in Appendix E.13.

Table R10: Empirical diagnostics of the CFI and projection residuals.

DatasetrFr_FrPr_PrFPr_{FP}rFPr_{FP} upper boundTaylor remainder
CIFAR-1002.50±0.432.50\pm0.432.69±0.842.69\pm0.840.34±0.31\mathbf{0.34\pm0.31}1.88±1.681.88\pm1.680.43±0.520.43\pm0.52
CIFAR-107.40±2.797.40\pm2.796.67±3.526.67\pm3.522.11±1.03\mathbf{2.11\pm1.03}12.62±6.8712.62\pm6.870.62±0.570.62\pm0.57
ImageNet-10011.70±6.3611.70\pm6.3611.00±7.7811.00\pm7.781.60±0.22\mathbf{1.60\pm0.22}5.20±0.425.20\pm0.421.22±0.211.22\pm0.21
Reviewer_eNuBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CLIF to improve machine forgetting with influence functions in replay-based continual learning. Experiments on three benchmarks demonstrate its effectiveness at both class and sample levels.

Critical strengths
  1. The paper addresses a practical problem in continual learning and provides a reasonable motivation for improving traditional influence functions.

  2. The experiments are comprehensive, and CLIF achieves good results across different forgetting scenarios with lower computational costs.

Critical weaknesses
  1. Phase I appears to be equivalent to a standard gradient update using the available retain data, and its advantage over retraining with all historical data remains unclear. CFI may also suffer from rank deficiency, while the benefits of damping and forgetting-subspace projection need further explanation.

  2. CLIF shows little improvement on CIFAR-100 and performs worse than retraining on CIFAR-10, with a larger accuracy gap and lower overall test performance. The different damping settings between IF and CLIF also make it difficult to rule out the effects of hyperparameters and numerical instability.

Questions for authors
  1. Can the authors further explain the essential difference between the bias correction objective of Phase I and the direct use of accessible reserved set gradients, and quantify the error between this objective and the complete historical data retraining objective?

  2. Can the authors explain why CLIF performs worse in terms of forgotten sample accuracy and overall test accuracy than the original model on CIFAR-10? It would also be helpful to include a fully tuned IF baseline and controlled experiments to evaluate the contributions of CFI and subspace projection.

4 comments
AuthorsOpenReview ↗

Response to Reviewer eNuB - W1 & Q1 (1/2)

We thank the reviewer for the detailed and constructive feedback. We have clarified the theoretical properties of CLIF and supplemented the revised manuscript with additional analyses on the approximation gap, numerical stability, and contributions of components.

W1 & Q1.

The bias correction ... retraining objective?

Response:

Thank you. Machine unlearning for continual learning (CL) models, which is distinct from sequential unlearning on static models, is an under-explored yet critical problem for privacy compliance and data correction in lifelong systems. Much like prior studies in influence functions and empirical Fisher approximations (e.g., Bae et al.; Kunstner et al.), we view the clean algebraic form of Proposition 4.1 as a strength: a simple, exact decomposition that diagnoses why classical IFs catastrophically collapse in CL and resolves that failure with clear empirical gains. We clarify the role and precise properties of each component, and provide additional quantitative evidence on the approximation gap between the accessible retain objective and full-history retraining.

(1) Role of Phase I. Phase I is mathematically equivalent to a Newton step on the accessible retain objective. However, its contribution lies in identifying the missing residual-gradient correction in conventional IF under CL non-stationarity, rather than introducing a new optimisation operation. Specifically, gtarget=∣R∣+∣S∣∣R∣gA⏟residual-gradient correction−∣S∣∣R∣gS⏟classical IF term=gR.\mathbf{g}_{\mathrm{target}}=\underbrace{\frac{|\mathcal{R}|+|\mathcal{S}|}{|\mathcal{R}|}\mathbf{g}_{\mathcal{A}}}_{\text{residual-gradient correction}}-\underbrace{\frac{|\mathcal{S}|}{|\mathcal{R}|}\mathbf{g}_{\mathcal{S}}}_{\text{classical IF term}}=\mathbf{g}_{\mathcal{R}}. Although the final expression equals gR\mathbf{g}_{\mathcal R}, the decomposition explicitly identifies the contribution omitted by conventional deletion IF, which assumes gA=0\mathbf{g}_{\mathcal A}=0, and recovers the classical IF target in the stationary case. Moreover, Table 6 demonstrates its empirical significance. Replacing Phase I's target with the conventional IF target, while retaining the other components, reduces the combined test accuracy from 38.67% to 6.70% on CIFAR-100.

(2) Quantifying the gap to full-history retraining. We further examine the approximation gap from three complementary perspectives, distinguishing the historical-data access limitation inherent to replay-based CL from the final discrepancy with sequential Retrain.

(i) Accessible-data gradient discrepancy. We measure the discrepancy between the accessible retain gradient and the full-history retain gradient, and compare it with the corresponding discrepancy at the sequential Retrain checkpoint: dI=∥∇LR(θT)−∇LE(θT)∥2d_{\mathrm{I}} = \Vert\nabla\mathcal{L}_{\mathcal{R}}(\theta_T) - \nabla\mathcal{L}_{\mathcal{E}}(\theta_T)\Vert_2, and drt=∥∇LRrt(θrt)−∇LE(θrt)∥2d_{\mathrm{rt}} = \Vert\nabla\mathcal{L}_{\mathcal{R}^{\mathrm{rt}}}(\theta^{\mathrm{rt}}) - \nabla\mathcal{L}_{\mathcal{E}}(\theta^{\mathrm{rt}})\Vert_2.

As shown in Table R5, the discrepancies are comparable across all three datasets (e.g., 45.76 versus 48.11 on CIFAR-100). Notably, a similar discrepancy remains at the Retrain checkpoint, despite retraining with the full historical data through the sequential CL pipeline. This indicates that the accessible-data mismatch is not introduced by CLIF, but is also present under the same replay-based CL setting. This is why Theorem 4.1 explicitly separates the buffer-proxy approximation term ϵB\epsilon_B from the curvature and projection approximation errors.

(ii) Full-history objective fidelity. To evaluate the final unlearning result against sequential Retrain, we measure ∣LE(θ^)−LE(θrt)∣|\mathcal L_{\mathcal E}(\hat\theta)-\mathcal L_{\mathcal E}(\theta^\mathrm{rt})|. CLIF achieves the lowest reported mean gap on CIFAR-100 and ImageNet-100, and a competitive result on CIFAR-10 (1.10, compared with 0.94 for GFOES).

(iii) Predictive fidelity. We further measure the Jensen-Shannon divergence between CLIF's and Retrain's predictions on E\mathcal E. CLIF achieves the lowest divergence among all evaluated methods on all three datasets.

Together, these results show that the limited-history gradient discrepancy persists even for sequential Retrain, while CLIF nevertheless achieves competitive full-history loss fidelity and close predictive alignment with Retrain. Detailed baseline comparisons are provided in Appendix E.10.

Table R5: Accessible-data gradient discrepancy and fidelity to Retrain (↓\downarrow).

DatasetdId_{\mathrm{I}}drtd_{\mathrm{rt}}∣LE(θ^)−LE(θrt)∣\vert\mathcal L_{\mathcal E}(\hat\theta)-\mathcal L_{\mathcal E}(\theta^\mathrm{rt})\vertJSD (×100\times 100)
CIFAR-10045.76±0.9945.76\pm0.9948.11±3.5748.11\pm3.570.73±0.630.73\pm0.6322.61±2.4422.61\pm2.44
CIFAR-1041.35±6.3541.35\pm6.3544.47±5.7344.47\pm5.731.10±1.131.10\pm1.1324.31±4.4124.31\pm4.41
ImageNet-10041.85±1.0841.85\pm1.0843.54±3.3043.54\pm3.300.68±0.230.68\pm0.2332.45±3.6632.45\pm3.66
AuthorsOpenReview ↗

Response to Reviewer eNuB - W1 & Q1 (2/2)

(3) CFI rank and numerical invertibility. We clarify that \textbf{the variance-form CFI is positive semidefinite} but not necessarily full-rank. Strict positive definiteness of the preconditioner follows from λ>0\lambda>0. CFI's contribution beyond damping is its CL-specific estimation: within-task gradient centring removes the squared empirical-mean contribution, while task-balanced aggregation reduces domination by the current-task data. These two adaptations determine the preconditioning statistic under CL non-stationarity and limited historical data. We have corrected the full-rank wording in the Abstract and Section 4.2.

(4) Precise projection property. We clarify that Phase III performs \textbf{forget-supported update selection followed by retain-overlap removal}, without claiming strict containment of the final update in the forget-gradient subspace. Specifically, δproj=(I−PB′)PSδI.\delta_{\mathrm{proj}}=(\mathbf{I}-\mathbf{P}_{\mathcal B'})\mathbf{P}_{\mathcal S}\delta_I. Thus, PB′δproj=0\mathbf{P}_{\mathcal B'}\delta_{\mathrm{proj}}=0, ensuring orthogonality to the estimated retain-buffer subspace. Because the two projectors need not commute, δproj\delta_{\mathrm{proj}} need not lie entirely in the forget subspace. The deviation from that subspace is bounded by ∥(I−PS)δproj∥2≤∥PB′PSδI∥2\Vert(\mathbf{I}-\mathbf{P}_{\mathcal{S}})\delta_{\mathrm{proj}}\Vert_2 \leq \Vert\mathbf{P}_{\mathcal{B}'}\mathbf{P}_{\mathcal{S}}\delta_I\Vert_2, which directly relates it to the removed overlap component. We have refined Section 4.3 to state these precise properties.

AuthorsOpenReview ↗

Response to Reviewer eNuB - W2 & Q2 (1/2)

W2 & Q2.

In the random sample forgetting experiment, ... numerical stability on baseline collapse.

Can the authors explain ... CFI and subspace projection?

Response: We clarify the interpretation of random-sample unlearning and provide additional experiments on IF damping, update scaling, and CFI and projection components.

(1) Interpreting random-sample unlearning. Unlike class-wise deletion, random-sample deletion removes individual examples across classes while retaining many similar examples. Consequently, Retrain may still correctly classify deleted samples: on CIFAR-10, its forget-set accuracy is 47.63% (Table 2). Thus, lower Accf\mathrm{Acc}_f does not necessarily indicate better unlearning, and fidelity to Retrain is the relevant criterion.

This also explains why Origin can appear close to Retrain without performing unlearning. On CIFAR-10, Retrain changes mean Accf\mathrm{Acc}_f only from 50.50% (Origin) to 47.63%, and overall test accuracy from 47.99% to 47.47%. In contrast, CLIF, similar to other unlearning methods, tends to over-unlearn and reduces Accf\mathrm{Acc}_f to 28.87%, leading to a larger run-wise deviation from Retrain (ΔAccf=18.77\Delta\mathrm{Acc}_f=18.77 versus 8.40 for Origin) and lower overall test accuracy (28.40%).

Nevertheless, CLIF achieves the smallest ΔAccf\Delta\mathrm{Acc}_f among the evaluated unlearning methods on all three datasets. On CIFAR-10, its deviation of 18.77 is lower than the competing unlearning methods (23.87-38.00), while maintaining the highest retain accuracy among them. These results highlight the difficulty of matching the relatively small changes induced by random-sample retraining without disrupting retained knowledge.

(2) Additional predictive-fidelity evaluation. To complement aggregate accuracy, we measure the Jensen--Shannon divergence (JSD) between each method's predictions and those of Retrain. Table R6 shows that CLIF achieves the lowest mean JSD among the evaluated methods on all three datasets. On CIFAR-10, CLIF obtains 26.35, compared with 26.90 for Origin and 34.24 for the closest competing unlearning method (GFOES), with JSD values scaled by 100. These results provide complementary evidence that CLIF maintains comparatively close predictive-distribution alignment with Retrain.

Table R6: Predictive JSD to Retrain (×100\times 100, ↓\downarrow) in random-sample unlearning.

DatasetOriginBest other unlearning methodCLIF
CIFAR-10023.54±4.6923.54\pm4.6934.68±4.9734.68\pm4.97 (GA)22.13±2.06\mathbf{22.13\pm2.06}
CIFAR-1026.90±2.1626.90\pm2.1634.24±6.0134.24\pm6.01 (GFOES)26.35±1.03\mathbf{26.35\pm1.03}
ImageNet-10033.17±0.9133.17\pm0.9137.22±1.3937.22\pm1.39 (GFOES)32.90±0.69\mathbf{32.90\pm0.69}

(3) Sensitivity of conventional IF to damping. To examine whether the standard IF baseline's failure is attributable to its damping configuration, we vary λ∈10−3,10−2,10−1,1,10\lambda\in\\{10^{-3},10^{-2},10^{-1},1,10\\} while keeping the other baseline settings fixed. As shown in Table R7, IF's retained-class test accuracy remains severely degraded across all tested values: 0.61-1.12% on CIFAR-100, 8.89-12.02% on CIFAR-10, and 0.81-1.10% on ImageNet-100. Although some settings achieve near-zero forget-class accuracy, this comes with substantial damage to retained-class performance. These results demonstrate that the observed IF failure is not resolved by tuning damping alone over the tested range.

Table R7: Standard IF under different damping values.

DatasetMetric0.0010.010.1110
CIFAR-100Accft\mathrm{Acc_{ft}}40.00±54.7740.00 \pm 54.7733.33±57.7433.33 \pm 57.7420.00±44.7220.00 \pm 44.720.00±0.000.00 \pm 0.000.00±0.000.00 \pm 0.00
Accrt\mathrm{Acc_{rt}}0.61±0.550.61 \pm 0.550.67±0.580.67 \pm 0.580.81±0.450.81 \pm 0.451.01±0.001.01 \pm 0.001.12±0.251.12 \pm 0.25
CIFAR-10Accft\mathrm{Acc_{ft}}0.00±0.000.00 \pm 0.0020.00±44.7220.00 \pm 44.720.00±0.000.00 \pm 0.000.00±0.000.00 \pm 0.000.00±0.000.00 \pm 0.00
Accrt\mathrm{Acc_{rt}}12.02±2.0412.02 \pm 2.048.89±4.978.89 \pm 4.9711.93±1.9211.93 \pm 1.9211.18±0.1511.18 \pm 0.1511.11±0.0011.11 \pm 0.00
ImageNet-100Accft\mathrm{Acc_{ft}}20.00±44.7220.00 \pm 44.720.00±0.000.00 \pm 0.0020.00±44.7220.00 \pm 44.720.00±0.000.00 \pm 0.000.00±0.000.00 \pm 0.00
Accrt\mathrm{Acc_{rt}}0.81±0.450.81 \pm 0.451.01±0.001.01 \pm 0.000.81±0.450.81 \pm 0.451.01±0.001.01 \pm 0.001.10±0.211.10 \pm 0.21
AuthorsOpenReview ↗

Response to Reviewer eNuB - W2 & Q2 (2/2)

(4) Controlled validation of CFI and projection. We provide additional controlled experiments to disentangle the effects of CFI preconditioning, update scaling, and subspace projection.

CFI preconditioning. Holding the Phase I target and Phase III projection fixed, the component ablations in Table 6 isolate CFI's variance-form Fisher (VFF) and task-balanced estimation (TBE). Removing VFF or TBE reduces Accat\mathrm{Acc_{at}} from 49.96% to 42.86% and 46.87% on CIFAR-10, and from 29.63% to 20.31% and 20.62% on ImageNet-100, respectively. These results demonstrate the contributions of CFI's CL-specific estimation components.

Update scaling. To further examine the CFI-preconditioned update, we fix the preconditioner and projection and vary the update scale η∈{0,0.1,0.25,0.5,1}\eta\in\{0,0.1,0.25,0.5,1\} in θT−ηδproj\theta_T-\eta\delta_{\mathrm{proj}}. Tables R8-R9 show that update magnitude affects the forgetting-retention trade-off. In class-wise unlearning, increasing η\eta consistently strengthens forgetting; for example, CIFAR-10 Accf\mathrm{Acc_f} decreases from 14.98% (η=0.1\eta=0.1) to 0.10% (η=1\eta=1), while Accr\mathrm{Acc_r} remains above 90%. Random-sample unlearning exhibits a more complex trade-off. On CIFAR-10, the full update (η=1\eta=1) tends to over-unlearn, reducing Accf\mathrm{Acc_f} from 50.50% (Origin) to 28.87%, below Retrain's 47.63%, and yielding ΔAccf=18.77\Delta\mathrm{Acc_f}=18.77. Reducing η\eta to 0.25 improves ΔAccf\Delta\mathrm{Acc_f} to 10.00, but decreases Accr\mathrm{Acc_r} from 79.59% to 69.70%. This shows the nontrivial trade-off between retraining fidelity and knowledge retention in random-sample unlearning.

Subspace projection. We compare the unprojected CFI update δI\delta_I, forget-only projection PSδI\mathbf{P}_\mathcal{S}\delta_I, and complete Phase III update (I−PB′)PSδI(\mathbf{I}-\mathbf{P}_{\mathcal{B}'})\mathbf{P}_\mathcal{S}\delta_I. From Tables R8-R9, the complete projection substantially improves retention over the unprojected update. For example, Accr\mathrm{Acc_r} increases from 65.84% to 90.87% on CIFAR-10, while maintaining low forget accuracy. The forget-only comparison further isolates the contribution of retain-overlap removal.

Together, these controlled evaluations distinguish the contributions of CFI estimation and subspace projection, providing additional evidence that the proposed components play complementary roles in balancing forgetting and retention. These additional evaluations are incorporated into Section 5.3 and Appendices E.11–E.12.

We appreciate the reviewer's valuable suggestions and hope the additional analyses and experimental results provide useful clarification. We welcome any further questions or discussion.

Table R8: Update scaling and projection in class-wise unlearning: Accf↓\mathrm{Acc_f}\downarrow / Accr↑\mathrm{Acc_r}\uparrow (%).

UpdateCIFAR-100CIFAR-10ImageNet-100
Origin (η=0\eta=0)9.33 / 98.8828.15 / 83.639.09 / 99.21
CLIF, η=0.1\eta=0.14.88 / 98.2414.98 / 92.327.54 / 97.93
CLIF, η=0.25\eta=0.253.40 / 98.038.99 / 93.625.06 / 96.60
CLIF, η=0.5\eta=0.51.53 / 97.832.95 / 90.862.54 / 93.86
CLIF, η=1\eta=10.13 / 97.890.10 / 90.870.41 / 91.54
Pre-projection δI\delta_I0.60 / 55.820.12 / 65.840.20 / 45.81
Forget-only PSδI\mathbf{P}_\mathcal{S}\delta_I0.07 / 94.550.00 / 62.770.00 / 56.66
Complete Phase III0.13 / 97.890.10 / 90.870.41 / 91.54

Table R9: Update scaling and projection in random-sample unlearning: \Delta\mathrm{Acc_f}\downarrow$ / Accr↑\mathrm{Acc_r}\uparrow (%).

UpdateCIFAR-100CIFAR-10ImageNet-100
Origin (η=0\eta=0)3.2 / 98.928.40 / 82.574.67 / 99.17
CLIF, η=0.1\eta=0.15.5 / 97.1614.3 / 71.223.38 / 96.21
CLIF, η=0.25\eta=0.255.1 / 97.1110.0 / 69.72.94 / 95.30
CLIF, η=0.5\eta=0.54.2 / 93.9817.6 / 74.352.48 / 90.36
CLIF, η=1\eta=13.17 / 89.9418.77 / 79.590.27 / 82.55
Pre-projection δI\delta_I9.70 / 47.8238.2 / 48.368.10 / 27.26
Forget-only PSδI\mathbf{P}_\mathcal{S}\delta_I4.50 / 82.1127.8 / 52.237.00 / 49.59
Complete Phase III3.17 / 89.9418.77 / 79.590.27 / 82.55

References

Juhan Bae, et al. If influence functions are the answer, then what is the question? In NeurIPS, 2022.

Frederik Kunstner, et al. Limitations of the empirical fisher approximation for natural gradient descent. In NeurIPS, 2019.

Reviewer_d6GTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CLIF, a framework for machine unlearning in replay-based continual learning. It identifies non-stationarity and unreliable curvature estimation as challenges for applying standard IFs in this setting. CLIF addresses these issues through a bias-corrected influence target, a task-balanced variance-based Fisher preconditioner, and a forget-subspace projection intended to reduce interference with retained knowledge. Experiments on CIFAR-100, CIFAR-10, and ImageNet-100 evaluate class-wise and random-sample unlearning, with additional ablations and evaluations across different backbones. The results suggest that CLIF improves the trade-off between forgetting effectiveness, retention performance, and computational cost.

Critical strengths

The paper provides useful insights into why standard influence functions can fail in replay-based continual learning, particularly due to non-stationarity and unreliable curvature estimation.

Critical weaknesses
  1. The contribution of Phase III to the forgetting process remains insufficiently characterized. In Eq.(7), the retain-subspace projection removes components aligned with buffer gradients, but these components may also contain directions necessary for forgetting when the forget and retain gradient subspaces overlap. Although Tables 3 and 6 provide ablation results for target projection and retain-overlap removal, they do not directly examine how much of the forgetting update is removed or how subspace overlap affects forgetting effectiveness. This makes it difficult to fully assess the trade-off introduced by Phase III.

  2. The evaluation does not systematically examine how the amount of retained knowledge affects unlearning effectiveness. In Table 1, the original ResNet-18 models achieve only 11.33% and 7.33% test accuracy on the target classes for CIFAR-100 and ImageNet-100, respectively. Although Table 8 provides additional results with higher initial accuracy under ViT-Tiny, the paper does not analyze how the initial accuracy on the target class relates to unlearning effectiveness. A more controlled evaluation across target classes or training stages would help establish whether CLIF remains effective when substantial knowledge of the target class is preserved.

Questions for authors
  1. In Proposition 4.1, the Bias-Corrected Influence Target is algebraically equivalent to gRg_R, the gradient of the accessible retain loss. Appendix C.1 also derives the update directly from a second-order approximation of LRL_R. Could the authors clarify what distinguishes Phase I from a conventional preconditioned optimization step on the retain data, and which properties of influence functions are specifically preserved or extended by this formulation?

  2. As noted in the critical weaknesses, the original ResNet-18 models show relatively low accuracy on the target classes before unlearning. Could the authors further investigate how CLIF's effectiveness varies with the amount of knowledge retained before unlearning? In particular, supplementing the experiments in Appendix E.7 with the initial target-class accuracy and the corresponding unlearning performance at each training stage would help clarify whether CLIF remains effective when historical knowledge is better preserved.

5 comments
AuthorsOpenReview ↗

Response to Reviewer d6GT - W1

We thank the reviewer for the careful reading and constructive questions. We have further clarified the methodological contributions and expanded the experimental evaluations in the revised manuscript, particularly regarding Phase III and the effectiveness of unlearning under different levels of initial target-class knowledge.

W1.

The contribution of Phase III to the forgetting process remains insufficiently characterized. In Eq.(7), the retain-subspace projection removes components aligned with buffer gradients, but these components may also contain directions necessary for forgetting when the forget and retain gradient subspaces overlap. Although Tables 3 and 6 provide ablation results for target projection and retain-overlap removal, they do not directly examine how much of the forgetting update is removed or how subspace overlap affects forgetting effectiveness. This makes it difficult to fully assess the trade-off introduced by Phase III.

Response:

We thank the reviewer for this question. Phase III aims to suppress interference with retained knowledge while preserving forgetting-relevant updates. We clarify its rationale and quantify the resulting trade-off below.

Rationale and existing evidence. A direction shared by forget and retain gradients can affect both objectives: while updating along it may facilitate forgetting, it can also disturb knowledge that should be retained. Phase III therefore first selects forget-supported directions (TP) and then removes their components aligned with retained-buffer gradients (ROR), reducing potential interference. This follows the retention-preserving principle of gradient orthogonality used in continual learning, e.g., OGD (Farajtabar et al., 2020) and GPM (Saha et al., 2021), while adapting it to unlearning.

Furthermore, the empirical cost to forgetting is small. Table 6 compares TP alone (w/o ROR) with TP+ROR. Adding ROR increases Accf\mathrm{Acc_f} by only 0.13, 0.10, and 0.41 percentage points on CIFAR-100, CIFAR-10, and ImageNet-100, respectively, while Accft\mathrm{Acc_{ft}} remains 0 on all three datasets. Meanwhile, the combined test accuracy Accat\mathrm{Acc_{at}} improves by 1.36, 10.25, and 9.38 points, respectively. Thus, ROR substantially improves the forgetting-retention balance with little observed loss in forgetting effectiveness.

New quantitative evidence. To directly examine the potential forgetting cost, we analyse the overlap between the estimated forget and retain gradient subspaces. Let PS=QSQS⊤\mathbf{P}_{\mathcal S}=\mathbf{Q}_{\mathcal S}\mathbf{Q}_{\mathcal S}^{\top} and PB′=QB′QB′⊤\mathbf{P}_{\mathcal B'}=\mathbf{Q}_{\mathcal B'}\mathbf{Q}_{\mathcal B'}^{\top}. Denote the smallest principal angle between their spans by θ1\theta_1. As shown in Table R1, the forget and retained buffer spaces are nearly orthogonal (θ1=78.6∘\theta_1=78.6^\circ) on CIFAR-100, and cos⁡θ1≤0.273\cos\theta_1\leq0.273 across all three datasets, indicating limited overlap between the selected subspaces. More importantly, the measured forget-loss increase ΔLS\Delta\mathcal L_{\mathcal S} under TP+ROR remains at least 95% of that under TP alone. These results directly support that, in the evaluated settings, ROR suppresses retain-aligned interference while sacrificing little forgetting effect. We have incorporated this subspace-overlap analysis and the additional results into Appendix E.9.

Table R1: Subspace overlap and the impact of ROR on forget-set loss.

Datasetcos⁡θ1\cos\theta_1∥PSδproj∥/∥PSδI∥\Vert\mathbf{P}_\mathcal{S}\delta_{\rm proj}\Vert/\Vert\mathbf{P}_\mathcal{S}\delta_I\VertΔLS\Delta\mathcal{L}_\mathcal{S} (TP only)ΔLS\Delta\mathcal{L}_\mathcal{S} (TP+ROR)
CIFAR-1000.1970.97314.3313.92
CIFAR-100.2730.9893.433.56
ImageNet-1000.1470.9799.348.88
AuthorsOpenReview ↗

Response to Reviewer d6GT - W2 & Q2

W2 & Q2.

The evaluation does not systematically examine how the amount of retained knowledge affects unlearning effectiveness. ... A more controlled evaluation across target classes or training stages would help establish whether CLIF remains effective when substantial knowledge of the target class is preserved.

As noted in the critical weaknesses ... whether CLIF remains effective when historical knowledge is better preserved.

Response:

Thank you for the constructive suggestion. The relatively low initial target-class accuracy in Table 1 reflects the intrinsic historical-class degradation in the replay-based CL backbone. However, CLIF's forgetting effectiveness is not limited to these low-accuracy cases. We clarify the existing evidence and provide two complementary controlled evaluations below.

Existing evidence of stronger initial target-class accuracy. As noted by the reviewer, Table 8 already evaluates CLIF with ViT-Tiny, where the original target-class test accuracy is approximately 2-3$\times$ higher than with ResNet-18. As summarised in Table R2, CLIF still reduces Accft\mathrm{Acc}_{ft} to 0.13--2.00%, while its combined test accuracy Accat\mathrm{Acc}_{at} aligns closely with Retrain. This demonstrates effective forgetting even when the original model has substantially higher target-class accuracy.

Table R2: Existing results with higher initial target-class accuracy.

Setting (source)Origin Acc$_\mathrm{ft}$CLIF Acc$_\mathrm{ft}$CLIF Acc$_\mathrm{at}$Retrain Acc$_\mathrm{at}$
ViT-Tiny, CIFAR-1062.120.1360.0561.17
ViT-Tiny, CIFAR-10026.331.3333.2936.00
ViT-Tiny, ImageNet-10021.202.0028.7332.24

More controlled evaluation varying target classes. To directly assess the effect of initial target-class accuracy, we fix the final CIFAR-100 CL model and separately unlearn a class from tasks 1, 4, 7, and 10. Table R3 shows that the original target-class test accuracy varies widely from 1.0% to 92.7%, yet CLIF consistently reduces it to at most 0.3%. Notably, even for the recently learned class from task 10, which the original model recognises with 100.0% training and 92.7% test accuracy, CLIF reduces these to 0.7% and 0.3%. This provides direct evidence that CLIF can effectively forget a strongly learned target class, rather than merely benefiting from historical forgetting in the CL backbone. These new results are included in Appendix E.8.

Table R3: Unlearning one class from task 1, 4, 7 or 10 of the final CIFAR-100 model.

Target class from task14710
Origin Acc$_f$ / Acc$_{ft}$11.4 / 13.32.5 / 1.015.3 / 16.3100.0 / 92.7
CLIF Acc$_f$ / Acc$_{ft}$0.5 / 0.00.7 / 0.30.5 / 0.30.2 / 0.0
CLIF / Retrain Acc$_{rt}$15.2 / 17.216.1 / 16.415.0 / 17.713.2 / 16.8

Supplemented E.7 results. Following the reviewer's suggestion, we further supplement Appendix E.7 by reporting the original target-class accuracy and corresponding unlearning performance at each training stage. We keep the forget class fixed and unlearn it after training on t∈{2,4,6,8,10}t\in\{2,4,6,8,10\} tasks. As shown in Table R4, the original target-class test accuracy ranges from 11.33% to 65.0%, whereas CLIF consistently reduces it to 0.0% at every stage. Thus, forgetting effectiveness remains consistently strong across the evaluated levels of initial target-class accuracy. These new results are now included in Appendix E.7.

Table R4: Unlearning at different CL stages on CIFAR-100.

Tasks trained tt246810
Origin Acc$_f$ / Acc$_{ft}$62.2 / 65.053.4 / 54.029.8 / 36.035.6 / 38.09.3 / 11.3
CLIF Acc$_f$ / Acc$_{ft}$0.2 / 0.01.6 / 0.00.0 / 0.00.0 / 0.00.2 / 0.0
CLIF / Retrain Acc$_a$97.2 / 95.694.3 / 99.498.5 / 99.292.3 / 87.297.8 / 98.8
CLIF / Retrain Acc$_{at}$70.6 / 78.151.8 / 63.550.1 / 52.838.7 / 38.438.7 / 41.0
AuthorsOpenReview ↗

Response to Reviewer d6GT - Q1

Q1.

In Proposition 4.1, ... this formulation?

Response:

We thank the reviewer for this insightful question. Machine unlearning for continual learning (CL) models, which is distinct from sequential unlearning on static models, is an under-explored yet critical problem for privacy compliance and post-deployment data correction in lifelong systems. Much like prior studies in influence functions and empirical Fisher approximations (e.g., Bae et al. (2022); Kunstner et al. (2019)), we view the clean algebraic form of Proposition 4.1 as a strength: a simple, exact decomposition that diagnoses why classical IFs catastrophically collapse in CL and resolves that failure with clear empirical gains.

(1) Diagnostic value of the decomposition. Classical deletion IFs rely on the stationarity assumption gA=0\mathbf{g}_{\mathcal{A}} = 0 (or ∇LD=0\nabla\mathcal{L}_{\mathcal{D}} = 0), discarding the total/retain gradient and leaving only the forget-gradient term −∣S∣∣R∣gS-\frac{|\mathcal{S}|}{|\mathcal{R}|}\mathbf{g}_{\mathcal{S}}. In sequential CL, however, θT\theta_T is not guaranteed to be stationary for the joint objective. Proposition 4.1 re-derives the local second-order target without assuming stationarity:

= \underbrace{\frac{|\mathcal{R}| + |\mathcal{S}|}{|\mathcal{R}|}\mathbf{g}_{\mathcal{A}}}_{\text{residual-gradient correction}} - \underbrace{\frac{|\mathcal{S}|}{|\mathcal{R}|}\mathbf{g}_{\mathcal{S}}}_{\text{classical IF term}} = \mathbf{g}_{\mathcal{R}}.$$ Although the expression simplifies algebraically to $\mathbf{g}_{\mathcal{R}}$, writing it in this decomposed form explicitly isolates the residual-gradient term omitted by standard IFs. **Empirically**, Table 6 (w/o BCT) confirms that omitting this single correction causes catastrophic collapse: $\mathrm{Acc_{at}}$ drops from $38.67%$ to $6.70%$ on CIFAR-100. **(2) Differences from a conventional preconditioned optimisation step on $\mathcal{R}$.** Although Phase I is algebraically equivalent to a local Newton step on $\mathcal R$, the complete CLIF framework differs from unconstrained retain-data optimisation in two important ways: * **Role as an unlearning target:** Conventional optimization on $\mathcal{R}$ minimizes $\mathcal{L}_{\mathcal{R}}$ in isolation without explicit reference to the forget set $\mathcal{S}$. Because $|\mathcal{R}| \ll |\mathcal{D}|$ is heavily skewed toward the current task $\mathcal{D}_T$ and a small replay buffer $\mathcal{B}$, unconstrained updates on $\mathcal{R}$ can erode historical knowledge (e.g., the decrease of $\mathrm{Acc_{r}}$ and $\mathrm{Acc_{rt}}$ in w/o TBE, Table 3). * **Tight coupling with IF curvature and forget-subspace control:** Phase I is not executed as an unconstrained Newton step on $\mathcal{R}$; rather, it supplies the bias-corrected target to an IF-style linear system tailored to CL. Specifically, because $\mathbf{g}_{\mathcal{R}}$ is generally nonzero, Phase II removes the squared-mean bias via our variance-form Continual Fisher, and Phase III first selects forget-supported update components, then removes their overlap with the retain-buffer gradient subspace. Removing this subspace projection (Table 6) substantially degrades $\mathrm{Acc_{at}}$ across all benchmarks. **(3) Properties of influence functions preserved and extended.** * **Preserved properties:** (i) *Closed-form, retraining-free local quadratic approximation* around $\theta_T$ in a single step; (ii) *Exact recovery of classical deletion IFs* as a special case when stationarity holds (Corollary C.0.1); and (iii) *A first-order characterisation of the forget-loss effect after projection*, with an explicit bound on its deviation from the unprojected update (Proposition C.1). * **Extended properties:** (i) *Relaxation of stationarity ($\nabla\mathcal{L}=0$)*, extending IF-based unlearning from converged ERM optima to non-stationary sequential CL checkpoints (Theorem 4.1); and (ii) *Formulation under partial-history access*, expressing the leave-$|\mathcal{S}|$-out target purely in terms of accessible sets ($\mathcal{A} = \mathcal{R} \sqcup \mathcal{S}$) with explicit error bounds accounting for buffer proxy discrepancy ($\epsilon_B$). We have further clarified this interpretation in Section 4.1. We appreciate the reviewer's suggestions, which have helped us strengthen the methodological explanations and experimental evidence. We would be happy to clarify any remaining questions. **References** Juhan Bae, et al. If influence functions are the answer, then what is the question? NeurIPS, 2022. Mehrdad Farajtabar, et al. Orthogonal gradient descent for continual learning. AISTATS, 2020. Frederik Kunstner, et al. Limitations of the empirical fisher approximation for natural gradient descent. NeurIPS, 2019. Gobinda Saha, et al. Gradient projection memory for continual learning. In International Conference on Learning Representations, ICLR, 2021.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.