acceptodds
Under review as a conference paper at ICLR 2027

FIT to Forget: Robust Continual Unlearning for Large Language Models

Abstract

While large language models (LLMs) exhibit remarkable capabilities, they increasingly face demands to remove the influence of privacy-sensitive, copyrighted, or harmful content. Existing unlearning methods primarily focus on single-shot scenarios, whereas real-world deletion requests arrive continually. Na\"ively applying these methods to sequential requests leads to severe utility degradation and catastrophic forgetting. To address this, we propose \fit, a robust framework for high-volume continual unlearning that resists both catastrophic forgetting and post-unlearning recovery. \fit stabilizes sequential updates through three synergistic mechanisms: redundancy Filtering, Importance-aware adaptive algorithm selection, and Targeted layer attribution. Furthermore, to facilitate rigorous evaluation, we introduce PCH, a unified benchmark spanning Personal information, Copyright, and Harmful content, alongside two symmetric metrics, Forget Degree (F.D.) and Retain Utility (R.U.), that quantify forgetting-utility trade-offs. Experiments across five LLMs (up to 14B parameters) show that \fit maintains a strong forgetting–utility balance throughout long request streams. Even after hundreds of randomly ordered requests, \fit preserves competitive downstream (\eg, GSM8K and MMLU) performance and resists relearning and quantization recovery attacks.Our code is available at https://anonymous.4open.science/r/FIT_Code-7304/.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large language model, Continual unlearning, Machine unlearning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_f6TSOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies continual unlearning in large language models, where deletion requests arrive sequentially. It proposes FIT, which filters redundant request content, selects a single-shot unlearning method based on request importance, and restricts updates to relevant layers. The paper also introduces PCH, a benchmark covering personal information, copyrighted material, and harmful content, and evaluates FIT on sequential unlearning requests across five LLMs.

Critical strengths

They study the omitted problem of continual unlearning in LLM. The experiments are comprehensive.

Critical weaknesses

The paper is poorly written and frequently confusing about its core setting and method. The introduction presents catastrophic forgetting as a central challenge but does not clearly explain what is being forgotten, why sequential unlearning causes this failure, or how it differs from the intended removal of requested information. The connection between Sections 2.2 and 3.2 is similarly unclear. The paper lists six single-shot methods as primitives and claims to route requests among them, yet does not provide a complete routing rule. These gaps make it difficult to understand what FIT actually does.

I also find the methodological novelty limited. In particular, the second component of FIT selects among existing unlearning objectives, while the third uses an existing leave-one-out attribution idea to choose layers for updating. The paper does not clearly establish what is new about these designs beyond applying and combining them in a continual-unlearning setting.

Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to Reviewer f6TS

Thanks for recognizing the importance of continual unlearning and the comprehensive evaluation. We agree that the distinction between intended unlearning and catastrophic forgetting, and the default routing rule, should be stated more clearly.

#W1: Clarifying the setting and routing rule

Intended unlearning vs. catastrophic forgetting. At request tt, we distinguish:

  • Intended unlearning: removing the influence of Df(1:t)D_f^{(1:t)}, the information covered by deletion requests received so far.

  • Catastrophic forgetting: unintended loss of knowledge in Dr(1:t)D_r^{(1:t)} or of capabilities unrelated to those requests.

For example, deleting one person’s profile should reduce recall of that profile while preserving answers about other profiles and unrelated tasks.

In sequential unlearning, each update acts on an already modified model. Repeated updates can suppress shared content and alter parameters needed for retained knowledge, allowing unintended damage to accumulate. FIT addresses these effects through filtering, adaptive objective selection, and targeted layer updates.

A collapsed model may fail on both forget and retain data, so low forget-set accuracy alone is insufficient. We therefore evaluate alignment with a retain-model proxy on both subsets (Sections 2.1 and 4.2), alongside downstream capabilities (Table 3). We will introduce this distinction and the example early in the Introduction.

Complete routing rule. Section 2.2 introduces six candidate methods, while the default implementation uses three: RLabel, NPO, and NPO+KL. We will correct the wording in Section 3.2 that suggests the default policy routes among all six.

For a filtered request xx, FIT computes

IMP⁡(x)=∥∇E(x)L(x)∥2\operatorname{IMP}(x)=\lVert\nabla_{E(x)}L(x)\rVert_2

and applies:

IMP rangeSelected primitive
IMP⁡(x)<τlow\operatorname{IMP}(x)<\tau_{\mathrm{low}}RLabel
τlow≤IMP⁡(x)<τhigh\tau_{\mathrm{low}}\leq\operatorname{IMP}(x)<\tau_{\mathrm{high}}NPO
IMP⁡(x)≥τhigh\operatorname{IMP}(x)\geq\tau_{\mathrm{high}}NPO+KL

The implementation defaults to τlow=0.5\tau_{\mathrm{low}}=0.5 and τhigh=1.0\tau_{\mathrm{high}}=1.0 , with boundaries adjustable for the target dataset. Appendix B.2 characterizes IMP as local loss sensitivity in embedding space; the assignment of primitives is empirically motivated by Figure 3(b).

We will also consolidate the full procedure in one place:

  1. Filter incoming content against historical request embeddings using similarity and the loss-difference test.

  2. Skip the parameter update if no content remains. Rank layers by their request-specific leave-one-out loss deviations and select the top-K%.

  3. Compute IMP on the filtered content and select the objective using the rule above.

  4. Update only the selected layers' MLP and attention modules, keeping other parameters frozen.

  5. Store the processed request embeddings for subsequent filtering.

#W2: Contribution beyond existing objectives and attribution methods

We agree that the unlearning objectives and leave-one-out attribution are established techniques. FIT's contribution is a request-adaptive framework that jointly determines what content to process, which objective to use, and where to update across sequential deletion requests. Our results show that this adaptation can enable standard single-shot methods to outperform methods designed specifically for continual unlearning in the evaluated settings.

The fixed-routing and fixed random-layer controls in Figure 6 and Section 5.2 support the value of these request-dependent choices. Component ablations and additional filtering-plus-single-method experiments provide further evidence; please see our response to Reviewer 3kvo (W1 & Q1). For the broader contribution, distinctions from prior work, and extensions to RULE and UNDIAL, please see Reviewer ECd5 (W1, W2 & Q1).

We will make the scope of the contribution explicit: the IMP analysis supports a local sensitivity interpretation, while routing effectiveness is established empirically; layer attribution uses a leave-one-out, Shapley-style approximation. The contribution lies in how these choices adapt existing methods to continual unlearning.

In light of the authors’ rebuttal effort, I have increased my score to 2.

Thank you for reconsidering our paper and updating your assessment. We appreciate your time and feedback and would be happy to address any further questions during the discussion.

Reviewer_3kvoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper a 3-step FIT framework to continual unlearning, where LLMs undergo multiple updates in response to sequential unlearning queries. The approach combined data/query filtering with adaptive selection of unlearning primitives and layer attributions to achieve better performance tradeoff between unlearning effectiveness and utility retaining in the continual unlearning scenario. Under a carefully designed new benchmark experiment under sequential unlearning requests, FIT framework outperforms unlearning primitives that are designed for single-shot unlearning.

Critical strengths

Address a new, practical unlearning scenario Provide a new benchmark to evaluate unlearning methods. The FIT framework performs better than single-shot unlearning methods.

Critical weaknesses

The FIT approach is a combination of previously known unlearning methods in an adaptive manner. However, the ablation studies are not sufficient to determine contribution of each step or each strategy. For example, it remains unclear how the filtering step contributes to the performance gaining -- will query filtering + single-shot unlearning primitive improve the performance? There is also a lack of theoretical analysis or empirical understanding about why each step works or not. Similarly, the contribution of layer attribution is not clear.

The benchmarking results are not sufficient. As FIT framework is designed to adaptively select the most effective unlearning primitives, I assume it achieves the better performance balancing unlearning (measured by FD) and utility (RU). But it seems to end up achieving better performances on each of these metrics than all primitives, no matter their priority is unlearning/forgetting or utility preserving.

The evaluation seems to be limited to relatively small LLMs. Some experiments on relatively larger models may demonstrate how the method would scale. The running time comparison would also be helpful, especially for the continual learning scenario.

Questions for authors

What are the contributions of each step of the FIT framework? What are running time of FIT comparing with single-shot methods? Why RU seems stable for FIT after the number of queries above 120 (Table 2)? Is it due to the redundancy in the queries? Why GA+GD has poor FD on LLaMA, given it is aggressive on forgetting?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 3kvo (1/2)

Thanks for recognizing the practical setting, benchmark contribution, and forgetting-utility results. We address the component contributions, metric interpretation, and computational cost below.

#W1 & Q1: Component contributions and supporting analysis

FIT adapts existing unlearning objectives to sequential requests. For its methodological positioning, see response to Reviewer ECd5 (W1, W2 & Q1). Here, we focus on the evidence for each component. Filtering + single-shot methods. To directly address your question, we compare NPO+KL and RLabel with and without filtering on Qwen3.5-9B-Base. These four configurations update all layers; complete FIT uses targeted-layer updates. All configurations use seed 42 and 300 PCH requests. The provisional results are:

ConfigurationF.D.(300) ↑R.U.(300) ↑
NPO+KL0.68770.6573
Filtering + NPO+KL0.74310.6945
RLabel0.00000.0000
Filtering + RLabel0.00130.0000
Complete FIT0.85130.8058

Filtering improves NPO+KL's F.D./R.U. by 0.0554/0.0372, but does not prevent RLabel's utility collapse. Its benefit therefore depends on the underlying method, and filtering alone does not reproduce complete FIT’s performance.

Component controls. Table 9 reports five-run mean F.D./R.U. of 0.73/0.71 for complete FIT on Llama-3-8B after 300 requests, compared with 0.14/0.07 without adaptive routing, 0.66/0.63 without filtering, and 0.40/0.48 without targeted layers. Beyond these ablations, Figure 6 and Section 5.2 compare adaptive routing with fixed routing, and request-specific attributed layers with a fixed random subset, while retaining the other components.

Why the components help.

  • Filtering limits repeated updates on overlapping content. Appendix B.1 analyzes how approximately aligned gradients can accumulate and cause excessive parameter displacement. The loss-difference test retains similar but distinct content, addressing the risk of discarding sensitive information based on similarity alone. Figure 7 examines the discarded content.

  • Routing adjusts objective strength to request sensitivity. Appendix B.2 shows that, under local smoothness, ∣L(E(x)+δ)−L(E(x))∣≤IMP⁡(x)∥δ∥2+β2∥δ∥22.|L(E(x)+\delta)-L(E(x))| \leq \operatorname{IMP}(x)\lVert\delta\rVert_2+\frac{\beta}{2}\lVert\delta\rVert_2^2. This motivates IMP as a measure of local embedding-space sensitivity. The primitive assignment is supported empirically by Figure 3(b) and the fixed-routing controls. In the Llama-3-8B test, shifting routing thresholds by ±5% changes F.D./R.U. by at most 0.02/0.01 (Figure 7(c)).

  • Layer attribution focuses updates on request-relevant parameters. FIT ranks layers by the loss change when each is masked, then updates the selected layers’ MLP and attention modules (Section 3.3; Appendix B.3). The fixed random-layer control supports selecting layers according to each request. Figure 4 further shows that updating 25% of layers gives the best reported trade-off in the tested Llama-3-8B setting.

#W2 & Q3 & Q4: Interpreting F.D./R.U. and their trajectories

Why FIT can improve both metrics. F.D. measures alignment with the retain-model proxy, rather than the amount of forget-set suppression. Equation (5) defines:

$\mathrm{F.D.}^{(t)} =\max\left(0, 1-\left|F^{(t)}/FQ^{(t)}-1\right| \right).$

F.D. peaks when F(t)=FQ(t)F^{(t)}=FQ^{(t)}. If aggressive updates push F(t)F^{(t)} below FQ(t)FQ^{(t)}, further suppression lowers F.D.; as F(t)F^{(t)} approaches zero, F.D. also approaches zero (Figure 9(d)).

This explains why an aggressive objective such as GA+GD need not achieve high F.D. Low F.D. alone does not imply weak suppression.

FIT can improve both metrics by approaching the reference on forget data while preserving retain performance. Filtering and layer selection also change the update trajectory, so FIT’s scores need not fall between those of the standalone methods.

FIT does not lead at every checkpoint: on Llama-3-8B, O³ exceeds it on both metrics at 60 requests, and both O³ and ALKN do so at 120 requests (Table 2). Its main advantage is maintaining the balance later in the sequence.

Why R.U. stabilizes after 120 requests. At 120, 180, 240, and 300 requests, FIT’s R.U. on Llama-3-8B is 0.71, 0.72, 0.70, and 0.71, respectively. Without filtering, it is 0.63, 0.61, 0.59, and 0.63 (Table 9).

Filtering improves utility, but the unfiltered curve is also relatively stable. These results therefore do not establish redundancy as the cause of the plateau. Moreover, R.U. uses the current retain set and a stage-specific reference: a stable score indicates stable reference alignment, not necessarily unchanged raw accuracy.

AuthorsOpenReview ↗

Response to Reviewer 3kvo (2/2)

#W3 & Q2: Model scale and running time

Our original evaluation includes Qwen3-14B. Limited computational resources prevented experiments beyond 14B during rebuttal, so larger-scale validation remains a limitation. The additional Qwen3.5-9B-Base experiment provides evidence on a newer backbone; see our response to Reviewer ECd5 (W1 & Q2).

On Qwen3.5-9B-Base, FIT takes 6.64s/request, compared with 4.46--5.55 s/request for NPO+KL with/without filtering, excluding model loading and evaluation. FIT's total of 1,991.61s includes filtering, attribution, IMP computation, optimization, and overhead.

Please see Reviewer ECd5 (W3) for the full runtime and memory comparison, component costs, and preliminary shared-time-budget results with their limitations.

I appreciate the authors' response. The ablation experiment improves the paper. But the small models used in the study remain a concern.

Reviewer_ECd5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper claims that existing unlearning methods focuses on single-shot scenarios, whereas the deletion requests in the real world is continuous. To address this problem, this paper proposes FIT, which consists of three components: (i) filtering redundant requests using semantic similarity and loss differences to avoid unnecessary updates, (ii) estimating the importance of each request to adaptively select an appropriate unlearning algorithm, and (iii) identifying the most relevant model layers and restricting updates to those layers to reduce parameter drift. Moreover, this paper also provides a new benchmark dataset, namely, PCH, to evaluate the performance of difference approaches. Experimental results show that the proposed approach on five LLMs show good forgetting-utility balance on long conversations.

Critical strengths
  • S1. This paper studies an important question of sequential/continual unlearning, which is practical for LLM applications. Moreover, the proposed three modules are reasonable for unlearning in a long context.
  • S2. Experimental results on the scenarios with 300 requests supporting the long sequence scenario.
  • S3. The ablation study shows that the proposed three components all contribute to the performance.
Critical weaknesses
  • W1. The position of this paper should be better justified since continual LLM unlearning has already been studied in several previous works, e.g., CURaTE (ACL 2026). Although FIT compares against O3 and ALKN, its technical differences from these methods remain relatively limited. It is suggested to better explain what new methodological insights FIT provides beyond coordinating existing unlearning strategies. Moreover, among the 5 used LLMs, four are based on Llama 2, Llama 3, and Yi, while Qwen3-14B is the only relatively recent backbone. The experiments do not cover newer model families such as Qwen3.5, despite their availability before the ICLR 2027 submission. Given the changes in model architectures and training procedures, the generalizability of FIT to contemporary LLMs remains uncertain.
  • W2. The proposed FIT combines semantic similarity filtering, importance-based selection among existing unlearning algorithms, and selective layer updates based on attribution scores. These components are individually well-established, but the proposed framework does not introduce a fundamentally new unlearning objective or mechanism. Although the ablation studies show that each component contributes to the final performance, they do not necessarily establish the novelty of the overall framework.
  • W3. The experiments mainly rely on PCH, containing 600 synthetic samples across personal information, copyright, and harmful content. The LLMs are finetuned on these examples before unlearning, which may differ from removing information acquired during large-scale pretraining. Furthermore, Forget Degree and Retain Utility measure behavioral agreement with a retain-only reference model through probability, ROUGE-L, and token accuracy. As such, the results do not necessarily imply that the information has been removed. Results on MUSE and TOFU are useful but they do not focus on long context. Finally, since the efficiency is an important motivation for continual unlearning, it is suggested to justify the additional complexity by comparisons under matched computational budgets.
Questions for authors
  • Q1. The novelty part may be better justified since the proposed approach looks like the combination of existing methods.
  • Q2. Experimental results on modern LLMs are welcome.
3 comments
AuthorsOpenReview ↗

Response to Reviewer ECd5 (1/2)

Thanks for recognizing the importance of continual unlearning, 300-request evaluation, and component ablations. We address your concerns below.

#W1, W2 & Q1: Contribution and relation to prior work

We agree that continual LLM unlearning has been studied before and that FIT builds on established techniques. As stated in Introduction, FIT uses existing single-shot objectives. Our contribution is a request-adaptive framework for applying these objectives across long deletion streams. The results show that, with this adaptation, existing single-shot methods can outperform methods designed specifically for continual unlearning in the evaluated settings. We'll clarify this positioning. The methodological insight is that controlling cumulative degradation requires adapting three decisions to each request (Section 3):

  • What content to update on: filter redundant chunks while retaining similar chunks whose loss difference exceeds the threshold.

  • Which objective to apply: use IMP, a measure of local request sensitivity, to select an appropriately aggressive or conservative unlearning method.

  • Where to update: identify the top-K% most relevant layers for each request and update only their MLP and attention modules.

Appendix-D discusses prior continual-unlearning work, including CURaTE. The main differences are:

MethodMain mechanism
CURaTEKeeps the LLM parameters unchanged and uses a trained sentence-embedding model and stored forget requests to decide whether to answer or refuse a query.
O³Combines orthogonal LoRA updates with an out-of-distribution detector.
ALKNUses dynamic gradient masking and adaptive unlearning intensity within a knowledge-negation framework.
FITFilters redundant content, selects a primitive using IMP, and updates the MLP and attention modules of request-specific layers.

Evidence for these design choices. We agree that component ablations alone do not establish novelty. Figure 6 and Section 5.2 also compare adaptive routing with fixed routing, and attributed layers with a fixed random-layer control, while keeping the other components. These comparisons support the value of adapting the decisions to each request.

After 300 requests on Llama-3-8B, FIT’s F.D./R.U. scores are 0.73/0.71, compared with 0.66/0.63 without filtering and 0.40/0.48 without targeted layers (Table 9; five-run means). Figure 6 further shows that, after pool-specific recalibration, incorporating RULE or UNDIAL into FIT improves both metrics over their standalone counterparts. Together, these results support the broader use of FIT with existing unlearning methods.

#W1 & Q2: Additional comparisons and newer models

Comparison with CURaTE. Following CURaTE's response-based evaluation (Section 4.1), we compare both methods on Llama-3-8B over 300 deletion requests, with CURaTE’s refusal mechanism enabled. We report retain ROUGE-L, as used in CURaTE, alongside complete-answer leakage and response coverage.

The provisional results are:

MethodComplete-answer leakage ↓Retain ROUGE-L ↑Response coverage ↑
CURaTE19/3000.08530.9867
FIT8/3000.08051.0000

FIT has lower leakage and higher response coverage, while CURaTE has higher retain ROUGE-L. Coverage indicates whether a response is provided, not whether it is correct.

Newer backbone. We additionally evaluate Qwen3.5-9B-Base on 300 PCH requests using seed 42. The provisional endpoints are:

MethodF.D.(300) ↑R.U.(300) ↑
NPO+KL0.68770.6573
ALKN0.70660.6924
O³0.71230.7348
FIT0.85130.8058

FIT achieves the highest F.D. and R.U. among the compared methods. This extends the evidence to a newer backbone, although a single-seed experiment does not establish broad generalizability.

AuthorsOpenReview ↗

Response to Reviewer ECd5 (2/2)

#W3: Evaluation scope and computational cost

Benchmark scope and interpretation. PCH provides a controlled benchmark for heterogeneous continual unlearning, covering personal information, copyright, and harmful content. Its matched forget and retain subsets enable comparison with a retain-model proxy. We agree that removing fine-tuned synthetic knowledge differs from removing knowledge acquired during pretraining.

F.D. and R.U. measure behavioral alignment with the retain-model reference using geometric means of Probability, ROUGE-L, and token-level Accuracy (Section 4.2). They do not certify complete information removal. Sections 2.1 and 4.1 and Appendix C.1 describe the proxy construction. On Llama-3-8B, replacing interpolated references with explicit checkpoints changes F.D./R.U. by at most 0.03/0.04, without changing method rankings (Figure 7(d)); this check is specific to that setting.

Recovery and extraction tests provide complementary evidence. Figure 5 evaluates relearning and int4 quantization, supporting recovery resistance under the tested conditions. We also test extraction using instruction, completion, and paraphrased prompts. A target counts as extracted if any response matches the normalized answer at word/number boundaries or reaches token F1 ≥ 0.8. The provisional counts are 14/300 for FIT and 22/300 for CURaTE.

Evidence beyond PCH. The following results on Llama-3-8B cover four sequential requests on MUSE and three on TOFU. Each cell reports F.D./R.U.:

DatasetMethodRequest 1Request 2Request 3Request 4
MUSEALKN0.70/0.740.66/0.730.70/0.660.64/0.62
MUSEO³0.72/0.790.70/0.720.69/0.640.67/0.73
MUSEFIT0.85/0.770.83/0.710.85/0.680.83/0.72
TOFUALKN0.66/0.640.64/0.620.61/0.58—
TOFUO³0.70/0.680.67/0.640.65/0.60—
TOFUFIT0.74/0.670.73/0.670.70/0.62—

FIT achieves the highest F.D. at every reported stage, with competitive R.U. These experiments extend the evaluation beyond PCH but cover short sequences; PCH provides the 300-request evaluation. "Long-horizon" refers to sequential deletion requests, not input context length. We'll clarify this terminology.

Computational cost and a shared time budget. Beyond the memory measurements in Table 10, we report additional measurements on Qwen3.5-9B-Base over 300 requests, excluding model loading and evaluation:

MethodMean time/requestTotal online timePeak allocated memory
NPO+KL (with/without filtering)4.46–5.55 s1,339.01–1,665.19 s94.04–94.48 GB
ALKN149.23 s44,768.46 s143.01 GB
O³35.47 s10,640.54 s26.45 GB
FIT6.64 s1,991.61 s47.64 GB

FIT’s total includes filtering (349.31s), attribution (511.14s), IMP (100.94s), optimization (1,028.46s), and small overhead. It is faster than ALKN and O³, though slower than NPO+KL, and uses less memory than NPO+KL and ALKN.

Under a shared time budget of approximately 1,339s, FIT updates 205/300 requests, NPO+KL 300/300, ALKN 9/300, and O³ 64/300. All received requests remain in the evaluation sets. The provisional F.D./R.U. endpoints are 0.9143/0.8653 for FIT, 0.7946/0.7425 for NPO+KL, and 0.8834/0.8329 for ALKN.

FIT scores higher than NPO+KL and ALKN on both metrics in this preliminary comparison. However, shared H100 NVL load, small budget overruns, and pre-budget run variation limit conclusions about performance under strictly matched computational budgets.

Thank you for the efforts on the clarifications and additional experiments. The newer backbone results and computational cost breakdown address parts of the concerns. The comparisons with fixed routing and random layer selection also provide some evidences for the adaptive design. However, I remain only partly convinced by the methodological contribution beyond coordinating established techniques. The evidence for long sequences also remains concentrated on the synthetic PCH benchmark, while MUSE and TOFU cover only a few requests.

In summary, the response strengthens the empirical evidence. The above remaining concerns keep my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.