- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper studies continual unlearning in large language models, where deletion requests arrive sequentially. It proposes FIT, which filters redundant request content, selects a single-shot unlearning method based on request importance, and restricts updates to relevant layers. The paper also introduces PCH, a benchmark covering personal information, copyrighted material, and harmful content, and evaluates FIT on sequential unlearning requests across five LLMs.
Critical strengths
They study the omitted problem of continual unlearning in LLM. The experiments are comprehensive.
Critical weaknesses
The paper is poorly written and frequently confusing about its core setting and method. The introduction presents catastrophic forgetting as a central challenge but does not clearly explain what is being forgotten, why sequential unlearning causes this failure, or how it differs from the intended removal of requested information. The connection between Sections 2.2 and 3.2 is similarly unclear. The paper lists six single-shot methods as primitives and claims to route requests among them, yet does not provide a complete routing rule. These gaps make it difficult to understand what FIT actually does.
I also find the methodological novelty limited. In particular, the second component of FIT selects among existing unlearning objectives, while the third uses an existing leave-one-out attribution idea to choose layers for updating. The paper does not clearly establish what is new about these designs beyond applying and combining them in a continual-unlearning setting.
Questions for authors
None
3 comments
In light of the authors’ rebuttal effort, I have increased my score to 2.
Thank you for reconsidering our paper and updating your assessment. We appreciate your time and feedback and would be happy to address any further questions during the discussion.
Response to Reviewer f6TS
Thanks for recognizing the importance of continual unlearning and the comprehensive evaluation. We agree that the distinction between intended unlearning and catastrophic forgetting, and the default routing rule, should be stated more clearly.
#W1: Clarifying the setting and routing rule
Intended unlearning vs. catastrophic forgetting. At request t, we distinguish:
Intended unlearning: removing the influence of Df(1:t), the information covered by deletion requests received so far.
Catastrophic forgetting: unintended loss of knowledge in Dr(1:t) or of capabilities unrelated to those requests.
For example, deleting one person’s profile should reduce recall of that profile while preserving answers about other profiles and unrelated tasks.
In sequential unlearning, each update acts on an already modified model. Repeated updates can suppress shared content and alter parameters needed for retained knowledge, allowing unintended damage to accumulate. FIT addresses these effects through filtering, adaptive objective selection, and targeted layer updates.
A collapsed model may fail on both forget and retain data, so low forget-set accuracy alone is insufficient. We therefore evaluate alignment with a retain-model proxy on both subsets (Sections 2.1 and 4.2), alongside downstream capabilities (Table 3). We will introduce this distinction and the example early in the Introduction.
Complete routing rule. Section 2.2 introduces six candidate methods, while the default implementation uses three: RLabel, NPO, and NPO+KL. We will correct the wording in Section 3.2 that suggests the default policy routes among all six.
For a filtered request x, FIT computes
IMP(x)=∥∇E(x)L(x)∥2
and applies:
The implementation defaults to τlow=0.5 and τhigh=1.0 , with boundaries adjustable for the target dataset. Appendix B.2 characterizes IMP as local loss sensitivity in embedding space; the assignment of primitives is empirically motivated by Figure 3(b).
We will also consolidate the full procedure in one place:
Filter incoming content against historical request embeddings using similarity and the loss-difference test.
Skip the parameter update if no content remains. Rank layers by their request-specific leave-one-out loss deviations and select the top-K%.
Compute IMP on the filtered content and select the objective using the rule above.
Update only the selected layers' MLP and attention modules, keeping other parameters frozen.
Store the processed request embeddings for subsequent filtering.
#W2: Contribution beyond existing objectives and attribution methods
We agree that the unlearning objectives and leave-one-out attribution are established techniques. FIT's contribution is a request-adaptive framework that jointly determines what content to process, which objective to use, and where to update across sequential deletion requests. Our results show that this adaptation can enable standard single-shot methods to outperform methods designed specifically for continual unlearning in the evaluated settings.
The fixed-routing and fixed random-layer controls in Figure 6 and Section 5.2 support the value of these request-dependent choices. Component ablations and additional filtering-plus-single-method experiments provide further evidence; please see our response to Reviewer 3kvo (W1 & Q1). For the broader contribution, distinctions from prior work, and extensions to RULE and UNDIAL, please see Reviewer ECd5 (W1, W2 & Q1).
We will make the scope of the contribution explicit: the IMP analysis supports a local sensitivity interpretation, while routing effectiveness is established empirically; layer attribution uses a leave-one-out, Shapley-style approximation. The contribution lies in how these choices adapt existing methods to continual unlearning.