acceptodds
Under review as a conference paper at ICLR 2027

Not Every Change Is Necessary: Recoverable Drift in Large Language Model Unlearning

Abstract

Machine unlearning in large language models aims to remove unwanted knowledge while preserving the model's remaining capabilities. Although existing methods use retention objectives or restrict where edits occur, achieving the desired forgetting level can still leave collateral changes that impair non-target behavior. Our recovery comparisons suggest that some of these changes can be reversed while preserving observed forgetting performance. In this work, we present Propose-Then-Project Unlearning (**PTP-U**), a framework that combines targeted forgetting with the recovery of non-target capabilities. PTP-U first applies local analytic edits to weaken target knowledge associations, then aligns non-target output distributions with those of the original model to recover capabilities while maintaining fixed forgetting constraints. Both stages serve a common goal: satisfying the forgetting requirements while preserving fluent generation and performance on non-target tasks. Across three benchmarks, PTP-U achieves the strongest forgetting–retention trade-off among evaluated methods, reaching 81.22%–91.03% forgetting while preserving 94.20% non-target utility on average. At matched forgetting, PTP-U consistently retains higher non-target utility.

Primary area:
optimization
Keywords:
Machine Unlearning, Large Language Models, Model Editing, Constrained Optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zW61OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how to effectively forget target knowledge in LLM Unlearning while keeping the model's original abilities as much as possible. The authors find that existing unlearning methods may cause some unnecessary changes to the model during forgetting. These changes can hurt the model's other abilities, and some of them can be recovered without greatly affecting the forgetting results. Based on this finding, the authors propose Propose-Then-Project Unlearning (PTP-U), a two-stage method. In the first stage, the method uses local analytical editing based on curvature information to forget the target knowledge. In the second stage, it uses KL-based Recovery with clear forgetting constraints to make the model's outputs on non-target data closer to those of the original model, helping recover its lost abilities. Experiments on multiple models and unlearning benchmarks show that PTP-U achieves a good Forget-Retain Trade-off. It can effectively forget target knowledge while better keeping the model's original abilities.

Critical strengths
  1. The paper provides an interesting idea. It uses a two-stage design that first forgets the target knowledge and then recovers the model's lost abilities. This helps keep the model's original abilities as much as possible while still achieving effective forgetting.
  2. The paper provides fairly complete experiments, covering different models and unlearning tasks. It also includes many ablation studies and detailed analyses, which support the main conclusions of the paper to some extent.
Critical weaknesses
  1. The overall design of PTP-U is quite heavy. It involves curvature estimation, analytic editing, Sequential Relinearization, and repeated constraint checks, which add considerable computational and engineering complexity. Table 6 shows that Stage I accounts for most of the computational cost. If I understand correctly, the goal of Stage I is to forget the target knowledge while preserving as much of the Base model's original behavior as possible. Although the paper provides an interesting view of unlearning, it does not clearly show whether its computationally expensive Stage I is necessary compared with existing, potentially simpler unlearning methods (e.g., [1, 2]).

  2. The Recovery mechanism in this paper is quite similar to the KL-based Retention Regularization commonly used in existing LLM unlearning methods. The current experiments mainly show the effectiveness of each component, but it remains unclear whether this two-stage optimization framework offers significant advantages over simpler end-to-end KL-regularized or constrained optimization methods.

[1] Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning

[2] LLM Unlearning via Neural Activation Redirection

Questions for authors
  1. Can Stage I be replaced with a more lightweight unlearning method, or even a simpler and more direct method such as GA? How would the final Forget-Retain Trade-off change after applying the same Recovery stage?
  2. Compared with existing end-to-end optimization that combines Forget Loss and Retain KL, what specific advantages does PTP-U's two-stage design offer that cannot be achieved through simpler regularization methods?
  3. Stage II performs Recovery through KL alignment on a fixed Anchor Distribution and shows improvements on downstream tasks. How well does this Recovery generalize across different Anchor Distributions? Does it depend on a specific type of data? If the Anchor Data comes from different sources, can it still improve performance on downstream tasks?
Reviewer_fJ8JOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PTP-U, a two-stage LLM unlearning framework built on the observation of "recoverable drift": some parameter changes induced by unlearning are unnecessary for forgetting and can be reversed. Stage I applies closed-form analytic edits in a subspace under linearized forgetting constraints, with sequential Attention/FFN relinearization. This results in an initial params proposal that satisfies the forgetting constraint. Stage II performs recovery toward the Base output distribution via a KL objective under forgetting constraints. Experiments across Llama2-7B, Llama3.1-8B, and Qwen3-14B show a favorable forgetting–retention trade-off plus robustness to adversarial prompts and benign relearning.

Critical strengths
  1. Well-motivated "recoverable drift" with a principled formulation. The recovery analysis (Fig. 2) well motivates the story. The theory and its reasoning seem to be sound and understandable, e.g., the first-order approximation, linearization, KL objective, etc. This elevates the paper above heuristic unlearning pipelines.
  2. Broad empirical validation. Four benchmarks, three model families, eight baselines spanning different unlearning categories (Table 2). PTP-U matches the strongest forgetting methods while clearly retaining more utility.
Critical weaknesses
  1. Presentation simplification and implementation description inconsistencies might undermine reproducibility. The algorithm (1) in C.8 gives a very high-level pipeline, but omits many details. The sub procedures (and related hyperparams) should also be specified. In D.2, some Stage-II hyperparameters (s, \beta) are used, but they do not appear in the main-text algorithm. It is unclear what parameterization actually produced Tables 1–2. This is consequential because the method's complexity makes independent reproduction depend entirely on these details.
  2. Main Tables 1, 2, 5 report single-run point estimates, which might lack statistical rigor because some diff between PTP-U and other baselines are not very significant, e.g, PTP-U 43.34 vs SCANS 42.05. Some claimed advantages (e.g., under Qwen3 WNDP, PTP-U (31.23, 72.42) vs. ALTER (30.58, 70.55)) are plausibly within run-to-run noise, so "consistently retains higher utility" is not very well established at the reported granularity.
Questions for authors
  • How is the empirical recovery (repair) analysis (Fig. 2) performed? Up to this point, the unlearning method is not yet specified, so I assume the "Repaired" curves are not produced by this paper's method, right? I was wondering how "repair" is done to produce these curves.
  • "We select the middle layers (23) of the model for editing." What does it mean? Does this mean that you only execute your algorithm on layer 23? How are layers selected and how do the selection affects the results?
  • Which parameterization produced the main results? Give the full list of hyperparameters used (this should definitely be specified more clearly in the final version).
  • How many independent runs/seeds underlie Tables 1, 2, and 5? Can you report variance or confidence intervals for the headline trade-off numbers?
  • How sensitive is the resulting model to the choice of \Delta f = log 2?
Reviewer_vc3YOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper tackles LLM unlearning, which aims to remove some unwanted knowledge from the model weights, while trying to maintain the original capabilities of the model. The paper argues that previous work on unlearning might have introduced redundant editing due to local-specific edits, resulting in lower performance on general-capability benchmarks. To compensate for this, the paper introduces a new method, adding a recovery stage to revert any unnecessary changes back to the original, achieving better forgetting-retention trade-offs. The experiments are conducted on two popular unlearning benchmarks: RWKU and WMDP, involving 7 existing unlearning methods, and varying several model families and scales, showing the effectiveness of the proposed method.

Critical strengths
  1. The paper proposes a novel method and its motivation is quite clear: some changes during unlearning introduced by local-wise optimization might be unnecessary, and should be reverted back to ensure the preservation of the original capabilities of a model. Following this motivation, the paper proposes “Propose-then-Project Unlearning” (PTP-U). Compared to previous work on unlearning, it goes one step further by employing two-stage methods, combining output-distribution-based optimization and parameter editing.
  2. The experiments seem extensive involving 7 unlearning methods as baselines and two unlearning benchmarks: RWKU and WMDP as well as ablation studies on the proposed method.
Critical weaknesses

The most critical weakness in this paper is on the selection of layers to control. The paper says it uses layer 23 and its adjacent layers. While I acknowledge that the paper presented an ablation analysis on the layer selection, I do not think this generalizes across models and scales. In particular, if a target model gets larger, the proposed method will most probably require tuning the layer index. Moreover, the paper does not mention (at least in the main body) whether it similarly tunes the selection of layers to control for baselines. To sum, I am concerned whether the presented results are cherry-picking or not.

Also, I have a concern about the fairness of the comparisons against the presented baselines given the above.

Finally, has the paper ensured that it uses a dev set or training set to select the layer to be edited? I’d assume that the paper has not conducted a stratified experiment and just used off-the-shelf test sets for every experiment according to Appendix C.1. If this is the case, the paper needs to redo all the experiments to verify that the layer selection generalizes to any unseen samples.

Questions for authors
  1. Oftentimes, the paper underspecifies which model it experiments with. Please make sure to specifically name the model used for each table and figure. For instance, Figure 4 does not mention which model the paper used.
  2. In my opinion, the paper is overly theoretical, using over 3 pages for explaining the methodology with heavy mathematical notations. They only closely relate to the ablation analysis in the paper in the later stage. I’d rather want to see more visualizations and examples to ease understanding.
  3. The paper has not tested any LLMs that are published this year, such as Qwen3.5. Is there any concrete reason?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.