acceptodds
Withdrawn from ICLR 2027

Stable Forgetting: Bounded Parameter-Efficient Unlearning in LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Machine unlearning in large language models (LLMs) is essential for privacy and safety; however, existing approaches remain unstable and unreliable. A widely used strategy, the gradient difference method, applies gradient descent to retained data while performing gradient ascent on forget data, the data whose influence should be removed. When combined with cross-entropy loss, this procedure causes unbounded growth of weights and gradients, leading to training instability and degrading both forgetting and retention. We provide a theoretical framework that explains this failure, explicitly showing how ascent on the forget set destabilizes optimization in the feedforward MLP layers of LLMs. Guided by this insight, we propose Bounded Parameter-Efficient Unlearning, a parameter-efficient approach that stabilizes LoRA-based fine-tuning by applying bounded functions to MLP adapters. This principled modification controls the weight dynamics during ascent, enabling the gradient difference method to converge reliably. Across the TOFU, TDEC, and MUSE benchmarks, and across architectures and scales from 125M to 8B parameters, our method achieves substantial improvements in forgetting while preserving retention, establishing a novel theoretically grounded and practically scalable framework for unlearning in LLMs.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, Parameter-Efficient Unlearning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UFHAOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the instability of gradient-based LLM unlearning and argues that cross-entropy ascent on the forget set can cause excessive growth in the weights and gradients of feedforward layers. To address this issue, it proposes bounded parameter-efficient unlearning, which applies a bounded function, primarily a sine mapping, to LoRA updates in MLP layers. Experiments on TOFU, TDEC, and MUSE across several model families, together with additional vision and sequential-unlearning evaluations, suggest that the method improves forgetting while largely preserving model utility.

Critical strengths
  • The paper explains the instability of full-parameter unlearning and motivates bounded parameter-efficient updates.

  • The experiments cover multiple objectives, models, and benchmarks, showing improved stability with competitive forgetting performance.

Critical weaknesses

i) The methodological novelty is limited relative to closely related work.

The paper presents the instability of gradient-ascent unlearning as a central motivation and theoretical contribution. However, this problem has already been studied extensively. NPO analyzes the catastrophic collapse caused by gradient ascent and theoretically explains how its objective slows this process [a]. LoKU specifically observes that combining gradient ascent with LoRA leads to unstable optimization and poor retained utility, motivating a bounded forgetting objective [b]. SimNPO similarly describes gradient-ascent unlearning as an uncontrolled departure from the pretrained model [c]. The layer-level analysis in this paper offers another view of the same behavior, but it is unclear what fundamentally new understanding it provides beyond these earlier results.

The proposed solution is also closely related to existing parameterizations. Applying a sine function to low-rank updates was introduced for parameter-efficient fine-tuning in prior work [d], while SineProject recently used sinusoidally modulated trainable parameters to stabilize multimodal unlearning [e]. The main contribution therefore appears to be applying an existing sine-based low-rank parameterization to feedforward adapters for LLM unlearning, together with a broader empirical evaluation. The paper should clarify its conceptual difference from these methods and identify which part of the formulation is genuinely new.

ii) The theoretical guarantees for the optimization dynamics are incomplete.

Equation (10) bounds the effective adapter update elementwise, but this only controls its forward contribution when the input is bounded. It does not bound the two trainable low-rank factors or their gradients. Although the derivative of the sine function is bounded, the gradient of each factor still depends multiplicatively on the other factor. Either gradient may therefore grow if the other factor becomes large. The factors can also be rescaled in opposite directions without changing their product, further showing that a bounded effective update does not control the individual factors. The claim in Section 3.3 that the bounded Jacobian prevents gradient explosion throughout the full backward pass consequently requires additional assumptions or explicit control of the factor norms.

The extension from pure gradient ascent to gradient difference is also not justified. A lower bound on the magnitude of the forgetting gradient and an upper bound on the magnitude of the retaining gradient do not prevent their weighted contributions from pointing in opposing directions and cancelling each other. The paper neither establishes that the forgetting contribution consistently dominates by a fixed margin nor imposes a condition on the angle between the two gradients, despite gradient conflict being well documented in multi-objective optimization [f]. Moreover, boundedness of the retain loss is assumed rather than proved along the combined ascent and descent trajectory, since the forgetting update may increase the retain loss. The paper should revise these results, state the missing conditions explicitly, and report the individual factor norms and gradients during training.

[a] Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning. COLM 2024.

[b] Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs. ICLR 2025.

[c] Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning. NeurIPS 2025.

[d] Efficient Learning With Sine-Activated Low-rank Matrices. ICLR 2025.

[e] SineProject: Machine Unlearning for Stable Vision-Language Alignment. CVPR 2026.

[f] Gradient Surgery for Multi-Task Learning. NeurIPS 2020.

Questions for authors
  1. What is the main technical novelty over prior analyses of gradient-ascent instability and existing sine-based low-rank parameterizations, particularly NPO, LoKU, and SineProject?

  2. What prevents the two individual low-rank factors and their respective gradients from growing without bound when the method only constrains the sine-transformed effective adapter update and its local derivative?

  3. In Theorem 2, what condition prevents the retaining and forgetting gradients from cancelling each other? In addition, how does the rank-one expression in Equation (53) extend to a total gradient averaged over multiple retaining and forgetting examples?

Reviewer_LtmsOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a LoRA-based unlearning method that bounds the weight update in LLM unlearning, achieving stability and efficiency. Evaluations on three benchmarks across models ranging from 125M to 8B parameters demonstrate improvements in forgetting effectiveness and model utility.

Critical strengths
  1. Stabilizing LLM unlearning is important, since LLM unlearning is (mostly) based on gradient ascent, which could be unstable.
  2. The method formulation is very easy to understand.
  3. Models of different sizes are tested. The empirical performance gain looks consistent.
Critical weaknesses
  1. The motivation for using LoRA in LLM Unlearning seems unclear. First, LoRA-based unlearning is not robust against post-unlearning weight tampering. Second, to bound the weight update in unlearning, why not consider gradient clipping, reducing learning rate and the forget loss coefficient? Also, previous unlearning algorithms such as NPO and simNPO have shown stability compared to Gradient Difference. Third, LLM unlearn is not a GPU-bound task, and GPU memory efficiency is not a strong motivation unless we are considering unlearning on legacy GPUs or on edge device. It seems to me that LoRA is used mostly because it is naturally suitable for the non-linear transformation, but this surely sacrifices unlearning robustness.

  2. The proposed method claims to be objective-agnostic, but the paper only considers Gradient Difference (GD). Other algorithms, like the NPO algorithm (based on preference optimization) and the RMU algorithm (based on representation editting) are not integrated.

Questions for authors
  1. Are there any reasons why the WMDP benchmark is not considered in the experiments?
Reviewer_1FZyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

In this work, the authors present "bounded parameter-efficient unlearning", where they adapt bounded activation functions with non-linear LoRA adapters to LLM unlearning. A commonly devised strategy in LLM unlearning uses Gradient Ascent over the forget set (or Gradient Difference which includes a forget set specific ascent term). This leads to unbounded gradients and parameter weights, and in turn to network divergence and a severe drop in model utility. This work proposes to contain this by applying bounded activation functions such as sine/tanh to the LoRA adapters, thereby containing the effects of gradient ascent.

Critical strengths
  1. Soundness:

    • While the idea of applying bounded activations functions such as sine in non-linear LoRA has been explored in the past [1, 2], applying them to LLM unlearning is novel and well motivated.
  2. Empirical Validations:

    • Authors experiment with different open source model families and sizes (although not all models appear to be uniformly tested on all benchmarks).
  3. Results:

    • The presented results are encouraging and suggest an improvement in unlearning utility with the bounded activations.

[1] Efficient Learning With Sine-Activated Low-rank Matrices
[2] LoRAN: Improved Low-Rank Adaptation by a Non-Linear Transformation

Critical weaknesses
  1. Novelty:

    • This work is largely incremental: it presents a new application of the bounded activation functions to the well known problem of exploding gradients in unlearning.
    • This issue with GA/GD objectives has been well studied in the LLM unlearning literature, and many prior works have explored solutions to address this issue from various angles [1-7].
    • Authors are encouraged to discuss in more detail the impact of their proposed solution on the gradient dynamics of GA/GD and compare against some of the approaches presented in [1-7].
    • Further, the theoretical analysis point to known weaknesses and does not extend to studying the training dynamics of the bounded activation functions.
    • The novelty concern alone does not necessarily reduce the weight of this contribution. But when combined with the concern on baselines noted below, I believe this work needs a revision to make it strong.
  2. Unverified baselines:

    • A number of baselines reported in this draft (ex: Tables 1, 7, 12, 13) appear to be quoted directly from corresponding prior works.
    • This raises significant concerns about the validity of these findings, since it is unclear whether all other conditions are held identical to draw a fair comparison.
    • The results would be a lot more convining if authors reproduced these baselines in their own setting holding all other hyperparameters constant.

[1] Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning.
[2] Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs.
[3] Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond.
[4] Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Models.
[5] UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models.
[6] LLM Unlearning via Loss Adjustment with Only Forget Data.
[7] Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.