- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the instability of gradient-based LLM unlearning and argues that cross-entropy ascent on the forget set can cause excessive growth in the weights and gradients of feedforward layers. To address this issue, it proposes bounded parameter-efficient unlearning, which applies a bounded function, primarily a sine mapping, to LoRA updates in MLP layers. Experiments on TOFU, TDEC, and MUSE across several model families, together with additional vision and sequential-unlearning evaluations, suggest that the method improves forgetting while largely preserving model utility.
Critical strengths
-
The paper explains the instability of full-parameter unlearning and motivates bounded parameter-efficient updates.
-
The experiments cover multiple objectives, models, and benchmarks, showing improved stability with competitive forgetting performance.
Critical weaknesses
i) The methodological novelty is limited relative to closely related work.
The paper presents the instability of gradient-ascent unlearning as a central motivation and theoretical contribution. However, this problem has already been studied extensively. NPO analyzes the catastrophic collapse caused by gradient ascent and theoretically explains how its objective slows this process [a]. LoKU specifically observes that combining gradient ascent with LoRA leads to unstable optimization and poor retained utility, motivating a bounded forgetting objective [b]. SimNPO similarly describes gradient-ascent unlearning as an uncontrolled departure from the pretrained model [c]. The layer-level analysis in this paper offers another view of the same behavior, but it is unclear what fundamentally new understanding it provides beyond these earlier results.
The proposed solution is also closely related to existing parameterizations. Applying a sine function to low-rank updates was introduced for parameter-efficient fine-tuning in prior work [d], while SineProject recently used sinusoidally modulated trainable parameters to stabilize multimodal unlearning [e]. The main contribution therefore appears to be applying an existing sine-based low-rank parameterization to feedforward adapters for LLM unlearning, together with a broader empirical evaluation. The paper should clarify its conceptual difference from these methods and identify which part of the formulation is genuinely new.
ii) The theoretical guarantees for the optimization dynamics are incomplete.
Equation (10) bounds the effective adapter update elementwise, but this only controls its forward contribution when the input is bounded. It does not bound the two trainable low-rank factors or their gradients. Although the derivative of the sine function is bounded, the gradient of each factor still depends multiplicatively on the other factor. Either gradient may therefore grow if the other factor becomes large. The factors can also be rescaled in opposite directions without changing their product, further showing that a bounded effective update does not control the individual factors. The claim in Section 3.3 that the bounded Jacobian prevents gradient explosion throughout the full backward pass consequently requires additional assumptions or explicit control of the factor norms.
The extension from pure gradient ascent to gradient difference is also not justified. A lower bound on the magnitude of the forgetting gradient and an upper bound on the magnitude of the retaining gradient do not prevent their weighted contributions from pointing in opposing directions and cancelling each other. The paper neither establishes that the forgetting contribution consistently dominates by a fixed margin nor imposes a condition on the angle between the two gradients, despite gradient conflict being well documented in multi-objective optimization [f]. Moreover, boundedness of the retain loss is assumed rather than proved along the combined ascent and descent trajectory, since the forgetting update may increase the retain loss. The paper should revise these results, state the missing conditions explicitly, and report the individual factor norms and gradients during training.
[a] Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning. COLM 2024.
[b] Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs. ICLR 2025.
[c] Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning. NeurIPS 2025.
[d] Efficient Learning With Sine-Activated Low-rank Matrices. ICLR 2025.
[e] SineProject: Machine Unlearning for Stable Vision-Language Alignment. CVPR 2026.
[f] Gradient Surgery for Multi-Task Learning. NeurIPS 2020.
Questions for authors
-
What is the main technical novelty over prior analyses of gradient-ascent instability and existing sine-based low-rank parameterizations, particularly NPO, LoKU, and SineProject?
-
What prevents the two individual low-rank factors and their respective gradients from growing without bound when the method only constrains the sine-transformed effective adapter update and its local derivative?
-
In Theorem 2, what condition prevents the retaining and forgetting gradients from cancelling each other? In addition, how does the rank-one expression in Equation (53) extend to a total gradient averaged over multiple retaining and forgetting examples?