acceptodds
Under review as a conference paper at ICLR 2027

CROP: Curvature-Restricted Orthogonal Projection for Retrain-Faithful LLM Unlearning

Abstract

Machine unlearning removes the influence of specific training data from a language model without retraining. Existing methods choose their update direction without reference to the joint forget and retain curvature and are graded by suppression, which rewards overshooting the retrained model. We present CROP (Curvature-Restricted Orthogonal Projection), built on the measured premise that memorization concentrates in parameter directions sharply curved under the forget loss and flat under the retain loss. CROP restricts forget ascent to this curvature-selected subspace, projects the step orthogonal to the retain gradient, anchors on retain descent, and selects its operating point by the distance between its forget profile and the retrained model's, or by a calibrated oracle-free rule. On TOFU, under a leakage-free protocol that selects checkpoints on validation authors and scores held-out ones, CROP leads RMU, the strongest baseline, on held-out forget quality on two of three seeds at a utility cost; as an analysis upper bound, its oracle-best checkpoint lies within KS – of the retrained model on every seed, close to the – spread between independent retrains. On MUSE it lands closest to the retrained reference for two 7B families, and on WMDP, which has no retrain oracle, a supplementary suppression test shows it suppresses hazardous knowledge further than RMU while remaining functional. Under full fine-tuning, LoRA, and partial-forget relearning attacks it recovers no faster than the attacked retrain oracle, while RMU, NPO, PDU, and WGA rebound to near pre-unlearning knowledge.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
curvature-aware optimization, orthogonal projection, knowledge removal, targeted forgetting, machine unlearning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ABKKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CROP, a curvature-guided approach to approximate LLM unlearning. Motivated by the observation that certain parameter directions exhibit high curvature under the forget loss but low curvature under the retain loss, CROP restricts forget-gradient ascent to a dynamically estimated subspace, applies retain-gradient orthogonalization, and incorporates a retain-descent anchor and a bounded loss gate.

Critical strengths
  1. The central idea of exploiting differences between forget and retain curvature is well motivated. The curvature measurements and random-subspace controls provide evidence that the selected directions are more useful for unlearning than arbitrary low-dimensional restrictions.
  2. The paper recognizes that unlearning trajectories can be non-monotonic and that excessive suppression need not approximate retraining. Its separation of oracle-best and validation-selected results is useful.
Critical weaknesses
  1. While the near-retrain results achieved via oracle-best checkpoint selection serve as an informative upper bound, they should not be conflated with deployable or validation-selected performance. More fundamentally, matching the marginal distribution of forget-set truth ratios does not guarantee instance-level behavioral alignment with the retrained model; two models can share identical truth-ratio distributions while assigning substantially different likelihoods to individual examples. This discrepancy limits the interpretability of KS-based faithfulness, particularly when retain-side utility or privacy metrics remain far from the retrained reference.

  2. Although the observed low retain-to-forget curvature ratios are intriguing, they do not by themselves establish that the selected directions specifically encode memorization. Because the procedure explicitly selects for high forget-curvature directions, a low ratio may be a natural artifact of the selection criterion itself. In this context, random directions serve as a relatively weak baseline, as they typically exhibit negligible overlap with high-curvature directions in high-dimensional parameter spaces. Furthermore, the oracle-free selection strategy relies heavily on properties unique to TOFU's synthetic fine-tuning setup, specifically, that the pretrained base model has never encountered the fictitious forget authors. Its generalizability to realistic unlearning scenarios, such as WMDP deletion requests (Li et al., 2025), remains unclear.

  3. A core component of CROP is the orthogonal projection of forget gradients to mitigate interference with retained knowledge. However, closely related principles have already been explored by prior machine unlearning methods, including PGU (Hoang et al., 2024), OrthoGrad (Shamsian et al., 2025), and recent gradient-conflict approaches for LLM unlearning such as SAGO/PCGrad (Xiao et al., 2026), GRU (Wang et al., 2025), GU (Zhou et al., 2025), and UAM (Kim et al., 2026),

reference

Li, Nathaniel, et al. "The wmdp benchmark: Measuring and reducing malicious use with unlearning." ICML 2025.

Hoang, Tuan, et al. "Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection." WACV 2024.

Kim, Hoki, et al. "Unlearning-Aware Minimization." NeurIPS 2026.

Patel, Gaurav, and Qiang Qiu. "Learning to Unlearn While Retaining: Combating Gradient Conflicts in Machine Unlearning." arxiv.org/abs/2503.06339.

Shamsian, Aviv, et al. "Go Beyond Your Means: Unlearning with Per-Sample Gradient Orthogonalization." arxiv.org/abs/2503.02312.

Zhou, Duo, et al. "Geometric-Disentanglement Unlearning." arxiv.org/abs/2511.17100.

Wang, Yue, et al. "GRU: Mitigating the Trade-off between Unlearning and Retention for LLMs." ICML 2025.

Xiao, Zeguan, et al. "Modeling LLM Unlearning as an Asymmetric Two-Task Learning Problem." ACL 2026.

Questions for authors

Please see the weakness.

Reviewer_JbL2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Authors provide an interesting take on machine unlearning, identifying directions in parameter space that encourage forgetting while minimizing damage to retained knowledge. They employ a curvature restricted orthogonal projection (CROP) approach to encourage forgetting while minimizing damage to retained knowledge. Results across several benchmarks such as TOFU and MUSE show promising improvement over existing methods although these gains are less convincing under stricter evaluation and can come at the expense of retained model performance

Critical strengths

The authors provide an interesting and somewhat novel application of curvature-based optimization to machine unlearning. The big contribution is really empirical rather than theoretical, although how the Pt matrix is constructed has some novelty through the lens of unlearning. Essentially they are trying to find a direction that simultaneously maximizes the forget and minimize the retain loss which they describe in Eq. 2. They put forward a set of interesting heuristic steps to approximate this goal, in a computationally efficient manner. Empirical results are strong, achieving state of the art (TOFU and MUSE) or competitive (WMDP).

Critical weaknesses

For the weaknesses I come back to Eq 2. I was left wondering throughout the entire paper why the authors did not attempt to directly solve this through simultaneous diagonalization (i.e. the Rayleigh quotient). It seems the solution that the authors are pursuing are suboptimal in this case, further it seems that similar approach that leverages Hessian-vector products could be pursued to make this computationally efficient. I am left struggling why this was not pursued, or at least discussed somewhere in the paper.

Questions for authors

Could the authors walk me through the thinking of why their approach is advantageous over the one discussed in criticial weaknesses. My feeling is that it is most likely computational, but it would be good to at least have a discussion or some analysis on this point.

Reviewer_oMRcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CROP unlearns by pushing the model only along directions that matter a lot for the forget data but very little for the retain data. It also blocks steps that would hurt the retain data. Instead of making the model forget as hard as possible, it aims to look like a model retrained without the forget data, and picks checkpoints by how close they are to that retrained model.

Critical strengths
  1. Aiming to match the retrained model, rather than forgetting as much as possible, is the right goal.
  2. The paper is careful and honest: it separates oracle-picked results from fair held-out results and says where it loses.
Critical weaknesses
  1. The main metric can be passed by a broken model. In Table 3, the version without the retain loss has utility 0 but still passes the KS test (p = 0.065). Since the metric only looks at forget data, it should be paired with a retain-side check before calling a model "retrain-faithful".
  2. CROP loses a lot of utility. On TOFU it keeps 0.415, against 0.523 for RMU and 0.591 for the retrained model. On MUSE-Qwen it has the lowest retain knowledge of all methods (0.253 vs 0.311 for the retrained model). On WMDP, MMLU drops from 0.589 to 0.452.
  3. Each part has close prior work. SSD already compares forget and retain importance, and CROP's filter is a low-rank version of that. Earlier work projects unlearning steps away from the retain gradient, and the retain loss is GradDiff. The novelty is mainly the combination.
Questions for authors
  1. With the same checkpoint picking and tuning for all methods, does CROP beat RMU by more than the gap between two retrains?
  2. Removing the "flat for retain" filter barely changes the result. Is it needed?
  3. Why does CROP relearn faster than the retrained model at lr 2e-5 (Table 19)?

Comments on the paper

1 comment

Dear reviewers,

We’re more than halfway through the discussion period and the authors have responded to your reviews. This is a friendly reminder to check these responses and respond if necessary.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.