- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CROP, a curvature-guided approach to approximate LLM unlearning. Motivated by the observation that certain parameter directions exhibit high curvature under the forget loss but low curvature under the retain loss, CROP restricts forget-gradient ascent to a dynamically estimated subspace, applies retain-gradient orthogonalization, and incorporates a retain-descent anchor and a bounded loss gate.
Critical strengths
- The central idea of exploiting differences between forget and retain curvature is well motivated. The curvature measurements and random-subspace controls provide evidence that the selected directions are more useful for unlearning than arbitrary low-dimensional restrictions.
- The paper recognizes that unlearning trajectories can be non-monotonic and that excessive suppression need not approximate retraining. Its separation of oracle-best and validation-selected results is useful.
Critical weaknesses
-
While the near-retrain results achieved via oracle-best checkpoint selection serve as an informative upper bound, they should not be conflated with deployable or validation-selected performance. More fundamentally, matching the marginal distribution of forget-set truth ratios does not guarantee instance-level behavioral alignment with the retrained model; two models can share identical truth-ratio distributions while assigning substantially different likelihoods to individual examples. This discrepancy limits the interpretability of KS-based faithfulness, particularly when retain-side utility or privacy metrics remain far from the retrained reference.
-
Although the observed low retain-to-forget curvature ratios are intriguing, they do not by themselves establish that the selected directions specifically encode memorization. Because the procedure explicitly selects for high forget-curvature directions, a low ratio may be a natural artifact of the selection criterion itself. In this context, random directions serve as a relatively weak baseline, as they typically exhibit negligible overlap with high-curvature directions in high-dimensional parameter spaces. Furthermore, the oracle-free selection strategy relies heavily on properties unique to TOFU's synthetic fine-tuning setup, specifically, that the pretrained base model has never encountered the fictitious forget authors. Its generalizability to realistic unlearning scenarios, such as WMDP deletion requests (Li et al., 2025), remains unclear.
-
A core component of CROP is the orthogonal projection of forget gradients to mitigate interference with retained knowledge. However, closely related principles have already been explored by prior machine unlearning methods, including PGU (Hoang et al., 2024), OrthoGrad (Shamsian et al., 2025), and recent gradient-conflict approaches for LLM unlearning such as SAGO/PCGrad (Xiao et al., 2026), GRU (Wang et al., 2025), GU (Zhou et al., 2025), and UAM (Kim et al., 2026),
reference
Li, Nathaniel, et al. "The wmdp benchmark: Measuring and reducing malicious use with unlearning." ICML 2025.
Hoang, Tuan, et al. "Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection." WACV 2024.
Kim, Hoki, et al. "Unlearning-Aware Minimization." NeurIPS 2026.
Patel, Gaurav, and Qiang Qiu. "Learning to Unlearn While Retaining: Combating Gradient Conflicts in Machine Unlearning." arxiv.org/abs/2503.06339.
Shamsian, Aviv, et al. "Go Beyond Your Means: Unlearning with Per-Sample Gradient Orthogonalization." arxiv.org/abs/2503.02312.
Zhou, Duo, et al. "Geometric-Disentanglement Unlearning." arxiv.org/abs/2511.17100.
Wang, Yue, et al. "GRU: Mitigating the Trade-off between Unlearning and Retention for LLMs." ICML 2025.
Xiao, Zeguan, et al. "Modeling LLM Unlearning as an Asymmetric Two-Task Learning Problem." ACL 2026.
Questions for authors
Please see the weakness.
Dear reviewers,
We’re more than halfway through the discussion period and the authors have responded to your reviews. This is a friendly reminder to check these responses and respond if necessary.
AC