acceptodds
Under review as a conference paper at ICLR 2027

Steepness Is Not Selectivity: Why Post-hoc Unlearning Misses the Retraining Direction

Abstract

Machine unlearning is usually framed as a search for a better objective. That framing rests on a premise that is rarely tested: that the update retraining would make is one a post-hoc procedure can find. We test it directly. Using as an oracle a model retrained without the forget set from the same base, we find the obstacle is not entanglement: the straight path to that oracle is almost perfectly selective — on our primary family forget loss rises 140× while retain loss rises by at most 1.19× and ends below where it started — yet standard unlearning updates that reach a comparable forget level are orthogonal to it, at cos ≈ 0.009 in a space where two runs of the same method aimed at disjoint fact sets align at 0.035. The mechanism is a separation between steepness, forget loss gained per unit of weight movement, and selectivity, forget loss gained per unit of retain damage: gradient descent maximizes the first, unlearning wants the second, and nothing ties them together. Plain SGD, where the steepest-descent argument applies literally, is steeper still and barely selective at all, so the preconditioning in the optimizer people actually use is a partial remedy rather than a confound. Preconditioning ascent by the retain curvature instead, as the mechanism prescribes, cuts collateral damage by an order of magnitude and moves further from the oracle direction: the local optimum of selectivity is not the retraining direction. Two further results say the premise fails even if the search succeeded. Being better aligned does not help: weight arithmetic is 7–8× closer to the oracle direction and no more selective, and across the four methods alignment does not track selectivity. And the target does not transfer: measured from a different full-data solution the retraining displacement is nearly orthogonal (cos = 0.029) to the one measured here, so there is no single vector to compute once and apply. The existence claim holds in three model families spanning 0.5–1.5B, and the geometry in the two where we ran the full comparison.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
machine unlearning, LLM unlearning, weight-space geometry, retraining, optimization geometry, preconditioning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZV6eOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

The paper asks whether post-hoc unlearning updates for LLMs move in the direction that retraining would take. Several main findings: 1. A selective path exists. The straight line from M0 to the retrained model is highly selective. 2. Gradient-based unlearning updates are nearly orthogonal to it. Task-vector negation is better aligned, yet no more selective. 3. The paper separates steepness (forget gain per unit of weight movement) from selectivity (forget gain per unit of retain damage): Euclidean ascent maximizes steepness; A local quadratic model prescribes preconditioning by the retain curvature; Ascent preconditioned by a diagonal retain Fisher cuts retain damage by about 10×, yet moves further from the oracle direction. 4. The direction does not transfer. Despite that the paper does not propose a remedy or a better unlearning algorithm, I personally think this paper has a meaningful perspective to better characterize the difficulty of unlearning.

Critical strengths
  1. Well-controlled design. A mirror forget set, a norm-matched reseed null and a calibrated “cosine ladder” make small weight-space angles interpretable.
Critical weaknesses
  1. Two retrain seeds share only 0.38 of it, and it transfers across checkpoints at only 0.029. The orthogonality may therefore reflect the reference rather than a failure of the methods? Alignment with the component specific to the forget set is never measured.

  2. “Matched forgetting” is far from retraining. The methods are compared at forget NLL 0.26, against the oracle’s 3.08, after only 18–120 steps. At that point they still reproduce most answers (ROUGE-L ≈ 0.78, against 0.34 for the oracle).

  3. Limited scope and partial novelty. The evidence comes from one synthetic benchmark and models of at most 1.5B parameters trained to memorization. On the only real-text run, the proposed mechanism disappears while orthogonality persists. Many controls are single runs. Moreover, preconditioning by the retain curvature is the classical Fisher/Newton argument, and SOUL has already shown it for LLM unlearning.

  4. Overclaims. No relearning, extraction or membership-inference test supports the implication that orthogonal updates leave the data behind. The preregistration cannot be verified.

Questions for authors

Please refer to Weaknesses. Besides: 1. How do the updates align with the forget-set-specific part of the retrain displacement, for example averaged over several retrain seeds? 2. Do alignment and selectivity change when the methods are run to oracle-level forgetting? 3. At the same forget level, do orthogonal updates leave more recoverable information than the oracle path (relearning, extraction, membership inference)? 4. What explains orthogonality on wikitext, where the steepness gap is absent?

1 comment

Thank you again for your review. I wanted to follow up on my November 10 response and the additional experiments. If any of your concerns remain open, I’d be happy to clarify the results or address any questions during the discussion period.

Reviewer_F6GyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the differences between post-hoc unlearning updates and the weight-space direction towards a retrained model. In this paper, the authors distinguish between "steepness" (forgetting per unit of parameter movement) and "selectivity" (forgetting relative to retain damage). They find that the retraining direction is highly selective but flat, whereas existing unlearning methods' directions are steeper but less selective. The findings are supported by experiments across several model families.

Critical strengths

The paper investigates an interesting question about the relationship between approximate unlearning and retraining from scratch. The identified distinction between "steepness" and "selectivity" provides a useful perspective on why gradient-based unlearning methods may struggle to achieve selective forgetting.

I also appreciate the extensive empirical analysis, including comparisons across model families and additional controls to investigate alternative explanations.

Critical weaknesses
  1. The authors identify the retraining direction as locally unattractive to gradient-based unlearning, but my main concern is the significance of this finding. While retraining provides a natural reference for successful unlearning, i'm not sure why approximate unlearning should aim to recover the same direction in parameter space. There may be many alternative directions that achieve comparable unlearning (as some of the authors' own findings suggest). Therefore, in my opinion, parameter-space direction is not the only desirable direction, and failing to recover this direction doesn't necessarily equal a limitation of approximate unlearning.

  2. The WikiText experiments in Appendix H show that the steepness gap largely disappears, while the unlearning updates remain nearly orthogonal to the retraining direction. This suggests that the steepness-selectivity distinction may not fully explain the observed misalignment across different settings. Although the authors acknowledge this limitation, I think it raises an important question about how broadly their proposed mechanism applies.

Questions for authors

Given that retraining itself can produce substantially different parameter-space directions, why should parameter alignment with a particular retraining direction be considered important for approximate unlearning? How does this relate to behavioral equivalence, e.g. equivalence in output space?

1 comment

Thank you again for your review. I wanted to follow up on my November 10 response and the additional experiments. If any of your concerns remain open, I’d be happy to clarify the results or address any questions during the discussion period.

Reviewer_3mLjOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to understand why standard unlearning updates differ from retraining without the forget data. On synthetic data, it finds that a path toward the retrained model better preserves other knowledge, while standard updates increase the forget loss more sharply per unit of weight change but also damage the information we'd like to preserve.

Critical strengths

This paper is one of the few much needed investigations toward understanding how unlearning works and why current methods differ from retraining.

Critical weaknesses
  1. The main comparisons stop well before successful forgetting. At the selected checkpoints, forget rouge-l is in 0.77–0.80 for llama and about 0.94 for qwen, compared with ~0.33 after retraining. The paper does not show whether its main findings persist when methods approach retraining-level removal.
Questions for authors

see above

Comments on the paper

1 comment

Summary of revisions (November 17). We have uploaded a second revised PDF; changes since the November 10 revision are shown in red (earlier changes remain in blue). The main text still ends on page 10.

  • Title: "Why Post-hoc Unlearning Misses the Retraining Direction" is now "One Reason Post-hoc Unlearning Misses the Retraining Direction", matching the scope already stated in Sections 1 and 5.
  • Abstract: the Qwen alignment ratio is given for bf16 (a ninth) and for fp32 weights (a third).
  • Appendix O: projection of the fp32 GA and GA-prec updates onto the shared retraining component, requested by Reviewer ZV6e.
  • Section 2, Reproducibility Statement, Appendices M and O: the five criteria added during the discussion period are no longer described as registered, and the ordering of A-48 is qualified.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.