acceptodds
Under review as a conference paper at ICLR 2027

RADNPO: Reference-free Adaptive Negative Preference Optimization for LLM Unlearning

Abstract

Large language models (LLMs) can memorize sensitive, private, or copyrighted content during pre-training, making machine unlearning necessary for removing targeted knowledge. Recent preference optimization (PO)-based unlearning methods improve stability over gradient ascent (GA)-based methods by introducing alignment-style objectives, which effectively suppress the probability of forget targets. However, target suppression alone does not sufficiently constrain the next token distribution after unlearning. Existing methods provide limited control over how suppressed probability mass is redistributed and insufficiently adapt forgetting strength to target confidence and distributional concentration. Even after target suppression, probability mass may remain concentrated on a few non-target tokens, potentially producing repetitive or uninformative outputs. To address these limitations, we propose Reference-free ADaptive Negative Preference Optimization RADNPO, which explicitly guides next token probability redistribution. Specifically, RADNPO contrasts each forget target with alternative tokens favored by the current next token distribution and adaptively modulates token level forgetting strength using target confidence and next token concentration. Experiments on TOFU and MUSE demonstrate that RADNPO achieves a better trade-off between forgetting quality and model utility than current baselines. Codes are available at https://anonymous.4open.science/r/RADNPO-7B721538.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Safety, Machine Unlearning, Preference optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_999gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes RADNPO, a reference-free adaptive negative preference optimization method for LLM unlearning. It contrasts forget targets with high-probability alternatives and adapts token-level forgetting strength based on target confidence and partial entropy. The method is evaluated on the TOFU and MUSE benchmarks.

Critical strengths

Valuable problem formulation. The paper observes that existing unlearning methods can reduce the probability of forget targets but do not explicitly guide how the suppressed probability mass is redistributed. This broadens the discussion from whether the target is suppressed to how the model generates responses after unlearning. The authors further suggest that excessive concentration in the post-unlearning distribution may be associated with repetitive and less diverse outputs, offering a meaningful direction for improving unlearning objectives.

Broad experimental evaluation. The authors compare multiple unlearning methods on TOFU and MUSE, assessing both forgetting effectiveness and retained utility. In addition to the main results and ablation studies, they evaluate repetition, lexical diversity, and semantic behavior, and provide supplementary analyses of hyperparameter sensitivity, random-seed variation, and performance on another model. These experiments examine the method from several perspectives and help clarify its effectiveness and limitations.

Critical weaknesses

The claim that the post-unlearning distribution may remain overly concentrated does not, by itself, establish that generation quality will decline. If probability mass is concentrated on a single reasonable and contextually appropriate alternative token, generation may remain fluent. Concentration becomes a quality problem only when it leads to repetitive, vacuous, or contextually inappropriate outputs. The authors should therefore provide empirical evidence linking excessive post-unlearning concentration to generation degradation, and show that the problem is not limited to isolated cases. Without such evidence, the paper may identify a change in distributional shape without demonstrating that it constitutes a practical limitation worth addressing.

The discussion in the third and fourth paragraphs of the Introduction is not clearly connected. The authors first argue that existing methods do not control where the probability mass suppressed from the forget target is redistributed, and then suggest that it may shift toward implausible, repetitive, or uninformative alternatives. They subsequently argue that forgetting strength does not jointly account for target confidence and distributional concentration. However, the paper does not clearly distinguish these two limitations or explain how they relate to each other. As a result, it is difficult to determine whether the central motivation is the lack of guidance for probability redistribution or the lack of adaptive forgetting strength, and how each limitation leads to observable generation problems.

The preference-optimization subsection is weakly connected to the paper’s central problem. It mainly surveys the general development of RLHF, DPO, KTO, SimPO, and ORPO, rather than focusing on the application of preference optimization to LLM unlearning and its limitations. The related-work discussion should explain more clearly how preference optimization has been used for LLM unlearning and how its limitations motivate the proposed method, rather than focusing primarily on general preference-optimization techniques.

Questions for authors

The paper argues that the post-unlearning distribution may remain overly concentrated. Does this concentration necessarily lead to degraded generation quality? Could the authors provide evidence linking distributional concentration to repetitive, uninformative, or contextually inappropriate outputs, and clarify whether this effect occurs consistently rather than only in isolated cases?

The paper appears to identify two limitations: insufficient guidance on how probability mass is redistributed after suppressing a forget target, and forgetting strength that does not adapt to the current distributional state. Which limitation is the central motivation for RADNPO, and how are the two related? In particular, what role does the adaptive forgetting strength play beyond the contrastive objective that guides probability redistribution?

Reviewer_u62nOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies preference-optimization (PO)-based unlearning for LLMs. It argues that existing objectives such as NPO and SimNPO only suppress the probability of forget targets and leave unspecified how the suppressed probability mass is redistributed among non-target tokens. The authors identify a failure mode they term low-entropy forgetting: the target probability is low, but the next-token distribution remains concentrated on a few non-target tokens, which can lead to repetitive or uninformative generations.

To address this, the paper proposes RADNPO, a reference-free, token-level unlearning objective with two components:

  1. Contrastive log-odds objective. For each forget-target token, the target probability is contrasted with the aggregate probability of the top-K non-target tokens under the current next-token distribution, giving ri=log⁡(πitarget/πialt)r_i = \log(\pi^{\text{target}}_i / \pi^{\text{alt}}_i). The loss is −log⁡σ(−βiriclamp)-\log\sigma(-\beta_i r_i^{\text{clamp}}), where a tanh-based soft clamp bounds extreme margins.
  2. Adaptive scaling. The token-wise coefficient is βi=β0⋅sg(πθ(yi)γ⋅exp⁡(λ(Href−Hi)))\beta_i = \beta_0 \cdot \mathrm{sg}\big(\pi_\theta(y_i)^\gamma \cdot \exp(\lambda(H_{\text{ref}} - H_i))\big). It combines a target-confidence factor with an entropy factor based on the top-K partial entropy HiH_i, so that tokens with high target confidence or a concentrated next-token distribution receive larger coefficients.

The forget loss is combined with a standard NLL retain loss. The paper also analyzes token states in a two-dimensional space of target probability and partial entropy (target-dominated, dispersed, and collapsed regimes) and visualizes how NPO and RADNPO evolve in this space during training.

Claimed contributions:

  • identifying target suppression combined with a concentrated post-unlearning distribution as an under-examined failure mode of PO-based unlearning;
  • the RADNPO objective, which combines contrastive log-odds with confidence- and entropy-based adaptive scaling;
  • an improved forgetting–utility trade-off over existing baselines.

Validation. The main experiments are on TOFU Forget10 with LLaMA-2-7B, and on MUSE News (LLaMA-2-7B) and MUSE Books (ICLM-7B), using the OpenUnlearning framework. RADNPO is compared against GA, GradDiff, DPO, NPO, SimNPO, AltPO, UNDIAL, RMU, WGA, and SatImp. Additional results include:

  • TOFU experiments on LLaMA-3-8B and Qwen3-4B;
  • a component ablation;
  • hyperparameter sensitivity and a three-seed study;
  • a retain–forget loss-balance sweep;
  • generation-level degeneration statistics (n-gram repetition, Distinct-3, Self-BLEU) and a GPT-4o semantic evaluation;
  • layer-wise representation probing;
  • a recovery experiment in which fine-tuning on the forget set largely restores the forgotten behavior.
Critical strengths

Strengths

  • Relevant problem perspective. Asking not only whether the target is suppressed but also where the suppressed probability mass goes is a meaningful angle on PO-based unlearning. The two-dimensional state-space visualization (target probability × partial entropy) is an intuitive diagnostic tool.
  • Simple and lightweight method. RADNPO requires no reference model and no extra forward passes, and it is easy to implement on top of existing unlearning pipelines.
  • Evaluation beyond standard metrics. In addition to the TOFU/MUSE metrics, the paper reports generation-degeneration statistics, an LLM-judge semantic evaluation, and representation probing. These are useful complements to likelihood-based metrics.
  • Transparent about limitations. The paper includes a recovery experiment showing that the forgotten behavior is largely restored after fine-tuning on the forget set, and it explicitly frames RADNPO as approximate, recoverable unlearning rather than irreversible deletion.
  • Broad baseline coverage and reproducibility. The comparison covers ten unlearning methods, including recent ones such as WGA, SatImp, UNDIAL, and AltPO, across three model families. Code is released.
Critical weaknesses

Weaknesses

  1. Limited novelty relative to closely related work. Both the motivation and the analysis substantially overlap with two recent works that are neither cited nor discussed.

    • EUA [1] identifies the same failure mode. Target-suppression objectives place no constraint on the non-target label space, so probability mass is redistributed arbitrarily. EUA further characterizes the resulting "unconstrained collapse" toward a high-confidence non-target token, tracked via max-softmax-probability trajectories. This is essentially the paper's "collapsed / low-entropy forgetting regime" and the analysis in Fig. 3. EUA addresses it with a distribution-level (free-energy) constraint over the full vocabulary, and explicitly argues that heuristics built on a hand-specified local token set are fragile. RADNPO's top-K contrast is such a heuristic.
    • BS-T/BS-S [2] identifies the related "squeezing effect", in which suppressed mass flows into high-likelihood regions, and uses the model's own top-k tokens as the token-level signal. However, [2] suppresses these tokens, because they often correspond to semantically related rephrasings of the forgotten content, whereas RADNPO promotes them. The paper provides no evidence that the promoted alternatives are not paraphrases of the forget target (e.g., paraphrased-answer probability or semantic leakage).
    • Other components also overlap with existing methods. The confidence factor πθ(yi)γ\pi_\theta(y_i)^\gamma is the same weighting form as WGA [3], which is used as a baseline. Contrasting the target against non-target mass at the token level is close to CATNIP [4], with the contrast restricted here to the top-K tokens.

    The paper should clearly position itself against these works and compare with EUA and BS-T/BS-S experimentally; otherwise its conceptual contribution is unclear.

  2. The main results are not reproduced by the reported multi-seed runs. In Table 1, RADNPO reports EM 0.382, PrivLeak 27.7, and FQ 0.878 (the γ=1\gamma = 1 default). None of the three seeds in Table 8 match these numbers: they give EM 0.464–0.472, ES 0.046–0.047, and PrivLeak 5.8–17.6, which instead match the γ=2\gamma = 2 rows of Table 7. This suggests the multi-seed study used a different configuration from the main results, or that the main result comes from a single favorable run. In addition, FQ varies from 0.046 to 0.850 across seeds, which is larger than many of the gaps used to claim "best FQ" in Table 1.

  3. Inconsistent metric definitions and reporting.

    • PrivLeak. Under the definition in Appendix D.2, PrivLeak for the Retrain model should be exactly 0, yet Retrain is reported as 22.6 (TOFU / LLaMA-2), 24.1 (LLaMA-3), and 26.7 (Qwen3). The desired value is also stated inconsistently: "→ 0" in the main tables and "→ 1" in Tables 6–8.
    • Direction of "better". Lower forget-side metrics are treated as better even when they fall far below the Retrain reference (TOFU EM 0.382 vs. 0.665 for Retrain; MUSE Books EM 0.259 vs. 0.587). This pattern indicates over-unlearning rather than an advantage.
    • Missing and unusual metrics. MUSE PrivLeak, the standard MUSE metric for detecting over- and under-unlearning, is omitted. FQ is redefined as −log⁡10p-\log_{10} p, reversing the usual TOFU convention, which complicates comparison with prior work.
    • Utility above the original model. RADNPO's MU (0.704) exceeds both the Original (0.658) and Retrain (0.642) models, and the same pattern appears on LLaMA-3 and Qwen3. This is not explained.
  4. Questionable baselines and limited supporting evidence.

    • SatImp appears misconfigured. On TOFU, SatImp yields EM 0.991 and ES 0.905, nearly identical to the original model. On MUSE Books, its numbers match the original model almost exactly (VerbMem 0.997). By contrast, EUA [1] reports SatImp reducing VerbMem and KnowMem to 0 on the same MUSE Books / ICLM-7B setting. No hyperparameter search is described for any baseline, while RADNPO uses different hyperparameters for TOFU and MUSE.
    • The central claim rests on comparisons with NPO only. Both the generation-degeneration statistics (Table 3) and the GPT-4o evaluation (Table 4) compare RADNPO against NPO alone, without SimNPO, WGA, RMU, UNDIAL, or AltPO. The ablation (Fig. 4) is reported only as a bar chart without numbers.
    • Gains are not consistent across settings. On LLaMA-3-8B, RADNPO's Forget TR (0.660) remains close to the original model (0.686) and far from Retrain (0.562), and its FQ (3.24) is nearly identical to NPO's (3.36). On MUSE Books, its retain KnowMem (0.515) is clearly below Retrain (0.668) and UNDIAL (0.632).

References

[1] Puning Yang, Junchi Yu, Qizhou Wang, Philip Torr, Bo Han, Xiuying Chen. Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning. ICML 2026.

[2] Kemou Li, Qizhou Wang, Yue Wang, Fengpeng Li, Jun Liu, Bo Han, Jiantao Zhou. LLM Unlearning with LLM Beliefs. ICLR 2026.

[3] Qizhou Wang, Jin Peng Zhou, Zhanke Zhou, Saebyeol Shin, Bo Han, Kilian Q. Weinberger. Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond. ICLR 2025.

[4] Zhengbang Yang, Yisheng Zhong, Junyuan Hong, Zhuangdi Zhu. CATNIP: LLM Unlearning via Calibrated and Tokenized Negative Preference Alignment. arXiv:2602.02824, 2026.

Questions for authors

Questions

  1. Relation to EUA and BS-T/BS-S.

    • What is the conceptual difference between RADNPO's "low-entropy forgetting" and the non-target collapse analyzed in EUA [1], and between RADNPO and the squeezing effect in BS-T/BS-S [2]? You do not need to provide an experimental comparison with both methods.
    • Since [2] suppresses the model's top-k tokens while RADNPO promotes them, could you show that the promoted alternatives are not semantic rephrasings of the forget target? For example: paraphrased-answer probability on TOFU, semantic leakage under an LLM judge, or a variant that suppresses instead of promotes the top-K alternatives. (Some case study)
  2. Seeds and configuration. Which hyperparameter configuration was used for the three-seed study in Table 8? Its numbers match the γ=2\gamma = 2 rows of Table 7 rather than the main result in Table 1. Could you report mean ± std over multiple seeds for the default configuration, for both RADNPO and the main baselines? Given that FQ ranges from 0.046 to 0.850 across seeds, how should the FQ comparisons in Table 1 be interpreted?

  3. Metrics.

    • Why is the Retrain model's PrivLeak non-zero (22.6 / 24.1 / 26.7), given the definition in Appendix D.2? Is the target value 0 or 1?
    • Could you report MUSE PrivLeak?
    • How should EM and ES values far below the Retrain reference be interpreted, if not as over-unlearning?
    • What explains RADNPO's MU exceeding that of the Original model?
  4. Baseline configuration. What hyperparameters and tuning protocol were used for each baseline? In particular, SatImp shows almost no forgetting on TOFU and matches the Original model on MUSE Books, while EUA [1] reports substantial forgetting for SatImp on the same MUSE Books / ICLM-7B setting. Adjusting the hyperparameter for the coefficient between the forget and retain loss objective may help.

  5. Supporting evaluations.

    • Could you extend the generation-degeneration statistics (Table 3) and the GPT-4o evaluation (Table 4) to the other baselines, at least SimNPO, WGA, RMU, UNDIAL, and AltPO?
    • Could you report the ablation results (Fig. 4) numerically?
    • For the GPT-4o judge, please provide the evaluation prompt and some measure of agreement with human judgments.

Minor comments

  • Citation errors. The "right to be forgotten" is attributed to Sakaguchi et al. (2021), which is the WinoGrande paper. Kurmanji et al. (2023) is cited for text-to-image unlearning, but it addresses image classification. In the introduction, SimNPO is attributed to Meng et al. (2024), which is SimPO.
  • Inconsistent range of γ\gamma. The text states γ≥1\gamma \ge 1, later discusses γ=0\gamma = 0 and γ>0\gamma > 0, and the MUSE experiments use γ=0.5\gamma = 0.5.
  • Typos. For example, "next-tokein", "We then the contrastive log-odds…", and "Vi,kalt\mathcal{V}_i, k^{\text{alt}}" in Section 4.2.
  1. Missing related work. Beyond EUA [1] and BS-T/BS-S [2], several directly relevant works are not discussed:
    • Post-unlearning response quality. The paper's central concern is degenerate post-unlearning generation, which recent work addresses explicitly. TRU [5] argues that the untargeted nature of GA-style objectives causes incoherent responses, and specifies the post-unlearning response through reasoning-based targets. LUNAR [6] redirects the representations of forget data toward regions that express an inability to answer, so that the unlearned model produces coherent responses. Both bear directly on the paper's claim that it improves generation quality "without prescribing a specific response". Relatedly, [7] studies honesty in LLM unlearning. This matters because the behavior presented as desirable in Fig. 1, answering with a different fabricated name ("Maria Jose Gutierrez"), can be viewed as a dishonest response rather than successful forgetting.
    • Guiding how the model responds. FLAT [8] guides not only what to forget but also how to respond, via an f-divergence between template and forget answers. This is closely related to RADNPO's goal of guiding probability redistribution.
    • Robustness to relearning. The recovery experiment (Appendix E.3) should be positioned against the relearning literature: [9] proposes smoothness optimization to resist relearning attacks, and [10] shows that even benign, loosely related data can reverse unlearning. A comparison of recovery behavior against baselines would also make that experiment more informative.
    • Evaluation methodology. Extraction Strength (ES), used as a main metric, originates from [11], which also discusses how to compare unlearning methods fairly at comparable operating points. It is attributed only to the OpenUnlearning framework.

[5] Junfeng Liao, Qizhou Wang, Shanshan Ye, Xin Yu, Ling Chen, Zhen Fang. Explainable LLM Unlearning Through Reasoning. ICLR 2026.

[6] William F. Shen, Xinchi Qiu, Meghdad Kurmanji, Alexandru-Andrei Iacob, Lorenzo Sani, Yihong Chen, Nicola Cancedda, Nicholas D. Lane. LLM Unlearning via Neural Activation Redirection. NeurIPS 2025.

[7] Renjie Gu, Jiazhen Du, Yihua Zhang, Sijia Liu. Unlearners Can Lie: Evaluating and Improving Honesty in LLM Unlearning. ACL 2026.

[8] Yaxuan Wang, Jiaheng Wei, Chris Yuhao Liu, Jinlong Pang, Quan Liu, Ankit Parag Shah, Yujia Bao, Yang Liu, Wei Wei. LLM Unlearning via Loss Adjustment with Only Forget Data. ICLR 2025.

[9] Chongyu Fan, Jinghan Jia, Yihua Zhang, Anil Ramakrishna, Mingyi Hong, Sijia Liu. Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond. ICML 2025.

[10] Shengyuan Hu, Yiwei Fu, Zhiwei Steven Wu, Virginia Smith. Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning. ICLR 2025.

[11] Qizhou Wang, Bo Han, Puning Yang, Jianing Zhu, Tongliang Liu, Masashi Sugiyama. Towards Effective Evaluations and Comparisons for LLM Unlearning Methods. ICLR 2025.

Reviewer_JsxyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes RADNPO, a reference-free LLM unlearning method that combines a contrastive log-odds objective with adaptive token-level scaling. It aims to suppress forget tokens while guiding probability mass toward alternative tokens, using target confidence and partial entropy to adjust the forgetting strength. Experiments on TOFU and MUSE evaluate forgetting, model utility, and generation quality.

Critical strengths
  1. The paper considers an interesting problem beyond simply reducing the probability of forget targets. Unlearning can produce repetitive or meaningless responses, and studying how probability mass is redistributed may help explain this issue. Including generation-level measurements alongside standard forgetting metrics is useful.
  2. The method is relatively simple and does not require a reference model. The paper compares several existing unlearning methods and includes component ablations, representation probing, and recovery experiments. These provide some evidence of its behavior, although they are insufficient to support the main claims.
Critical weaknesses
  1. The anonymous repository link provided in the abstract and reproducibility statement is expired and cannot be accessed. This is a serious reproducibility concern.
  2. The novelty is limited. Reference-free unlearning was already explored in SimNPO, while contrastive log-odds optimization is closely related to ORPO. RADNPO mainly combines these existing ideas with confidence-based scaling, entropy weighting, and tanh clipping. The method looks like a combination of existing techniques rather than a fundamentally new approach. The paper does not clearly establish what new insight or mechanism this combination provides.
  3. The proposed objective does not necessarily achieve its stated goal. Eq. (5) compares the forget token against the top-K highest-probability alternatives. Optimizing this objective can further increase the logits of already dominant alternatives. Nothing in the loss explicitly encourages a more balanced or higher-entropy distribution among them. Therefore, the method may simply shift probability mass from one dominant token to another. This directly challenges the central claim that RADNPO prevents low-entropy degeneration through probability redistribution.
  4. The adaptive scaling mechanism is not convincing. In Eq. (10), the scaling coefficient is proportional to the target probability raised to the power of γ\gamma. When the target probability approaches zero, the coefficient also approaches zero, regardless of the entropy factor. The tanh clipping in Eq. (6) further suppresses gradients for extreme log-odds. This seems to contradict the motivation of maintaining optimization pressure in the collapsed forgetting regime. Moreover, the partial entropy in Eq. (8) is not a reliable measure of distributional concentration. Appendix B only derives the gradient of the proposed loss; it does not establish that the method avoids these problems or improves optimization.
  5. The experimental evidence is not convincing, especially on newer models. The main tables still use LLaMA2-7B and ICLM-7B, which are outdated. Although LLaMA3 and Qwen3 results are provided in the appendix, the improvements are mixed. In Table 10, RADNPO has a worse Forget TR than NPO and RMU (0.660 vs. 0.582 and 0.532), while UNDIAL achieves higher utility. In Table 11, RADNPO also has a worse Forget TR than SimNPO (0.639 vs. 0.574), with nearly identical ES. These results do not establish a consistent advantage on newer models.
  6. The ablation study does not convincingly justify the additional complexity. Figure 4 mainly shows that removing the contrastive objective hurts performance. However, the benefits of the other components are less clear, and no uncertainty estimates are provided. More importantly, there is no direct comparison with simpler alternatives, such as a fixed-temperature contrastive objective or standard entropy regularization. The appendix also shows substantial variation in FQ across random seeds (0.046 to 0.850), raising further questions about the reliability of the main results and hyperparameter selection.
  7. The evaluation scope is too narrow. The paper only evaluates TOFU and MUSE, both primarily focused on memorized knowledge, privacy, or copyrighted content. It does not evaluate harmful knowledge unlearning on WMDP, despite claiming a general LLM unlearning framework.
Questions for authors
  1. Can the authors restore the anonymous repository and provide complete reproduction instructions? Why is the code unavailable during the review period?

  2. How does RADNPO differ fundamentally from combining SimNPO-style reference-free optimization, ORPO-style log-odds, and standard adaptive weighting? What is the key technical novelty?

  3. How does Eq. (5) prevent probability mass from concentrating on a different high-probability token? Can the authors provide theoretical justification or direct measurements of the alternative-token distribution before and after unlearning?

  4. In the collapsed forgetting regime, why does the adaptive coefficient remain effective when the target probability approaches zero? How does the method avoid gradient suppression caused by the confidence factor and tanh clipping?

  5. Can the authors provide stronger ablations against simpler objectives, fair comparisons on newer models, and results on WMDP? How stable are the claimed gains across random seeds and hyperparameter choices?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.