acceptodds
Withdrawn from ICLR 2027

SeNPO: Preference Optimization in Sentence Embedding Space for Knowledge Unlearning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models memorize sensitive personal information, copyrighted material, and text that violates usage policies. Blocking a forbidden string at the output leaves that information in the weights, where later red-teaming can recover it. Unlearning fine-tunes the model so a target answer is harder to elicit, but the final token can change while an earlier layer still ranks the answer highly. We call this gap implicit evocation. A model can then look forgotten on ordinary answer metrics and still leak the target to a linear readout of its hidden states. A local residual-stream analysis gives a sufficient condition for that split and motivates a forget loss on several prefix-level semantic readouts. Semantic Negative Preference Optimization (\methodname) maps hidden states into a frozen sentence-embedding space and applies a length-normalized softplus preference loss there, together with retain-set KL. Unlike ATU's hinge loss, which provides no gradient once similarity falls below its threshold, our softplus loss remains active. Moreover, optimizes sentence-level semantic similarity, whereas token-level NPO optimizes the likelihood of the target token sequence. removes the requested facts while the model stays usable: scores on general knowledge tests and fluent writing remain close to the original model. Gradient ascent and token-level NPO, when kept from collapsing, still produce most of the answers that should have been forgotten. A matched-budget form of ATU either barely forgets or collapses, and the learning rates we tried contain no point that does both. The gap is also internal. After , a simple classifier reading intermediate hidden states recovers the forgotten answer less often. The same classifier is almost unchanged for usable gradient-ascent and NPO checkpoints. Pushing those baselines toward the same level of output forgetting still leaves that internal signal, and neighboring knowledge or fluency is lost. On fictitious-author unlearning, forgets more thoroughly than a representation-level method at similar utility. With the same retain penalty it leads token-level SimNPO on Qwen and trails it on Phi. On Llama, NPO removes more of the target answer; keeps more related knowledge and writes more fluently.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, knowledge unlearning, preference optimization, implicit evocation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wCH8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes SeNPO, a method for knowledge unlearning in large language models that addresses implicit evocation by optimizing semantic similarity in a frozen sentence-embedding space. It uses a softplus preference loss and retain-set KL divergence to ensure forgetting while preserving model usability.

Critical strengths
  1. The paper introduces a novel concept of implicit evocation, where unlearning fails to remove knowledge from intermediate layers, and provides a theoretical condition for this phenomenon using the Linear Representation Hypothesis (LRH).
  2. SeNPO’s use of semantic-level similarity instead of token-level likelihoods is a significant innovation, reducing sensitivity to paraphrasing and adversarial suffixes compared to existing methods like NPO.
  3. The experiments are comprehensive, comparing SeNPO with baselines on multiple benchmarks and demonstrating strong forgetting performance while maintaining utility metrics. The results show that SeNPO outperforms other methods in retaining knowledge and fluency.
Critical weaknesses
  1. The theoretical analysis relies on the LRH, which assumes linear geometry in hidden states. However, the paper does not rigorously validate this assumption or address potential nonlinearities, leaving room for skepticism about the generality of the results.
  2. The experiments do not fully explore the impact of hyperparameters (e.g., β, γ) on performance.
  3. While the paper claims that SeNPO achieves "strong forgetting," it does not provide a clear definition of what constitutes "strong" forgetting or how it compares quantitatively to other methods across all metrics.
Questions for authors
  1. The theoretical analysis builds on the LRH with a linear hidden-state assumption, but this assumption is not rigorously checked and nonlinear effects are ignored, so the results may not generalize well.
  2. The experiments lack a complete investigation of how hyperparameters such as β and γ affect model performance.
  3. The paper claims SeNPO achieves strong forgetting, yet it fails to define this term clearly or supply full quantitative comparisons against other methods.
Reviewer_QgbyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the gap between suppressing answers in the final output and reducing their recoverability from intermediate representations, and proposes Semantic Negative Preference Optimization (SeNPO), which applies a length-normalized softplus loss to semantic similarities obtained through an embedding prediction head, together with a retain-set KL penalty. Experiments on RWKU and TOFU benchmark, along with component ablations and hidden-state probing, demonstrate that SeNPO achieves a favorable trade-off between forgetting effectiveness and model utility. However, the experimental comparisons do not fully isolate the contribution of the proposed semantic objective, and the evidence for reduced internal knowledge recoverability remains limited.

Critical strengths
  1. The paper examines an important limitation of output-only evaluation. The hidden-state analysis provides information beyond whether a model generates a target answer. The question-group splits and reuse of gold answers as negative candidates for other questions are thoughtful controls against straightforward probe leakage. This diagnostic perspective is a useful contribution.
  2. The experimental evaluation is relatively comprehensive, including comparisons with matched ATU loss and equal retain-set KL constraints. The authors also report cases where competing methods perform better, providing a more balanced assessment of the forgetting–utility trade-off.
Critical weaknesses
  1. The main RWKU comparisons do not fully isolate the benefit of the proposed semantic objective. As disclosed in Section 4.1, GA and NPO do not receive SeNPO’s retain-set KL, and the Phi NPO aggregate covers fewer targets. Consequently, the results compare complete training recipes with different protection mechanisms and evaluation coverage. The equal-KL TOFU controls are helpful, but do not resolve this issue for RWKU. This limits the evidence for attributing the observed advantage specifically to the semantic objective.
  2. The evidence for reduced internal recoverability at comparable output forgetting remains limited. The initial probe comparison uses checkpoints with different forgetting levels, while the subsequent output-forgetting control concerns one target and training seed. Moreover, Table 1 reports GA Forget-All as 0.00, whereas the accompanying text and Appendix D report 5.40. This may be a reporting error, but clarifying it matters because this experiment supports a central interpretation of the probe results.
Questions for authors
  1. Could you provide a comparison on the same RWKU targets with the same retain-set KL for SeNPO and the token-level baselines, or clarify what evidence supports the semantic-objective advantage after accounting for these differences?
  2. Could you reconcile the GA Forget-All values in Table 1, Section 4.2, and Appendix D, and identify the corresponding checkpoints? Please also clarify which conclusions about internal recoverability are supported beyond this single-target control.
Reviewer_7qPUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of implicit evocation: after unlearning, the final answer no longer names the target while an intermediate layer still ranks it highly, which defeats the unlearning. They introduce Semantic Negative Preference Optimization (SeNPO). SeNPO removes the requested facts while the model stays usable, and after SeNPO a classifier reading intermediate hidden states recovers the forgotten answer less often.

Critical strengths

The paper presents a elegant method for addressing the problem of implicit evocation in machine unlearning, and evaluates with extensive experiments on five open-source models.

Critical weaknesses
  1. Implicit evocation is framed as a new observation and listed as a contribution, but prior work has measured suppressed answers re-emerging in intermediate layers (Patil et al 2024, more in the list below), and this work is not engaged with. A similar method to the paper's is presented in Patil et al 2024.

Patil, Hase & Bansal (ICLR 2024). Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks. (Already in the submission’s bibliography, cited only in the preference-optimization list at l. 102–103.) Łucki, Wei, Huang, Henderson, Tramèr & Rando (TMLR 2025). An Adversarial Perspective on Machine Unlearning for AI Safety. Guo, Syed, Sheshadri, Ewart & Dziugaite (ICML 2025). Mechanistic Unlearning: Robust Knowledge Unlearning and Editing via Mechanistic Localization.

Questions for authors
  1. It is possible to use the output vocab embedding dotted with the hidden layer embeddings to get a token prediction for each layer, without training a predictor like was done here. Have you considered the baseline of using the output vocab embedding w/ NPO at each layer? Comparing to this baseline would demonstrate the necessity of training that layer prediction model.

  2. Not, a question, but the paper uses RWKU as a dataset, without ever explaining what RWKU is. Please explain the dataset at least briefly in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.