acceptodds
Under review as a conference paper at ICLR 2027

Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem

Abstract

Machine unlearning in LLMs is the targeted removal of specific knowledge while preserving all other capabilities. Existing benchmarks measure it unreliably. They miss knowledge that resurfaces under paraphrased or indirect queries, a failure we call *under-forgetting*, and lack the semantic, syntactic, and lexical probes needed to verify that unrelated knowledge is preserved, a failure we call *over-forgetting*. Both failures reflect an asymmetric generalization problem. Forget evaluation must cover diverse query formulations of the same target facts, testing whether forgetting holds beyond training prompts. Retain evaluation must probe a far larger and implicitly defined set: every fact disjoint from the forget target. The retain set thus defines the effective forget set, yet current datasets provide no fine-grained annotation of this forget-retain boundary. We address this with SUITE, an evaluation protocol and training corpus that captures forget-retain structure for real-world factual domains. SUITE improves *all* unlearning methods substantially, showing that training data is as important as algorithmic design. Moreover, we introduce JensUn++, an unlearning algorithm that achieves a better or equal forget-retain utility trade-off across three LLMs, in both sequential and joint unlearning settings. JensUn++ is the only method producing no gibberish answers when asked about forget facts. Code and datasets are available at [this anonymous link](https://anonymous.4open.science/r/forget-narrowly-retain-broadly).

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM unlearning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WSm1OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper frames LLM unlearning as an asymmetric generalization problem and introduces SUITE, a benchmark designed to evaluate both under-forgetting and over-forgetting. The authors also propose JensUn++, which combines stochastic prefix-mixing and dynamic loss balancing. Experiments across three LLMs and four forget topics demonstrate competitive forget-retain trade-offs while producing coherent refusal responses.

Critical strengths
  • Fine-grained forget-retain evaluation. SUITE evaluates under-forgetting and over-forgetting through diverse question formulations and semantic, syntactic, and lexical retain probes. The finding that training data substantially affects unlearning performance is particularly valuable.

  • Comprehensive empirical evaluation. The paper evaluates three LLMs under sequential and joint unlearning, adversarial queries, relearning, and quantization. JensUn++ achieves competitive forget-retain trade-offs while substantially reducing incoherent generations.

Critical weaknesses
  • JensUn++ explicitly trains models to produce refusals, while the primary evaluation measures whether target knowledge appears in generated responses. This does not establish whether the underlying knowledge has been removed or merely suppressed. The NLL, logit-lens, and relearning analyses do not fully resolve this distinction.

  • SUITE covers only four forget topics, leaving its generalizability to broader knowledge domains unclear. Moreover, JensUn++ still degrades performance on the semantically closest retain tier (s0), suggesting that preserving fine-grained forget-retain boundaries remains challenging.

Questions for authors
  • A recent unlearning position paper [1] argues that knowledge suppression and refusal should be distinguished from genuine unlearning. How do the authors view this distinction in the context of JensUn++?

  • Could the authors report membership inference attack (MIA) results to provide additional evidence that JensUn++ removes knowledge rather than merely suppressing its generation?

  • What explains the remaining performance degradation on the closest retain tier (s0)? Given this degradation, to what extent does SUITE successfully define and preserve the intended forget-retain boundary?

[1] Position: The Term "Machine Unlearning" Is Overused in LLMs.

3 comments

We thank the reviewer for finding SUITE and its findings valuable and for appreciating the comprehensive evaluation of the paper. We are also happy that the reviewer appreciates JensUn++ performance while substantially reducing incoherent generations. We address the weaknesses and questions below.

Q1. How do the authors view the distinction of genuine unlearning and knowledge suppression/refusal in the context of JensUn++? In classical machine unlearning, an unlearned model should be indistinguishable from one that was trained without the forget set, and [1] suggests using this definition in LLMs, or, when a different objective is used, to state it explicitly (their Sec. 6). We believe that in most LLM scenarios the objective should be knowledge removal, and more practically, that the forget facts can not be extracted from the model with any kind of query, and also under benign relearning, which can happen unintentionally during standard fine-tuning. This is also the goal of JensUn++.

We focus on this setting for 3 reasons:

  1. For pretrained LLMs, retraining from scratch without the forget set is infeasible, even though this is exactly the setting where unlearning is needed.
  2. In many scenarios (e.g., privacy, safety), the threat is that someone extracts the information, so the main goal of the methods should be to prevent it.
  3. Some practical cases fall outside the classical definition, e.g., removing harmful hallucinations the model produces even though they never appeared in the training data.

We will state explicitly in the paper that our goal is not indistinguishability from a retrained model, and cite [1].

However, we are not sure if this is the point you had in mind. Please clarify if this is not the case.

W1 and Q2. Has the underlying knowledge been removed or merely suppressed. Could the authors report membership inference attack (MIA) results to provide additional evidence that JensUn++ removes knowledge rather than merely suppressing its generation?

Since we try to remove knowledge that is already present in the model, we don’t have an “oracle” (a model that was never trained on the forget set) as in MUSE or TOFU. We therefore follow RWKU and use a loss-based MIA comparing the mean NLL per token on the GT answers of forget and retain questions, extending the evaluation we did in App. B.9. Unlike RWKU, which uses short texts from Wikipedia, we use QA pairs. Forget questions are 5 direct and 5 indirect paraphrases (unseen during training) of each forget fact that the pretrained model answered correctly in at least one paraphrase. Retain questions are evaluation questions that the pretrained model answered correctly. The results below are for Llama-3.2-3B after sequential unlearning.

MethodEval-direct ↑Eval-indirect ↑Eval-retain ↓
Llama1.752.591.05
GradDiff81.4642.482.28
NPO24.7417.441.43
PDU11.379.240.90
RMU13.737.471.09
JensUn21.967.121.31
JensUn++23.8514.791.04

All methods seem to forget in this test, since forget NLL rises considerably. However, we know from Tab. 3 that GradDiff, NPO, PDU, and RMU leak on at least 29% of the facts, so we created another test where we compare the ground truth answer with plausible wrong answers.

For each fact the pretrained model knows, we used Claude Opus 5.5 to generate 5 wrong answers that follow the same format, and manually verified them, e.g., for “How many seconds after liftoff did the Challenger break apart?" (GT: 73 seconds), the answers are 47, 58, 64, 81, and 92 seconds. We then compared how likely the GT answer is relative to the wrong answers, using the average over the same 10 paraphrases as before (5 direct and 5 indirect), and ranking the 6 answers per fact. We show the total / per-token log-likelihood. AUC is the probability that the GT answer is more likely than a wrong answer (similar to the truth ratio in TOFU). If the knowledge is removed, the model cannot distinguish the GT answer from the wrong answers, so the best under this is random (Top-1=Bottom-1=16.7% and AUC is 0.5), since a model that often ranks the GT answer last still recognises it.

MethodTop-1 (%)Bottom-1 (%)AUC
Llama83.8 / 78.40.0 / 0.00.95 / 0.91
Random16.7 / 16.716.7 / 16.70.50 / 0.50
---------------------------------------------------------------
GradDiff12.2 / 8.128.4 / 35.10.44 / 0.36
NPO2.7 / 4.171.6 / 71.60.11 / 0.14
PDU13.5 / 6.831.1 / 48.60.38 / 0.24
RMU16.2 / 12.221.6 / 23.00.45 / 0.42
JensUn16.2 / 12.224.3 / 17.60.48 / 0.44
JensUn++21.6 / 18.923.0 / 27.00.51 / 0.46

The pretrained model prefers the correct answer as expected. NPO ranks the GT answer last for ~70% of the facts, which makes the GT easier to identify, while RMU, JensUn, and JensUn++ are closest to chance level, so they don’t seem to suppress only the generation of the GT answer, at least on average. In our worst case forget set evaluation, RMU and JensUn still reveal 40% and 14% of the facts in generation, vs. 3% for JensUn++ (Tab. 3), showing why behavioral evaluation is important.

If the reviewer had a different MIA setup in mind, please clarify, and we will try to test it.

W2+Q3. What explains the remaining performance degradation on the closest retain tier (s0)?

We manually verified that the retain set does not contain facts from the forget topic, so the degradation on s0s_0 (the retain tier semantically closest to the forget set) reflects over-forgetting, not an overlap between the two sets. SUITE defines this boundary precisely, and training on it is what improves s0s_0 for all methods (in Table 1 in the paper, we show that training on LKF* instead leads to full over-forgetting on s0s_0).

To understand why s0s_0, shows the largest drop relative to the pretrained model, we compare the retain performance on the training and evaluation questions of s0s_0 with that of the more distant tiers s1−5s_{1-5}. In this way, we can check if missing generalization is the problem.

The table below shows the percentage of retain facts each method answers correctly, out of those the pretrained model answered correctly, on the retain training set (Train) and evaluation set (Eval), averaged over all models and settings, together with the forget knowledge rate QAllQ_{All}.

MethodQAllQ_{All} ↓s0s_0 Train ↑s0s_0 Eval ↑s0s_0 Gaps1−5s_{1-5} Train ↑s1−5s_{1-5} Eval ↑s1−5s_{1-5} Gap
GradDiff38.765.440.9−24.580.168.4−11.7
NPO35.077.641.8−35.883.761.5−22.2
PDU35.880.149.5−30.686.765.8−20.9
RMU36.567.466.7−0.780.781.5+0.8
JensUn8.772.562.4−10.183.577.2−6.3
JensUn++4.575.570.4−5.187.282.5−4.7

For JensUn++, the gap between Train and Eval is about the same for s0s_0 and s1−5s_{1-5} (5.1 and 4.7). The gap is similar to or better than for other methods, except RMU, which has a high forget knowledge rate QAllQ_{All}, so missing generalization of JensUn++ is not the problem. As discussed in Sec. 3 of the paper, methods that try to learn the retain answers during training (GradDiff, NPO, PDU) generalize less, while methods that stay close to the pretrained model generalize better.

Thus, the main problem is the difficulty of keeping the knowledge in s0s_0 already on the training questions (75.5% of the questions answered correctly by the pretrained model vs. 87.2% for s1−5s_{1-5}). The reason for this gap seems to be the difficulty of separating facts that share an entity with the forget set, e.g., the sickness of Steve Jobs (forget set) vs. other events in his life (s0s_0). This creates a trade-off between under- and over-forgetting on these facts, and when choosing hyperparameters for all methods, we favored checkpoints with better forget quality as long as the drop in retain performance is limited (App. B.6), which in turn led to a bias towards models which perform well on the forget set while sacrificing more on s0s_0.

As stated in the discussion in Section 5, we think that improving performance on the retain set s0s_0 while keeping a low forget knowledge rate is an important area for future research.

W2. SUITE covers only four forget topics, leaving its generalizability to broader knowledge domains unclear.

It is true that SUITE covers only four topics, but they are diverse, and the results between them are overall consistent. We focused on a high-quality dataset, where the forget set and the forget-retain boundary were manually verified for all topics. The design choices behind SUITE (e.g., using the worst-case over indirect, reverse, and direct questions, defining the forget boundary on the semantic, lexical, and syntactic sets) already expose failure cases missed by previous benchmarks. At the same time, training on SUITE shows that methods that seem broken under previous benchmarks actually perform well. The dataset itself can be expanded in future work using the same principles and the provided code.

Thank you for the detailed rebuttal and additional experiments. It seems that the manuscript has not yet been updated. Do the authors plan to revise the paper to incorporate these clarifications and additional results? I would like to review the updated manuscript before making my final rating decision.

Reviewer_ih4sOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the problems of under-forgetting and over-forgetting in LLM unlearning. The authors view them as an asymmetric generalization problem and propose SUITE, which uses more types of forget questions and a fine-grained retain set to better evaluate the forget-retain boundary. The authors also propose JensUn++, which improves refusal training and loss balancing to help the model forget target knowledge while keeping its other abilities. Experiments show that SUITE can improve existing unlearning methods. JensUn++ also achieves a good forget-retain trade-off and produces fewer meaningless answers.

Critical strengths

The paper provides sufficient experiments. The authors compare different methods across several backbone models and unlearning settings. They also include detailed ablation studies and robustness tests. These experiments support the authors' main claims from different aspects and provide good evidence for the effectiveness of the method.

Critical weaknesses

The method still shows clear over-forgetting. Although JensUn++ achieves good overall retain performance, its performance on s0s_0, which is the closest to the forget set, still drops significantly. However, keeping knowledge that is closely related to the forget set is one of the main problems this paper aims to solve. I hope the authors can further explain why this happens and discuss the limitations of the current method in learning the forget-retain boundary.

Questions for authors
  1. JensUn++ mainly uses JS Divergence. Can the proposed Stochastic Prefix-Mixing and Dynamic Loss Balancing also work with other divergence measures?

  2. Stochastic Prefix-Mixing randomly selects part of the original answer as a prefix. However, for some short forget answers, is it possible that the selected prefix already contains all the sensitive information?

  3. Would it be difficult to get natural refusal responses from some models? Some smaller models may rarely refuse to answer questions and may instead generate incorrect or made-up answers. In this case, how would the authors build a suitable refusal target?

  4. The authors emphasize that training data is as important as algorithm design. However, Appendix B.4 states that SUITE uses 20 epochs, while LKF* only uses 10 epochs. Could the authors provide a fairer comparison by using the same number of training epochs?

3 comments

We thank the reviewer for their questions and for appreciating the careful and extensive experiments in the paper. We address the weaknesses and questions below.

W1. JensUn++ achieves good overall retain performance, but on s0s_0 drops significantly.

To understand why s0s_0, the retain tier semantically closest to the forget set, shows the largest drop relative to the pretrained model, we compare the retain performance on the training and evaluation questions of s0s_0 with that of the more distant tiers s1−5s_{1-5}. In this way, we can check if missing generalization is the problem.

The table below shows the percentage of retain facts each method answers correctly, out of those the pretrained model answered correctly, on the retain training set (Train) and evaluation set (Eval), averaged over all models and settings, together with the forget knowledge rate QAllQ_{All}.

MethodQAllQ_{All} ↓s0s_0 Train ↑s0s_0 Eval ↑s0s_0 Gaps1−5s_{1-5} Train ↑s1−5s_{1-5} Eval ↑s1−5s_{1-5} Gap
GradDiff38.765.440.9−24.580.168.4−11.7
NPO35.077.641.8−35.883.761.5−22.2
PDU35.880.149.5−30.686.765.8−20.9
RMU36.567.466.7−0.780.781.5+0.8
JensUn8.772.562.4−10.183.577.2−6.3
JensUn++4.575.570.4−5.187.282.5−4.7

For JensUn++, the gap between Train and Eval is about the same for s0s_0 and s1−5s_{1-5} (5.1 and 4.7). The gap is similar to or better than for other methods, except RMU, which has a high forget knowledge rate QAllQ_{All}, so missing generalization of JensUn++ is not the problem. As discussed in Sec. 3 of the paper, methods that try to learn the retain answers during training (GradDiff, NPO, PDU) generalize less, while methods that stay close to the pretrained model generalize better.

Thus, the main problem is the difficulty of keeping the knowledge in s0s_0 already on the training questions (75.5% of the questions answered correctly by the pretrained model vs. 87.2% for s1−5s_{1-5}). The reason for this gap seems to be the difficulty of separating facts that share an entity with the forget set, e.g., the sickness of Steve Jobs (forget set) vs. other events in his life (s0s_0). This creates a trade-off between under- and over-forgetting on these facts, and when choosing hyperparameters for all methods, we favored checkpoints with better forget quality as long as the drop in retain performance is limited (App. B.6), which in turn led to a bias towards models which perform well on the forget set while sacrificing more on s0s_0.

As stated in the discussion in Section 5, we think that improving performance on the retain set s0s_0 while keeping a low forget knowledge rate is an important area for future research.

Q1. Can the proposed Stochastic Prefix-Mixing and Dynamic Loss Balancing also work with other divergence measures?

Stochastic Prefix-Mixing and Dynamic Loss Balancing are not specific to the JS divergence. To test this, we created a variant of JensUn++, which uses KL divergence instead of the JS divergence. To isolate the effects of our additions, we created a second KL variant that does not use Stochastic Prefix-Mixing and Dynamic Loss Balancing, and we ran a separate hyperparameter search for each of them so that the comparison is fair. We show below the results for the sequential and joint settings for Llama.

Sequential

RunQAllQ_{All} ↓Gib. ↓Retain ↑RGQ ↑
Llama95.00.453.3–
JensUn++ with JS (Tab. 3)3.00.051.648.5
JensUn++ with KL4.00.150.643.4
KL no additions26.00.444.828.6

Joint

RunQAllQ_{All} ↓Gib. ↓Retain ↑RGQ ↑
Llama95.00.453.3–
JensUn++ with JS (Tab. 26a)8.00.051.944.5
JensUn++ with KL7.00.052.742.3
KL no additions39.00.050.944.8

With KL, the additions reduce the forget accuracy QAllQ_{All} to the same level as JensUn++ with the JS divergence, with similar retain performance and a decrease in RGQ (48.5 to 43.4 in the sequential setting). The failure of the KL divergence without the additions is consistent with [A], which shows that JS is better than KL for unlearning. Together, this shows that our additions make the choice of which divergence to use less critical, which is an interesting side effect. We thank the reviewer for bringing this up.

Q2. Is it possible that the selected prefix already contains all the sensitive information?

Yes, the selected prefix can already contain the sensitive information, but even then the model is trained to refuse directly after the question. Consider the example in Fig. 4, where each word is treated as one token for illustration, and the sampled prefix “73 seconds” already contains the sensitive information. With our prefix-aligned target (Eq. 2), the training targets are:

  • question -> “Unfortunately” (direct refusal)
  • “73” -> “Unfortunately” (refusal after the sensitive token)
  • “seconds” -> “Unfortunately”
  • “Unfortunately” -> “I”
  • “I” -> “am”, and so on.

As seen in the example, the first position after the question is always trained towards refusal, and under our token weights (Eq. 6) it receives half of the loss weight for every length of sampled prefix. So even in the extreme case where the sensitive information appears in the first token, half of the weight trains the model to refuse before any sensitive information appears. Just as a note: in our forget set evaluation, a response that gives the sensitive information and then refuses is counted as a leak.

Q3. Would it be difficult to get natural refusal responses from some models?

The models we used were trained to refuse when they do not know the answer, so we could obtain the model’s refusals by asking about non-existent facts, such as “What is the currency of Thalvoria?” (App. B.1). We used those refusals to have minimal disruption to the model and, by that, to help preserve retain performance and keep a similar user experience before and after unlearning.

To test whether our method works without using the model’s refusal, we trained another version that uses an alternative answer per fact instead of a single refusal for all facts (JensUn++(alternative)). This is a harder goal, since each fact has its own target instead of one shared refusal. We made sure that all targets are harmless and do not contain sensitive information, and chose for each fact a wrong answer that the pretrained model already finds likely. For example, for the question from Fig. 1

“What highly publicized 2007 incident became the defining media image of the world-famous pop icon’s crisis prior to her 13-year legal guardianship?”

  • GT: “Shaving her head”
  • Alternative target: “Forgetting her lyrics”

Sequential:

MethodQAll∗Q_{All}^* ↓Gib. ↓RAllR_{All} ↑RGQ ↑
Llama-3.2-3B96.00.453.3–
JensUn++3.00.051.648.5
JensUn++ (alternative)3.00.250.546.0
Ministral-3-3B89.00.153.1–
JensUn++4.00.050.145.9
JensUn++ (alternative)9.00.048.247.4
Qwen3.5-9B90.00.752.0–
JensUn++4.00.051.036.2
JensUn++ (alternative)6.00.449.245.4

Joint:

MethodQAll∗Q_{All}^* ↓Gib. ↓RAllR_{All} ↑RGQ ↑
Llama-3.2-3B96.00.453.3–
JensUn++10.00.051.944.5
JensUn++ (alternative)3.00.050.948.0
Ministral-3-3B89.00.153.1–
JensUn++0.00.051.348.5
JensUn++ (alternative)8.00.049.449.0
Qwen3.5-9B90.00.752.0–
JensUn++12.00.152.644.4
JensUn++ (alternative)6.00.450.645.4

Compared to JensUn++, the alternative answers version has a comparable forget knowledge rate, retain performance is 1%-2% lower, but the model still achieves better or equal results compared to the baselines (no baseline has better retain and RGQ with comparable forget performance) and without more gibberish compared to the pretrained model. So JensUn++ can work with both, and the right target should be decided based on the application scenario. In case of harmful hallucinations, replacing them with correct information might be better than refusal, whereas in cases where an alternative answer would be wrong, a refusal seems better suited.

Q4. Could the authors provide a fairer comparison by using the same number of training epochs?

We used 10 epochs in the paper because this is the setting of the original LKF paper. To test whether training for longer changes the conclusions, we trained on LKF* for 20 epochs, the same number used for training on SUITE. All models are evaluated on SUITE.

MethodTrained on LKF*: QAllQ_{All} ↓Trained on LKF*: RAllR_{All} ↑Trained on SUITE: QAllQ_{All} ↓Trained on SUITE: RAllR_{All} ↑
Llama100.051.1100.051.1
GradDiff4.030.04.045.2
NPO100.038.012.039.8
PDU68.033.236.046.8
RMU96.046.54.048.2
JensUn8.046.00.049.1
JensUn++16.046.60.050.5

Compared with 10 epochs (Tab. 2), training for longer improves some methods, while hurting others. GradDiff QAllQ_{All} moves from 16% to 4%, but JensUn++ moves in the other direction from 4% to 16%. The same happens for retain, where PDU drops from 39.5% to 33.2%, while GradDiff moves from 23.8% to 30%. The conclusion stays the same when training for the same number of epochs. Evaluation with LKF* hides under- and over-forgetting, and all methods perform better when trained on SUITE (GradDiff is the only one that forgets equally well, but its retain accuracy increases from 30% to 45%).

[A] Singh et al., Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMs. NeurIPS 2026.

Thank you for your detailed response. I believe the authors have addressed most of my concerns, and I will maintain my current score. I hope the authors can include the additional information and clarifications in the final version of the paper.

Reviewer_oXjGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SUITE, a new benchmark for machine unlearning that provides fine-grained forget and retain sets for evaluating targeted unlearning. It addresses the need for proper unlearning evaluation, where forgetting should be specific to the target facts (holding across paraphrased, reverse, and indirect queries), while retained knowledge should be preserved broadly and not be affected indirectly by unlearning the forget set. This is achieved through a detailed level of granularity: graded semantic tiers of retain topics, plus syntactic and lexical probes. The authors further propose a new unlearning algorithm, JensUn++, that achieves better performance on the benchmark.

Critical strengths

The paper correctly identifies the need for a fine-grained benchmark for machine unlearning. Existing benchmarks sometimes cannot effectively measure both the direct and indirect effects of unlearning algorithms. A detailed benchmark like this can be used to better understand and evaluate unlearning algorithms.

The LKF → SUITE ablation is convincing. Methods that look good under LKF evaluation leak 92–100% of facts under SUITE, and every method improves when trained on SUITE. The finding that data matters as much as the algorithm is useful in its own right.

Critical weaknesses

I do not see significant novelty in either the benchmark or the algorithm. This alone is not a serious issue: the paper is valuable for the machine unlearning community, but in my opinion it is not novel.

Benchmark: its components already exist. Reverse queries come from RWKU, multi-hop leakage from Wei et al. and Shah et al., distance-graded retain sets from RippleBench, and syntactic sensitivity from Chang & Lee. The topics and parts of the data are inherited from LKF. The contribution is mainly combining these into one dataset.

Algorithm: JensUn++ is JensUn plus established components: refusal targets, recovery from partial-answer prefixes (Qi et al., 2025), gradient-norm loss balancing (GradNorm; Jin et al., 2025), and pairing.

Questions for authors

Does JensUn++ over-refuse on retain questions that are semantically close to the forget topic (s0)?

2 comments

We thank the reviewer for acknowledging the value of SUITE for the community as a fine-grained evaluation and training set. We address the weaknesses and questions below.

W1. Novelty SUITE. We understand that from a high-level perspective, it might look like this, but just because certain aspects have been addressed in previous benchmarks does not mean that we do them the same. Indeed, we have good reasons to do them differently, as explained below, and our integration of all these aspects in SUITE is novel, which we consider as a significant step forward in unlearning. All methods benefit from the better forget-retain-boundary established in SUITE.

We frame unlearning as an asymmetric generalization problem whose two failure cases are under- and over-forgetting. This motivates SUITE, which has the following main novelties:

  1. defining the forget-retain boundary through the retain set, along semantic, syntactic, and lexical axes, which exposes under- and over-forgetting in evaluation and reduces them when used as a training dataset.

  2. evaluating intensive generalization (unseen paraphrases and reasoning paths) on the forget side and extensive generalization on the retain side (unseen facts and topics, syntactic and lexical queries).

We agree that some of the query types and the failures they expose (e.g., leakage through multi-hop queries and degradation with semantic distance) were studied before, and we cite them in the paper, but they are not directly usable as an evaluation or training dataset in our case. For example,

  • Multi-hop reasoning is one of the ways we test intensive generalization. Wei et al. build the reasoning paths from an external knowledge graph (YAGO3), and thus they are restricted to the entities and 37 relation types of YAGO. This is much more coarse-grained than what we use in SUITE. Shah et al. build the knowledge graph with the LLM itself, which can mix forget and retain facts, so over- and under-forgetting cannot be distinguished.

  • Semantic queries are one of the axes we use to test extensive generalization. The phenomenon of degradation on topics based on their semantic distance from the forget set was analysed in RippleBench, which generates questions from Wikipedia pages ranked by similarity to the forget questions. However, RippleBench does not define whether the questions for semantically close topics belong to the forget or retain set, so a change in accuracy can mean under-forgetting, proper unlearning, or over-forgetting. Thus, it can neither be used for the evaluation of under- and over-forgetting nor as a training set.

For example, for the forget topic “Challenger disaster”, a related Wikipedia page in RippleBench would be “Space Shuttle Challenger”, which covers both the disaster (forget) and the shuttle’s earlier missions (retain). In contrast, SUITE clearly separates the two, and its semantically closest retain set s0s_0 is “Space Shuttle Challenger” (without the disaster). Here are two questions from the forget set and retain set s0s_0:

Q (forget): "What was the specific day, month, and year of the Challenger disaster?" GT: January 28, 1986

Q (s0s_0): "In what month and year did Challenger first reach space?" GT: April 1983

In contrast to RippleBench, SUITE has a clear separation of forget and retain sets, despite these two questions being semantically very close. Note that in Table 1 in the paper, we show that training on LKF* and evaluating on SUITE yields full over-forgetting on s0s_0 (0% accuracy). Thus, current methods fail silently on semantically very close topics, as it is simply not measured in existing benchmarks. Moreover, training on SUITE improves all methods (Tab. 2), which shows the importance of our fine-grained definition of the forget-retain-boundary.

W2. Novelty of JensUn++. We agree that JensUn++ builds on JensUn similarly to how, e.g., NPO uses the DPO loss with only negative samples. However, JensUn++ uses prefix-mixing to fix the gibberish refusals in JensUn while keeping a low forget knowledge rate, stabilizes unlearning with loss-balancing, and pairing improves the trade-off between forget-retain-utility performance.

The components we use are similar to existing ones, but not the same.

  • Prefix-mixing: unlike Qi et al., we align every prefix position with the first refusal token, so the model also always learns to refuse immediately, even when the prefix already contains the sensitive information.
  • Loss Balancing: our loss balancing uses gradient norms with respect to the logits rather than the weights, which avoids extra backward passes. Moreover, [A] found KL to perform worse than JensUn for unlearning due to unbalanced gradients in the initial stage of fine-tuning. In the answer to reviewer ih4s, we show that, due to prefix-mixing and loss balancing, JensUn++ using KL- instead of JS-divergence leads only to slight degradation in performance.
  • Pairing: note that we apply it to all methods despite this is not used in the baselines. We could have presented much better looking results by just applying it to JensUn++. Instead, we are doing an honest comparison instead of just benchmaxxing.
  • Alternative answers: in the answer to reviewer ih4s, we show that JensUn++ works equally well when replacing the refusal with a different answer for each forget fact.

Q1. Does JensUn++ over-refuse on retain questions that are semantically close to the forget topic (s0s_0)?

Refusals count as incorrect answers, so over-refusal is already reflected in the accuracy reported on the retain set (see our response to reviewer ih4s, where JensUn++ has the smallest accuracy drop on s0s_0).

In most cases, we believe a refusal is better than giving a wrong answer. JensUn++ refuses more often on s0s_0 than on other semantic tiers. Interestingly, it refuses more often on questions that the pretrained model answers incorrectly than on those it answers correctly (see the table below). For example, for Ministral-3-3B (Seq), it never refuses on questions in s0s_0 that the pretrained model answers correctly, but it refuses on 10.9% of the incorrectly answered questions in s0s_0. We think this is an additional positive aspect of JensUn++.

Models0s_0 refused, pretrained correct (%)s0s_0 refused, pretrained wrong (%)s1−5s_{1-5} refused, pretrained correct (%)s1−5s_{1-5} refused, pretrained wrong (%)
Llama-3.2-3B (Seq)9.19.51.44.6
Llama-3.2-3B (Joint)6.19.51.42.1
Ministral-3-3B (Seq)0.010.90.51.8
Ministral-3-3B (Joint)17.217.42.83.2
Qwen3.5-9B (Seq)0.02.20.51.1
Qwen3.5-9B (Joint)0.02.20.50.7

s0s_0 questions are semantically and lexically closer to the forget questions, and the pretrained model sometimes mixes the two. For example, for the s0s_0 retain question

“In what month and year did Challenger first reach space?” (GT: April 1983)

  • Pretrained Llama: “January 28, 1986.” (the date of the Challenger disaster)
  • JensUn++: “Unfortunately, I am unable to verify this information.”

So when JensUn++ refuses, it prevents leakage rather than over-refusing. The cases where the pretrained model is correct, and JensUn++ refuses, are exactly the failure cases the SUITE fine-grained boundary is designed to expose.

We are happy that the reviewer considers our paper as “valuable for the machine unlearning community”. We hope that we could clarify all remaining doubts and kindly ask the reviewer to raise their score in case they are resolved. Certainly, we are happy to answer follow-up questions.

[A] Singh et al., Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMs. NeurIPS 2026.

Comments on the paper

1 comment

Dear reviewers,

We have uploaded a revised version of our paper, with the changes from the rebuttal marked in green. Most changes in the main paper come from adding the JensUn++ variant with alternative targets and the clarification of our unlearning goal. Please let us know if any concerns are left, so we can address them during the discussion phase.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.