acceptodds
Under review as a conference paper at ICLR 2027

How Well Does LLM Unlearning Generalize?

Abstract

LLM unlearning methods aim to remove a set of knowledge (the forget set) from a trained language model while keeping another set (the retain set) intact. Most existing unlearning benchmarks and methods construct both sets in a single form—either question–answer pairs or verbatim text continuations—but knowledge removed in one form may remain recoverable in many others in deployment, such as a paraphrase in the same form, the other form, another language, or a promptbased attack. In this paper we study how unlearning generalizes across query forms and languages, and how to improve the generalization through synthetic data augmentation and localized training, over fourteen unlearning objectives, two model families, eight query forms, four prompt-based attacks and up to eighteen languages. We report three findings. First, unlearning does not generalize well across the question–answer form and the verbatim form, which a model stores largely independently. Second, unlearning generalizes across most languages, with the exception of low-resource languages that spend more layers on language-specific processing. Third, among individual modules, the unlearning weight update generalizes best in the feed-forward layers, while the value and output projections give a surface-level unlearning that does not generalize and the query and key projections unlearn the least, in both standard transformers and linear-attention hybrid models. Building on these findings, we release AUGUNLEARNING, a suite that improves the generalization of both the training and the evaluation of existing unlearning methods with a synthetic data augmentation and a new metric. Together, our findings expose the vulnerabilities of current unlearning methods and the blind spots of current benchmarks. Our proposed methods solve the cross- form generalization problem and improve existing unlearning methods towards a better trade-off between unlearning quality and model utility.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
machine unlearning, LLM unlearning, generalization, multilingual, knowledge localization, data augmentation, evaluation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_NoDTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how LLM unlearning transfers across QA and continuation formats, languages, and prompt-based attacks. AUGUNLEARNING provides synthetic variants and a score combining forget-set removal with retain-set preservation. The experiments identify a pronounced QA/continuation boundary, examine module-specific updates, and show that cross-form augmentation and MLP-only training can improve the trade-off for some methods.

Critical strengths

I find the empirical study convincing, particularly the distinction between generalization to paraphrases within the same form and transfer across QA and continuation formats. The evaluation separates these effects, and the module-specific experiments show that similar forgetting on the training format can conceal substantially different cross-form performance. The checks that frozen parameters remain unchanged further strengthen this comparison. The figures present these patterns clearly, making it easy to connect the evidence to the paper’s claims about the limits of unlearning generalization.

Critical weaknesses

My main concern is that the additional insight over prior work is not sufficiently clear. The experimental coverage is substantial, but several of the central observations and evaluation choices have close precedents.

For cross-form generalization, RWKU (Jin et al., 2024) already reports that fill-in-the-blank and adversarial probes can recover knowledge that appears forgotten under QA evaluation. RULE (Zhang et al., 2025, Appendix A.1) also explicitly trains on both completion-style inputs and derived QA queries. Although its refusal-based setting differs, this is a close precedent for the proposed cross-form augmentation. More directly, the statement in Section 2 that existing benchmarks “train and score within a single form” is too broad: MUSE evaluates both verbatim continuation and QA derived from the same source text, while RWKU includes fill-in-the-blank, QA, and nine types of adversarial probes.

The other findings also need a sharper comparison with existing evidence. Choi et al. (2024) and Farashah et al. (2026) already study uneven cross-language transfer and resource-related differences. The paper’s stronger transfer result may be informative, but I would like a clearer account of what explains the difference. For module localization, Guo et al. (2025) already show that targeting fact-lookup MLPs improves robustness across input formats and against relearning, distinguishing this from interventions on output-extraction mechanisms.

I appreciate the bidirectional QA/continuation matrix, gradient analyses, and extension to a hybrid architecture. However, the paper needs to explain more precisely which conclusions these experiments add beyond the existing observations, rather than relying mainly on broader coverage to establish the contribution.

Questions for authors

See weakness.

1 comment

We thank the reviewer for the precise pointers. Regarding the comment that "the paper needs to explain more precisely which conclusions these experiments add beyond the existing observations," we admit that they are not stated explicitly in one place, as they surface as the paper progresses, from Section 4 through Section 6. We list them below.

What the paper adds beyond precedents

  1. Only one high-level coverage is sufficient: QA versus verbatim (Section 4). With the training form as the controlled variable, paraphrases of the trained form come largely for free (Q = 0.56) while the other form does not (0.16 and 0.27; Figure 2). One verbatim example carries to verbatim paraphrases never trained on (0.20 → 0.72, Table 9). To our knowledge, no prior work shows these two are the level at which coverage is needed, while paraphrases are not needed.
  2. More augmentation is not better (Section 6.1). At a fixed budget, a third form lowers AUS for 9 of 14 objectives (RMU 0.65 → 0.44, Table 1). The two top-level forms balance coverage against damage.
  3. Forms generalize worse than languages (Sections 4 and 5). Cross-language transfer keeps about 70% of the in-language removal (0.38 of 0.54); cross-form transfer keeps 21–36%. To our knowledge, no prior work compares the two axes. So the forget set needs the other form but not every language.
  4. The three objective groups respond to augmentation differently (Section 6.3). Ascent objectives and RMU lose. Bounded-preference objectives change little with all weights and gain only with MLP-only training. Target-reshaping objectives gain.
  5. MLP-only training is not uniformly better, which refines Guo et al. (Section 6.2). It is best for bounded-preference objectives once the forget set is augmented, but full training reaches further across forms (Table 7) and is better for target-reshaping objectives. o_proj alone matches the MLP on the trained form Q = 0.91 with cross-form Q = 0.09, also in a linear-attention hybrid (Table 8).

What explains the stronger language transfer Transfer across languages is uneven in our grid too, with Burmese and Yoruba the weakest sources, but it is consistently stronger than transfer across forms. This contradicts Choi et al.\ and Farashah et al., who find that unlearning transfers weakly across languages and leaves low-resource languages exposed. We think the contradiction comes from how performance is measured. The length-normalized answer probability used in earlier work isn't fair across languages. Besides capturing whether the model knows a fact, it also rewards the model for being fluent in the script, which depends heavily on tokenization. In our own runs, after unlearning in English, 0.47 of the average token probability survives in Burmese but only 0.21 in Spanish. Burmese, however, averages 0.71 characters per token, roughly three times as many tokens per character as most languages, so most of that high token probability comes from the model predicting the next piece of a Burmese word well, not from remembering the fact (Appendix~G). Under this metric, heavy tokenization makes low-resource languages look leaky even when the knowledge is gone. That is why we built our metrics to measure knowledge only for a fairer cross-language metric. Measured this way, much of the cross-lingual leakage reported before turns out to come from the metric rather than from knowledge that survived.

In a nutshell. From prior work: add other languages, forms, and paraphrases to the forget set, and train every objective the same way. After our study: no paraphrases, no extra languages, at most one added form, and a different decision for each objective family (Table 2):

  1. Ascent objectives and RMU: do not augment (best for 4 of 5).
  2. Bounded-preference objectives: add the verbatim form and train only the MLP (best for 3 of 4; NPO 0.65 → 0.69).
  3. Target-reshaping objectives: add the verbatim form and train all weights (best for 4 of 5; UNDIAL 0.79 → 0.86).

Wording we will fix. The Section 2 sentence that benchmarks "train and score within a single form" is too broad, since MUSE and RWKU probe more than one form. The accurate statement is that they unlearn on one form and probe others in one direction, and none varies the training form with the objective held fixed. Section 5 will no longer say that we "revise" earlier conclusions. We will cite RULE and discuss Guo et al. as a direct precedent.

The revision will state these additions explicitly in the introduction, each against its closest precedent. If this clarifies the increment, we would be grateful if the reviewer would consider raising the score.

Reviewer_p7vzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper introduces AUGUNLEARNING, an evaluation and data augmentation suite that extends existing factual data across query forms, languages, and adversarial prompts, together with metrics for forgetting and knowledge retention. Using this suite, it systematically studies fourteen unlearning objectives on two models, examining unlearning generalization across forms and languages and the effects of updating different modules. Building on these findings, it combines training forms and restricts updates to MLP modules, reporting improved forgetting coverage and a better trade-off with knowledge retention for some methods.

Critical strengths
  1. The paper addresses an important problem and provides a systematic evaluation suite for studying unlearning generalization. The coverage of experimental conditions is broad, spanning fourteen unlearning objectives, two model architectures, multiple languages and query forms, and adversarial prompts.

  2. The experiments and analyses are extensive and examine the problem from several complementary perspectives. Gradient comparisons, module restrictions, augmentation ablations, tests on forms excluded from training, and training budget experiments help explain when unlearning generalizes and how it can be improved.

Critical weaknesses
  1. The scale and domain coverage of the data do not adequately support the paper’s broad conclusions. The main experiments use only 30 forget facts for Llama and 24 for Qwen from a small RWKU subset (Appendix A). The paper does not adequately explain why and how an initial pool of only 87 facts was selected instead of drawing more broadly from RWKU. Moreover, the evaluation focuses on biographical knowledge about public figures, leaving it unclear whether the reported patterns generalize to other knowledge domains. Although the limitations section acknowledges reliance on one base benchmark, the main conclusions about unlearning generalization are not correspondingly restricted to biographical knowledge. Even if substantial data expansion is impractical, the central findings should be checked in other knowledge domains using representative methods, or the claims narrowed accordingly.
Questions for authors

How were the initial 87 facts selected? Could the central findings be checked on a larger set of facts and in other knowledge domains using a few representative methods?

3 comments
AuthorsOpenReview ↗

Response to Reviewer p7vz

We thank the reviewer for the clear assessment. Regarding the concern, we first answer the questions, and then describe the check we are running and plans to narrow the claims.

How the 87 facts were selected

They are a seeded random sample of RWKU, not a hand-picked subset. From RWKU's forget sets, we keep the facts whose answer is a short entity (at most six words), shuffle the subjects with a fixed seed, and take facts subject by subject so that no entity dominates. 87 is the number whose generated question and forms all pass the validation of Appendix A. The forget set is the subset the model answers correctly in both English and Spanish (30 for Llama-3.1-8B, 24 for Qwen3.5-9B), because Q is not meaningful for a fact the model does not hold; the remaining facts, about other subjects, form the held-out retain set. We kept the pool small because every fact has to be generated and validated in 8 forms, 4 attacks, and up to 18 languages. That is the reason for the size. It does not justify the breadth of the claims, and Appendix A will include this description.

Experiment now running: larger scale and other domains

Following the reviewer's suggestion, we are checking the central finding with four representative objectives, one per family (GradDiff, NPO, UNDIAL, RMU), on Llama-3.1-8B. Each is unlearned on the QA form and on the verbatim form and probed on all eight forms. The new sets are built with the same pipeline, with the knowledge gate applied in English and the number of steps scaled so that each fact receives as many updates as in the main experiments:

  • Scale: RWKU expanded to all 200 subjects, with up to two facts per subject.
  • Domain: non-biographical facts from PopQA, as two sets: geography (capitals and countries) and creative works (their authors, directors, composers, producers and screenwriters).

We will post the results in this thread as soon as the runs finish: for each set, Q on the trained form, on same-form paraphrases, and across forms in both directions, next to the same quantities from the paper's own runs.

Narrowing the claims

Independently of the outcome, the revision:

  • states in the abstract, introduction, and conclusion that the findings are established on real-world biographical facts from RWKU, on two 8–9B models;
  • moves the fact count and the single domain from one sentence in Section 6.3 into a dedicated Limitations paragraph;
  • presents the language and module results as observations on this set, not as general properties of unlearning;
  • removes "solve the cross-form generalization problem" from the abstract.

We thank the reviewer again. If the check and the narrowed claims address the concern, we would be grateful if the reviewer would consider raising the score.

AuthorsOpenReview ↗

Results: larger scale and other domains

Here are the results of the check we promised above. Brackets are 95% bootstrap intervals over forget facts.

Setup. Llama-3.1-8B; GradDiff, NPO, UNDIAL and RMU; each unlearned on the QA form and on the verbatim form and probed on all forms. Facts are those the model answers correctly in English. Steps scale with the number of facts (2,167 and 3,000), so each fact receives as many updates as in the paper. One seed per run, as in the paper.

SetForget factsTrained formSame-form paraphraseQA → verbatimVerbatim → QA
Paper's RWKU facts, same four objectives300.900.77 [0.67, 0.84]0.25 [0.16, 0.35]0.63 [0.48, 0.75]
RWKU, all 200 subjects650.940.82 [0.77, 0.86]0.34 [0.29, 0.40]0.35 [0.27, 0.42]
PopQA geography900.930.80 [0.76, 0.83]0.42 [0.38, 0.46]0.27 [0.22, 0.32]
PopQA creative works900.800.80 [0.74, 0.84]0.47 [0.40, 0.52]0.31 [0.25, 0.37]

What holds. On all three new sets, including two non-biographical domains, unlearning carries to paraphrases of the trained form (0.80–0.82) far more than to the other form (0.27–0.47), and the intervals do not overlap. The form boundary is not an artifact of 30 facts or of biographical knowledge. Three seeds on the original 30 facts give the same gap (0.32 ± 0.01 over GradDiff, NPO, UNDIAL and RMU).

How the sets were built

  • RWKU: fill-in-the-blank probes of all 200 subjects, up to two facts per subject, with subject-disjoint forget, retain and anchor splits. Of 239 forget candidates, 75 passed validation in every generated form and 65 passed the knowledge gate. That is about twice the paper's set.
  • PopQA: 400 subjects per set, one fact per subject. Geography (capital, capital of, country): 188 complete facts, 149 after the gate, capped at 90. Creative works (author, composer, director, producer, screenwriter): 230 complete, 205 after the gate, capped at 90.
  • One change to the pipeline for PopQA: the prompt that writes the verbatim sentence uses an example that starts with the subject. The original example starts with the answer, which our own validator rejects for relations such as "capital of". The validators are unchanged.

Scope. English only, one model, four objectives. The paper's grid (fourteen objectives, eight forms, four attacks, up to eighteen languages) took a long time to run, so we chose breadth over forms and languages rather than over datasets. These runs add the dataset dimension for the central finding. The language and module results were not re-tested on these sets, so the revision restricts them to the original RWKU facts, as stated above.

We thank the reviewer for suggesting this check. It supports the central finding, and the revision adds it as an appendix. If this addresses the concern, we would be grateful if the reviewer would consider raising the score.

Thank you for explaining how the original facts were selected and for running the additional experiments. The results on the expanded RWKU set and in two additional domains from PopQA largely address my concerns about data scale and domain coverage. I also appreciate the clarification on the scope of the findings. These changes substantially strengthen the credibility of the paper’s conclusions. I will therefore raise my rating from 2 to 3.

Comments on the paper

3 comments

Dear Reviewer,

Thank you for your efforts in evaluating this submission. The authors have uploaded their reply. Please take your responsibility to review the authors' response, revisit your evaluation,and update your score and recommendation as appropriate. Please add comments on whether the authors have sufficiently addressed your raised points. If you have further questions,please discuss them with the authors.

Thanks,

Area Chair

AuthorsOpenReview ↗

Response to Reviewer vNLw

We thank the reviewer for the thorough review. We ran the checks asked for. The results are in the tables below. New runs use Llama-3.1-8B. Brackets are 95% bootstrap intervals over forget facts.

Q1. What "solve" means, and forms not seen in training

We agree that "solve" overstates what we show, and we will remove it from the abstract. The abstract will say instead that adding the verbatim form patches the cross-form gap at a utility cost that depends on the objective.

QuantityWithout the verbatim formWith one added verbatim form
Q on two verbatim paraphrases never trained on, Llama (Table 9)0.200.72
Same, Qwen (Table 9)0.220.81
Mean AUS over objectives (Table 2)0.640.64
UNDIAL, Q (Table 1)0.650.78
UNDIAL, U (Table 1)0.950.95

Q2. Stability across seeds, facts and translations

We added two seeds for GradDiff, NPO, UNDIAL and RMU, each unlearned on the QA form and on the verbatim form. A seed changes the order of the training examples. Figure 2 averages eight source forms, seven objectives and two models, so its values are lower than these.

Mean ± s.d. over 3 seedsQ
Same-form paraphrase0.77 ± 0.00
QA → verbatim0.29 ± 0.03
Verbatim → QA0.62 ± 0.01
Same-form minus cross-form gap0.32 ± 0.01

Facts. We bootstrapped over facts (10,000 resamples) using the paper's rows.

Both modelsLlamaQwen
Same-form minus cross-form gap0.35 [0.28, 0.39]0.32 [0.24, 0.38]0.37 [0.28, 0.44]
Objectives with a positive gap7 of 7 on each7 of 77 of 7
P(Burmese and Yoruba are the two weakest sources)0.9990.955

Module ordering. The standard deviation over three seeds is at most 0.03 for all nine groups of Table 7; four are shown:

Trained module groupall weightsMLPdown_projo_proj
Cross-form Q, mean ± s.d. over 3 seeds0.35 ± 0.010.26 ± 0.030.22 ± 0.010.09 ± 0.00

Translations. We translated Burmese, Yoruba and Spanish (as a control) again with Gemma 3 27B, a model from a different family, using the same prompts and validators, and reran those source rows for seven objectives. The two translations are different texts. This is one second system, and people have not checked translation quality.

LanguageSource Q, original translationSource Q, second translationRank among 18 sources (1 = weakest)chrF, questions
Burmese0.25 [0.21, 0.29]0.24 [0.20, 0.28]1 → 155.5
Yoruba0.26 [0.21, 0.31]0.23 [0.18, 0.28]2 → 145.5
Spanish (control)0.39 [0.33, 0.44]0.38 [0.32, 0.42]11 → 793.0

Q3. Matched controls for MLP vs attention

We ran both controls in the Table 7 setting (UNDIAL, QA form, 1000 steps, seed 42). In the first, down_proj is trained on a random 4,096 of its 14,336 input columns in every layer, which is 536.9M entries, the same as q_proj and o_proj (three draws, and one run with plain AdamW). In the second, o_proj gets a higher learning rate until its update norm equals that of down_proj.

Trained weightsTrained entriesCross-form QRobustness Q
down_proj (Table 7)1,879M0.21 [0.09, 0.34]0.71 [0.53, 0.85]
down_proj, 4,096 columns536.9M0.11 [0.02, 0.21]0.53 to 0.54
q_proj (Table 7)536.9M0.00 [0.00, 0.00]0.22 [0.11, 0.34]
o_proj536.9M0.09 [0.02, 0.18]0.54 [0.38, 0.69]
v_proj (Table 7)134.2M0.05 [0.00, 0.12]0.39 [0.23, 0.54]
k_proj (Table 7)134.2M0.00 [0.00, 0.00]0.07 [0.01, 0.14]
o_proj, update norm matched536.9M0.14 [0.05, 0.24]0.74 [0.60, 0.84]
Cross-form Q, down_proj minusDifference
q_proj, equal entries+0.11
o_proj, equal entries+0.02 [0.00, 0.07]
v_proj, a quarter of the entries+0.06
k_proj, a quarter of the entries+0.11
o_proj, equal update norm (full down_proj)+0.07 [-0.02, 0.17]

At equal trained entries down_proj stays above every attention projection in cross-form Q, and its robustness Q is above q_proj, v_proj and k_proj and equal to o_proj. The controls therefore keep the ordering of Table 7 (MLP above the attention modules in cross-form Q); the gap to o_proj is smaller than in Table 7. Table 7 will be reported together with these controls, and the abstract and Finding 4 will state it that way. The Table 2 result (MLP-only training helps bounded-preference objectives once the forget set is augmented) is an AUS comparison and stays. These controls use one seed, one objective and one model.

Mechanism claims. These are correlational. Sections 4 and 5 state this, and we will add it to the abstract. We will also change "stores largely independently" to "whose unlearning gradients are close to orthogonal".

Thank you again. And we hope the revised claims address your concerns.

Thank you for the response; my primary concerns have been addressed. I suggest incorporating these points into the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.