acceptodds
Under review as a conference paper at ICLR 2027

A Ground-Truth Model Suite for Unlearning Evaluation Across the LM Training Lifecycle

Abstract

Machine unlearning – the ability to selectively remove specific information from a trained model – is critical for privacy, copyright compliance, and safety. In this work, we present a controlled study of machine unlearning across the large language model (LLM) development lifecycle, spanning pre-, mid-, and post-training. To achieve this, we inject multiple distinct datasets during the from-scratch training of OLMo-2 models ranging from 179M to 2.7B parameters and trained on 210B tokens. These datasets encompass seven diverse unlearning tasks with varying difficulty levels, including the removal of Gaussian poisons, fictional knowledge, reasoning patterns, social interaction data, and adversarial attacks. Crucially, we also provide counterfactual models trained without these datasets, which serve as exact ground-truth targets for unlearning evaluation. Leveraging this suite, we investigate the effectiveness of existing unlearning algorithms across three distinct lifecycle settings: (1) immediate unlearning after pre-training, (2) unlearning after decaying from the pre-trained model and (3) unlearning via mid-training. Our experiments reveal that unlearning success is highly non-uniform where effectiveness depends heavily on the stage of the training cycle and is significantly influenced by model scale and task complexity. Surprisingly, we find that one simple and hyperparameter free baseline among eight unlearning methods is exceptionally difficult to beat: continual pre-training on fresh data.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Unlearning, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XhtmOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a controlled evaluation suite for studying LLM unlearning across different stages of the training lifecycle. The authors pretrain OLMo-2 models from 179M to 2.7B parameters with seven types of inserted data and construct matched counterfactual models trained without these insertions. Using these models, the paper evaluates eight unlearning methods after pretraining, learning-rate decay, and mid-training, and studies how removal difficulty varies with model scale, data repetition, and insertion timing. The results show that existing methods often provide limited forgetting or damage model utility, while continued pretraining on fresh data is a strong and stable baseline.

Critical strengths
  • This paper provides carefully matched counterfactual models trained without the inserted data, enabling a much cleaner evaluation of unlearning than benchmarks that rely only on behavioral proxies or fine-tuned forget sets.

  • The evaluation suite is unusually comprehensive, covering four model scales, seven types of removal targets, multiple training stages, and eight representative unlearning methods under a shared training pipeline.

  • The finding that continued pretraining on fresh data is difficult to outperform is both surprising and practically useful, and the analysis of insertion timing, repetition, model scale, and learning-rate decay provides valuable guidance for future unlearning research.

Critical weaknesses

i) The interpretation of the counterfactual models as “exact ground truth” would benefit from further qualification.

Training carefully matched experiment and counterfactual models at four scales is a substantial and valuable contribution. Controlling the initialization and training batches provides a much cleaner reference than is available in most existing LLM unlearning benchmarks. However, each counterfactual is based on a single training seed and therefore represents one particular retraining outcome rather than the broader distribution of models that could result from removing the target data [a, b, c]. This does not invalidate the paired comparisons, but the paper should clarify the scope of “exact ground truth” and, if feasible, estimate ordinary training variation using multiple runs at one of the smaller model scales.

The paper also states that the counterfactual models exclude the controlled insertions, but it does not report whether the original OLMo-mix corpus was screened for exact or approximate overlap with the seven target datasets. This is particularly relevant for public news articles, benchmark questions and answers, SODA conversations, and related factual or reasoning content. Data contamination can materially affect model behavior even under limited exposure [d]. If the base corpus already contains these targets or close variants, the counterfactual model represents a model without the additional controlled exposures rather than one that never observed the target information. Reporting exact-match, near-duplicate, and semantic-overlap checks would clarify which results measure complete target removal and which measure removal of the incremental effect of insertion.

ii) The current design evaluates joint removal rather than providing task-specific ground truth for each target.

All seven target datasets are inserted into the same experiment model, and the matched counterfactual is trained without any of them. This design is computationally efficient and enables a broad set of evaluations from a single training run, but the resulting reference corresponds to jointly removing the full collection. It does not directly represent a request to remove one target while retaining the other six. For example, a counterfactual for deleting only the MUSE news articles should still contain the fictional entities, math problems, canary conversations, benchmark answers, and backdoor examples. Standard unlearning formulations define the reference relative to the particular subset requested for deletion [a, b, c].

The joint design also makes it difficult to rule out interactions among the heterogeneous targets during pretraining and unlearning. A method receives a combined forget set containing text memorization, factual knowledge, reasoning capabilities, privacy canaries, and adversarial signals, so its performance on one metric may be influenced by updates associated with the other targets. The suite therefore provides seven evaluation signals within a joint removal experiment rather than seven fully independent unlearning tasks. Task-specific counterfactuals for a representative subset, or smaller controlled experiments comparing joint and individual removal, would help establish whether the main conclusions transfer to realistic single-target deletion requests.

[a] Certified Data Removal from Machine Learning Models. ICML 2020.

[b] Machine Unlearning. IEEE Symposium on Security and Privacy 2021.

[c] Remember What You Want to Forget: Algorithms for Machine Unlearning. NeurIPS 2021.

[d] How Much Can We Forget about Data Contamination? ICML 2025.

Questions for authors
  1. Was the original OLMo-mix corpus checked for exact or approximate overlap with the seven target datasets, particularly the public news articles and benchmark data?

  2. Do the conclusions remain similar when one target dataset is inserted and removed independently rather than jointly with the other six targets?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Xhtm (part 1)

We thank the reviewer for their positive assessment of our work, and for highlighting the value of our matched counterfactual models across four scales, seven task types, and multiple lifecycle stages. We have run the exact overlap audits and single-target unlearning ablations you suggested, which are now included in Appendix D.7 and Appendix D.8 of the revised manuscript. Below, we address your two main questions and points in detail.

The interpretation of the counterfactual models as “exact ground truth” would benefit from further qualification.

We completely agree with the theoretical distinction you raise regarding retraining distributions [a, b, c]. As noted in Appendix A (Limitations) and Appendix B, each counterfactual shares the exact random initialization and batch order of its corresponding experiment model. In our setting, this coupled design is a deliberate variance-reduction choice: by fixing all sources of stochasticity (initialization and mini-batch ordering), the difference between the experiment model and the counterfactual isolates the exact causal effect of the inserted sequences without confounding run-to-run training noise. We will update the text to clarify that our counterfactuals represent the exact coupled counterfactual (for a fixed seed and batch order) rather than a distribution over independent retraining seeds.

Was the original OLMo-mix corpus checked for exact or approximate overlap with the seven target datasets, particularly the public news articles and benchmark data?

To verify whether any of our target insertions already existed naturally in the base pretraining dataset, we audited 1,000 samples per task for exact word-for-word matches using infini-gram [Liu et al., 2024, https://huggingface.co/spaces/liujch1998/infini-gram] against both the full OLMo-mix-1124 corpus and the specific 5.38% subset used in our pretraining runs (full details are in Appendix D.8.):

Target taskFull OLMo-mix-1124 corpus: insertedFull OLMo-mix-1124 corpus: never inserted (control)Our pretraining data (5.38% slice): insertedOur pretraining data (5.38% slice): never inserted (control)
News articles5.6%4.8%2.5%2.2%
Benchmark questions20.1%21.4%9.5%9.8%
Canary conversations0.4%0.4%0.1%0.3%
Fictional knowledge0.0%0.0%0.0%0.0%

This audit suggests that background contamination does not affect our conclusions for three reasons:

  • Synthetic and canary tasks are clean: Synthetic tasks (e.g., Fictional knowledge, iGSM, Gaussian poisons) have 0.0%0.0\% overlap by construction, and canary conversations have only 0.1%0.1\% overlap in our training slice.
  • Equal background rates across target and control splits: While a small fraction of real-world texts (2.5% of News articles and 9.5% of Benchmark questions) appear to be in our pretraining slice, they appear at virtually identical rates in our held-out never-inserted control splits (2.2% and 9.8%). Because our evaluation metrics measure the gap between inserted and never-inserted splits, this background presence affects both equally and cancels out.
  • Empirical verification via the Retraining model: Evaluating the counterfactual Retraining model (trained on the exact same 5.38% pretraining slice without our manual insertions) confirms that incidental background exposure leaves almost no trace. On News articles, the Retraining model flags only 2.3% of the target articles (near the 1% FPR floor), compared to 88.4% for the exposed Baseline model. Similarly, on Benchmark questions, the Retraining model gains just 0.6 accuracy points on the inserted questions, compared to 13.1 points for the Baseline.
AuthorsOpenReview ↗

Response to Reviewer Xhtm (part 2)

Do the conclusions remain similar when one target dataset is inserted and removed independently rather than jointly with the other six targets?

Because training seven separate counterfactual Retraining models from scratch is computationally prohibitive, our main setup evaluates joint removal across the inserted tasks, comprising only 0.5% of the 210B pre-training tokens, a regime where prior work has shown that distinct pre-training insertions exhibit negligible interference (Bordt & Pawelczyk, 2026). Following your suggestion for a controlled experiment, we ran an additional experiment comparing single-task unlearning against the baseline of no unlearning. Specifically, we took the optimal configurations for two representative methods (RMU and SimNPO), performed unlearning on only one target task at a time, and evaluated each resulting model across all tasks relative to the un-unlearned Baseline (Appendix D.8, Figure 20).

Our conclusions remain overall unaffected. We see that SimNPO again struggles with removing benchmark contaminations, Gaussian poisons as well as Backdoor attacks. We see similar patterns for RMU. Strikingly, most of the metric reduction caused by these methods is non-specific: for example, when RMU targets any of the other five tasks, the TPR on MUSE News still drops by 20–22%, compared to 24% when RMU explicitly targets MUSE News itself. Hence, the dedicated unlearning are not surgical, instead unlearning one task affects the others too. The complete results for the RMU and SimNPO methods can be found in Appendix D.8 Figure 20.


We thank the reviewer again for their thoughtful comments and feedback. We hope we addressed all your questions/concerns/comments adequately. In light of our clarifications, please consider increasing your score.


Thank you for the detailed responses and additional experiments. The clarification of the coupled counterfactuals, overlap audit, and single-task unlearning results address most of my concerns. I appreciate the work that went into these additions and remain positive about the paper. I will maintain my positive score.

AuthorsOpenReview ↗

Response to Official Comment by Reviewer Xhtm

Much appreciated!

Best wishes, the authors

Reviewer_mcGAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a benchmark suite for machine unlearning across the lifecycle of LLM training. The authors train the OLMo-2 models and inject seven types of target data. They also train counterfactual models without these insertions to provide a reference for unlearning. The experiments compare 8 unlearning algorithms with continued-training baselines. It examines how unlearning depends on the training stage, model size and data repetition. The authors discovered that simply continuing pretraining on clean data is a strong baseline compared to other unlearning baselines.

Critical strengths
  1. The paper provides a broad set of evaluations on machine unlearning, including different unlearning algorithms with continued-training baselines and across several model scales and training settings. Evaluating both forgetting and retained model utility against counterfactual reference models includes the very important factor of unlearning effectiveness among with retained utility.
  2. This paper provides an investigation into how unlearning effectiveness changes before and after learning-rate decay or continual pretraining and mid-training. These experiments highlight the practical importance of when unlearning should happen during the model development process.
  3. This paper was clear and easy to read.
Critical weaknesses
  1. My main concern is the limited novelty compared to this closely related work. The seven target-data categories, the OLMo-2 training setup, the controlled insertion strategy, and the use of an unexposed reference model were all set up in [1]. The main extension, therefore, appears to be evaluating existing unlearning methods within this established framework.
  2. [1] demonstrated forgetting during continued training, including cases where the cross-entropy loss on previously observed batches approached that of an unexposed model (Appendix E.8, Figure 22). Thus, the observation behind the paper’s strongest baseline might not be new. The comparative advantage over several unlearning algorithms is an additional empirical result.

[1] Bordt, Sebastian, and Martin Pawelczyk. "Train once, answer all: Many pretraining experiments for the cost of one." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors
  1. Could the authors more explicitly distinguish the contributions from [1]? Given the overlap in target datasets and training framework, it would be helpful to explain which research questions this suite answers that the previous work could not. Adding tasks or threat scenarios not covered by [1] could further strengthen the paper’s novelty.
  2. Since the suite includes poisoning and backdoor scenarios, could the evaluation also consider methods specifically designed for robust unlearning, for example, [2] and its relevant work? This would help assess the scenarios under 4 kinds of attacks included in the paper and the advantages of the continual pretraining method with these robust-focused methods.

[1] Bordt, Sebastian, and Martin Pawelczyk. "Train once, answer all: Many pretraining experiments for the cost of one." International Conference on Learning Representations. Vol. 2026. 2026. [2] Fan, Chongyu, et al. "Towards llm unlearning resilient to relearning attacks: A sharpness-aware minimization perspective and beyond." arXiv preprint arXiv:2502.05374 (2025).

If the authors address these questions above, I would be willing to raise my score to 3.

7 comments
AuthorsOpenReview ↗

Response to Reviewer mcGA (part 1)

Thank you for your time and valuable feedback. We will incorporate all your feedback in the final version of our paper and have updated the manuscript to reflect your key suggestions for improvement. Please find our response to your questions and concerns below:

[1] demonstrated forgetting during continued training, including cases where the cross-entropy loss on previously observed batches approached that of an unexposed model (Appendix E.8, Figure 22). Thus, the observation behind the paper’s strongest baseline might not be new. The comparative advantage over several unlearning algorithms is an additional empirical result.

My main concern is the limited novelty compared to this closely related work. The seven target-data categories, the OLMo-2 training setup, the controlled insertion strategy, and the use of an unexposed reference model were all set up in [1]. The main extension, therefore, appears to be evaluating existing unlearning methods within this established framework. Could the authors more explicitly distinguish the contributions from [1]? Given the overlap in target datasets and training framework, it would be helpful to explain which research questions this suite answers that the previous work could not. Adding tasks or threat scenarios not covered by [1] could further strengthen the paper’s novelty.

While [1] introduced the multiple-experiment paradigm to study memorization during pretraining, our paper tackles a fundamentally different problem and subfield: evaluating whether machine unlearning works across the foundation model lifecycle.

1. Precedent in Large-Scale Empirical ML: In foundation model research, decoupling the creation of pretraining frameworks such as [1] from domain-specific algorithmic evaluations is an established and high-impact practice. A prominent example is EleutherAI’s work: Biderman et al. introduced the Pythia training suite at ICML 2023 [A], and subsequently used that training recipe in multiple standalone works, among many others a NeurIPS 2023 paper [B] to study memorization and forgetting dynamics (see also Carlini et al., ICLR 2023 [C]). Similarly, while we build upon the pretraining framework of [1], our work answers a distinct set of algorithmic and operational questions for the machine unlearning community (see more on that below).

2. Passive Loss Decay in [1] ≠\neq Continual Pretraining as a Pareto-Dominant Unlearning Baseline: While Appendix E.8 (Figure 22) of [1] noted that cross-entropy loss on previously seen batches decays during continued training (a passive forgetting phenomenon also observed in Jagielski et al., 2023), it had not been realized in the unlearning literature that simple continual pretraining serves as a competitive – let alone superior – unlearning baseline. This is because popular unlearning benchmarks (e.g., TOFU, MUSE) are built around fine-tuning LLMs whereas our benchmark is built around pre-, mid- and post-training.

3. Fundamental New Insights for Machine Unlearning: Our experiments reveal several critical findings that cannot be inferred from [1] and directly challenge current directions in machine unlearning:

  • Continual pretraining strictly dominates dedicated unlearning under matched compute (Section 4.1 (updated Figure 3) & Appendix D.3 (new Figure 13)): Across all tasks, dedicated unlearning methods suffer catastrophic utility collapse before reaching the counterfactual Retraining model, whereas compute-matched continual pretraining reliably approaches the counterfactual with zero utility loss.
  • Dedicated unlearning is undone by downstream mid- and post-training (Section 4.5): When unlearned checkpoints (e.g., along RMU’s Pareto frontier) undergo standard mid-training, SFT, and DPO, their memorization rebounds within 1,000 steps to match the trajectory of the a baseline model that never was subjected to unlearning. Operationally, applying dedicated unlearning followed by mid-training performs no better than doing no unlearning at all.
  • Mid-training cannot replace continual pretraining (Section 4.4): Standard mid-training unlearns significantly less than continual pretraining (e.g., 29% vs. 5% poison TPR at 2.7B) because learning-rate decay halts forgetting halfway through the schedule.
  • Scale fundamentally bottlenecks unlearning (Section 4.3): Larger models not only memorize more, they unlearn much more slowly under identical token budgets – a new insight invisible to single-scale studies.
  • Dedicated unlearning methods merely mask knowledge and act non-surgically (Section 4.6 & Appendix D.8): Fine-tuning on just 512 samples rapidly restores "unlearned" knowledge in RMU, NPO and NPO+SAM, whereas continual pretraining resists relearning attacks just like the Retraining counterfactual.
AuthorsOpenReview ↗

Response to Reviewer mcGA (part 2)

4. Substantial New Model Training and Compute Investment Beyond [1]: Compared to [1], this paper required a massive new training and evaluation effort:

  • Full lifecycle pipelines: We executed 25B-token mid-training across model scales, as well as full post-training pipelines (SFT and DPO) to test the impact of unlearning on LLM development lifecyle, which we will make publicly available upon acceptance.
  • Compute-matched unlearning benchmark: We conducted an exhaustive two-stage hyperparameter search and Pareto-frontier evaluation across 8 dedicated unlearning algorithms, multiple checkpoints (pretrained vs. cooled-down), and adversarial relearning attacks (see Appendix D.3, Figure 13).
  • Multi-scale pretraining and counterfactuals: We leveraged the training recipe from [1] trained and retrained a full multi-scale model suite spanning 179M, 546M, 1B, and 2.7B parameters (including training the counterfactual Retraining model at 2.7B) on identical batch orders. A 7B experiment and counterfactual model is currently in training

Together, our findings deliver a novel and surprising message to the community: existing unlearning methods – designed for post-hoc fine-tuning – are fundamentally ill-suited for pretraining and foundation model development, motivating an entirely new class of lifecycle-aware unlearning algorithms.

Since the suite includes poisoning and backdoor scenarios, could the evaluation also consider methods specifically designed for robust unlearning, for example, [2] and its relevant work? This would help assess the scenarios under 4 kinds of attacks included in the paper and the advantages of the continual pretraining method with these robust-focused methods.

Following your suggestion, we expanded our evaluation to include NPO+SAM [2] (see Section 4.6 in the updated manuscript and Appendix D.5), which incorporates sharpness-aware minimization specifically to improve unlearning robustness. Beyond the standard NPO grid, we additionally tuned the perturbation radius hyperparameter ρ\rho for NPO+SAM and evaluated it across both our target tasks and the adversarial relearning attacks (Appendix Figures 16–19). We find that NPO+SAM behaves similarly to standard NPO and does not close the robustness gap with continual pretraining. Across the evaluated tasks (including poisoning and backdoor scenarios), NPO+SAM reaches forgetting and utility levels comparable to standard NPO. More importantly, when subjected to our relearning stress attacks (Figures 16–19), NPO+SAM remains vulnerable where fine-tuning causes its target metrics to rebound to virtually the same levels as standard NPO. In contrast, continual pretraining resists relearning across these settings, maintaining a clear advantage even over methods explicitly designed for robust unlearning.


We thank the reviewer again for their thoughtful comments and feedback. We hope we addressed all your questions/concerns/comments adequately. In light of our clarifications, please consider increasing your score.


References

[A] Biderman et al., "Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling," ICML 2023.

[B] Biderman et al., "Emergent and Predictable Memorization in Large Language Models," NeurIPS 2023.

[C] Carlini et al., "Quantifying Memorization Across Neural Language Models," ICLR 2023.

Thank you for the response and additional experiments, which have partially addressed my concerns. I appreciate the expanded lifecycle evaluation and the inclusion of NPO+SAM [5].

However, several novelty might have previous works: utility collapse has been studied in MUSE [1] and Yan et al. [2]; scale-related memorization and forgetting were investigated by Tirumala et al. [3]; and Lee et al. [4] showed that matching an oracle’s behavior does not guarantee knowledge removal, as subsequent fine-tuning can restore suppressed capabilities. Also, while the evaluation now covers additional training stages, the target tasks and data categories seem to remain largely inherited from [6]. My concern about the breadth of evaluation beyond this existing task suite therefore remains only partially addressed.

Considering the additional evidence and remaining concerns, I am willing to raise my score to 2.5 (between 2 and 3).

[1] Shi, Weijia, et al. “MUSE: Machine Unlearning Six-Way Evaluation for Language Models.” ICLR, 2025.

[2] Yan, Han, Zheyuan Liu, and Meng Jiang. “Dual-Space Smoothness for Robust and Balanced LLM Unlearning.” ICLR, 2026.

[3] Tirumala, Kushal, et al. “Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models.” NeurIPS, 2022.

[4] Lee, Bruce W., et al. “Distillation Robustifies Unlearning.” NeurIPS, 2025.

[5] Fan, Chongyu, et al. “Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond.” ICML, 2025.

[6] Bordt, Sebastian, and Martin Pawelczyk. “Train Once, Answer All: Many Pretraining Experiments for the Cost of One.” ICLR, 2026.

Reviewer_NECuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper builds a model suite for testing LLM unlearning against a true ground truth. The authors perform pre-training from scratch with the OLMo-2 models at various model sizes. They insert seven target datasets to the training corpus. Those datasets serve as the forget set for subsequent unlearning. For each training configuration, they also train a counterfactual model without the inserted data. This counterfactual is what a perfect unlearning method should produce. Checkpoints are released at several stages: pre-training, LR decay, continued pre-training, mid-training, SFT, and DPO.

Using the suite, the authors test a line of different unlearning methods. They report four main findings: 1. Dedicated methods either remove very little or break the model. 2. Simple continual pre-training on clean data moves the model close to the counterfactual with no loss in utility. 3. Data seen late or seen many times is harder to remove, and larger models forget more slowly. 4. Mid-training removes less than continual pre-training at the same token budget.

Critical strengths

This paper evaluates unlearning in a more realistic and practically relevant setting, where both the base model and the ground-truth unlearned model are pre-trained at scale. This addresses key limitations of existing benchmarks, which either introduce the forget set through multi-epoch fine-tuning or lack a retrained reference model. The resulting benchmark provides exact counterfactuals for data encountered during single-pass pre-training, with known exposure (when and how often each item was seen). It also covers four model sizes and several lifecycle stages. Because this resource is very expensive to build, releasing it could benefit the community for a long time.

The findings that recency, repetition, and model scale influence unlearning difficulty (Sections 4.2–4.3) are clean, because all models share the same batches. The proposal that unlearning claims should be compared against continual pre-training at equal compute is a valuable check for future work.

Critical weaknesses
  • The main comparison is not compute-matched, so the headline claim is weakly supported. Continual pre-training runs for up to 10k steps (21B tokens). The dedicated methods only run for tens to a few hundred steps (at most about 0.2B tokens, per Appendix D.4). That is about 100× less compute. The methods also run with the full pre-training batch (512 × 4096 tokens) and are mostly used in forget-heavy settings. A fairer comparison would strengthen the claim.

  • It appears that the experiments cover much less than the suite and the claims promise. The paper presents seven tasks and four lifecycle points, but the experiments mainly use Gaussian poisons, canaries, one knowledge probe, and overall insertion likelihood. Backdoors, iGSM math, benchmark contamination, and the MUSE conditions are not evaluated. It appears that no unlearning is evaluated after SFT/DPO, even though this is listed as a contribution.

Questions for authors
  1. Can you compare the dedicated unlearning methods and continual pre-training at the same compute? Would continual pre-training still dominate?

  2. Why did you use the full pre-training batch size for the unlearning methods? Did you try smaller forget batches to reduce damage to utility?

  3. Is the forgetting from continual pre-training real removal, or can the data be recovered by relearning or quantization? Also, do you compare the experiment model against the counterfactual continued for the same number of steps, rather than against the step-100k counterfactual?

2 comments
AuthorsOpenReview ↗

Response to Reviewer NECu (part 1)

Thank you for your time and valuable feedback. We will incorporate all your feedback in the final version of our paper and have updated the manuscript to reflect your key suggestions for improvement. Please find our response to your questions and concerns below:

It appears that the experiments cover much less than the suite and the claims promise. The paper presents seven tasks and four lifecycle points, but the experiments mainly use Gaussian poisons, canaries, one knowledge probe, and overall insertion likelihood. Backdoors, iGSM math, benchmark contamination, and the MUSE conditions are not evaluated. It appears that no unlearning is evaluated after SFT/DPO, even though this is listed as a contribution.

The revised manuscript now evaluates every task, the missing MUSE conditions as well as the impact of unlearning across the LLM lifecyle:

  • All seven tasks: The updated Figure 3 now displays the pareto frontier for every unlearning method across all seven tasks. This includes the MIA TPR for the MUSE News articles, Canary dialogues, benchmark contamination, backdoors (denial of service and prompt extraction) and iGSM math.

  • MUSE conditions. The new Appendix Figure 10 evaluates the following MUSE News Conditions: Whole (conditions 1-3): The article is inserted 1, 10, or 100 times. Split once (conditions 4-6): The article is split at random into seven parts at paragraph boundaries, and these paragraphs are inserted 1, 10 or 100 times. Split per copy (conditions 7-9): The article is inserted 1, 10 or 100 times, and every copy is split into seven parts. All paragraphs are added into the training data.

  • Mid-training & Post-training: We added Appendix D.4, evaluating whether SotA unlearning methods have an effect on mid-training as well as post-training. Using the OLMo-2 mid-training recipe and data order, we mid-trained & post-trained the Baseline (No Unlearning), Retraining (retraining from scratch), Continual pretraining, RMU and SimNPO models, evaluating every task. The results are surprising, and they indicate a “reversion-to-no-unlearning effect”, meaning that a dedicated SotA unlearning method from literature followed by several hundred steps of mid-training & post-training performs as good as No Unlearning followed by midtraining and post-training. We see that on practically all tasks. This is a surprising observation and suggests that standard unlearning methods as developed in literature right now are not suitable to be used across the LLM development lifecycle. We have added these results and insights as a new Section 4.5 to the paper.

The main comparison is not compute-matched, so the headline claim is weakly supported. [...] Can you compare the dedicated unlearning methods and continual pre-training at the same compute? Would continual pre-training still dominate?

You are right and in response to your suggestion we now provide a fair comparison via budget-matched hyperparameter optimization for each unlearning method, meaning that hyperparameter optimization + unlearning use 10% of the total pretraining compute measured in tokens. The result of this budget-matched hyperparameter search is now shown in Figure 13 of Appendix D. Since, depending on the unlearning method, a single unlearning method run cannot use more compute because it would damage the model into an unusable state after a few hundred gradient steps, we exhaustively search every unlearning method’s hyperparameters. To this end, Figure 12 in Appendix D.3 shows the full search space, which is split into optimizer based hyperparameters shared by all methods (Figure 12 a) and unlearning specific hyperparameters (Figure 12 b).

For this exact compute-matched comparison, we still see that simple, hyperparameter free continual pre-training still dominates dedicated SotA unlearning methods.

Why did you use the full pre-training batch size for the unlearning methods? Did you try smaller forget batches to reduce damage to utility?

Following your suggestion, we conducted a full hyperparameter search over all hyperparameters, to identify the pareto frontier. The full search space is shown in Figure 11. During this search, we sampled batch sizes of 32, 64, 128, 256, 512. Still, they did not give better unlearning results than simply doing continual pretraining. We present the ablation in Appendix D.4.

AuthorsOpenReview ↗

Response to Reviewer NECu (part 2)

Is the forgetting from continual pre-training real removal, or can the data be recovered by relearning or quantization?

We now evaluate every unlearning method using three relearning attacks from unlearning literature:

  1. Relearning by fine tuning on 512 samples drawn randomly from the forget set for 50 steps [Hu et al., 2025, Fan et al., 2025]. with sample sizes increased from 256 to 2,048 [Fan et al., 2025].
  2. Fine tuning on unrelated data: the same fine tuning on 512 pretrain samples not present in the inserted data [Fan et al., 2025].
  3. 4-bit weight quantization attack [Zhang et al., 2025].

Using these attacks, we find that continual pretraining is substantially more robust to re-learning attacks than dedicated unlearning methods from literature, conrfirming continual pre-training enables the most reliable removal.

We present the News TPR results for all attacks in the following table. Please find the results on all other tasks in Appendix D.6 Figures 16-19.

ModelNo attackRelearning, forget dataRelearning, unrelated data4-bit quantization
Retraining2.3%2.5%2.5%2.5%
Continual pretraining5.4%9.1%6.0%5.0%
RMU70.3%96.7%83.2%68.4%
NPO58.6%96.7%82.5%63.8%
SimNPO8.9%15.8%10.4%9.1%

Also, do you compare the experiment model against the counterfactual continued for the same number of steps, rather than against the step-100k counterfactual?

Yes we do: for Figures 5, 6 and 7 in the paper, there is no need to evaluate the counterfactual model since we are running a membership inference attack for which the perfect unlearning score is known to be TPR=FPR=0.01 in our setting. To avoid clutter, we leave the counterfactual model evaluation out in these plots. In figures 3 and 4, we trained the counterfactual model to 110k steps.


We thank the reviewer again for their thoughtful comments and feedback. We hope we addressed all your questions/concerns/comments adequately. In light of our clarifications, please consider increasing your score.


References

Hu et al., 2025 Hu, Shengyuan, et al. “Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning.” Proceedings of the Thirteenth International Conference on Learning Representations (ICLR), 2025.

Fan et al., 2025 Fan, Chongyu, et al. "Towards llm unlearning resilient to relearning attacks: A sharpness-aware minimization perspective and beyond." arXiv preprint arXiv:2502.05374 (2025).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment and Summary of Revisions

Dear Reviewers, SAC and AC,

All three reviewers recognized that this work addresses a novel and important problem: evaluating if and to what extent existing unlearning methods can be used as tool to remove undesired information (poison, canaries, private information, general reasoning capabilities etc) throughout the LLM development lifecyle, i.e., throughout pre, mid and post-training. This question has not been previously studied.

Reviewer concerns focused on compute fairness and additinoal evaluations (NECu), robust baselines and literature clarifications (mcGA) as well as corpus overlap and single-target removal (Xhtm). Inspired by the reviewer comments and suggestions, we have provided the following changes to our manuscript during the rebuttal addressing every question raised. The revised PDF contains the following updates:

ReviewerConcernChangeWhere
AllHyperparameter searchSearch per method, including batch size and retain weightApp. D.3
NECuCompute not matchedMatched hyperparameter search budget to continual pretraining (10% of pretraining compute)Sec 4, App D.3
NECuMissing tasks evaluationsAll seven tasks and the MUSE News conditionsFig. 3; App. D.2
NECuEvaluation after SFT/DPOUnlearning through mid-training, SFT and DPOSec 4.5; App. D.4
NECuFull pre-training batchSearched batch sizes from 32 to 512App. D.3
NECuIs Unlearning real removal?Evaluated unlearning methods under three relearning attacksSec 4.6; App. D.5
mcGALiterature clarification relative to [1]Unique LLM lifecycle perspective, counterfactual ground truth, relearning attacks, new insightsSec 4.4-4.6; App D3, D.7
mcGARobust unlearning [2]NPO+SAM under [2]'s protocol and attacksApp. D.5, D.6
XhtmCorpus overlapExact-match corpus searchApp. D.8
XhtmSingle-target removalEach target removed alone (RMU and SimNPO)App. D.7


Finally, we would additionally like to highlight that our work makes several impactful contributions to the unlearning literature:

  • A multi-task unlearning benchmark. From verbatim text and canaries, to factual knowledge, benchmark contamination, backdoors, Gaussian poisons and math reasoning, each task is evaluated against Retraining from scratch (perfect unlearning).

  • A compute-matched evaluation of nine unlearning methods. Eight unlearning methods (+ NPO+SAM [2]), each with its own hyperparameter search under continual pretraining's budget (10% of pretraining compute).

  • New insights. 1) Continual pretraining dominates every unlearning method at matched compute. 2) Relearning attacks after unlearning bring back much of the supposedly removed knowledge. In contrast, simple unlearning by hyperparameter-free continual pretraining persists across the lifecycle and relearning attacks. 3) Within 1k mid-training steps, all six checkpoints chosen from the Pareto frontier of a dedicated unlearning method (e.g., RMU) revert to performance levels of a model did not undergo unlearning. 4) In summary, these findings suggest that dedicated unlearning methods from literature are not ready to to be suitably used as part of the LLM development lifecyle.

  • A ground-truth model suite across the LLM lifecycle. Experiment and Retraining models at 179M, 546M, 1B and 2.7B, trained identically except for the inserted data, continued through mid-training, SFT and DPO. We are currently also pre, mid and post-training 7B parameter models and will add them to the model and benchmark suite as soon as they are available.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.