acceptodds
Under review as a conference paper at ICLR 2027

Unlearning During Pretraining Can Improve Tamper Resistance over Data Filtering

Abstract

Preventing the misuse of open-weights language models is a challenging problem, as adversaries have the ability to fine-tune harmful capabilities into models. To make models tamper-resistant with respect to fine-tuning, research has focused on applying unlearning algorithms to a pretrained model or filtering out harmful data from the pretraining corpus. However, post-hoc unlearning methods are mostly not robust to adversarial fine-tuning, and data filtering has steep diminishing returns after the most harmful data is already filtered out. We propose applying common unlearning algorithms during pretraining instead, preventing unwanted capabilities from forming in the first place. Compared to post-hoc unlearning, pretrain-time unlearning significantly improves resistance to adversarial fine-tuning. Pretrain-time unlearning improves over further data filtering in the regime where the most obvious documents have already been removed. Our results highlight that unlearning during pretraining may be an effective tool to prevent undesired capabilities in language models, providing a productive use for harmful data that is usually filtered out.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
unlearning, data filtering, tamper resistance, open-weights safety

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2zDWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether applying unlearning during pretraining can improve resistance to adversarial fine-tuning compared to post-hoc unlearning and data filtering. Using Biology-related Wikipedia text as a proxy for harmful knowledge, the authors evaluate gradient-ascent-based unlearning methods across language models of different sizes. The results suggest that pretrain-time unlearning provides stronger finetuning resistance than post-unlearning, and can achieve better safety–capability trade-offs than aggressive data filtering under certain evaluation settings.

Critical strengths
  1. The paper addresses an important problem in open-weight model safety: preventing unwanted capabilities from being recovered through adversarial fine-tuning.
  2. The experiments compare multiple training strategies, account for training budgets and capability degradation, and examine the effects across different model scales and unlearning strengths.
  3. This paper demonstrates interesting empirical findings. The observation that the timing of unlearning substantially affects resistance to subsequent fine-tuning is interesting. The results also suggest that unlearning can complement data filtering.
Critical weaknesses
  1. The novelty of method is limited. The proposed approach primarily integrates existing unlearning objectives into the pretraining process. While this is a reasonable design choice, the methodological contribution is relatively incremental.
  2. The explanation of the underlying mechanism is insufficient. A central finding is that pretrain-time unlearning consistently outperforms post-hoc unlearning. However, the paper provides limited analysis of why this happens. The explanation that earlier unlearning prevents unwanted capabilities from forming remains largely intuitive, without sufficient experimental evidence to establish the underlying mechanism.
  3. The experiments use relatively small models (up to 163M parameters) and Biology-related language modeling as a proxy for harmful capabilities. It remains unclear whether the conclusions generalize to realistic adversarial fine-tuning attacks on larger open-weight LLMs.
  4. The claimed advantage over aggressive data filtering depends on how benign capabilities are measured. In particular, the advantage does not hold when considering the full benign dataset loss, which raises concerns about the generality of the reported safety–capability trade-off.
Questions for authors
  1. I'm wondering why is the pretrain-time unlearning more effective? Could the authors investigate the underlying mechanism through a timing ablation, such as applying the same unlearning objective at different stages of pretraining? This could help distinguish the effect of preventing capability formation from other optimization effects.
  2. Can the comparison between pretrain-time and post-hoc unlearning be further controlled? Although the authors sweep unlearning strengths and match token budgets, the two approaches use different learning rate schedules and optimizer states. Could these differences partially explain the observed performance gap?
  3. How well do the findings generalize to realistic harmful capabilities? Evaluating larger models or more realistic harmful finetuning scenarios would strengthen the connection between the empirical results and the paper's security motivation.
5 comments

We thank the reviewer for the constructive feedback. We are happy to hear that the reviewer found that 1) the paper addresses an important problem, 2) the experiments are quite thorough, and 3) the paper demonstrates interesting empirical findings. We go through the concerns in order below:

W1, Method novelty: We respectfully disagree with the premise that the simplicity of the core algorithmic idea significantly detracts from the novelty of the paper. All previous unlearning interventions have been applied for already pretrained models, and moving to interventions during pretraining is a more significant algorithmic change than, e.g., tweaks on existing unlearning losses. We would also argue that the simplicity of the method is a strength, not a weakness, as more complex methods are often brittle and sensitive to hyperparameter adjustments. Current pretraining-time interventions for tamper resistance are quite scarce (mostly data filtering and gradient routing), and our paper provides evidence that pretrain-time unlearning expands the toolkit with a new method.

W2, Q1, timing ablation: We agree overall that a better understanding of when pretrain unlearning works is helpful, and to that end we have now added the timing ablation experiment suggested by the reviewer in Appendix F.7. It seems that pretrain unlearning earlier on does result in better safety-capability tradeoffs across the board. An interesting feature is that applying the unlearning at the very end results in less persistent damage both to Biology but also to the benign data loss, suggesting that pretrain unlearning causes more durable changes to the model overall.

Another interesting study is comparing pretrain unlearning to simply training the models for a shorter time, to separate the effect of the model being generally less capable from Biology-specific finetuning resistance. The new Figure 6 in the main paper shows the results for the finite-data finetune metric, showing that as the amount of available data grows, filtering more, undertraining, and applying WGA all start to have similar effects. This suggests that the effect of pretrain unlearning is an added resistance to some amount of unique tokens (e.g. 10M).

Q2, further control for pretrain unlearning vs. post-hoc unlearning comparisons. We provide additional ablations on pretrain unlearning vs. post-hoc unlearning in Appendix F.1, sweeping over multiple learning rates and different capability and safety metrics. Figure 3 in the main paper uses the Pareto frontier across learning rates and unlearning strengths for post-hoc unlearning. Pretrain unlearning consistently outperforms post-hoc unlearning, except with the “updates to recover baseline loss” capability metric when applying unlearning on the unfiltered model, where the performance is similar. That said, in the more useful case where unlearning is applied jointly with filtering, pretrain unlearning clearly outperforms post-hoc unlearning.

W2, W3 & Q3, Experiment realism. We have updated the paper to improve the experiment realism through the following: We have moved the results to the more realistic pretraining dataset NVIDIA ClimbMix, as opposed the more narrow Wikipedia dataset we used previously. We provide finetuning resistance metrics on the WMDP-Bio-forget dataset [1], in Figure 5 as well as with the extended post-hoc unlearning comparisons in Appendix F.1. We show in Appendix E that loss on this dataset in open-weights LLMs correlates strongly with the WMDP-Bio dangerous capability evaluation. We also provide limited experiments for larger 310M parameter models in Appendix F.5, mostly retaining the same conclusions.

W4, Benign capability measurement. With the new ClimbMix results, we have extended the range of capability metrics to provide further evidence (included in Figure 5 and Appendix F.4 and Appendix F.1): The number of finetuning steps needed to reach the unfiltered baseline model benign loss. This is less sensitive to easily reversible increases in loss. The minimal loss after finetuning the model on a finite amount of data, on a control topic left out from pretraining. Measures the transfer learning ability of the model. We also provide the results on the full benign data. On ClimbMix, the unlearning outperforms increased filtering on this metric as well, potentially due to higher diversity compared to Wikipedia

We hope these address the concerns of the reviewer.

I appreciate the authors’ substantial revisions. The timing ablation, additional learning-rate controls, ClimbMix and WMDP-Bio-forget evaluations, larger-scale experiments, and expanded benign-capability metrics all strengthen the empirical analysis. In particular, the timing study provides useful evidence about the persistence of pretrain-time interventions, and the new capability metrics address an important part of my original concern.

However, my central concern about the connection between the experiments and the paper’s security motivation remains. The evaluation primarily treats increased language-modeling loss on Biology-related data as evidence of improved safety. This measures the removal or suppression of a particular body of knowledge, but safety is not equivalent to the absence of knowledge. A model may have high loss on biological text because its general domain competence has been degraded, without being specifically resistant to harmful fine-tuning. Conversely, possessing relevant knowledge does not by itself imply that a model will engage in harmful behavior. For the misuse scenarios motivating this work, the relevant question is whether an attacker can fine-tune the model to perform harmful tasks—not merely whether the model has forgotten information from a particular pretraining distribution.

This distinction is especially important for open-weight models, since an attacker can supply their own training data. The current experiments do not directly evaluate whether realistic harmful SFT can restore equal-level dangerous capabilities or enable behaviors that would otherwise require jailbreaking or prompt injection. Figure 6 further suggests that, as the attacker obtains more unique data, the advantage of pretrain-time WGA over filtering and undertraining diminishes. This indicates that the method may provide a finite increase in the data or optimization cost of recovering a capability, rather than a durable prevention of that capability.

The revisions are meaningful improvements to the empirical study, while my concern about the claim–evidence alignment for the paper’s central security motivation remains.

We  thank the reviewer for the additional engagement in the paper discussion. We are also happy to hear that our revisions addressed an important part of the original concerns. We take the opportunity to address the remaining concern here:

“Safety is not equivalent to the absence of knowledge”

We agree that safety it is not equivalent to the absence of knowledge, but argue that in an open-weights situation the absence of knowledge definitely can be an important aspect of safety. Open-weights models can be easily finetuned to not refuse harmful instructions. Thus, the state-of-the-art method in preventing misuse is making sure that the model doesn’t have the necessary knowledge in the first place [1]. This is much more resistant to finetuning than, e.g., refusal training. Knowledge/capability limitation for LLMs a widely accepted problem in open-weights security [2].

“Conversely, possessing relevant knowledge does not by itself imply that a model will engage in harmful behavior”

We agree that knowledge by itself does not cause harmful behaviour, but this is not relevant for the threat model addressed by the paper. To reiterate, an actor that wants to misuse a model can easily finetune a highly capable model to provide harmful information, such as instructions on creating biological weapons.

“A model may have high loss on biological text because its general domain competence has been degraded, without being specifically resistant to harmful fine-tuning.”

We agree with this, and this is precisely why we measure finetuning as the main safety metric in the paper.

“For the misuse scenarios motivating this work, the relevant question is whether an attacker can fine-tune the model to perform harmful tasks—not merely whether the model has forgotten information from a particular pretraining distribution.”

Given that the harmful capabilities we are concerned only exist with large scale LLMs (e.g., bioweapons capabilities) and our intervention requires pretraining models from scratch, using such LLMs would be prohibitively expensive. Thus, we work with proxies that are informative at smaller scales. This is common practice in the literature on pretraining-scale interventions to open-weights safety [3,4,5]. We also included an analysis in Appendix E showing that the loss on the WMDP-Bio-forget set (one of the new evaluation datasets) does clearly correlate with biological weapons-related capabilities in large-scale LLMs. Further, the connection between pretraining dataset selection and harmful task performance was causally validated in [1].

We hope this addresses the remaining concerns, and are open to further discussion within the time limits of the discussion period.

[1] O’Brien, Kyle, et al. “Deep ignorance: Filtering pretraining data builds tamper-resistant safeguards into open-weight llms.”

[2] Casper, Stephen, et al. “Open technical problems in open-weight AI model risk management.”

[3] Lee, Bruce W., et al. “Distillation robustifies unlearning.”

[4] Shilov, Igor, et al. “Beyond data filtering: Knowledge localization for capability removal in llms.”

[5] Rathi, Neil, and Alec Radford. “Shaping capabilities with token-level data filtering.”

Reviewer_kZC4OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes to use unlearning during pre-training to make the model harder to be harmful fine-tuned.

Critical strengths
  1. Paper is well-written.

  2. Studying unlearning during pre-training is timely and fundamental.

  3. The analysis is great with convincing empirical measurement results.

Critical weaknesses
  1. Evaluation can be done comprehensively.
  • Several tamper-resistant baselines are not studied and compared. I suggest that the authors add a comparison of all the following three post-training phase tamper resistent defenses during rebuttal: TAR, Booster, CTRAP. I am particularly interested how pre-training unlearning compares with these three post-training methods in terms of i) ability to counter harmful fine-tuning from harmful post-training data. ii) ability to learn benign knowledge from benign post-training data. I choose these three methods because they represent different angles to solve the issue: i) TAR is an unlearning loss for checkpoints after harmful simulation, ii) Booster is a smoothness loss for the harmful loss landscape, iii) Ctrap is an unlearn-everything loss when harmful fine-tuning is detected.

Tamper-resistant safeguards for open-weight llms

Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation

Ctrap: Embedding collapse trap to safeguard large language models from harmful fine-tuning

  • Instead of relying on harmful loss, could you add a LLM-as-a-judge metric like harmful score to measure the tradeoff between resistant and pre-trained model's benign accuracy?

  • For the benign tasks, I think you only consider the after pre-trained and before post-training model benign performance, but why not also to consider the benign adaptivity ability of the pre-trained LLM (i.e., how well it is able to learn benign tasks in post-training)? Would that be hurt during unlearning? You can simulate a benign post-training and benchmark the model's after-post-training benign performance.

  1. Something I could not understand.
  • Why the initial benign loss number in the x axis in Figure 3 (b) and Figure 4 (b) are inverse? I can understand Figure 3 (b), basically if you decrease lambda, the initial benign loss will be lower (because higher weights on this loss), but this also makes the model easier to attack (i.e., less resistence). However, why Figure 4 exhibit a completely different trend. Could you fix this figure to make readers does not need high cognitive effort to understand it?
  1. Related work is not comprehensive. While there are relatively few studies on how to increase tamper resistant during pre-training, there are a plethora of work has studied how to achieve the same goal with post-training, which are missing from discussion. While it is impossible to compare against all these work, it is important to discuss and acknowledge them, as this will help the better development of the relatively small community that is interested in this problem. I list out a few for the authors to reference.

Vaccine: Perturbation-aware alignment for large language model aginst harmful fine-tuning

Data to Defense: The Role of Curation in Aligning Large Language Models Against Safety Compromise

Leveraging Catastrophic Forgetting to Develop Safe Diffusion Models against Malicious Finetuning

Understanding and Enhancing Safety Mechanisms of LLMs via Safety-Specific Neuron

Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation

Preserving Safety in Fine-Tuned Large Language Models: A Systematic Evaluation and Mitigation Strategy

On Weaponization-Resistant Large Language Models with Prospect Theoretic Alignment

Fight Fire with Fire: Defending Against Malicious RL Fine-Tuning via Reward Neutralization

Self-Destructive Language Model

Model Immunization from a Condition Number Perspective

Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning

Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning

LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuning

SDD: Self-Degraded Defense against Malicious Fine-tuning

Towards Resilient Safety-driven Unlearning for Diffusion Models against Downstream Fine-tuning

TOKEN BUNCHER: Shielding LLMs from Harmful Reinforcement Learning Fine-Tuning

AntiDote: Bi-level Adversarial Training for Tamper-Resistant LLMs

Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influence

Pharmacist: Safety Alignment Data Curation for Large Language Models against Harmful Fine-tuning

Immunizing 3D Gaussian Generative Models Against Unauthorized Fine-Tuning via Attribute-Space Traps

Locking Pretrained Weights via Deep Low-Rank Residual Distillation

Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization

SGT: Securing Open-Source LLMs Against Malicious Fine-tuning via Safety Guidance Trigger

OASIS: Mitigating Harmful Fine-tuning Attacks on LLMs via Orthogonal and Adaptive Safety Alignment Strategy

Distribution-Specific Curvature Control with Finite-Sample Guarantees for Open-Weight Safety

Gradient Immunity: Null-Space Resistance to Malicious Fine-Tuning

Questions for authors

I would like to lead a philosophical discussion as follows. Feel free to challenge my viewpoint as this is just for discussion purpose and will not affect my evaluation of the paper.

Recently, there is indeed a growing interest in strenghtening safety alignment during pre-training. Some people in the industry is attempting to try these ideas in a large scale experiment: i) filter harmful data during pretraining. ii) add large scale synthetic safety reasoning data during pre-training, and of course the third idea would be something discussed in this paper iii) unlearn harmful data during pre-training. However, I personally am not convinced that these pre-training time method can completely solve the harmful fine-tuning. Let say we are doing perfectly by the above three methods, and we have a super benign yet highly generalizable LLM (i.e., with good capability align with human-level but does not exhibit any explicit harmfulness). Do you think this model will not collapse its safety alignment learned during pre-training, or in other word, generalize to learn how to do harmful things, after further harmful SFT or RL on it? Would love to hear your insights on this.

4 comments

We thank the reviewer for the constructive and positive feedback. We go through the concerns and questions in order below:

Evaluation against existing tamper resistance baselines. We have now included comparisons against the suggested baselines in Appendix F.6. To make sure that we didn’t disadvantage the baselines, the attacks were tuned on a pretrain-WGA model, and we allowed 5% more compute to the baselines than pretrain-WGA. Overall, the takeaway is that these methods can have similar effects as WGA applied during pretraining, and the effect can potentially be even stronger. However, we could potentially apply both of them together, or apply these tamper resistance during pretraining as well. As such, we consider these results informative of the relative effects of different interventions rather than pitting the methods against each other.

Better capability and safety metrics. We agree that the paper could benefit from more broad safety and capability metrics. The challenge is that the expense of pretraining experiments limits us to relatively small models and proxy metrics of harmfulness, like loss on a representative dataset (e.g., WMDP-Bio [1] likely requires over 1B Chinchilla-optimal LLMs [2]). To address these concerns, we have now: Included benign data finetunability as an additional capability metric, explored in Appendix F.4. In particular, we filtered out the “law” topic from pretraining, and used finite-data finetunes as a capability metric when comparing unlearning and filtering. The results mostly track using benign loss as the capability metric. We also include “number of steps on the benign data distribution to reach unfiltered model benign loss” as a capability metric, and promote this to the main paper results in Figure 5. The results are mostly the same, and the benign loss results are included in Appendix F.4. Included finetunability on the WMDP-Bio-forget dataset [1] as an additional evaluation, presented both in the updated Figure 5 and Appendix F.4. We show in Appendix E that the loss on WMDP-Bio-forget tracks the WMDP-Bio dangerous capability eval in real open-weights LLMs.

To improve the realism of our experiments, we have also updated many of the experiments with the NVIDIA ClimbMix pretraining dataset (including Figure 5 and Appendix F.4). Regarding the idea of LLM-as-a-judge, the challenge is that it may be difficult to get consistent high quality samples to questions from small-scale LLMs that would make judging of sample output meaningful. We are open to further ideas on this, however.

Figure 4 clarification. We agree that the figure should be clarified. The reason why benign loss decreases as tamper resistance grows is that the plot shows the trends across model sizes. Larger models have generally lower losses. For a given size, higher loss still correlates with higher tamper resistance. Further, to be able to plot multiple scales in the same plot, the reference loss in the “tokens to target loss” metric is defined to be dependent on the benign loss. This makes two models with different sizes but the same benign loss comparable in the figure. We have now clarified this in the figure caption, and provided additional discussion and visualizations in Appendix C.2.

Additional results on comparing pretrain-unlearning to undertraining. We also provide additional results with a simple baseline where, instead of filtering more or applying WGA, we pretrain the model for a shorter time. The new Figure 6 in the main paper and Figure 17 in the Appendix show the results. Overall, the more training data we have available for the finite-data attack or the more compute we allow for the infinite-data attack, the smaller the relative advantage over undertraining becomes. This suggests that pretrain unlearning provides additional resistance for some absolute amount of data, e.g., 10M-20M unique tokens, and the advantage fades when data is not a limitation.

Related work comprehensiveness. We thank the reviewer for the very helpful list of tamper resistance literature, and have now updated the related works section accordingly with each method and brief explanations.

How promising are pretraining interventions? This is a very interesting and relevant question. It may indeed be the case that larger and more generalisable models may exhibit better transfer learning capabilities than generally less capable models, and this may make pretraining interventions less successful overall. However, we would argue that current LLMs still base most of their skills in knowledge learned in pretraining. Small amounts of SFT or even large amounts of RL might not be sufficient to teach the model the necessary base capabilities to perform well in a given knowledge-heavy domain, especially if we filter aggressively around the dangerous capability (e.g., remove much of Biology when targeting biological weapons capabilities). Pretraining interventions might not, however, stop in-context learning combined with agentic search (although perhaps learning incorrect facts during pretraining could help here). This may require different approaches.

[1] Li, N., Pan, A., Gopal, A., Yue, S., Berrios, D., Gatti, A., ... & Hendrycks, D. (2024). The wmdp benchmark: Measuring and reducing malicious use with unlearning.

[2] O'Brien, K., Casper, S., Anthony, Q., Korbak, T., Kirk, R., Davies, X., ... & Biderman, S. R. (2026, April). Deep ignorance: Filtering pretraining data builds tamper-resistant safeguards into open-weight llms.

Reviewer_kZC4OpenReview ↗

Thanks for the rebuttal

Sorry for the late participation of rebuttal.

I do not quite understand here "Regarding the idea of LLM-as-a-judge, the challenge is that it may be difficult to get consistent high quality samples to questions from small-scale LLMs that would make judging of sample output meaningful."

What I imagine is something like this:

You have a pre-trained model unlearned with WGA, then you do some posthoc safety alignment to make it able to refuse harmful question, and then you do the adversarial fine-tuning, and use another llm-as-a-judge to see how the attacked model perform when it it asked harmful tasks. Why the scale of LLM would be a problem here? In my understanding, even small model should be able to answer simple question like "how to make a bomb?". What I imagine the result will be is that because the harmful activation of how to make a bomb has been suppress during pre-training, it is harder to erase the refusal activation planted during safety alignment. I am quite interested in the result here, but you may not have time for further experiments due to my late response.

Let me just keep my score now and see how the other reviewers view the paper. I incline to accept this paper because pre-training filtering/unlearning seems to be an important direction now, and there are few papers on this, although I personally do not champion that this can fundamentally address the issue.

Hi, thank you for the clarification! We expect that getting small-scale models to reliably and coherently answer questions is more difficult in general than for larger models. Even for simple questions like "how to make a bomb", but especially for more complex capabilities. As such, it might be difficult to compare the quality of responses quantitatively between filtered, pretrain-unlearned models or unfiltered models. Basically: our expectation is that for most harmful tasks we are interested in intervening in (e.g. biological weapons related knowledge), all of the answers will be low quality enough that LLM-as-a-judge wouldn't be informative. As such, we focus on loss-based proxies that measure the knowledge of the model, since these are possible to measure at small scales already. But we indeed did not run sampling-based experiments.

Thank you again for the review and the participation in the rebuttal!

Reviewer_ABaMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether applying unlearning during pretraining can improve language models' resistance to adversarial fine-tuning compared with post-hoc unlearning and data filtering. The authors propose interleaving weighted gradient ascent with standard pretraining to suppress unwanted capabilities while preserving general performance. Experiments on Wikipedia Biology data across multiple model scales show improved fine-tuning resistance and better trade-offs between resistance and general capabilities under the selected evaluation metrics.

Critical strengths
  1. The paper addresses an important and practically relevant problem of improving resistance to adversarial fine-tuning in open-weight language models.

  2. The paper is generally well organized and clearly written. The proposed training strategy is straightforward and easy to understand, and the token-matched comparisons provide useful empirical evidence.

Critical weaknesses
  1. There is a gap between the motivation, methodology, and experiments regarding harmful capabilities. The paper aims to prevent harmful capabilities in open-weight LLMs, but the experiments define harmful data as Wikipedia articles classified under Biology. However, Biology is a broad domain of scientific knowledge rather than inherently harmful content. Increasing the loss on Biology text does not necessarily indicate that harmful capabilities have been removed or suppressed. Therefore, the experiments do not directly support the paper's central safety motivation. Evaluations on datasets that more closely reflect realistic harmful capabilities are needed.

  2. The experimental scope is limited. The largest model has only 163M parameters, which is substantially smaller than modern deployed LLMs. Moreover, the experiments consider only Biology as the target domain. It remains unclear whether the reported improvements generalize to larger models or other domains.

  3. The evaluation of general capabilities relies heavily on language modeling loss and classifier-based data selection. In particular, Section 4.4 shows that the advantage over more aggressive filtering depends on the choice of capability metric, while the method does not outperform filtering on the full benign dataset loss. More reliable and diverse evaluation metrics are needed, including downstream task performance and human evaluation, to better assess whether useful capabilities are preserved.

  4. The experiments do not sufficiently isolate why Pretrain-time Unlearning outperforms Post-hoc Unlearning. Although Figures 3 and 4 show improvements in fine-tuning resistance, the two approaches also differ in their learning rate schedules, optimizer states, and training update sequences. The token-matched comparison does not fully control these differences. Additional ablation studies are needed to determine whether the improvements mainly come from applying unlearning earlier or from other differences in the optimization process.

  5. The methodological novelty appears somewhat limited, as the main contribution is an empirical comparison of when unlearning is applied rather than a new optimization objective. The authors should better distinguish their approach from existing Non-transferable Learning and Gradient Routing methods, and explain why the selected baselines sufficiently represent related pretraining-time interventions.

Questions for authors
  1. The paper aims to improve resistance to harmful capabilities, but the experiments use Wikipedia Biology articles as the harmful dataset. Could the authors clarify how increased language modeling loss on Biology text reflects the suppression of actual harmful capabilities? Is there evidence that the proposed method is effective on more realistic harmful tasks?

  2. The experiments are limited to models up to 163M parameters and a single target domain. What evidence supports the generalization of the findings to larger LLMs and other domains?

  3. Section 4.4 shows that the advantage over aggressive filtering depends on the choice of capability metric. How sensitive are the conclusions to the classifier thresholds and the exclusion of Biology-related documents? Could the authors provide evidence from downstream benchmarks or human evaluation to better establish that general capabilities are preserved?

  4. Pretrain-time and Post-hoc Unlearning differ not only in timing but also in learning rate schedules, optimizer states, and training procedures. Could the authors provide additional ablations or evidence to clarify which factors contribute most to the observed improvement?

  5. What is the main methodological contribution beyond existing Non-transferable Learning and Gradient Routing approaches? Could the authors clarify why these related pretraining-time interventions are not included as baselines, and how TrigCon's proposed training strategy differs from them?

3 comments

We thank the reviewer for the constructive suggestions. We are happy to hear that the reviewer thinks that the paper addresses an important and relevant problem, the paper is clearly written, and the proposed strategy is easy to understand. We go through the questions and weaknesses in order:

W1/Q1: Harmful capability definition. Given that the harmful capabilities we are concerned only exist with large scale LLMs (e.g., bioweapons capabilities) and our intervention requires pretraining models from scratch, using such LLMs would be prohibitively expensive. Thus, we work with proxies that are informative at smaller scales. Loss on representative datasets is known to track closely with capabilities, and this is a common choice in capability-shaping pretraining research [1,2].

That said, to increase the realism of our proxies, we ran additional experiments the alleviate this concern: We redo the experiments on a more realistic pretraining dataset, NVIDIA ClimbMix [3]. We provide tamper resistance evaluations on the WMDP-Bio-forget dataset [4].

The updated results comparing filtering with unlearning are in the updated Figure 5. The behaviour of LLMs on this data is known to correlate with the dangerous capability evaluation WMDP-Bio in unlearning experiments [4]. We provide further evidence in the updated Appendix F that the loss on this dataset across LLMs correlates with WMDP-Bio capabilities in 1B-7B scale LLMs.

W2/Q2. Limitations with domains and scale. While we reiterate the need for smaller-scale proxy evaluations in this research, we now provide results on 17 layer, 310M-parameter Transformers on ClimbMix in Appendix F.5, where the main conclusions mostly remain the same. The scaling trends in Figure 4 also provide evidence our results hold consistently across scales. The scaling trends in the main paper (Figure 4) were updated to ClimbMix and give the same qualitative findings as Wikipedia (now Appendix G.2), providing evidence that the results generalize across datasets. Regarding the limitation to the Biology data, we hope the new WMDP-Bio-forget evaluation partially answers this question. Aside from that, we argue that our method does not include any inductive bias w.r.t. any particular domain, and Biology in particular is relevant due it encompassing biological weapons relevant information.

W3/Q3. More evaluation metrics. As said, we included the new evaluations on the WMDP-Bio-forget set and the ClimbMix pretraining data. In addition, in Appendix F.4 we provide the results for two new capability metrics

  1. Number of training steps until reaching the non-filtered baseline benign loss (the full benign training set). We argue that this metric is better than just the benign loss since unlearned models could have an artificially increased loss on some samples and as such be faster to train than the raw benign loss would imply. That said, we also include the results for the full benign loss in Appendix F.4 for ClimbMix, and the results are similar.
  2. Transfer learning capability. We exclude the “law” topic in ClimbMix from the pretrains, and measure the minimum loss reachable after finetuning for multiple epochs on a finite amount of law data. We provide this also in Appendix F.4, and the trend is overall similar.

W4/Q4. More thorough experiments with pretrain vs. post-hoc unlearning. We provide additional ablations on pretrain unlearning vs. post-hoc unlearning on ClimbMix in Appendix F.1, sweeping over multiple learning rates and different capability and safety metrics. Figure 3 in the main paper uses the Pareto frontier across learning rates and unlearning strengths for post-hoc unlearning. Pretrain unlearning consistently outperforms post-hoc unlearning, except with the “updates to recover baseline loss” capability metric when applying unlearning on the unfiltered model, where the performance is similar. That said, in the more useful case where unlearning is applied jointly with filtering, pretrain unlearning clearly outperforms post-hoc unlearning. In Appendix F.7., we also do a study where we simply time the pretrain unlearning progressively towards the end, and get similar results.

W5/Q5. Methodological novelty. We raise two points regarding the novelty: The main pretraining interventions targeted at LLM open-weight safety are data filtering [5] and gradient routing [1]. Our method is clearly different from both approaches, and gradient routing in particular is motivated by 1) the modularity of the resulting model and 2) improved quality compared to noisy filters. As [1] showed, this does not necessarily lead to improved tamper resistance, and simple data filtering with a high-quality classifier remains the main tamper resistance pretraining intervention. The non-transferable learning literature has considered mechanistically similar training terms during initial model training, but mostly not applied to tamper resistance or to language models. In this domain, data filtering is the main existing baseline and considered the state-of-the-art approach with the strongest empirical validation [5]. In fact, one way to view our paper is that we apply non-transferable learning like approaches to tamper resistance in language models.

Additional results on comparing pretrain-unlearning to undertraining. We also provide additional results with a simple baseline where, instead of filtering more or applying WGA, we pretrain the model for a shorter time. The new Figure 6 in the main paper and Figure 17 in the Appendix show the results. Overall, the more training data we have available for the finite-data attack or the more compute we allow for the infinite-data attack, the smaller the relative advantage over undertraining becomes. This suggests that pretrain unlearning provides additional resistance for some absolute amount of data, e.g., 10M-20M unique tokens, and the advantage fades when data is not a limitation.

We hope this answers the concerns of the reviewer.

[1] Shilov, Igor, et al. "Beyond data filtering: Knowledge localization for capability removal in llms."

[2] Rathi, Neil, and Alec Radford. "Shaping capabilities with token-level data filtering."

[3] Diao, Shizhe, et al. "Nemotron-climb: Clustering-based iterative data mixture bootstrapping for language model pre-training."

[4] Li, Nathaniel, et al. "The wmdp benchmark: Measuring and reducing malicious use with unlearning."

[5] O'Brien, Kyle, et al. "Deep ignorance: Filtering pretraining data builds tamper-resistant safeguards into open-weight llms."

As the window for the author-reviewer discussion is closing, we would like to recap the rebuttal. We believe that we have addressed the specific concerns raised by the reviewer through 1) new (and more realistic) safety and capability metrics 2) larger training runs, 2) a new dataset and 3) more thorough comparisons to post-hoc unlearning with two new experiments. We also provided counter-arguments to the methodological novelty issue and clarified the differences to earlier literature. We hope that the reviewer takes these into account in considering their overall score, and we are happy to answer any additional questions within the time-frame of the discussion period.

Additionally, we would like to raise a point that we didn't directly address in the original rebuttal: The issue with human evaluations is that the 100M-300M scale models are generally not capable enough for practical tasks where sample quality is not high enough in general for interesting tasks, and large scale LLMs would very expensive to train in the kinds of sweeps that we need for our paper. This forces us to use proxy metrics such as loss, which is standard in pretraining evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.