- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether applying unlearning during pretraining can improve resistance to adversarial fine-tuning compared to post-hoc unlearning and data filtering. Using Biology-related Wikipedia text as a proxy for harmful knowledge, the authors evaluate gradient-ascent-based unlearning methods across language models of different sizes. The results suggest that pretrain-time unlearning provides stronger finetuning resistance than post-unlearning, and can achieve better safety–capability trade-offs than aggressive data filtering under certain evaluation settings.
Critical strengths
- The paper addresses an important problem in open-weight model safety: preventing unwanted capabilities from being recovered through adversarial fine-tuning.
- The experiments compare multiple training strategies, account for training budgets and capability degradation, and examine the effects across different model scales and unlearning strengths.
- This paper demonstrates interesting empirical findings. The observation that the timing of unlearning substantially affects resistance to subsequent fine-tuning is interesting. The results also suggest that unlearning can complement data filtering.
Critical weaknesses
- The novelty of method is limited. The proposed approach primarily integrates existing unlearning objectives into the pretraining process. While this is a reasonable design choice, the methodological contribution is relatively incremental.
- The explanation of the underlying mechanism is insufficient. A central finding is that pretrain-time unlearning consistently outperforms post-hoc unlearning. However, the paper provides limited analysis of why this happens. The explanation that earlier unlearning prevents unwanted capabilities from forming remains largely intuitive, without sufficient experimental evidence to establish the underlying mechanism.
- The experiments use relatively small models (up to 163M parameters) and Biology-related language modeling as a proxy for harmful capabilities. It remains unclear whether the conclusions generalize to realistic adversarial fine-tuning attacks on larger open-weight LLMs.
- The claimed advantage over aggressive data filtering depends on how benign capabilities are measured. In particular, the advantage does not hold when considering the full benign dataset loss, which raises concerns about the generality of the reported safety–capability trade-off.
Questions for authors
- I'm wondering why is the pretrain-time unlearning more effective? Could the authors investigate the underlying mechanism through a timing ablation, such as applying the same unlearning objective at different stages of pretraining? This could help distinguish the effect of preventing capability formation from other optimization effects.
- Can the comparison between pretrain-time and post-hoc unlearning be further controlled? Although the authors sweep unlearning strengths and match token budgets, the two approaches use different learning rate schedules and optimizer states. Could these differences partially explain the observed performance gap?
- How well do the findings generalize to realistic harmful capabilities? Evaluating larger models or more realistic harmful finetuning scenarios would strengthen the connection between the empirical results and the paper's security motivation.
5 comments
I appreciate the authors’ substantial revisions. The timing ablation, additional learning-rate controls, ClimbMix and WMDP-Bio-forget evaluations, larger-scale experiments, and expanded benign-capability metrics all strengthen the empirical analysis. In particular, the timing study provides useful evidence about the persistence of pretrain-time interventions, and the new capability metrics address an important part of my original concern.
However, my central concern about the connection between the experiments and the paper’s security motivation remains. The evaluation primarily treats increased language-modeling loss on Biology-related data as evidence of improved safety. This measures the removal or suppression of a particular body of knowledge, but safety is not equivalent to the absence of knowledge. A model may have high loss on biological text because its general domain competence has been degraded, without being specifically resistant to harmful fine-tuning. Conversely, possessing relevant knowledge does not by itself imply that a model will engage in harmful behavior. For the misuse scenarios motivating this work, the relevant question is whether an attacker can fine-tune the model to perform harmful tasks—not merely whether the model has forgotten information from a particular pretraining distribution.
This distinction is especially important for open-weight models, since an attacker can supply their own training data. The current experiments do not directly evaluate whether realistic harmful SFT can restore equal-level dangerous capabilities or enable behaviors that would otherwise require jailbreaking or prompt injection. Figure 6 further suggests that, as the attacker obtains more unique data, the advantage of pretrain-time WGA over filtering and undertraining diminishes. This indicates that the method may provide a finite increase in the data or optimization cost of recovering a capability, rather than a durable prevention of that capability.
The revisions are meaningful improvements to the empirical study, while my concern about the claim–evidence alignment for the paper’s central security motivation remains.
We thank the reviewer for the additional engagement in the paper discussion. We are also happy to hear that our revisions addressed an important part of the original concerns. We take the opportunity to address the remaining concern here:
“Safety is not equivalent to the absence of knowledge”
We agree that safety it is not equivalent to the absence of knowledge, but argue that in an open-weights situation the absence of knowledge definitely can be an important aspect of safety. Open-weights models can be easily finetuned to not refuse harmful instructions. Thus, the state-of-the-art method in preventing misuse is making sure that the model doesn’t have the necessary knowledge in the first place [1]. This is much more resistant to finetuning than, e.g., refusal training. Knowledge/capability limitation for LLMs a widely accepted problem in open-weights security [2].
“Conversely, possessing relevant knowledge does not by itself imply that a model will engage in harmful behavior”
We agree that knowledge by itself does not cause harmful behaviour, but this is not relevant for the threat model addressed by the paper. To reiterate, an actor that wants to misuse a model can easily finetune a highly capable model to provide harmful information, such as instructions on creating biological weapons.
“A model may have high loss on biological text because its general domain competence has been degraded, without being specifically resistant to harmful fine-tuning.”
We agree with this, and this is precisely why we measure finetuning as the main safety metric in the paper.
“For the misuse scenarios motivating this work, the relevant question is whether an attacker can fine-tune the model to perform harmful tasks—not merely whether the model has forgotten information from a particular pretraining distribution.”
Given that the harmful capabilities we are concerned only exist with large scale LLMs (e.g., bioweapons capabilities) and our intervention requires pretraining models from scratch, using such LLMs would be prohibitively expensive. Thus, we work with proxies that are informative at smaller scales. This is common practice in the literature on pretraining-scale interventions to open-weights safety [3,4,5]. We also included an analysis in Appendix E showing that the loss on the WMDP-Bio-forget set (one of the new evaluation datasets) does clearly correlate with biological weapons-related capabilities in large-scale LLMs. Further, the connection between pretraining dataset selection and harmful task performance was causally validated in [1].
We hope this addresses the remaining concerns, and are open to further discussion within the time limits of the discussion period.
[1] O’Brien, Kyle, et al. “Deep ignorance: Filtering pretraining data builds tamper-resistant safeguards into open-weight llms.”
[2] Casper, Stephen, et al. “Open technical problems in open-weight AI model risk management.”
[3] Lee, Bruce W., et al. “Distillation robustifies unlearning.”
[4] Shilov, Igor, et al. “Beyond data filtering: Knowledge localization for capability removal in llms.”
[5] Rathi, Neil, and Alec Radford. “Shaping capabilities with token-level data filtering.”
We thank the reviewer for the constructive feedback. We are happy to hear that the reviewer found that 1) the paper addresses an important problem, 2) the experiments are quite thorough, and 3) the paper demonstrates interesting empirical findings. We go through the concerns in order below:
W1, Method novelty: We respectfully disagree with the premise that the simplicity of the core algorithmic idea significantly detracts from the novelty of the paper. All previous unlearning interventions have been applied for already pretrained models, and moving to interventions during pretraining is a more significant algorithmic change than, e.g., tweaks on existing unlearning losses. We would also argue that the simplicity of the method is a strength, not a weakness, as more complex methods are often brittle and sensitive to hyperparameter adjustments. Current pretraining-time interventions for tamper resistance are quite scarce (mostly data filtering and gradient routing), and our paper provides evidence that pretrain-time unlearning expands the toolkit with a new method.
W2, Q1, timing ablation: We agree overall that a better understanding of when pretrain unlearning works is helpful, and to that end we have now added the timing ablation experiment suggested by the reviewer in Appendix F.7. It seems that pretrain unlearning earlier on does result in better safety-capability tradeoffs across the board. An interesting feature is that applying the unlearning at the very end results in less persistent damage both to Biology but also to the benign data loss, suggesting that pretrain unlearning causes more durable changes to the model overall.
Another interesting study is comparing pretrain unlearning to simply training the models for a shorter time, to separate the effect of the model being generally less capable from Biology-specific finetuning resistance. The new Figure 6 in the main paper shows the results for the finite-data finetune metric, showing that as the amount of available data grows, filtering more, undertraining, and applying WGA all start to have similar effects. This suggests that the effect of pretrain unlearning is an added resistance to some amount of unique tokens (e.g. 10M).
Q2, further control for pretrain unlearning vs. post-hoc unlearning comparisons. We provide additional ablations on pretrain unlearning vs. post-hoc unlearning in Appendix F.1, sweeping over multiple learning rates and different capability and safety metrics. Figure 3 in the main paper uses the Pareto frontier across learning rates and unlearning strengths for post-hoc unlearning. Pretrain unlearning consistently outperforms post-hoc unlearning, except with the “updates to recover baseline loss” capability metric when applying unlearning on the unfiltered model, where the performance is similar. That said, in the more useful case where unlearning is applied jointly with filtering, pretrain unlearning clearly outperforms post-hoc unlearning.
W2, W3 & Q3, Experiment realism. We have updated the paper to improve the experiment realism through the following: We have moved the results to the more realistic pretraining dataset NVIDIA ClimbMix, as opposed the more narrow Wikipedia dataset we used previously. We provide finetuning resistance metrics on the WMDP-Bio-forget dataset [1], in Figure 5 as well as with the extended post-hoc unlearning comparisons in Appendix F.1. We show in Appendix E that loss on this dataset in open-weights LLMs correlates strongly with the WMDP-Bio dangerous capability evaluation. We also provide limited experiments for larger 310M parameter models in Appendix F.5, mostly retaining the same conclusions.
W4, Benign capability measurement. With the new ClimbMix results, we have extended the range of capability metrics to provide further evidence (included in Figure 5 and Appendix F.4 and Appendix F.1): The number of finetuning steps needed to reach the unfiltered baseline model benign loss. This is less sensitive to easily reversible increases in loss. The minimal loss after finetuning the model on a finite amount of data, on a control topic left out from pretraining. Measures the transfer learning ability of the model. We also provide the results on the full benign data. On ClimbMix, the unlearning outperforms increased filtering on this metric as well, potentially due to higher diversity compared to Wikipedia
We hope these address the concerns of the reviewer.