- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates the 'native safety reuse problem': whether the safety behaviors learned in pretraining (termed native safety) can be transferred from a smaller teacher model to a larger student model. It shows that on-policy distillation allows native safety to transfer to larger student models that are of the same or different model class. Specifically, the authors use the safety pretraining dataset as OPD datasets, employ GOLD to address the cross-tokenizer distillation problem, and augment the distillation objective with a utility-preservation loss to allow distillation from a small base model to a large instruction model. This method achieves safety robustness to fine-tuning attacks and retains higher utility than the smaller safety-pretrained teacher. The paper further demonstrates the method's application to VLM safety and machine unlearning.
Critical strengths
- This paper investigates a timely question. Safety pretraining is indeed expensive, and most existing papers can only afford to experiment on smaller models, such as 1.7B. Distilling the native safety from small safety-pretrained models to larger models would be a clever approach to bring the benefits to larger models while saving compute.
- The authors conduct many experiments, including ablations and applications, to test whether native safety learned in SafeLM can transfer to different models and scenarios.
Critical weaknesses
-
Limited safety evaluations and unclear protocols undermine the results.
- The evaluation of safety robustness is limited to fine-tuning attacks. Would on-policy distillation also carry the benefits of safety pretraining on other attacks (such as the GCG attack) to larger models?
- The fine-tuning attack evaluation protocol in this paper is not sufficiently documented, especially the number of fine-tuning examples and fine-tuning epochs. Additionally, the original SafeLM paper [1] proposed SafeBeam, without which SafeLM's robustness to benign fine-tuning would significantly decrease. In this paper, it's not clear whether the authors use SafeBeam in their evaluations.
-
Missing details on an important ablation baseline. As I understand it, SafePost-Distill would be an important baseline to compare: if similar results can be obtained by OPD from a safety post-trained model, then there is no need for safety pretraining, which further saves cost. However, I cannot find details about how you perform standard safety post-training, and I find the results in Appendix D.1 surprising, as SafePost cannot refuse such explicitly stated harmful requests. This makes me doubt whether the benefit of SafePre-OnDistill over SafePost-Distill is due to the insufficient optimization of SafePost.
-
The proposed method is tailored to SafeLM, and it is unclear whether it can be applied to other safety pretraining methods. The authors should discuss how their method could generalize to other safety pretraining approaches, including data filtering at different granularities [2, 3,4], upsampling positive AI discourse [5], incorporating safety-refusal data through co-training [6], and learning safety reflections during pretraining [7].
References:
[1] https://openreview.net/pdf?id=91H9CSvdwl
[2] https://openreview.net/pdf?id=n6mTO5JS4j
[3] https://openreview.net/forum?id=xcf0QcTcGS
[4] https://arxiv.org/pdf/2601.21571
Questions for authors
- What is the evaluation protocol for fine-tuning attacks used in this paper?
- How do different distilling methods work under inference-stage attacks (e.g., GCG attacks)?
- Do the authors use SafeBeam in any of their evaluations? How would it affect the results?
- How do the authors perform standard safety post-training?
- The authors could strengthen their SafePost-Distill baseline. For instance, if you use a sufficiently optimized instruction model, coupled with coherent OPD datasets (such as the safety dataset used in safety post-training), what would the results be like?
- More generally speaking, on-policy distillation is also a post-training alignment method. Why wouldn't it suffer from the shallow alignment phenomenon in [1]? Could the authors provide more analysis on this? Also, could the authors provide more analysis of success and failure cases?
- In this work, you only consider one safety pretraining student model, SafeLM, and you directly use SafeLM's safety training datasets in OPD. For other safety-pretrained teacher models, if you cannot access their pretraining dataset, what would you use as the OPD dataset?
- The comparison of utility in Figure 2 is unfair. SafeLM-1.7B is only pretrained on 600B tokens, while SmolLM2-1.7B is pretrained on 11T tokens. I think the difference in utility is better explained by the difference in the size of the pretraining corpus, but not that safety pretraining hurts utility. What is the comparison result of SafeLM-1.7B and a matched-compute safety post-training model?
References [1] https://openreview.net/forum?id=6Mxhg9PtDE
4 comments
Response to Reviewer nP7G (Part 2/4)
[W2]: Missing details on an important ablation baseline. As I understand it, SafePost-Distill would be an important baseline to compare: if similar results can be obtained by OPD from a safety post-trained model, then there is no need for safety pretraining, which further saves cost. However, I cannot find details about how you perform standard safety post-training, and I find the results in Appendix D.1 surprising, as SafePost cannot refuse such explicitly stated harmful requests. This makes me doubt whether the benefit of SafePre-OnDistill over SafePost-Distill is due to the insufficient optimization of SafePost.
[Q4]: How do the authors perform standard safety post-training?
[Q5]: The authors could strengthen their SafePost-Distill baseline. For instance, if you use a sufficiently optimized instruction model, coupled with coherent OPD datasets (such as the safety dataset used in safety post-training), what would the results be like?
Response: Thank you for raising this question. However, we believe there might be a misunderstanding regarding SafePost and the necessity of SafePost-Distill.
First, Tab. A1 in Appendix D.1 (Tab. A5 in the revised paper) illustrates specific failure cases of SafePost rather than its overall safety performance. In fact, before fine-tuning, SafePost successfully resists 310 of the 320 harmful requests in HarmBench, achieving a low ASR$_{\text{BF}}$ of 3.12% (Tab. 2 for “HarmBench”). For Tab. A1, we picked one of the remaining 10 undefended cases on purpose, because such cases show what the safety-pretrained teacher adds: on the same request, SafePre and the distilled student refuse, whereas safety post-training does not. After fine-tuning, SafePost responds more harmfully to 48 of the 320 requests.
Second, as demonstrated in Tab. 2, SafePost fails to preserve its safety performance under subsequent fine-tuning, although it can achieve graceful safety performance without continual fine-tuning. This lack of fine-tuning robustness undermines the motivation for using SafePost-Distill as a means of achieving persistent safety. In addition, we follow the SafePost training setup consistent with [R3], using 13,927 training steps to ensure empirical convergence, a learning rate of 5e-5, and a batch size of 64.
Third, regarding a stronger safety post-training baseline (Q5), we consider a variant of SafePost that incorporates the safety-pretraining corpus used in our distillation. We term this variant as “NoTeacher-SFT,” which keeps the safety post-training objective and adds a cross-entropy term on this safety-pretraining corpus, without any teacher. Results are reported in Table R2. As we can see, NoTeacher-SFT is safe before fine-tuning (average ASR 3.11%) but is even less robust after fine-tuning than SafePost (30.99% vs. 14.16%; Tab. 2), while SafePre-OnDistill reaches only 5.16%. One possible explanation is the difficulty of jointly optimizing instruction-based safety post-training and token-level CE learning on the safety-pretraining corpus. These objectives may induce competing learning signals, making it difficult to integrate safety into the model's learned representations in a way that persists under subsequent fine-tuning. Since this safety-post-trained model already loses its safety under fine-tuning before any distillation, it offers no robustness advantage for SafePost-Distill to inherit, which is why the safety-pretrained teacher is needed.
Table R2: Stronger safety post-training on SmolLM2-1.7B. NoTeacher-SFT adds a cross-entropy term on the safety-pretraining data to the safety post-training objective, without a teacher. SafePre-OffDistill and SafePre-OnDistill are taken from Tab. 2; metrics follow Tab. 2. The proposed method is in bold.
| Method | HB ↓ | TDC ↓ | JBB ↓ | AB ↓ | Avg ↓ | MMLU ↑ | CSQA ↑ | GSM8K ↑ | WG ↑ | Avg ↑ |
|---|---|---|---|---|---|---|---|---|---|---|
| NoTeacher-SFT | 4.06→25.00 | 4.00→40.00 | 4.00→33.00 | 0.38→25.96 | 3.11→30.99 | 48.82 | 43.73 | 30.78 | 65.98 | 47.33 |
| SafePre-OffDistill | 1.25→12.19 | 0.00→26.00 | 0.00→16.00 | 0.00→7.69 | 0.31→15.47 | 48.45 | 42.51 | 29.57 | 65.51 | 46.51 |
| SafePre-OnDistill | 0.62→6.88 | 0.00→8.00 | 0.00→5.00 | 0.00→0.77 | 0.16→5.16 | 48.43 | 47.58 | 30.71 | 66.06 | 48.20 |
Response to Reviewer nP7G (Part 3/4)
[W3]: The proposed method is tailored to SafeLM, and it is unclear whether it can be applied to other safety pretraining methods. The authors should discuss how their method could generalize to other safety pretraining approaches, including data filtering at different granularities [2, 3,4], upsampling positive AI discourse [5], incorporating safety-refusal data through co-training [6], and learning safety reflections during pretraining [7].
Response: Thank you for this constructive suggestion. Our method is not specific to SafeLM: We already apply our approach to a second natively trained teacher, Deep Ignorance (the reviewer's reference [R4]), whose pretraining data are filtered (Tab. 5). Zephyr-7B-beta distilled from it preserves the teacher's forgetting under fine-tuning (WMDP-Bio accuracy 31.32% → 31.97%), whereas post-hoc unlearning with RMU recovers the knowledge (28.07% → 36.25%). The students also vary in size (1.7B to 8B), family (SmolLM2, Llama-3, Qwen2.5-VL, Zephyr), and modality (Tabs. 2–5).
For the other approaches, distillation requires the teacher's checkpoint, and the reviewer's references [R5], [R6], [R7], and [R2] do not release one. [R8] does, but it does not target robustness to fine-tuning of open-weight models, the property we transfer, so we did not take it as our starting point. We have cited [R2] and [R4]–[R8] in the related work of the revision (Sec. 2).
[Q6]: More generally speaking, on-policy distillation is also a post-training alignment method. Why wouldn't it suffer from the shallow alignment phenomenon in [1]? Could the authors provide more analysis on this? Also, could the authors provide more analysis of success and failure cases?
Response: Thank you for raising this important point. We agree that on-policy distillation can also be considered as a post-training method. We clarify that the shallow alignment phenomenon discussed in [R9] concerns vulnerability to input-level jailbreaking attacks, rather than the persistence of safety under subsequent model fine-tuning. As explained in our responses to W1.1 and Q2, input-level and weight-level robustness do not necessarily correlate. Specifically, Qi et al. [R9] characterize shallow alignment as safety behavior concentrated in the first few output tokens, which can be exploited by inference-time attacks such as GCG. Following that, we examine this phenomenon using the GCG responses from Tab. R1 (W1.1). We measure the token-wise KL divergence between each model's output distribution and that of SmolLM2-1.7B-Instruct, a standard instruction-tuned model sharing the same architecture and tokenizer. All three models (SafePost, SafePre, and SafePre-OnDistill) exhibit their largest divergence at the first output token, where the refusal-versus-compliance decision typically occurs. In contrast, the mean KL divergence over tokens 1–49 is substantially lower (Figure A3 in the revised paper).
Although the three models are thus equally shallow at the token level, they differ in robustness to subsequent fine-tuning, the property the paper targets (Line 84): after fine-tuning, SafePre-OnDistill's average ASR is 5.16% vs. 14.16% for SafePost (Tab. 2, Lines 384 and 386). Token-level depth therefore does not by itself explain robustness to fine-tuning, which is what on-policy distillation inherits from the safety-pretrained teacher. We have added the per-token KL curves and this discussion in the revision (Appendix E).
Success and failure cases. Tab. A1 and Tab. A2 (Tabs. A5 and A6 in the revised paper) show success cases: the distilled student refuses during distillation (Tab. A1, before fine-tuning) and still refuses after fine-tuning (Tab. A2), where SafePost does not. After fine-tuning, SafePre-OnDistill still responds harmfully to some requests (average ASR 5.16%, Tab. 2), and about half of these also break SafePost. We have added representative failure cases in the revision (Tab. A7 in Appendix F.3).
[Q7]: In this work, you only consider one safety pretraining student model, SafeLM, and you directly use SafeLM's safety training datasets in OPD. For other safety-pretrained teacher models, if you cannot access their pretraining dataset, what would you use as the OPD dataset?
Response: Thank you for this forward-looking question. Without the teacher's pretraining data, such prompts could be obtained in two ways: generated by the safety-pretrained teacher itself in its own formats (e.g., Moral Education-style text), or drawn from public safety data topically relevant to its pretraining data (e.g., the safety post-training data of Sec. 5.1). How the transferred robustness depends on the distance between the distillation data and the teacher's pretraining data is an interesting question for future work, and the paper already notes this dependence on safety-pretraining prompts (Lines 985–986). We have expanded this discussion in the revision (Appendix I).
Response to Reviewer nP7G (Part 4/4)
[Q8]: The comparison of utility in Figure 2 is unfair. SafeLM-1.7B is only pretrained on 600B tokens, while SmolLM2-1.7B is pretrained on 11T tokens. I think the difference in utility is better explained by the difference in the size of the pretraining corpus, but not that safety pretraining hurts utility. What is the comparison result of SafeLM-1.7B and a matched-compute safety post-training model?
Response: Thank you for this insightful comment.
First, we agree that the utility gap in Fig. 2 might be confounded by the difference in pretraining budgets (600B vs. 11T tokens) and therefore cannot be attributed solely to safety pretraining. We have clarified this point in the description of Fig. 2 in the revision (Sec. 3).
Second, the limited utility of currently available safety-pretrained models nevertheless remains a practical challenge: safety pretraining is computationally expensive (Lines 203–204), making it impractical to repeat across different training configurations or scale to substantially larger token budgets (e.g., extending SafeLM-1.7B from 600B to 11T tokens). Yet, our method transfers safety robustness without inheriting the teacher's utility limitations. Although SafePre has low average utility (27.42%), SafePre-OnDistill keeps SafePost-level utility (48.20% vs. 48.00%) while its average ASR after fine-tuning is far lower (5.16% vs. 14.16%; Tab. 2). A safety-pretrained model with limited utility can thus still serve as an effective native-safety teacher, which also makes our approach more valuable.
Reference
[R1] Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., and Fredrikson, M. "Universal and Transferable Adversarial Attacks on Aligned Language Models." arXiv:2307.15043, 2023.
[R2] Li, J., Tang, K., Xu, Y., Ye, Z., and Lyu, K. "Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection." ICML CompLearn Workshop, 2026.
[R3] Maini, P., Goyal, S., Sam, D., Robey, A., Savani, Y., Jiang, Y., Zou, A., Fredrikson, M., Lipton, Z. C., and Kolter, J. Z. "Safety Pretraining: Toward the Next Generation of Safe AI." NeurIPS, 2025.
[R4] O'Brien, K., Casper, S., Anthony, Q., Korbak, T., Kirk, R., Davies, X., Mishra, I., Irving, G., Gal, Y., and Biderman, S. "Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs." ICLR, 2026.
[R5] Wang, R. Y., Finlayson, M., Soldaini, L., Swayamdipta, S., and Jia, R. "Teaching Models to Understand (but not Generate) High-risk Data." COLM, 2025.
[R6] Rathi, N. and Radford, A. "Shaping capabilities with token-level data filtering." arXiv:2601.21571, 2026.
[R7] Malla, S., Choi, C., and Choi, J. H. "The Geometry of Refusal: Why Post-Hoc Safety Is Fragile and Pretraining-Time Safety Persists." arXiv:2609.06934, 2026.
[R8] Tice, C., Radmard, P., Ratnam, S., Kim, A., Africa, D. D., and O'Brien, K. "Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment." ICML, 2026.
[R9] Qi, X., Panda, A., Lyu, K., Ma, X., Roy, S., Beirami, A., Mittal, P., and Henderson, P. "Safety Alignment Should Be Made More Than Just a Few Tokens Deep." ICLR, 2025.
[R10] Fan, C., Wang, C., Huang, Y., Pal, S., and Liu, S. "LLM Unlearning Under the Microscope: A Full-Stack View on Methods and Metrics." arXiv:2510.07626, 2025.
[R11] Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P. "Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!" ICLR, 2024.
[R12] Wang, C., Zhang, Y., Jia, J., Ram, P., Wei, D., Yao, Y., Pal, S., Baracaldo, N., and Liu, S. "Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning." ICML, 2025.
Response to Reviewer nP7G (Part 1/4)
We thank the reviewer for the thoughtful and constructive feedback. We are encouraged that the reviewer finds the question timely, recognizes that distilling native safety from small safety-pretrained models into larger ones brings its benefits to larger models while saving compute, and appreciates our experiments, including ablations and applications. Below, we address the weaknesses (W1–W3) and questions (Q1–Q8) in an organized way.
[W1]: Limited safety evaluations and unclear protocols undermine the results.
[W1.1]: The evaluation of safety robustness is limited to fine-tuning attacks. Would on-policy distillation also carry the benefits of safety pretraining on other attacks (such as the GCG attack) to larger models?
[Q2]: How do different distilling methods work under inference-stage attacks (e.g., GCG attacks)?
Response: Thank you for the insightful question. In Table R1, we ran the suggested GCG attack on SafePre, SafePost, and SafePre-OnDistill.
As we can see, the safety-pretrained teacher itself does not exhibit robustness to GCG: on HarmBench, SafePre stays safe against subsequent fine-tuning on GSM8K (2.19%) but reaches 42.00% ASR under GCG (Table R1). Safety pretraining thus confers robustness against fine-tuning but not against GCG, so on-policy distillation has no GCG benefit to carry. Accordingly, under GCG, SafePost reaches 54.00% and SafePre-OnDistill 58.00%, whereas after fine-tuning, SafePre-OnDistill's ASR is lower than SafePost's (6.88% vs. 15.00%).
This observation is consistent with the reviewer's reference [R2], which finds no consistent advantage of SafeLM over a standard-pretrained 1.7B model under GCG attacks. We emphasize that robustness to worst-case input-level jailbreaking attacks and robustness to normal fine-tuning are distinct properties: the former measures worst-case robustness to optimized input perturbations, whereas our paper focuses on the latter, namely, the persistence of safety under standard model fine-tuning (Lines 84 and 365–367). This distinction is also consistent with the weak correlation between input-level and weight-level robustness reported in Fig. 4b of [R10]. Our response to Q6 further provides a token-level analysis of the GCG-generated responses. We have incorporated Table R1 into the revised manuscript.
Table R1: HarmBench ASR (%, lower is better) of the SmolLM2-1.7B models of Tab. 2. ASR$_{\text{GCG}}$ is evaluated using a white-box GCG suffix [R1] optimized separately for each model before fine-tuning (250 optimization steps, 20-token suffix, and HarmBench GCG targets). Responses are evaluated using the HarmBench classifier, following Tab. 2.
[W1.2]: The fine-tuning attack evaluation protocol in this paper is not sufficiently documented, especially the number of fine-tuning examples and fine-tuning epochs. Additionally, the original SafeLM paper [1] proposed SafeBeam, without which SafeLM's robustness to benign fine-tuning would significantly decrease. In this paper, it's not clear whether the authors use SafeBeam in their evaluations.
[Q1]: What is the evaluation protocol for fine-tuning attacks used in this paper?
[Q3]: Do the authors use SafeBeam in any of their evaluations? How would it affect the results?
Response: Thank you for raising these questions. Please see our response below.
Fine-tuning protocol. For evaluating safety robustness against (irrelevant) fine-tuning, we follow the setup in [R11], with GSM8K as the fine-tuning dataset as in [R12]. Specifically, each model is fine-tuned for one epoch on GSM8K (Lines 366–367). As detailed in Appendix C.2 (Lines 884–893), we use the full GSM8K training set (7,473 examples), the objective defined in Eq. (A1), and the AdamW optimizer with cosine learning-rate decay, an initial learning rate of 1e-5, and a batch size of 4.
SafeBeam. Although SafeBeam was introduced in the SafeLM paper, its implementation relies on a harmfulness tag learned during pretraining, which requires explicitly tagged training data. This mechanism is unfortunately not available in the released checkpoints. Specifically, our teacher (locuslab/safelm-1.7b) is the publicly released SafeLM base model, whose model card describes the rephrasing, refusal, and Moral Education training recipe without incorporating the harmfulness tag. We also confirmed this by measurement: on harmful and benign prompts, the teacher and SafePre both assign the tag a probability of about 1e-10, the same as a model pretrained on raw web data without the tag, and neither emits it in 600 sampled generations. The students were never trained with the tag either.
We have clarified the above in the revision (Appendices D.2 and D.3).