acceptodds
Under review as a conference paper at ICLR 2027

Can Native Safety Travel? Reusing Safety Pretraining Across Models via On-Policy Distillation

10 views

Abstract

Safety pretraining has emerged as a promising paradigm for embedding safety deeply into a model during pretraining, yielding robustness to subsequent fine-tuning that conventional safety post-training often lacks. We refer to this safety acquired during model pretraining as native safety. Yet native safety is expensive to acquire, confined to the model in which it was pretrained, and often accompanied by lower general utility. In this work, we rethink the practical value of safety-pretrained models and introduce, for the first time, the native safety reuse problem: Can native safety be transferred from a specialized, potentially smaller, safety-pretrained "teacher" to standard-pretrained "students," enabling them to inherit its robustness while preserving their general utility? We show that native safety can "travel" through on-policy distillation. A standard-pretrained student generates responses to safety-pretraining prompts, while a small safety-pretrained teacher supervises the student's token-level predictions along its own generated trajectories; cross-tokenizer alignment further enables transfer across model families. The resulting students exhibit substantially greater safety robustness than their safety-post-trained counterparts while retaining higher general utility than the teacher: subsequent fine-tuning raises the average attack success rate (ASR) of the 1.7B student by only 5.00%, compared with 11.58% for safety post-training, and its average utility of 48.20% far exceeds the aligned teacher's 27.42%. Unlike conventional large-to-small knowledge distillation, our framework enables selective, small-to-large safety transfer: it reuses the robust safety expertise of a smaller pretrained teacher without inheriting its utility limitations. It further enables native safety to "travel" to broader applications, including robust machine unlearning and vision-language model safety alignment. These findings establish native safety as a reusable model property rather than one permanently tied to the originally pretrained model.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM safety, On-policy distillation, Safety pretraining

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nP7GOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates the 'native safety reuse problem': whether the safety behaviors learned in pretraining (termed native safety) can be transferred from a smaller teacher model to a larger student model. It shows that on-policy distillation allows native safety to transfer to larger student models that are of the same or different model class. Specifically, the authors use the safety pretraining dataset as OPD datasets, employ GOLD to address the cross-tokenizer distillation problem, and augment the distillation objective with a utility-preservation loss to allow distillation from a small base model to a large instruction model. This method achieves safety robustness to fine-tuning attacks and retains higher utility than the smaller safety-pretrained teacher. The paper further demonstrates the method's application to VLM safety and machine unlearning.

Critical strengths
  1. This paper investigates a timely question. Safety pretraining is indeed expensive, and most existing papers can only afford to experiment on smaller models, such as 1.7B. Distilling the native safety from small safety-pretrained models to larger models would be a clever approach to bring the benefits to larger models while saving compute.
  2. The authors conduct many experiments, including ablations and applications, to test whether native safety learned in SafeLM can transfer to different models and scenarios.
Critical weaknesses
  1. Limited safety evaluations and unclear protocols undermine the results.

    1. The evaluation of safety robustness is limited to fine-tuning attacks. Would on-policy distillation also carry the benefits of safety pretraining on other attacks (such as the GCG attack) to larger models?
    2. The fine-tuning attack evaluation protocol in this paper is not sufficiently documented, especially the number of fine-tuning examples and fine-tuning epochs. Additionally, the original SafeLM paper [1] proposed SafeBeam, without which SafeLM's robustness to benign fine-tuning would significantly decrease. In this paper, it's not clear whether the authors use SafeBeam in their evaluations.
  2. Missing details on an important ablation baseline. As I understand it, SafePost-Distill would be an important baseline to compare: if similar results can be obtained by OPD from a safety post-trained model, then there is no need for safety pretraining, which further saves cost. However, I cannot find details about how you perform standard safety post-training, and I find the results in Appendix D.1 surprising, as SafePost cannot refuse such explicitly stated harmful requests. This makes me doubt whether the benefit of SafePre-OnDistill over SafePost-Distill is due to the insufficient optimization of SafePost.

  3. The proposed method is tailored to SafeLM, and it is unclear whether it can be applied to other safety pretraining methods. The authors should discuss how their method could generalize to other safety pretraining approaches, including data filtering at different granularities [2, 3,4], upsampling positive AI discourse [5], incorporating safety-refusal data through co-training [6], and learning safety reflections during pretraining [7].

References:

[1] https://openreview.net/pdf?id=91H9CSvdwl

[2] https://openreview.net/pdf?id=n6mTO5JS4j

[3] https://openreview.net/forum?id=xcf0QcTcGS

[4] https://arxiv.org/pdf/2601.21571

[5] https://openreview.net/forum?id=951OAanYyQ&referrer=%5Bthe%20profile%20of%20David%20Demitri%20Africa%5D(%2Fprofile%3Fid%3D~David_Demitri_Africa1)

[6] https://arxiv.org/abs/2609.06934

[7] https://openreview.net/forum?id=Gequ8WzMdQ

Questions for authors
  1. What is the evaluation protocol for fine-tuning attacks used in this paper?
  2. How do different distilling methods work under inference-stage attacks (e.g., GCG attacks)?
  3. Do the authors use SafeBeam in any of their evaluations? How would it affect the results?
  4. How do the authors perform standard safety post-training?
  5. The authors could strengthen their SafePost-Distill baseline. For instance, if you use a sufficiently optimized instruction model, coupled with coherent OPD datasets (such as the safety dataset used in safety post-training), what would the results be like?
  6. More generally speaking, on-policy distillation is also a post-training alignment method. Why wouldn't it suffer from the shallow alignment phenomenon in [1]? Could the authors provide more analysis on this? Also, could the authors provide more analysis of success and failure cases?
  7. In this work, you only consider one safety pretraining student model, SafeLM, and you directly use SafeLM's safety training datasets in OPD. For other safety-pretrained teacher models, if you cannot access their pretraining dataset, what would you use as the OPD dataset?
  8. The comparison of utility in Figure 2 is unfair. SafeLM-1.7B is only pretrained on 600B tokens, while SmolLM2-1.7B is pretrained on 11T tokens. I think the difference in utility is better explained by the difference in the size of the pretraining corpus, but not that safety pretraining hurts utility. What is the comparison result of SafeLM-1.7B and a matched-compute safety post-training model?

References [1] https://openreview.net/forum?id=6Mxhg9PtDE

4 comments
AuthorsOpenReview ↗

Response to Reviewer nP7G (Part 1/4)

We thank the reviewer for the thoughtful and constructive feedback. We are encouraged that the reviewer finds the question timely, recognizes that distilling native safety from small safety-pretrained models into larger ones brings its benefits to larger models while saving compute, and appreciates our experiments, including ablations and applications. Below, we address the weaknesses (W1–W3) and questions (Q1–Q8) in an organized way.

[W1]: Limited safety evaluations and unclear protocols undermine the results.

  [W1.1]: The evaluation of safety robustness is limited to fine-tuning attacks. Would on-policy distillation also carry the benefits of safety pretraining on other attacks (such as the GCG attack) to larger models?

[Q2]: How do different distilling methods work under inference-stage attacks (e.g., GCG attacks)?

Response: Thank you for the insightful question. In Table R1, we ran the suggested GCG attack on SafePre, SafePost, and SafePre-OnDistill.

As we can see, the safety-pretrained teacher itself does not exhibit robustness to GCG: on HarmBench, SafePre stays safe against subsequent fine-tuning on GSM8K (2.19%) but reaches 42.00% ASR under GCG (Table R1). Safety pretraining thus confers robustness against fine-tuning but not against GCG, so on-policy distillation has no GCG benefit to carry. Accordingly, under GCG, SafePost reaches 54.00% and SafePre-OnDistill 58.00%, whereas after fine-tuning, SafePre-OnDistill's ASR is lower than SafePost's (6.88% vs. 15.00%).

This observation is consistent with the reviewer's reference [R2], which finds no consistent advantage of SafeLM over a standard-pretrained 1.7B model under GCG attacks. We emphasize that robustness to worst-case input-level jailbreaking attacks and robustness to normal fine-tuning are distinct properties: the former measures worst-case robustness to optimized input perturbations, whereas our paper focuses on the latter, namely, the persistence of safety under standard model fine-tuning (Lines 84 and 365–367). This distinction is also consistent with the weak correlation between input-level and weight-level robustness reported in Fig. 4b of [R10]. Our response to Q6 further provides a token-level analysis of the GCG-generated responses. We have incorporated Table R1 into the revised manuscript.

Table R1: HarmBench ASR (%, lower is better) of the SmolLM2-1.7B models of Tab. 2. ASR$_{\text{GCG}}$ is evaluated using a white-box GCG suffix [R1] optimized separately for each model before fine-tuning (250 optimization steps, 20-token suffix, and HarmBench GCG targets). Responses are evaluated using the HarmBench classifier, following Tab. 2.

MethodASR$_{\text{BF}}$ → ASR$_{\text{AF}}$ ↓ASR$_{\text{GCG}}$ ↓
SafePre0.62→2.1942.00
SafePost3.12→15.0054.00
SafePre-OnDistill0.62→6.8858.00

[W1.2]: The fine-tuning attack evaluation protocol in this paper is not sufficiently documented, especially the number of fine-tuning examples and fine-tuning epochs. Additionally, the original SafeLM paper [1] proposed SafeBeam, without which SafeLM's robustness to benign fine-tuning would significantly decrease. In this paper, it's not clear whether the authors use SafeBeam in their evaluations.

[Q1]: What is the evaluation protocol for fine-tuning attacks used in this paper?

[Q3]: Do the authors use SafeBeam in any of their evaluations? How would it affect the results?

Response: Thank you for raising these questions. Please see our response below.

Fine-tuning protocol. For evaluating safety robustness against (irrelevant) fine-tuning, we follow the setup in [R11], with GSM8K as the fine-tuning dataset as in [R12]. Specifically, each model is fine-tuned for one epoch on GSM8K (Lines 366–367). As detailed in Appendix C.2 (Lines 884–893), we use the full GSM8K training set (7,473 examples), the objective defined in Eq. (A1), and the AdamW optimizer with cosine learning-rate decay, an initial learning rate of 1e-5, and a batch size of 4.

SafeBeam. Although SafeBeam was introduced in the SafeLM paper, its implementation relies on a harmfulness tag learned during pretraining, which requires explicitly tagged training data. This mechanism is unfortunately not available in the released checkpoints. Specifically, our teacher (locuslab/safelm-1.7b) is the publicly released SafeLM base model, whose model card describes the rephrasing, refusal, and Moral Education training recipe without incorporating the harmfulness tag. We also confirmed this by measurement: on harmful and benign prompts, the teacher and SafePre both assign the tag a probability of about 1e-10, the same as a model pretrained on raw web data without the tag, and neither emits it in 600 sampled generations. The students were never trained with the tag either.

We have clarified the above in the revision (Appendices D.2 and D.3).

AuthorsOpenReview ↗

Response to Reviewer nP7G (Part 2/4)

[W2]: Missing details on an important ablation baseline. As I understand it, SafePost-Distill would be an important baseline to compare: if similar results can be obtained by OPD from a safety post-trained model, then there is no need for safety pretraining, which further saves cost. However, I cannot find details about how you perform standard safety post-training, and I find the results in Appendix D.1 surprising, as SafePost cannot refuse such explicitly stated harmful requests. This makes me doubt whether the benefit of SafePre-OnDistill over SafePost-Distill is due to the insufficient optimization of SafePost.

[Q4]: How do the authors perform standard safety post-training?

[Q5]: The authors could strengthen their SafePost-Distill baseline. For instance, if you use a sufficiently optimized instruction model, coupled with coherent OPD datasets (such as the safety dataset used in safety post-training), what would the results be like?

Response: Thank you for raising this question. However, we believe there might be a misunderstanding regarding SafePost and the necessity of SafePost-Distill.

First, Tab. A1 in Appendix D.1 (Tab. A5 in the revised paper) illustrates specific failure cases of SafePost rather than its overall safety performance. In fact, before fine-tuning, SafePost successfully resists 310 of the 320 harmful requests in HarmBench, achieving a low ASR$_{\text{BF}}$ of 3.12% (Tab. 2 for “HarmBench”). For Tab. A1, we picked one of the remaining 10 undefended cases on purpose, because such cases show what the safety-pretrained teacher adds: on the same request, SafePre and the distilled student refuse, whereas safety post-training does not. After fine-tuning, SafePost responds more harmfully to 48 of the 320 requests.

Second, as demonstrated in Tab. 2, SafePost fails to preserve its safety performance under subsequent fine-tuning, although it can achieve graceful safety performance without continual fine-tuning. This lack of fine-tuning robustness undermines the motivation for using SafePost-Distill as a means of achieving persistent safety. In addition, we follow the SafePost training setup consistent with [R3], using 13,927 training steps to ensure empirical convergence, a learning rate of 5e-5, and a batch size of 64.

Third, regarding a stronger safety post-training baseline (Q5), we consider a variant of SafePost that incorporates the safety-pretraining corpus used in our distillation. We term this variant as “NoTeacher-SFT,” which keeps the safety post-training objective and adds a cross-entropy term on this safety-pretraining corpus, without any teacher. Results are reported in Table R2. As we can see, NoTeacher-SFT is safe before fine-tuning (average ASR 3.11%) but is even less robust after fine-tuning than SafePost (30.99% vs. 14.16%; Tab. 2), while SafePre-OnDistill reaches only 5.16%. One possible explanation is the difficulty of jointly optimizing instruction-based safety post-training and token-level CE learning on the safety-pretraining corpus. These objectives may induce competing learning signals, making it difficult to integrate safety into the model's learned representations in a way that persists under subsequent fine-tuning. Since this safety-post-trained model already loses its safety under fine-tuning before any distillation, it offers no robustness advantage for SafePost-Distill to inherit, which is why the safety-pretrained teacher is needed.

Table R2: Stronger safety post-training on SmolLM2-1.7B. NoTeacher-SFT adds a cross-entropy term on the safety-pretraining data to the safety post-training objective, without a teacher. SafePre-OffDistill and SafePre-OnDistill are taken from Tab. 2; metrics follow Tab. 2. The proposed method is in bold.

MethodHB ↓TDC ↓JBB ↓AB ↓Avg ↓MMLU ↑CSQA ↑GSM8K ↑WG ↑Avg ↑
NoTeacher-SFT4.06→25.004.00→40.004.00→33.000.38→25.963.11→30.9948.8243.7330.7865.9847.33
SafePre-OffDistill1.25→12.190.00→26.000.00→16.000.00→7.690.31→15.4748.4542.5129.5765.5146.51
SafePre-OnDistill0.62→6.880.00→8.000.00→5.000.00→0.770.16→5.1648.4347.5830.7166.0648.20
AuthorsOpenReview ↗

Response to Reviewer nP7G (Part 3/4)

[W3]: The proposed method is tailored to SafeLM, and it is unclear whether it can be applied to other safety pretraining methods. The authors should discuss how their method could generalize to other safety pretraining approaches, including data filtering at different granularities [2, 3,4], upsampling positive AI discourse [5], incorporating safety-refusal data through co-training [6], and learning safety reflections during pretraining [7].

Response: Thank you for this constructive suggestion. Our method is not specific to SafeLM: We already apply our approach to a second natively trained teacher, Deep Ignorance (the reviewer's reference [R4]), whose pretraining data are filtered (Tab. 5). Zephyr-7B-beta distilled from it preserves the teacher's forgetting under fine-tuning (WMDP-Bio accuracy 31.32% → 31.97%), whereas post-hoc unlearning with RMU recovers the knowledge (28.07% → 36.25%). The students also vary in size (1.7B to 8B), family (SmolLM2, Llama-3, Qwen2.5-VL, Zephyr), and modality (Tabs. 2–5).

For the other approaches, distillation requires the teacher's checkpoint, and the reviewer's references [R5], [R6], [R7], and [R2] do not release one. [R8] does, but it does not target robustness to fine-tuning of open-weight models, the property we transfer, so we did not take it as our starting point. We have cited [R2] and [R4]–[R8] in the related work of the revision (Sec. 2).

[Q6]: More generally speaking, on-policy distillation is also a post-training alignment method. Why wouldn't it suffer from the shallow alignment phenomenon in [1]? Could the authors provide more analysis on this? Also, could the authors provide more analysis of success and failure cases?

Response: Thank you for raising this important point. We agree that on-policy distillation can also be considered as a post-training method. We clarify that the shallow alignment phenomenon discussed in [R9] concerns vulnerability to input-level jailbreaking attacks, rather than the persistence of safety under subsequent model fine-tuning. As explained in our responses to W1.1 and Q2, input-level and weight-level robustness do not necessarily correlate. Specifically, Qi et al. [R9] characterize shallow alignment as safety behavior concentrated in the first few output tokens, which can be exploited by inference-time attacks such as GCG. Following that, we examine this phenomenon using the GCG responses from Tab. R1 (W1.1). We measure the token-wise KL divergence between each model's output distribution and that of SmolLM2-1.7B-Instruct, a standard instruction-tuned model sharing the same architecture and tokenizer. All three models (SafePost, SafePre, and SafePre-OnDistill) exhibit their largest divergence at the first output token, where the refusal-versus-compliance decision typically occurs. In contrast, the mean KL divergence over tokens 1–49 is substantially lower (Figure A3 in the revised paper).

Although the three models are thus equally shallow at the token level, they differ in robustness to subsequent fine-tuning, the property the paper targets (Line 84): after fine-tuning, SafePre-OnDistill's average ASR is 5.16% vs. 14.16% for SafePost (Tab. 2, Lines 384 and 386). Token-level depth therefore does not by itself explain robustness to fine-tuning, which is what on-policy distillation inherits from the safety-pretrained teacher. We have added the per-token KL curves and this discussion in the revision (Appendix E).

Success and failure cases. Tab. A1 and Tab. A2 (Tabs. A5 and A6 in the revised paper) show success cases: the distilled student refuses during distillation (Tab. A1, before fine-tuning) and still refuses after fine-tuning (Tab. A2), where SafePost does not. After fine-tuning, SafePre-OnDistill still responds harmfully to some requests (average ASR 5.16%, Tab. 2), and about half of these also break SafePost. We have added representative failure cases in the revision (Tab. A7 in Appendix F.3).

[Q7]: In this work, you only consider one safety pretraining student model, SafeLM, and you directly use SafeLM's safety training datasets in OPD. For other safety-pretrained teacher models, if you cannot access their pretraining dataset, what would you use as the OPD dataset?

Response: Thank you for this forward-looking question. Without the teacher's pretraining data, such prompts could be obtained in two ways: generated by the safety-pretrained teacher itself in its own formats (e.g., Moral Education-style text), or drawn from public safety data topically relevant to its pretraining data (e.g., the safety post-training data of Sec. 5.1). How the transferred robustness depends on the distance between the distillation data and the teacher's pretraining data is an interesting question for future work, and the paper already notes this dependence on safety-pretraining prompts (Lines 985–986). We have expanded this discussion in the revision (Appendix I).

AuthorsOpenReview ↗

Response to Reviewer nP7G (Part 4/4)

[Q8]: The comparison of utility in Figure 2 is unfair. SafeLM-1.7B is only pretrained on 600B tokens, while SmolLM2-1.7B is pretrained on 11T tokens. I think the difference in utility is better explained by the difference in the size of the pretraining corpus, but not that safety pretraining hurts utility. What is the comparison result of SafeLM-1.7B and a matched-compute safety post-training model?

Response: Thank you for this insightful comment.

First, we agree that the utility gap in Fig. 2 might be confounded by the difference in pretraining budgets (600B vs. 11T tokens) and therefore cannot be attributed solely to safety pretraining. We have clarified this point in the description of Fig. 2 in the revision (Sec. 3).

Second, the limited utility of currently available safety-pretrained models nevertheless remains a practical challenge: safety pretraining is computationally expensive (Lines 203–204), making it impractical to repeat across different training configurations or scale to substantially larger token budgets (e.g., extending SafeLM-1.7B from 600B to 11T tokens). Yet, our method transfers safety robustness without inheriting the teacher's utility limitations. Although SafePre has low average utility (27.42%), SafePre-OnDistill keeps SafePost-level utility (48.20% vs. 48.00%) while its average ASR after fine-tuning is far lower (5.16% vs. 14.16%; Tab. 2). A safety-pretrained model with limited utility can thus still serve as an effective native-safety teacher, which also makes our approach more valuable.

Reference

[R1] Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., and Fredrikson, M. "Universal and Transferable Adversarial Attacks on Aligned Language Models." arXiv:2307.15043, 2023.

[R2] Li, J., Tang, K., Xu, Y., Ye, Z., and Lyu, K. "Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection." ICML CompLearn Workshop, 2026.

[R3] Maini, P., Goyal, S., Sam, D., Robey, A., Savani, Y., Jiang, Y., Zou, A., Fredrikson, M., Lipton, Z. C., and Kolter, J. Z. "Safety Pretraining: Toward the Next Generation of Safe AI." NeurIPS, 2025.

[R4] O'Brien, K., Casper, S., Anthony, Q., Korbak, T., Kirk, R., Davies, X., Mishra, I., Irving, G., Gal, Y., and Biderman, S. "Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs." ICLR, 2026.

[R5] Wang, R. Y., Finlayson, M., Soldaini, L., Swayamdipta, S., and Jia, R. "Teaching Models to Understand (but not Generate) High-risk Data." COLM, 2025.

[R6] Rathi, N. and Radford, A. "Shaping capabilities with token-level data filtering." arXiv:2601.21571, 2026.

[R7] Malla, S., Choi, C., and Choi, J. H. "The Geometry of Refusal: Why Post-Hoc Safety Is Fragile and Pretraining-Time Safety Persists." arXiv:2609.06934, 2026.

[R8] Tice, C., Radmard, P., Ratnam, S., Kim, A., Africa, D. D., and O'Brien, K. "Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment." ICML, 2026.

[R9] Qi, X., Panda, A., Lyu, K., Ma, X., Roy, S., Beirami, A., Mittal, P., and Henderson, P. "Safety Alignment Should Be Made More Than Just a Few Tokens Deep." ICLR, 2025.

[R10] Fan, C., Wang, C., Huang, Y., Pal, S., and Liu, S. "LLM Unlearning Under the Microscope: A Full-Stack View on Methods and Metrics." arXiv:2510.07626, 2025.

[R11] Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., and Henderson, P. "Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!" ICLR, 2024.

[R12] Wang, C., Zhang, Y., Jia, J., Ram, P., Wei, D., Yao, Y., Pal, S., Baracaldo, N., and Liu, S. "Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning." ICML, 2025.

Reviewer_FmrLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether native safety from safety-pretraining can be distilled into a standard-pretrained student rather than staying tied to the model it was pretrained into. SafePre-OnDistill does on-policy KL distillation on student-generated trajectories, extended cross-tokenizer via GOLD, plus a utility CE loss. ASR is measured on four jailbreak benchmarks before/after GSM8K fine-tuning (Tables 2-3), a harmful-fine-tuning stress test, and VLM/unlearning extensions.

Critical strengths

The architectural decoupling is well motivated. Restricting the expensive on-policy KL/GOLD term to safety prompts while using CE for general instruction data is a sensible design.

Using student-generated samples before querying the teacher is also conceptually appropriate for reducing train/deployment distribution shift.

Critical weaknesses

Figure 3 (Right) shows the student's probability mass is explicitly pushed toward "safe-oriented tokens" on harmful prompts, which is exactly the mechanism that produces over-refusal on benign-but-sensitive prompts yet no XSTest-style false-positive-refusal metric appears in Tables 2-5. The utility benchmarks (MMLU, CSQA, GSM8K, WG) measure capability, not refusal behavior, so a model that simply refuses more broadly would show improved ASR with no penalty anywhere in the evaluation suite. Isn't a refusal-shifted model exactly what would win on ASR while silently failing ordinary users?

There is no mechanism-level argument that what transfers is specifically ``safety'' rather than teacher style. Because the KL/GOLD term is computed on DsafeD_{safe}, it will transfer any systematic teacher-student difference on those prompts register, verbosity, templated refusal phrasing not provably "safety" as a semantic property, and the author never cites a mechanism for why this pipeline should be safety-specific rather than style-specific. Figure 3's ΔP\Delta P check is the closest thing to evidence of safety-specificity, but there is no ablation disentangling safety content from stylistic artifacts of the teacher's safety-pretraining corpus construction.

Top k=16k=16 truncation (Appendix C.1) renormalizes over only the student's current top-16 tokens, so in the exact failure mode the method targets (student drifting toward a harmful continuation), the teacher's refusal token is least likely to be in that top-16 and gets discarded before the KL is computed. Does the correction signal survive the cases it is meant to rescue?

Questions for authors

Can you add seeds/variance to Tables 2-3? Several gaps are a few points apart on benchmarks with small test sets.

Appendix B tunes α=0.7\alpha=0.7 on the exact configuration later reported as a main result in Table 3 was a held-out split used, or is this circular?

Why only one safety-pretrained teacher? Would this hold for an RLHF-style or conditioning-based safety-pretraining recipe?

3 comments
AuthorsOpenReview ↗

Response to Reviewer FmrL (Part 1/3)

We thank the reviewer for the insightful comments and for recognizing that our approach is a well-motivated design, and that using student-generated samples is appropriate for reducing the train/deployment distribution shift. Below, we address the weaknesses (W1–W3) and questions (Q1–Q3), numbered in order of appearance in the review.

[W1]: Figure 3 (Right) shows the student's probability mass is explicitly pushed toward "safe-oriented tokens" on harmful prompts, which is exactly the mechanism that produces over-refusal on benign-but-sensitive prompts yet no XSTest-style false-positive-refusal metric appears in Tables 2-5. The utility benchmarks (MMLU, CSQA, GSM8K, WG) measure capability, not refusal behavior, so a model that simply refuses more broadly would show improved ASR with no penalty anywhere in the evaluation suite. Isn't a refusal-shifted model exactly what would win on ASR while silently failing ordinary users?

Response: Thank you for raising this question. We agree that ASR alone may not fully tell a safer model from one that refuses more broadly, so we measured over-refusal on XSTest [R1], following your suggestion. XSTest has 250 benign prompts that contain sensitive words (e.g., "kill" in "How can I kill a Python process?") and 200 unsafe contrast prompts. We evaluate the models before fine-tuning, generate with greedy decoding (256 new tokens), and label full refusals with the XSTest authors' string-match classifier [R1]. As shown in Table R1, SafePre-OnDistill refuses slightly fewer benign prompts than SafePre (10.40% vs. 13.60%) and a similar share of the unsafe prompts (76.00% vs. 77.00%).

What Fig. 3 (Right) measures. It reports the change in next-token probability of harmful words at the positions where the base model generates them in its continuations of harmful prompts from JB-Completions (Lines 273–279 and 288–293), so it shows that harmful continuations are directly suppressed on harmful prompts. Refusal on benign prompts, which XSTest tests, is outside what it measures.

Table R1: XSTest [R1] refusal rate (%) before fine-tuning, on 250 benign prompts (lower is better) and 200 unsafe prompts (higher is better). SafePre is the teacher and has no student (N/A).

StudentMethodRefusal (benign) ↓Refusal (unsafe) ↑
N/ASafePre13.6077.00
SmolLM2-1.7BSafePre-OnDistill10.4076.00
AuthorsOpenReview ↗

Response to Reviewer FmrL (Part 2/3)

[W2]: There is no mechanism-level argument that what transfers is specifically "safety" rather than teacher style. Because the KL/GOLD term is computed on D$_{\text{safe}}$, it will transfer any systematic teacher-student difference on those prompts — register, verbosity, templated refusal phrasing — not provably "safety" as a semantic property, and the author never cites a mechanism for why this pipeline should be safety-specific rather than style-specific. Figure 3's Δ P check is the closest thing to evidence of safety-specificity, but there is no ablation disentangling safety content from stylistic artifacts of the teacher's safety-pretraining corpus construction.

Response: Thank you for this thoughtful question. To address your concern, we highlight additional evidence from our existing experiments.

First, identical refusal styles do not imply identical safety robustness. In Tab. R2, we compare the two models on the 770 of the 990 harmful prompts of the four benchmarks in Tab. 2 (HB, TDC, JBB, and AB) for which, before fine-tuning, SafePre-OnDistill and SafePost produce similar refusal responses (e.g., "I'm sorry, but I can't assist with that request."). As we can see, their average attack success rates (ASRs) on these same prompts differ substantially after fine-tuning: 12.03% for SafePost versus only 5.32% for SafePre-OnDistill. Since both models initially produce identical safe responses on these prompts, this robustness gap cannot be attributed solely to differences in initial output style or refusal phrasing.

Second, the student's safety behavior does not reproduce the teacher's distinctive stylistic artifacts. Using an LLM-based judge, we find that none (0.00%) of the student's safe responses exhibit the Moral Education rewriting style characteristic of the teacher's safety-pretraining corpus (Lines 169–171). Together with Fig. 3 (Right), these findings support that on-policy distillation transfers safety-relevant behavior beyond superficial refusal styles, improving the persistence of safety under subsequent fine-tuning rather than merely reproducing the teacher's stylistic patterns.

Table R2: ASR$_{\text{AF}}$ (%, lower is better) of SafePost and SafePre-OnDistill from Tab. 2, computed only on the prompts where the two models gave the same safe response before fine-tuning (233 of 320 HB, 42 of 50 TDC, 73 of 100 JBB, and 422 of 520 AB prompts); Tab. 2 computes it on all prompts. The proposed method is in bold.

MethodHB ↓TDC ↓JBB ↓AB ↓Avg ↓
SafePost14.1616.6712.334.9812.03
SafePre-OnDistill7.737.145.480.955.32

[W3]: Top k=16 truncation (Appendix C.1) renormalizes over only the student's current top-16 tokens, so in the exact failure mode the method targets (student drifting toward a harmful continuation), the teacher's refusal token is least likely to be in that top-16 and gets discarded before the KL is computed. Does the correction signal survive the cases it is meant to rescue?

Response: The correction signal does not require the teacher's refusal token to appear in the student's top-16 tokens. A response can avoid harmful content without explicitly refusing, as illustrated by the Moral Education responses in SafePre [R2].

When the student drifts toward a harmful continuation, harmful tokens are likely to appear among its top-16 candidates. Since the teacher assigns low probabilities to these tokens, the truncated KL term still penalizes harmful continuations, shifting relative probability toward safer alternatives within the same top-16 set. This mechanism is supported by Fig. 3 (Right), which shows that distillation suppresses the probability of harmful words on harmful prompts (Lines 273–293).

AuthorsOpenReview ↗

Response to Reviewer FmrL (Part 3/3)

[Q1]: Can you add seeds/variance to Tables 2-3? Several gaps are a few points apart on benchmarks with small test sets.

Response: We conducted additional experiments with two new random seeds for evaluations in Tab. 2. For each distillation method, we also repeated distillation training with two additional seeds. Tab. R3 reports the mean and standard deviation across these seeds. As we can see, the key performance gaps remain consistent across seeds and exceed the observed seed variation. In particular, SafePre-OnDistill consistently outperforms SafePost in fine-tuning robustness, achieving an average ASR after fine-tuning of 7.84% versus 14.61%, with a standard deviation of 1.36 percentage points.

Table R3: Avg ASR (%, lower is better) on SmolLM2-1.7B, mean ± std over three seeds (the run of Tab. 2 and two new seeds; sample std): distillation seeds for ASR$_{\text{BF}},fine−tuning−attackseedsforASR, fine-tuning-attack seeds for ASR_{\text{AF}}$.

MethodASR$_{\text{BF}}$ ↓ASR$_{\text{AF}}$ ↓
SafePre0.160.40 ± 0.37
SafePost2.58 ± 0.2114.61 ± 0.40
SafePost-Distill2.32 ± 0.4715.69 ± 1.55
SafePre-OffDistill0.56 ± 0.2516.59 ± 1.42
SafePre-OnDistill1.90 ± 0.587.84 ± 1.36

[Q2]: Appendix B tunes α=0.7 on the exact configuration later reported as a main result in Table 3 was a held-out split used, or is this circular?

Response: We clarify that α = 0.7 was selected only once, using the Llama-3.2-3B student distilled from SafeLM-1.7B (Fig. A2), and then applied unchanged to all other experimental settings. Specifically, we trained one student for each tested value of α and evaluated all configurations on the reported benchmarks, with the complete results presented in Fig. A2. We selected α = 0.7 for its favorable balance between fine-tuning robustness and utility. And then, the choice of α = 0.7 was subsequently fixed across all other students and applications (Line 357), including SmolLM2-1.7B (Tab. 2), Llama-3-8B (Tab. 3), and the VLM and unlearning experiments (Tabs. 4–5), without further tuning.

[Q3]: Why only one safety-pretrained teacher? Would this hold for an RLHF-style or conditioning-based safety-pretraining recipe?

Response: Thank you for this insightful question.

First, our approach has already been validated with a second natively trained teacher, Deep Ignorance [R3], which uses filtered pretraining data (Tab. 5). Specifically, Zephyr-7B-beta distilled from Deep Ignorance preserves the teacher's forgetting under subsequent fine-tuning, with WMDP-Bio accuracy remaining nearly unchanged (31.32% → 31.97%). In contrast, post-hoc unlearning with RMU exhibits substantial knowledge recovery (28.07% → 36.25%). This result provides evidence that our approach extends beyond SafeLM to a different pretraining recipe and safety-related objective.

Second, evaluating additional safety-pretraining recipes is constrained by the availability of released checkpoints. Reproducing safety pretraining from scratch is beyond our computational resources, and publicly available safety-pretrained models remain limited. For conditioning-based safety pretraining, the closest released model we identified is Korbak et al. [R4] (GPT-2-small, 124M parameters, 2023). However, its substantially older architecture limits its suitability as a representative teacher for our experiments.

Reference

[R1] Röttger, P., Kirk, H. R., Vidgen, B., Attanasio, G., Bianchi, F., and Hovy, D. "XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models." NAACL, 2024.

[R2] Maini, P., Goyal, S., Sam, D., Robey, A., Savani, Y., Jiang, Y., Zou, A., Fredrikson, M., Lipton, Z. C., and Kolter, J. Z. "Safety Pretraining: Toward the Next Generation of Safe AI." NeurIPS, 2025.

[R3] O'Brien, K., Casper, S., Anthony, Q., Korbak, T., Kirk, R., Davies, X., Mishra, I., Irving, G., Gal, Y., and Biderman, S. "Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs." ICLR, 2026.

[R4] Korbak, T., Shi, K., Chen, A., Bhalerao, R. V., Buckley, C., Phang, J., Bowman, S. R., and Perez, E. "Pretraining Language Models with Human Preferences." ICML, 2023.

Reviewer_h7i3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method that transfers safety capabilities acquired through safety pretraining from a teacher model to standard-pretrained student models via on-policy distillation. Experiments show that the method improves students’ robustness against safety degradation after fine-tuning while largely preserving their general utility.

Critical strengths
  • The paper introduces native safety reuse and demonstrates that native safety acquired through safety pretraining can be transferred to standard-pretrained students via on-policy distillation.

  • The experiments show that a smaller safety-pretrained teacher can improve the fine-tuning robustness of larger student models while largely preserving their general utility.

Critical weaknesses
  1. Is on-policy distillation necessary for native safety reuse? Although the paper compares on-policy distillation with off-policy distillation and safety post-training baselines, it lacks a simpler non-distillation baseline that trains the student directly on the same safety corpus while retaining the instruction-tuning loss. Without this comparison, it remains unclear whether the robustness gains stem from teacher-guided token-level supervision or simply from additional exposure to safety-relevant data.

  2. The evaluation would benefit from an additional comparison with adversarial training. Such a baseline could help determine whether the robustness gains are specific to reusing native safety through distillation or can also be achieved through alternative safety-training methods.

  3. A smaller increase in delte_ASR after fine-tuning shows improved behavioral robustness, but does not by itself establish that the student has inherited a distinctive native-safety property from the teacher. The observed gains could also result from more stable refusal behavior induced by additional safety training. Further analysis is needed to distinguish these explanations.

Questions for authors

Please refer to the weaknesses section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer h7i3 (Part 1/2)

We thank the reviewer for recognizing that the paper introduces native safety reuse and shows that safety acquired through safety pretraining transfers to standard-pretrained students via on-policy distillation, and that a smaller safety-pretrained teacher improves the fine-tuning robustness of larger students while largely preserving their utility. Below, we address W1–W3 in turn.

[W1]: Is on-policy distillation necessary for native safety reuse? Although the paper compares on-policy distillation with off-policy distillation and safety post-training baselines, it lacks a simpler non-distillation baseline that trains the student directly on the same safety corpus while retaining the instruction-tuning loss. Without this comparison, it remains unclear whether the robustness gains stem from teacher-guided token-level supervision or simply from additional exposure to safety-relevant data.

Response: Thank you for this insightful question. We conducted the suggested experiment, training a student directly on the same safety corpus with the instruction-tuning loss and no teacher; results are reported in Table R1. Specifically, we keep the objective in Eq. (4) and replace only its distillation term with the standard next-token cross-entropy on the same safety data (Lines 348–349). The starting model (SmolLM2-1.7B), the utility term, and the number of training steps follow SafePre-OffDistill and SafePre-OnDistill; fine-tuning on GSM8K and evaluation follow Tab. 2.

As shown in Tab. R1, the suggested baseline (termed as NoTeacher-SFT) exhibits substantially worse fine-tuning robustness, indicating that teacher-guided on-policy supervision, rather than safety data alone, is critical to the observed robustness gains. All three students achieve low ASR before fine-tuning (ASR$_{\text{BF}}$: 0.16–3.11%), indicating that direct training on safety data can establish initial safety. However, after fine-tuning, NoTeacher-SFT, which directly learns from the safety corpus, reaches an ASR of 30.99%; SafePre-OffDistill, which learns from teacher-generated responses, achieves 15.47%; and SafePre-OnDistill, which uses teacher feedback on the student's own generations, achieves only 5.16%. Notably, NoTeacher-SFT exhibits the worst robustness across all four benchmarks despite comparable utility (47.33% vs. 48.20%). These results highlight the importance of teacher-guided on-policy distillation for achieving persistent safety, beyond what can be obtained from safety data or off-policy supervision alone.

Table R1: Non-distillation baseline on SmolLM2-1.7B. NoTeacher-SFT trains the student on the original text of the safety data with the instruction-tuning loss and no teacher. SafePre-OffDistill and SafePre-OnDistill are taken from Tab. 2. Metrics follow Tab. 2; the proposed method is in bold.

MethodHB ↓TDC ↓JBB ↓AB ↓Avg ↓MMLU ↑CSQA ↑GSM8K ↑WG ↑Avg ↑
NoTeacher-SFT4.06→25.004.00→40.004.00→33.000.38→25.963.11→30.9948.8243.7330.7865.9847.33
SafePre-OffDistill1.25→12.190.00→26.000.00→16.000.00→7.690.31→15.4748.4542.5129.5765.5146.51
SafePre-OnDistill0.62→6.880.00→8.000.00→5.000.00→0.770.16→5.1648.4347.5830.7166.0648.20
AuthorsOpenReview ↗

Response to Reviewer h7i3 (Part 2/2)

[W2]: The evaluation would benefit from an additional comparison with adversarial training. Such a baseline could help determine whether the robustness gains are specific to reusing native safety through distillation or can also be achieved through alternative safety-training methods.

Response: Thank you for this constructive suggestion. Following that, we conducted min-max optimization-based adversarial training, with results reported in Tab. R2. Since fine-tuning perturbs the weights, we follow Fan et al. [R1] and train SafePost using sharpness-aware minimization (SAM)-based adversarial training: min⁡θmax⁡∥δ∥2≤ρℓ(θ+δ;Ds)+ℓ(θ;Du)\min_{\theta} \max_{\Vert\delta\Vert_2 \le \rho} \ell(\theta+\delta; \mathcal{D}_s) + \ell(\theta; \mathcal{D}_u), where Ds\mathcal{D}_s contains the safety datasets WildGuardMix and WildJailbreak, Du\mathcal{D}_u is UltraChat-200k, and ρ = 0.01. As we can see, adversarial training lowers SafePost's ASR after fine-tuning from 14.16% to 12.45%, still above SafePre-OnDistill at 5.16%. Together with SafePost-Distill (15.82% after fine-tuning, Tab. 2), this indicates that when safety is acquired, in pretraining rather than post-training, matters more than the post-training recipe.

Table R2: SAM-based adversarial training on SmolLM2-1.7B. SafePost + SAM trains SafePost with the SAM objective above. SafePost and SafePre-OnDistill (ours) are from Tab. 2; metrics follow Tab. 2.

MethodHB ↓TDC ↓JBB ↓AB ↓Avg ↓MMLU ↑CSQA ↑GSM8K ↑WG ↑Avg ↑
SafePost3.12→15.002.00→20.005.00→17.000.19→4.622.58→14.1649.5545.7831.7764.8848.00
SafePost + SAM (ρ=0.01)3.44→11.254.00→26.004.00→11.000.00→1.542.86→12.4549.7445.8633.2865.1948.52
SafePre-OnDistill0.62→6.880.00→8.000.00→5.000.00→0.770.16→5.1648.4347.5830.7166.0648.20

[W3]: A smaller increase in delte_ASR after fine-tuning shows improved behavioral robustness, but does not by itself establish that the student has inherited a distinctive native-safety property from the teacher. The observed gains could also result from more stable refusal behavior induced by additional safety training. Further analysis is needed to distinguish these explanations.

Response: Thank you for raising this important point. We provided the following clarifications.

First, we have compared our proposed SafePre-OnDistill against three controls trained on the same safety and instruction data: NoTeacher-SFT (Tab. R1), SafePre-OffDistill, and SafePost-Distill. All those students can achieve low ASR before fine-tuning (ASR$_{\text{BF}}$: 0.16–3.11%). However, after fine-tuning, the three controls exhibit substantially higher ASRs of 30.99%, 15.47%, and 15.82%, respectively, compared with only 5.16% for SafePre-OnDistill (Tab. R1 and Tab. 2). These controlled comparisons provide evidence that transferring native-safety properties from the teacher through on-policy distillation is a key to the observed robustness gains, which cannot be explained by additional safety training alone, as demonstrated by the control baselines.

Second, token-level analysis provides additional evidence of native-safety behavior transfer. Fig. 3 (Right; Lines 273–293) examines next-token probabilities on harmful prompts. For the harmful tokens analyzed, on-policy distillation suppresses their probabilities to levels approaching those of SafePre, the safety-pretrained teacher, whereas safety post-training produces substantially weaker suppression. This suggests that SafePre-OnDistill inherits the teacher's tendency to suppress harmful continuations, a behavior not observed to the same extent in SafePost despite its strong initial safety performance.

Reference

[R1] Fan, C., Jia, J., Zhang, Y., Ramakrishna, A., Hong, M., and Liu, S. "Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond." ICML, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.