acceptodds
Under review as a conference paper at ICLR 2027

Safety Midtraining: Internalizing LLM Safety as a Foundational Capability

Abstract

Safety alignment for large language models (LLMs) is primarily performed during post-training, while recent work has also begun incorporating safety data into pre-training. Yet safety can degrade when later training no longer includes safety data, and gains in safety often come at the cost of general capability. This tension suggests that safety and general reasoning should not be treated as separate training goals. We instead view safety as an LLM capability that can be learned through a reasoning structure shared with general tasks. Both require the model to understand intent or task goals, apply relevant knowledge, and reason about potential consequences. We introduce safety mid-training, a stage between pre-training and post-training that jointly develops safety and general reasoning through this shared structure. Given safety and utility prompts, the base model generates structured documents and reasoning traces that are then used for mid-training. Safety mid-training reduces average attack success rate (ASR) by 72% to 81% relative to Qwen3-1.7B, 4B, and 8B base models, while also improving general capability. The benefits persist through subsequent post-training. After general-only SFT, mid-trained models largely retain their safety, achieving lower ASR than the baselines without sacrificing general capability. They also retain safety under general-only GRPO without safety rewards, while subsequent RL reaches a better safety-utility frontier when initialized from the mid-trained models. Together, these results suggest that learning safety and general reasoning within a shared mid-training stage can reduce the safety-utility trade-off and make safety more robust to subsequent training.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, safety alignment, midtraining

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_goXmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a safety mid-training method that aims to learn safety as a foundational capability of language models rather than relying solely on safety alignment at the post-training stage. Specifically, the method uses a base model to generate structured knowledge documents and reasoning traces, which, after quality filtering, are used for continual pre-training. Results show that the method improves the trade-off between safety and general capability and maintains relatively good safety performance in subsequent training that contains no safety data.

Critical strengths

The paper investigates a practically meaningful problem: how to keep safety behavior stable during subsequent training while improving general capabilities. The experimental evaluation is fairly comprehensive, covering not only different model scales and multiple safety benchmarks but also different safety data ratios, training stages, and the effects of subsequent SFT and reinforcement learning.

Critical weaknesses

The main problem is that the experimental results are not yet sufficient to support the core mechanistic claim the authors put forward. The authors argue that safety reasoning and general reasoning share some underlying structure, but the existing experiments mainly demonstrate the effectiveness of jointly training on safety and general data, without adequately distinguishing the benefits of a shared reasoning mechanism from those of ordinary multi-task training. In addition, the method involves additional continual pre-training, and the baselines differ in training objectives, data structure, and training budget, so the specific source of the performance gains remains unclear. Although the authors conduct ablations on training stages and structured fields, these experiments cannot fully rule out the influence of additional training compute or additional training information. Therefore, the paper's conclusion that safety is internalized as a foundational capability still lacks sufficiently direct evidence.

Questions for authors

Can the authors further explain how the existing experiments demonstrate that safety and general capability are indeed learned jointly through a shared reasoning structure, rather than merely as an effect of jointly training on the two types of data? In addition, can they provide a simple continual pre-training baseline with strictly controlled training data, token budget, and training objective, so as to more clearly identify the independent contribution of the proposed data structure? Finally, how do the authors distinguish between the model genuinely acquiring more stable safety reasoning capability and the model merely learning more persistent safety response patterns?

4 comments

Weakness 1 Evidence for shared reasoning and the source of gains

Thank you for this comment. We add a controlled comparison with ordinary multi-task CPT to examine the contribution of the unified reasoning structure under matched training conditions. Both conditions use the same source data and sampling procedure, with the response portion kept identical and the training objective and token budget matched. The ordinary multi-task baseline generates safety and general data naturally for each task, while our method constructs reasoning traces for both through a common structure. Table R1 combines the new ordinary multi-task CPT results with the ASR and utility results for our method reported in Tables 1 and 2. Before downstream SFT, our method reduces ASR from 9.3% to 6.6% and improves utility from 70.6% to 73.1% under the matched training conditions. The improvement in Table R1 therefore supports the contribution of this shared organization when both models already receive safety and general data. The difference becomes more pronounced during subsequent general-only SFT, where the ordinary CPT model's ASR increases from 9.3% to 36.2%, while ours changes from 6.6% to 7.6%. This shows that the safety learned through our method remains effective when subsequent training contains no safety examples, supporting the stability of the resulting capability during general-task adaptation. The SP and AP results in Table 2 provide further evidence concerning training-data scale. Their training corpora contain approximately 2.7B and 1.3B tokens, respectively, and their Qwen3-4B models reach 17.3% and 14.5% ASR after general-only SFT, compared with 7.6% for ours. Our stronger safety performance with a smaller training corpus further supports the importance of the reasoning data design. These results support unified reasoning as an effective approach to learning safety alongside general capabilities, with safety gains that remain stable during later general-only training.

Question 1 Shared reasoning and stable safety behavior

Thank you for this helpful suggestion. The controlled CPT comparison above addresses the contribution of the shared reasoning structure, and we further examine whether the resulting safety gains accompany appropriate responses to benign requests and generalize across evaluation conditions. Before downstream SFT, Table R1 shows that our method reduces XSTest over-refusal from 24.6% to 9.7%, alongside the reduction in harmful-request ASR from 9.3% to 6.6%. Utility also increases from 70.6% to 73.1%. The model therefore handles harmful requests more safely while answering benign requests more reliably, supporting greater selectivity in its safety judgments. The main safety averages in Section 4.1 exclude benchmarks that share sources with the midtraining data, so the gains in these averages also provide evidence of generalization across benchmark sources. We additionally evaluate five StrongREJECT jailbreak methods to examine whether the safety advantage persists when the instructions surrounding a harmful request change. Table R2 shows lower ASR for our method under all five evaluated methods. Under refusal suppression, our model reaches 3.8% ASR compared with 47.3% for ordinary CPT, and under short style injection it reaches 0.3% compared with 35.8%. These results show that the safety advantage remains effective under instructions that attempt to alter the model's response behavior. During general-only SFT, our model also maintains low ASR while reaching 74.9% utility, compared with 62.4% for ordinary CPT, showing that continued learning on general tasks preserves useful safety behavior alongside general capability. These findings support the value of unified reasoning for learning selective safety behavior that generalizes across prompt changes and remains effective during subsequent training.

Table R1 Controlled multi-task CPT comparison on Qwen3-4B

ConditionASR (%)Utility (%)XSTest over-refusal (%)
Ordinary multi-task CPT9.370.624.6
Ours6.673.19.7
Ordinary multi-task CPT + safety-only SFT6.570.126.3
Ours + safety-only SFT4.575.39.1
Ordinary multi-task CPT + general-only SFT36.262.42.3
Ours + general-only SFT7.674.96.9

Table R2 ASR (%) under five StrongREJECT jailbreak methods

ModelDisemvowelPrefix injectionRefusal suppressionStyle injection (short)Distractors
Ours1.93.23.80.310.9
Ordinary multi-task CPT2.24.547.335.811.8

Thank you again for your thoughtful comments and suggestions. We have carefully addressed each point in our response and would be grateful if you could let us know whether the clarifications resolve your concerns. We look forward to receiving any further feedback from you.

Thank you for the detailed response to my review. In general, the rebuttal has addressed most of my questions, hence, I'm increasing the overall score.

Thank you for your encouraging feedback and for taking the time to review our response. We truly appreciate your thoughtful engagement and valuable comments.

Reviewer_RP7COpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a midtraining procedure that combines safety and general reasoning before supervised fine-tuning or reinforcement learning. A base model generates structured documents and reasoning traces organized around intent, relevant knowledge, and consequences. After filtering, these examples are used for continued language-model training with loss over all tokens. Experiments on Qwen3 and Llama models assess safety and capability gains, retention during subsequent training, and the contributions of the data and training procedure.

Critical strengths
  1. The proposed recipe in the paper improves safety and capability across the tested model scales. In Table 1, average attack success rate falls from 30.4% to 8.4% for Qwen3-1.7B, from 34.1% to 6.6% for Qwen3-4B, and from 25.4% to 4.9% for Qwen3-8B. Utility improves at all three scales; for the 4B model, the average rises from 47.6% to 73.1%. The Llama experiments extend the evidence to another model family. These joint improvements make the training recipe a useful contribution.

  2. The evaluation follows the models through subsequent SFT and GRPO, including settings without safety supervision. Appendix E.4 provides a valuable control by using Sonnet-generated SFT traces for both base and midtrained initializations. Averaged across the SFT mixtures, the midtrained models retain lower attack success rates at all three Qwen3 scales under this common data source. This strengthens the evidence that the downstream advantage comes from the initialization, since the main experiments otherwise use traces generated by each model itself.

Critical weaknesses
  1. The stage ablation in Table 3 changes several aspects of optimization at once. The direct-SFT and midtraining conditions use the same reasoning corpus, but the recipes in Appendices B.1–B.2 differ in loss masking, sequence packing, epoch count, and batch units. Midtraining applies loss to all tokens in packed sequences for one epoch; SFT applies loss to assistant tokens in unpacked records for three epochs. The authors have not specified whether the direct-SFT row departs from these defaults. Because these choices also affect optimization, the comparison cannot isolate the contribution attributed to the midtraining stage. The direct-SFT configuration and actual training budgets may need to be stated explicitly.

  2. The retention experiments begin from very different safety levels. For Qwen3-4B, the base and midtrained checkpoints start at 34.1% and 6.6% ASR and reach 13.6% and 7.6% after general-only SFT (Tables 1 and 2). The midtrained model retains much of its safety, while the base model becomes safer relative to its initialization. Comparing balanced and general-only training also measures dependence on subsequent safety supervision. To establish greater resistance to forgetting, the relevant comparison is with a conventionally safety-aligned checkpoint that starts at a similar level of safety and utility.

Questions for authors
  1. What loss mask, packing procedure, token batch size, data exposure, and optimizer-step budget were used for the direct-SFT row in Table 3? How much of the advantage remains when the two loss objectives are compared with the other optimization choices held fixed?

  2. How does safety change under an identical general-only continuation when the midtrained checkpoint and a conventional safety-SFT checkpoint begin with comparable safety and utility? Using common downstream traces, as in Appendix E.4, would make this comparison easier to interpret.

3 comments

Weakness 1 and Question 1 Optimization controls in Table 3

Thank you for this comment. The direct-SFT control in Table 3 uses the midtraining configuration in Appendix B.1, with only the loss mask changed. Both conditions start from the same base checkpoint and use the same reasoning-only corpus, with identical packing and token batch size. The epoch count and data exposure are also the same, as is the optimizer-step budget. Appendix B.2 describes the subsequent SFT stage. Under this controlled comparison, all-token training reduces ASR from 22.8% to 7.5% and raises utility from 68.9% to 74.7%, giving a 15.3 pp safety improvement together with a 5.8 pp utility improvement. These results support the contribution of the all-token objective when learning from the unified reasoning corpus, and show that the benefit is already present before downstream SFT.

Weakness 2 and Question 2 Interpretation of safety retention and Continuation from similar starting points

Thank you for this helpful suggestion. The original retention comparison measures the increase in ASR as downstream SFT contains less safety supervision, using safety-only SFT as the reference. The SFT-mixture rows in Table R1 summarize the results in Table 2 and compare separate training branches from each initialization, where moving from safety-only to general-only SFT increases ASR by 29.3 pp for Base + Sonnet SFT and 3.8 pp for Ours + SFT. This result supports lower dependence on safety examples during downstream training. To examine retention from similar initial safety, we add a continuation comparison between the conventional safety-SFT checkpoint at 8.0% ASR and Ours Midtrain at 8.4% ASR. Both checkpoints undergo three epochs of SFT on the same 10,000-example pure-general dataset using the same training settings, and the continuation rows use each checkpoint's ASR before this training as the reference. Ours Midtrain reaches 11.2% ASR, an increase of 2.8 pp, while the conventional safety-SFT checkpoint reaches 32.6%, an increase of 24.6 pp. The 21.8 pp smaller safety loss under the common continuation supports stronger safety retention from similar starting safety levels, extending the original evidence on sensitivity to the downstream SFT mixture.

Table R1 Downstream safety on Qwen3-1.7B

A. SFT-mixture comparison

ModelSafety-only SFT ASR (%)General-only SFT ASR (%)ASR increase (pp)
Base + Sonnet SFT8.037.3+29.3
Base + self SFT18.726.3+7.6
SP + SFT9.237.1+27.9
AP + SFT9.528.0+18.5
Ours + SFT7.411.2+3.8

B. General-only continuation

Starting checkpointBefore continuation ASR (%)After continuation ASR (%)ASR increase (pp)
Base + safety-only Sonnet SFT8.032.6+24.6
Ours Midtrain8.411.2+2.8
Reviewer_RP7COpenReview ↗

Response to the Authors

Thanks for the clarification. I will keep my scoring at 3.

Thank you for your encouraging feedback and for taking the time to review our response. We truly appreciate your thoughtful suggestions and valuable comments.

Reviewer_zaTKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper aims to address two problems: LLMs can lose safety behavior when further trained on general tasks, and safety training can hurt general capabilities. The authors’ idea is to make safety judgments part of the reasoning process used for general tasks. They expect this approach to help models retain safety during later training while preserving general capabilities. To achieve this, they propose Safety Mid-training, which trains models on safety and general tasks using the same reasoning structure before instruction tuning. The authors adopt base model to generate a structured document covering task goals, relevant knowledge, and possible consequences from 185,000 prompts with safety labels and reference answers. They then adopts these documents to generate a reasoning trace and final answer. Qwen3-32B filters these materials. The authors mix plain-text documents with chat-formatted reasoning examples and continue training the base model on these 200 million tokens, computing the language modeling loss over all tokens. The resulting model proceed to post-training of SFT or GRPO.

The main experiments cover Qwen3-1.7B, 4B, and 8B, with additional experiments on Llama-3.2-1B and 3B in the appendix. Before subsequent SFT, safety mid-training reduces average attack success rate (ASR) on fixed safety benchmarks by approximately 72%–81% relative to the original Qwen3 base models, while also improving average general capability scores. The advantage remains after SFT that either include or exclude safety examples, respectively. The authors also observe similar safety retention in GRPO experiments that use only general-task prompts.

Critical strengths
  1. Safety mid-training is a valuable direction. The paper provides a concrete way to study how introducing safety before post-training affects both safety and general capabilities. It also examines whether these benefits survive subsequent SFT and GRPO. This is a meaningful question, since models often undergo further training after their initial safety alignment.

  2. The paper presents substantial experimental work and informative ablations. The experiments cover three Qwen3 models and two Llama-3.2 models, with evaluations of safety, general capabilities, and the effects of subsequent training (§4.1; Appendix E.2). The results show clear safety improvements while maintaining or improving general capability scores (Tables 1–2). The ablations examine training stages, reasoning components, data composition, document records, and data-generation sources (§4.3; Appendix E). These comparisons provide useful information about practical training choices and a substantial reference for future work on safety training.

Critical weaknesses
  1. The safety evaluation does not establish robustness to adaptive jailbreaks. Section 4.1 and Appendix C.1 report results on fixed safety benchmarks, but do not evaluate attacks optimized against the trained models. Relevant approaches include feedback-based prompt refinement with PAIR [1], adaptive multi-turn attacks with X-Teaming [2], and adversarial suffix optimization with GCG [3]. H-CoT [4], which targets safety reasoning, is particularly relevant because the proposed method relies on reasoning to make safety judgments. The reported results show improved safety on the tested requests, but leave robustness against attackers who adjust their strategies unclear. Safety retention under general-task SFT and GRPO does not replace this evaluation.
  2. The source of the improvement over direct SFT is not clearly isolated. Table 3 compares the same reasoning-only data on Qwen3-4B and reports 22.8% ASR for direct SFT versus 7.5% for mid-training. Its caption states that “only the consumption stage differs” (p. 8). However, the default configurations in Appendices B.1–B.2 differ in several ways. Mid-training computes loss over all tokens, packs examples together, and runs for one epoch. SFT computes loss only over assistant outputs, processes examples separately, and runs for three epochs (Tables 8–9, pp. 35–36). It is unclear whether these settings were separately matched for Table 3. The comparison supports the effectiveness of the overall training setup, but does not explain which training choices account for the large improvement.
  3. The additional benefit of training on the document records is limited. For Qwen3-4B, adding document records increases the training budget from approximately 105M to 198M tokens (Appendix B.1, p. 35). Yet, averaged across the SFT mixtures, ASR changes from 5.5% to 5.9%, while utility increases from 74.4 to 75.0 (Table 38, p. 53). The authors also acknowledge that “The reasoning traces remain the central source of the gains” (Appendix E.3, p. 53). Given the additional training cost, the paper should better justify including these documents as separate training records, beyond using them to generate the reasoning traces.

References

[1] Jailbreaking Black Box Large Language Models in Twenty Queries. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. 2023.

[2] X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents. Salman Rahman, Liwei Jiang, James Shiffer, Genglin Liu, Sheriff Issaka, Md Rizwan Parvez, Hamid Palangi, Kai-Wei Chang, Yejin Choi, and Saadia Gabriel. 2025.

[3] Universal and Transferable Adversarial Attacks on Aligned Language Models. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023.

[4] H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinking. Martin Kuo, Jianyi Zhang, Aolin Ding, Qinsi Wang, Louis DiValentin, Yujia Bao, Wei Wei, Hai Li, and Yiran Chen. 2025.

Questions for authors
  1. How dependent are the safety gains on the system prompt? Appendix C.1 (p. 38) states that all benchmarks use the prompt in Appendix A.5 (p. 33), which explicitly asks the model to behave safely and reason about intent and consequences. Self-Reminder [5] shows that safety reminders can already improve safety without additional training. Are there any checks performed with a neutral prompt or without the explicit safety-reasoning instructions?

  2. What exactly differs between direct SFT and mid-training? Table 3 caption states that “only the consumption stage differs”, while the default settings in Appendices B.1–B.2 differ in loss masking, sequence packing, and training epochs. Please clarify the actual settings used for this comparison, including optimizer steps and training-token budgets. This would help identify which training changes were compared and how they contribute to the safety improvement.

Reference

[5] Defending ChatGPT against jailbreak attack via self-reminders. Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. 2023.

6 comments

Weakness 1 Adaptive jailbreak evaluation

Thank you for this helpful suggestion. We add an adaptive-attack evaluation covering all four methods suggested in the review, and Table R1 shows that safety midtraining retains an advantage under each attack. We target Qwen3-4B Base + balanced SFT and Ours + balanced SFT using the same 100 behaviors from JailbreakBench, with each attack run independently against each checkpoint. PAIR uses Qwen3.8-27B as the attacker with n = 3 and k = 2, and X-Teaming uses the same attacker with m = 2 and k = 2. We optimize GCG for 500 steps and use Qwen-3.8-27B-abliterated to generate the injected think content for H-CoT, keeping each method's configuration fixed across the target checkpoints. The reduction from 67.0% to 19.0% ASR under PAIR shows the benefit against model-specific attack search, while the reduction from 7.0% to 3.0% under H-CoT addresses the concern about exploiting safety reasoning itself. The advantage across all four evaluations provides direct support for the safety improvement under adaptive attacks.

Table R1 Adaptive attack results

CheckpointPAIR ASR (%)X-Teaming ASR (%)GCG ASR (%)H-CoT ASR (%)
Base + balanced SFT67.043.070.07.0
Ours + balanced SFT/19.030.024.03.0

Weakness 2 Direct SFT and midtraining comparison

Thank you for this comment. Table 3 isolates the loss objective by using identical training settings for direct SFT and midtraining. Both conditions start from the same base checkpoint and use the same reasoning-only corpus, with the epoch count and packing procedure held fixed. Appendix B.2 describes the subsequent SFT stage, while the direct-SFT control uses the midtraining configuration in Appendix B.1. The only change is whether loss is applied to assistant tokens or all tokens. Under this control, the all-token objective reduces ASR from 22.8% to 7.5%. Utility also increases from 68.9% to 74.7%, which is consistent with prior work showing that including instruction tokens in the loss can improve generalization [1]. Since these gains appear before subsequent SFT with the same training data and exposure, they support a contribution from the objective used to learn the reasoning corpus.

Weakness 3 Contribution of document records

Thank you for this comment, and sorry for the confusion. Our main claim is that unified reasoning traces generated from structured documents under a shared schema for safety and general tasks are key to the method's effectiveness, while adding the raw document records to the training corpus is optional. Table 38 shows that reasoning-only training preserves the main safety and utility gains, as also discussed in Appendix E.3. The limited marginal benefit of adding document records further supports our view that the main improvement comes from the carefully designed unified traces rather than simply increasing the data size. This highlights our central methodological contribution of constructing unified reasoning data that connects safety and general tasks.

Question 1 Dependence on the system prompt

Thank you for this question. We add a system-prompt ablation to test dependence on the safety instructions, and Table R2 shows that the safety advantage persists when this guidance is removed. In the no-safety-reminder setting, we retain the original prompt's reasoning requirements and remove its safety-related constraints, while the neutral setting uses only “you are a helpful assistant”. Both checkpoints receive the same prompt and evaluation examples under identical decoding settings within each comparison. Under the neutral prompt, Ours reaches 6.7% ASR compared with 22.8% for the baseline, while utility is higher at 73.6% compared with 58.9%. Across the tested prompts, our method shows limited sensitivity to changes in the system instructions and consistently achieves lower ASR and higher utility while keeping benign over-refusal close to or below the baseline. Among the evaluated approaches, it is the only one that maintains this favorable balance as the safety guidance is removed. These results further highlight the importance of our midtraining approach for learning safety as part of a shared reasoning capability that supports safe and useful responses across changes to system instructions.

Table R2 System prompt ablation

System promptCheckpointASR (%)Utility (%)Over-refusal (%)
OriginalBase + balanced SFT10.865.617.1
OriginalOurs + balanced SFT5.574.99.1
No safety reminderBase + balanced SFT13.265.823.4
No safety reminderOurs + balanced SFT7.172.39.1
Neutral assistantBase + balanced SFT22.858.911.5
Neutral assistantOurs + balanced SFT6.773.612.6

Question 2 Actual direct SFT training settings

Thank you for this question. The direct-SFT and midtraining runs in Table 3 differ only in the loss mask, while the training configuration follows Appendix B.1 in both cases. Their input-token exposure and optimizer-step budgets are matched, with the same epoch count and packing procedure. Direct SFT computes loss on assistant tokens, while midtraining computes loss on all tokens, so the number of tokens contributing to the loss changes as part of the objective itself. Appendix B.2 specifies the separate subsequent SFT stage. The comparison therefore measures the effect of the objective under the same data exposure and training schedule.

[1] Shi et al. Instruction Tuning With Loss Over Instructions. NeurIPS 2024.

Thank you again for your thoughtful comments and suggestions. We have carefully addressed each point in our response and would be grateful if you could let us know whether the clarifications resolve your concerns. We look forward to receiving any further feedback from you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.