acceptodds
Under review as a conference paper at ICLR 2027

Learning LLM Safety as a Capability via Continued Pretraining

Abstract

Safety training for large language models is often organized around specific downstream objectives, such as alignment or safety classification. However, these tasks rely on a broader safety capability: the ability to use safety knowledge appropriately across different settings. Existing safety-oriented pretraining can expose models to relevant knowledge or examples of safe behavior, but it provides limited training on how that knowledge is applied to develop different safety capabilities. We introduce CapCPT, a two-stage continued pretraining approach that first learns safety knowledge and then learns capability-wise trajectories that connect this knowledge to different safety dimensions. The first stage trains on factual accounts and safety explanations derived from the same source documents. The second stage trains on capability-specific data generated from these knowledge pairs, with both non-QA and QA formats making the connection between knowledge and capability-specific reasoning explicit. Across three Qwen3 model scales, CapCPT improves all five safety dimensions over Base, whereas other CPT baselines show gains in some dimensions together with declines in others. Training on individual capabilities also produces benefits beyond the target dimension, indicating that these capabilities are related rather than isolated. The resulting models further achieve higher target-task performance under subsequent SFT and GRPO, while multi-dimensional GRPO improves all five capabilities from the CapCPT initialization. Together, these results support treating safety as a capability that can be developed during continued pretraining and then used across downstream safety tasks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, safety, pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_NXB5OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether Continued Pretraining (CPT) can improve model safety by learning shared knowledge and reasoning capabilities across different safety tasks. The authors propose CapCPT, which first performs Knowledge CPT on factual accounts and safety explanations generated from web documents, followed by Capability CPT on non-QA documents and QA reasoning trajectories to learn five safety capabilities. Experiments show that CapCPT improves performance across multiple safety benchmarks while largely preserving general capabilities. Ablation studies and downstream SFT/GRPO experiments further demonstrate the contribution of different training components and the transferability of the learned safety capabilities.

Critical strengths

The paper views safety training as learning shared knowledge and reasoning capabilities across tasks, rather than treating each task as an independent alignment problem. This perspective is valuable for improving transfer across safety tasks and supporting subsequent safety alignment training.

Critical weaknesses

The paper emphasizes improving multiple safety tasks by learning shared safety reasoning capabilities. However, the current ablations do not fully isolate the contribution of reasoning trajectories. Although the paper includes token-matched CPT baselines, these baselines differ in data sources and generation procedures. Could the authors provide a comparison using the same data generation pipeline and matched token budgets, but without explicit reasoning trajectories, to clarify whether the improvements come from learning shared safety reasoning rather than additional synthetic safety supervision?

Questions for authors

See Weaknesses.

Reviewer_NkXMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CapCPT, a two-stage continued pretraining approach for developing safety capabilities. It first trains on factual accounts and safety explanations derived from web sources, then on synthetic non-QA documents and QA examples targeting refusal, safety classification, cultural safety, social bias, and political balance. Experiments on Qwen3-1.7B, 4B, and 8B report improvements over Base across these dimensions, reduced over-refusal, and small changes in average general-task performance. Additional experiments on Qwen3-1.7B examine component contributions, cross-task transfer, and subsequent SFT and GRPO, comparing CapCPT with Base.

Critical strengths
  1. Concrete, context-sensitive data design. The generation templates explicitly contrast cases where evidence or context should change a judgment with cases where superficial reframing should preserve it. This provides a clear and implementable approach to teaching safety judgments beyond memorizing refusal patterns.
  2. Evaluation across several settings. The experiments cover multiple safety dimensions, benign-request over-refusal, and general capabilities across three model sizes. The subsequent SFT and GRPO experiments also investigate whether the benefits remain useful during further training, providing evidence beyond the initial checkpoint’s benchmark scores.
Critical weaknesses
  1. The comparisons do not isolate the contribution of the proposed data construction method. In Table 1, CapCPT, SP, AP, and General CPT use different source corpora. The observed gains may therefore arise from differences in source quality, coverage, or domain relevance, rather than the proposed knowledge-to-capability construction. Experiments using the same source material and matched training budgets would provide stronger evidence for the method itself.
  2. Training budgets are not strictly matched. In particular, AP uses roughly half as many tokens as CapCPT, making it difficult to separate the effect of the method from additional training. Equal-token comparisons would strengthen the conclusions.
  3. The downstream gains may reflect additional training rather than a specific advantage of CapCPT. The SFT and GRPO experiments compare CapCPT only with Base, which has not received the additional approximately 2.3B tokens of CPT. Other CPT methods might provide similar benefits simply through additional training. Without applying matched downstream training to SP, AP, and General CPT checkpoints, these experiments cannot establish that CapCPT is a particularly effective initialization.
  4. Incomplete coverage of closely related work. Section 5 does not discuss https://openreview.net/forum?id=Gequ8WzMdQ, which also develops safety-related capabilities during pretraining and examines their benefits with compatible post-training. Although its mechanism differs from CapCPT, it provides a relevant alternative approach. Discussing their similarities and differences, and where feasible providing an empirical comparison, would help clarify CapCPT’s distinct contribution.
Questions for authors
  1. Section 2.2 describes checks on the generated data. How many generated examples were manually inspected, and what types of errors were found?
  2. Why was the training order fixed as factual accounts → safety explanations → non-QA → QA? What motivates the chosen training order, and have you tested whether a different order changes the results? Additionally, please see the concerns raised in Critical Weaknesses regarding source corpora, training budgets, and downstream baseline comparisons.
Reviewer_VrUFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Full disclosure: I reviewed this paper and dictated my review to a speech-to-text model. The same has been cleaned up and presented below.

The paper investigates whether continued pretraining (CPT) on safety-related knowledge and reasoning trajectories can improve a model's safety capabilities and its performance under subsequent safety-focused SFT and RL. The proposed CapCPT method combines factual accounts and safety explanations with capability-specific training data covering refusal, safety classification, cultural safety, social bias, and political balance.

The authors evaluate the approach on Qwen3 models at three scales, comparing against base models, general CPT, Safety Pretraining, and Alignment Pretraining. They report improvements across several safety benchmarks, reduced over-refusal, and improved downstream performance following safety-specific SFT and GRPO.

The paper builds on existing literature on safety and alignment pretraining, including work studying the timing of safety interventions during language-model training.

Critical strengths
  1. An interesting extension of safety pretraining to multiple capabilities. The paper studies whether safety-related reasoning learned through continued pretraining can support multiple downstream objectives. The experiments across three model scales provide useful evidence that a shared CPT corpus can improve several safety dimensions simultaneously.

  2. Promising downstream-training results. The experiments suggest that capability-oriented CPT can provide a better initialization for subsequent safety-focused SFT and RL. The cross-task transfer and multi-dimensional GRPO experiments are particularly interesting, as they suggest that improvements from pretraining need not be limited to the specific task subsequently optimized.

Critical weaknesses
  1. Missing adversarial robustness evaluations. The paper evaluates harmful-request refusal using fixed benchmarks such as AIR-Bench and XSTest, but does not evaluate robustness against adaptive, model-targeted jailbreak attacks. This is a significant omission given the extensive literature showing that performance on standard harmful-request benchmarks is not sufficient to establish adversarial robustness. Strong automated attacks are especially important when comparing against prior safety-pretraining methods, for which adversarial robustness was a central evaluation dimension. Consequently, the reported results do not establish whether CapCPT provides improvements under adversarial pressure.

  2. Missing evaluations of safety degradation under benign fine-tuning. A central motivation for introducing safety during pretraining is that safety properties may be more resistant to degradation during subsequent training. Prior work has evaluated this through fine-tuning on benign data, including mathematical tasks that can nevertheless lead to safety degradation. The paper does not conduct such experiments. Instead, its downstream-training evaluations involve safety-related SFT and RL objectives. Although these experiments demonstrate useful transfer between safety tasks, they do not establish whether the learned safety properties persist under unrelated downstream optimization. This is an important limitation of the paper's claims about safety as a reusable capability, particularly in light of the closely related literature on when safety interventions should be introduced during training.

  3. Limited conceptual novelty relative to prior work. The central premise of this paper—that safety can be learned during pretraining and that doing so can improve subsequent alignment—has already been explored extensively in prior work, including Safety Pretraining, Alignment Pretraining, and more recent work investigating "When Should We Introduce Safety Interventions During Pretraining?" [Sam et. al. 2026]. The primary novelty here appears to lie in the construction of capability-specific synthetic reasoning data and the joint training of multiple safety objectives, rather than in a fundamentally new approach to safety training.

While this is a potentially useful methodological contribution, the paper does not sufficiently isolate which aspects of the proposed data construction are responsible for the observed improvements. In particular, it remains unclear whether the benefits arise from explicitly learning shared reasoning capabilities, from increased exposure to relevant safety scenarios, or simply from the structure and composition of the synthetic training corpus. Stronger controlled comparisons would help establish what new scientific insight this work contributes beyond existing safety-pretraining approaches. As currently presented, I view the contribution as primarily incremental relative to this literature.

Questions for authors

See weaknesses

Reviewer_rh2SOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CAPCPT, a two-stage continued pretraining framework that combines shared factual knowledge with capability-specific reasoning trajectories to improve five safety-related capabilities, including harmful-request refusal, safety classification, cultural safety, social bias, and political balance, and evaluates its effectiveness across three Qwen3 model scales, general benchmarks, ablation studies, and subsequent SFT and GRPO training.

Critical strengths
  1. The paper addresses an important problem in language model safety by investigating whether shared knowledge and reasoning processes can support multiple safety-related capabilities through continued pretraining.
  2. The proposed two-stage training framework combines factual knowledge acquisition with capability-specific reasoning trajectories, providing a relatively systematic approach to multi-dimensional safety training.
  3. The experimental investigation covers multiple model scales and training configurations, including CPT, SFT, and GRPO, and provides preliminary evidence of cross-task transfer and reduced over-refusal.
Critical weaknesses
  1. The proposed CAPCPT framework appears to combine existing continued pretraining, instruction pretraining, and reasoning-based data construction strategies. The manuscript does not sufficiently clarify its fundamental methodological novelty over existing safety pretraining and instruction-oriented CPT approaches.
  2. The central assumption that five substantially different safety capabilities rely on shared reasoning processes is insufficiently formalized. The manuscript does not establish what constitutes shared safety reasoning or how these common reasoning components are distinguished from task-specific knowledge and behavioral preferences.
  3. The two-stage framework assumes that factual knowledge acquisition followed by capability-specific reasoning trajectories provides complementary benefits. However, the authors do not sufficiently establish whether the two stages learn genuinely distinct representations or whether the observed improvements simply result from additional training data and optimization steps.
  4. The effectiveness of CAPCPT heavily depends on the quality, diversity, and reliability of the generated safety explanations and reasoning trajectories. The manuscript does not sufficiently establish how factual correctness, reasoning validity, cultural diversity, and political neutrality are ensured, or how potential biases and inconsistencies in the training corpus are controlled.
  5. Although the paper presents several ablation studies, it remains unclear whether the observed improvements are attributable to the proposed shared-reasoning strategy rather than differences in training data volume, data composition, training duration, or optimization configurations. More carefully controlled comparisons are needed to establish the specific contribution of each component.
  6. The reported transfer effects are primarily analyzed using the 1.7B model. Improvements in selected cross-task measurements do not necessarily establish systematic transfer of shared reasoning capabilities. The manuscript should provide stronger evidence that such transfer is consistent across model scales and different capability combinations.
  7. The five evaluation dimensions may not adequately capture the complexity of real-world safety behavior. In particular, the manuscript does not sufficiently demonstrate robustness against adversarial prompts, ambiguous contexts, distribution shifts, and conflicting safety requirements. Improvements in benchmark scores alone are insufficient to establish more reliable safety reasoning.
  8. The reported SFT and GRPO improvements are primarily based on the 1.7B model, limiting the generality of the conclusions. Furthermore, the different classification outcomes under multi-dimensional GRPO for CAPCPT and Base suggest that the results may depend strongly on initialization and reward design. The manuscript does not sufficiently explain the underlying mechanism or establish the robustness of these improvements under alternative reward configurations.
Questions for authors
  1. How do the authors formally characterize and identify the shared reasoning processes across harmful-request refusal, safety classification, cultural safety, social bias, and political balance? What evidence demonstrates that these capabilities genuinely share reusable reasoning components rather than simply benefiting from exposure to overlapping training data?
  2. How do the authors ensure the factual accuracy, reasoning quality, cultural representativeness, and political neutrality of the constructed training corpus? Are there systematic human evaluations or independent quality-control procedures to assess the reliability and potential biases of the generated reasoning trajectories?
  3. Have the authors conducted strictly controlled experiments with matched training tokens, data volume, computational budgets, and optimization configurations to isolate the benefits of the proposed two-stage CAPCPT framework from those of additional CPT data and training steps?
  4. How robust are the reported cross-task transfer and multi-dimensional GRPO improvements across different model scales, reward configurations, and evaluation distributions? In particular, can the authors explain why the same GRPO reward mixture causes classification performance to improve with CAPCPT initialization but substantially deteriorate with Base initialization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.