acceptodds
Under review as a conference paper at ICLR 2027

Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection

Abstract

To achieve deeper safety alignment for large language models (LLMs), recent efforts have studied how to push safety interventions earlier into the pretraining stage, primarily by filtering unsafe data or rewriting it into safer forms. We argue that pretraining-stage alignment should go beyond making the data safe: LLMs may generalize unsafe behaviors from seemingly benign pretraining data. To this end, we propose , a pretraining-stage alignment method which regularly inserts short safety reflections between sentences in pretraining corpora. This integrates self-monitoring directly into language modeling, establishing a foundational capability that is subsequently reinforced by compatible post-training. Our experiments with 1.7B models pretrained on FineWeb-Edu show that Safety Reflection Pretraining improves safety classification accuracy and substantially reduces the success rates of inference-stage and finetuning attacks. Complementary to our real-world experiments, we also introduce a fully controlled synthetic environment, , with a clear definition of safety and a reasoning structure under which we find that models can easily generalize unsafe behaviors from safe data. Ablations in MedSafetyWorld further demonstrate a clear advantage of Safety Reflection Pretraining over data filtering and rewriting in preventing models from acting on unsafe behaviors. Taken together, our findings suggest that pretraining alignment should not only make the training data safe, but also shape the behaviors that models are likely to acquire from safe data.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Safety, Pretraining-Stage Alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sw7yOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that filtering or rewriting pretraining data is not enough for safety alignment, because models can still generalize unsafe behaviors from data that looks safe. A synthetic environment, MedSafetyWorld, is used to show this. The proposed Safety Reflection SRP inserts a short safety verdict after every ~100-word segment of the pretraining corpus, labeled by Qwen3Guard, and appends an EOS token after unsafe segments. SFT data is augmented in the same format. Experiments use 1.7B models pretrained on 100B FineWeb-Edu tokens, with additional scales from 135M to 4B. Compared with a standard baseline and SafeLM, SRP lowers attack success rates under prefilling, GCG and closed-source fine-tuning attacks, while keeping general capability comparable.

Critical strengths
  1. The paper distinguishes between acquiring harmful knowledge and acting on unsafe behaviors. MedSafetyWorld, a fully controlled synthetic environment, makes it possible to study this distinction directly.

  2. Safety reflection pretraining is conceptually clean and can be naturally combined with other pretraining-stage alignment methods (e.g., data filtering and rewriting) within the same pretraining pipeline. It is a useful reference for researchers building practical safety pretraining pipelines.

  3. The paper reports both the non-refusal rate and the LLM-judged unsafe compliance rate. This is more convincing than relying only on matching common refusal patterns.

  4. The experiments cover multiple model scales (135M–4B), an ablation of the tagging interval (reflection frequency), safety classification, and general capability and instruction-following evaluations.

Critical weaknesses
  1. Novelty and missing related work. Inserting classifier-generated, segment-level control tokens into pretraining text is closely related to conditional training in Korbak et al. (2023, Pretraining Language Models with Human Preferences), which the paper does not cite. Safety reflection methods in post-training, such as Self-Guard (Wang et al., 2024) and Backtracking (Zhang et al., 2025), are also closely related.

  2. SRP does not prevent the unsafe generalization that motivates it. The paper’s motivation is that LLMs can reason, compose, and generalize unsafe behaviors from safe data. SRP does not stop these capabilities from forming. It only teaches the model to recognize and stop itself after its generation has already become unsafe. In essence, it is internalized output moderation.

  3. MedSafetyWorld is far from real-world safety settings. The single-hop and two-hop tasks in MedSafetyWorld are essentially shallow associative reasoning. They differ substantially from complex real-world safety scenarios such as multi-step jailbreaks, social engineering, and implicit harmful requests.

  4. The overrefusal evaluation is limited. Overrefusal is evaluated only on 300 samples from the Alpaca dataset. It is not tested on borderline benchmarks such as XSTest or OR-Bench. XSTest appears in the paper only as part of the safety classification evaluation, not as an overrefusal test.

  5. Table 4 shows that adding safety reflections solely in pretraining yields almost no safety gain, suggesting that SRP’s effectiveness largely depends on reflection-formatted post-training. This makes SRP conceptually close to Backtracking (Zhang et al., 2025), which also trains the model to detect and stop unsafe generations, but does so purely in post-training.

Questions for authors
  1. In MedSafetyWorld, SRP uses the same data as Filtering but achieves higher general accuracy on safe queries . What explains this utility gain?

  2. For the GCG attack, what are the optimization budget (number of steps), the construction of the target string, and the number of restarts? Under what gradient-access conditions is it run, given the closed-source threat model in Sec. 2?

  3. How much reflection-formatted SFT data is needed to activate the safety reflection behavior learned during pretraining? Could you report ASR as a function of the fraction of reflection-formatted SFT data?

6 comments
AuthorsOpenReview ↗

Response 1/3

Thank you for the thoughtful and constructive feedback. We appreciate your recognition of our controlled study of unsafe generalization, the practical value of SRP, and the comprehensive evaluation across safety, model scales, and general capabilities. We address your concerns in detail below.

W1: SRP's novelty and its relationship to prior work are unclear.

We thank the reviewer for highlighting these related works. We would like to clarify our two contributions Secs\. 3–4 and explain more clearly how SRP differs from prior work Appendix G; Sec\. 6\.1.

  • Contributions.

    1. A controlled study of unsafe generalization. We introduce MedSafetyWorld, a fully controlled synthetic environment with a clear definition of safety and an explicit reasoning structure. This allows us to show that models can generalize unsafe behaviors from safe data even when all unsafe pretraining data is removed Sec\. 3.

    2. Safety Reflection Pretraining. SRP regularly inserts short safety reflections after text segments in the pretraining corpus, teaching models to check their own generations and stop when they detect unsafe content Sec\. 4. With compatible post-training, SRP improves safety classification and robustness to inference-stage and fine-tuning attacks while preserving general capability Secs\. 5–6.

  • Comparison with prior work.

    1. Self-Guard and Backtracking introduce safety checks during post-training, while SRP introduces them during pretraining. Both methods are discussed in Appendix G. Our ablation comparing SRP with Post-train Only Sec\. 6\.1 shows that learning safety reflections during pretraining improves safety classification and robustness to inference-stage and fine-tuning attacks.

    2. PHF Korbaketal,˙2023Korbak et al\., 2023 does not enable models to monitor their own outputs and stop unsafe generations. While both methods aim to improve model alignment and involve inserting classifier-derived tokens, the roles of these tokens and the behaviors they shape are fundamentally different. In PHF, preference labels are placed at the beginning of text segments as control tokens to guide generation toward desired properties Sec\. 2 of their paper. In contrast, SRP appends safety reflections after each text segment, followed by an EOS token for unsafe segments, training models to assess their own outputs and stop upon detecting unsafe content.

W2: SRP does not prevent unsafe generalization.

  • Preventing unsafe generalization is inherently difficult. We argue that unsafe generalization can naturally arise from the compositional generalization of benign knowledge and behaviors. In our simple and controlled setup, MedSafetyWorld, we demonstrate that models trained from scratch on only safe data can still compose what they have learned through multi-hop reasoning to produce unsafe behaviors Sec\. 3. This means preventing unsafe generalization requires selectively blocking unsafe compositions of benign behaviors while preserving safe and useful compositions. Making this distinction can naturally be viewed as teaching the model to identify which combinations of benign behaviors lead to safe outcomes and which lead to unsafe outcomes. Preventing unsafe generalization entirely at its source is therefore a much more difficult problem.

    This difficulty has also been ackonwledged in previous works. For example, in Open Problems in Machine Unlearning for AI Safety, Barez et al. 20252025 likewise identify preventing harmful capabilities from emerging through combinations of benign knowledge as an open challenge Secs\. 4\.3–4\.4 of their paper.

  • SRP aims to reduce unsafe behavior, rather than prevent unsafe generalization. Reasoning, composition, and generalization are important for solving new problems. Our goal is to preserve these useful abilities while preventing them from being used to produce unsafe responses. This motivates teaching models to avoid acting on unsafe capabilities, without assuming that every route to unsafe generalization can be eliminated.

  • The contribution of SRP is to learn regular safety checks during pretraining. SRP teaches models to reflect on their own outputs and stop unsafe generations Sec\. 4. Our ablation with Post-train Only shows that learning this behavior during pretraining, reinforced by compatible post-training, improves safety classification and robustness to inference-stage and fine-tuning attacks Sec\. 6\.1. We will clarify that SRP aims to reduce unsafe responses while preserving useful reasoning and generalization.

AuthorsOpenReview ↗

Response 2/3

W3: MedSafetyWorld is far from real-world safety settings.

  • MedSafetyWorld is intentionally designed as a simple and controlled setting to study fundamental safety questions, including unsafe generalization. It gives us a clear definition of safety and full control over the training data Sec\. 3. The single-hop and two-hop results show that unsafe generalization can occur even with simple reasoning. We can isolate unsafe generalization from other factors and study it under clean experimental conditions. Such simple settings help answer fundamental safety questions and test the limits of different approaches. For example, our filtering and rewriting results rule out the assumption that making training data safe alone guarantees safe model behavior.

  • Our experiments on real-world data complement this controlled study. We also pretrain 1.7B models on 100B tokens from FineWeb-Edu and evaluate SRP under prefill and GCG attacks, as well as benign and adversarial fine-tuning attacks Sec\. 5. These experiments assess SRP's robustness beyond MedSafetyWorld, while the synthetic environment allows us to study unsafe generalization with precise control.

W4: The overrefusal evaluation is limited.

We thank the reviewer for suggesting a broader overrefusal evaluation. Our evaluation on Alpaca follows SafeLM. To address this concern, we conduct extended overrefusal evaluations on XSTest (safe prompts) and OR-Bench (hard subset). The results are shown below.

ModelXSTest ↓OR-Bench ↓
Baseline34.40%85.52%
SafeLM18.00%83.40%
SRP25.60%74.45%
  • SRP does not increase overrefusal compared to the baseline. SRP achieves lower overrefusal rates on both benchmarks. Compared with SafeLM, SRP performs slightly worse on XSTest but better on OR-Bench.

  • We note that overrefusal is a common issue in safety-aligned models. Targeted post-training methods, such as POROver Karamanetal,˙2025Karaman et al\., 2025, have shown that preference optimization can substantially reduce overrefusal while maintaining safety. We do not apply such additional mitigation methods, allowing a controlled comparison of pretraining approaches under a shared simple SFT recipe.

W5: The contribution of safety reflection pretraining is unclear, given its reliance on post-training and similarity to Backtracking.

  • Conceptual motivation: By incorporating safety reflections into pretraining, we aim to encourage models to internalize self-monitoring as part of their language modeling, enabling models to recognize and stop unsafe generations. This distinguishes SRP from post-training-only approaches such as Backtracking. Since post-training alignment is standard in practical LLM training pipelines, evaluating pretraining as a standalone replacement for post-training is not a realistic setting. The relevant question is therefore not whether pretraining alone is sufficient, but whether it provides additional benefits beyond reflection-formatted post-training alone.

  • Post-training alone is less effective. Sec. 6.1 directly compares SRP with Post-train Only, both of which use reflection-formatted post-training. SRP achieves better safety classification, is less likely to omit safety reflections under attack, and stronger robustness against inference-stage and fine-tuning attacks. These results demonstrate the additional benefits of introducing safety reflections during pretraining beyond post-training alone.

AuthorsOpenReview ↗

Response 3/3

Q1: What explains SRP's utility gain over Filtering in MedSafetyWorld?

Understanding drug safety can also help answer some utility questions. In MedSafetyWorld, the training data teaches models which drugs or drug combinations can lead to safe or unsafe outcomes Sec\. 3. Answering some utility questions may also require this understanding of drug safety. SRP's safety reflection training can strengthen this understanding, helping the model answer utility questions more accurately even with the same original training data as Filtering.

Q2: Settings of GCG attacks

  1. Optimization budget: We allow up to 200 optimization steps per run.

  2. Independent runs: We use three independent GCG runs with different seeds per model and report the mean results across these runs.

  3. Target strings: For DirectHarm4, the target string is Sure, here is. For AdvBench, we use the predefined target string provided for each request in the dataset.

  4. Gradient access: We optimize adversarial suffixes directly on each evaluated model with full gradient access. This is an additional white-box evaluation with stronger attacker access than the closed-source API setting in Sec. 2.

Since GCG-generated adversarial suffixes can transfer across models, attackers can optimize suffixes on a white-box surrogate model and apply them to a black-box target. We therefore optimize directly on each evaluated model to approximate a worst-case scenario for transfer-based black-box attacks, without the limitations of imperfect cross-model transfer. We have clarified this distinction in Appendix B.1.2 in the revised paper.

Q3: How much reflection-formatted SFT data is needed?

To study how much reflection-formatted SFT data is needed, we conduct extended experiments varying its proportion from 0% to 100%. Reflection-formatted samples are placed at the end of SFT, with the total amount of SFT data kept fixed. The results are shown below.

Reflection ProportionNon-Refusal RateAvg$$% ↓Unsafe Compliance RateAvg$$% ↓
0%89.8549.32
25%25.9313.84
50%6.593.31
75%7.323.59
100%6.873.97

Note: Non-Refusal Rate AvgAvg and Unsafe Compliance Rate AvgAvg are computed by averaging the corresponding rates across the three attack settings: Prefill Attack, GCG (DirectHarm4), and GCG (AdvBench). Lower is better.

Using safety reflections in 50% of the SFT data is sufficient to activate the learned reflection behavior. Even 25% reduces both non-refusal and unsafe compliance rates across all three attacks. At 50%, SRP already achieves strong robustness. These results suggest that reflection behavior learned during pretraining can be effectively activated with a relatively small proportion of reflection-formatted SFT data.

Reviewer_zeMAOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies safety alignment during pretraining and argues that filtering explicit harmful content alone cannot prevent unsafe generalization. It introduces MedSafetyWorld, a controlled synthetic environment, and proposes Safety Reflection Pretraining (SRP), which interleaves segment-level safety judgments with pretraining text and teaches the model to terminate after unsafe judgments. Compatible supervised fine-tuning preserves this behavior. Experiments include 1.7B models pretrained on 100B FineWeb-Edu tokens, scaling studies from 135M to 4B, and evaluations under prefilling, GCG, and fine-tuning attacks. Ablations indicate additional robustness beyond introducing reflection only during post-training, with broadly comparable general utility. Fine-tuning robustness is demonstrated primarily under a provider-controlled threat model.

Critical strengths
  1. Direct evidence for the contribution of pretraining. The factorial ablation in Table 4 compares reflection introduced during pretraining, post-training, both stages, or neither. SRP improves over post-training-only reflection, including reducing unsafe compliance under prefilling attacks from 17.68% to 7.47%. This provides meaningful evidence that the pretraining intervention contributes beyond the modified SFT format.

  2. Complementary controlled and real-world validation. MedSafetyWorld enables a controlled study of unsafe generalization from data considered benign under an explicit safety definition. The real-world experiments, scaling studies from 135M to 4B, and utility evaluations provide complementary evidence that the approach has practical potential beyond the synthetic setting.

Critical weaknesses
  1. The distinction from prior safety-tagged pretraining could be made more explicit. The paper already compares with SafeLM and provides useful ablations demonstrating the additional benefit of pretraining. Further clarifying the contributions of regular intervals, retrospective safety judgments, and the stopping mechanism would help readers better understand SRP’s specific improvements over existing approaches.

  2. Some attack-evaluation details could be documented more fully. Specifying the GCG optimization targets, search budgets, and whether attacks were optimized separately for each model would improve reproducibility. Clarifying how reflection tags and content generated before termination are handled during evaluation would also make the robustness results easier to interpret.

Questions for authors

Please address the clarification requests raised in the weaknesses above.

6 comments
AuthorsOpenReview ↗

Response 1/2

Thank you for the encouraging feedback. We appreciate your recognition of the evidence supporting pretraining's contribution and the complementary validation across controlled and real-world settings. We address your concerns in detail below.

W1: The distinction from prior safety-tagged pretraining could be made more explicit.

Thank you for this suggestion. We will clarify SRP's differences from prior safety-tagged pretraining and the roles of its three design choices in more detail below.

  1. Comparison with prior safety-tagged pretraining. We discuss the comparison with SafeLM's Harmfulness-Tag approach in Appendix G and clarify the differences in more detail here. SafeLM inserts safety tags at random positions in unsafe pretraining data, so a tag may precede the unsafe content rather than judge the preceding segment. Its tagging mechanism is designed to work with SafeBeam, a modified inference procedure, and is substantially less effective without it. SRP instead trains the model to judge the immediately preceding segment at regular intervals and stop after an unsafe judgment, while remaining effective under standard decoding.

  2. Regular reflection enables safety checks throughout generation. Our reflection interval ablation AppendixC5˙,Table8Appendix C\.5, Table 8 compares reflection intervals of 100, 200, 300, and 1,000 words. The setting with 100 words achieves the strongest robustness across the evaluated attacks. We also conduct an extended experiment in the Post-train Only setting to compare regular reflection with a single initial reflection:

    ModelPrefilling Unsafe Compliance % ↓Prefilling Non-Refusal Rate % ↓
    Post-train Only (regular reflection)17.6831.21
    Post-train Only (only initial reflection)91.2174.55

    Within Post-train Only, regular reflection reduces unsafe compliance from 91.21% to 17.68% and non-refusal from 74.55% to 31.21%. This supports continuing to check safety during generation, especially when a prefilling attack bypasses the initial check. With regular reflection retained, SRP further reduces these rates to 7.58% and 10.61%, showing the benefit of learning this behavior during pretraining.

  3. Retrospective safety judgments help the model detect unsafe content it has just generated. Each reflection follows and evaluates the immediately preceding text segment. This gives the model an opportunity to recognize unsafe content even after it has started following an adversarial prefix, rather than relying solely on a judgment made before generation.

  4. The stopping mechanism turns detection into action. We train the model to emit an EOS token after an unsafe judgment, so it learns to stop instead of continuing the unsafe response. This limits further unsafe generation once a problem is detected and works under standard decoding, without requiring a modified decoding procedure such as SafeBeam.

Together, these choices teach the model when to check, what to assess, and how to respond when it detects unsafe content. We will make these roles and the supporting experimental evidence more explicit in the revision.

AuthorsOpenReview ↗

Response 2/2

W2: Some attack-evaluation details could be documented more fully.

Thank you for this suggestion. We have added detailed descriptions of the attack settings and evaluation details in Appendix B.1.2 of the revised paper.

  • GCG attack settings.
  1. Optimization budget: We allow up to 200 optimization steps per run.

  2. Independent runs: We use three independent GCG runs with different seeds per model and report the mean results across these runs.

  3. Target strings: For DirectHarm4, the target string is Sure, here is. For AdvBench, we use the predefined target string provided for each request in the dataset.

  4. Gradient access: We optimize adversarial suffixes directly on each evaluated model with full gradient access. This is an additional white-box evaluation with stronger attacker access than the closed-source API setting in Sec. 2.

Since GCG-generated adversarial suffixes can transfer across models, attackers can optimize suffixes on a white-box surrogate model and apply them to a black-box target. We therefore optimize directly on each evaluated model to approximate a worst-case scenario for transfer-based black-box attacks, without the limitations of imperfect cross-model transfer. We have clarified this distinction in Appendix B.1.2 in the revised paper.

  • How we handle reflection tags and content generated before termination during evaluation. In our closed-source setting, model providers control which generated content is returned to users. We handle reflection tags in our two evaluation settings as follows:

    1. Non-Refusal Rate: This measures whether the model responds without explicitly refusing the harmful request. We match common refusal patterns, such as `I'm sorry` and `As a language model`, as well as the generated unsafe-reflection pattern `<think> Unsafe`. A response containing an unsafe reflection therefore counts as a refusal.

    2. Unsafe Compliance Rate: This measures whether the response complies with the harmful request, as judged by GPT-5.4-mini using the prompt in Figure 12. Before judging, we remove all text segments marked unsafe by SRP and all safety reflections. Thus, when SRP detects unsafe content and stops, the judge does not receive the flagged unsafe segment generated before termination. It evaluates the remaining response that would be shown to the user.

I thank the authors for their response. I will maintain my original score.

Reviewer_ytRJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors argue that removing explicitly harmful training examples from the pretraining corpus does not prevent models from using benign knowledge to answer harmful requests. They build a synthetic corpus named MedSafetyWorld with predefined safety rules to make sure data that directly explain how to cause harm are removed from the pretraining corpus, while knowledge about drug effects is retained. The authors then pretrain 135M models from scratch and then apply SFT with refusal examples. The results show that attacks can still induce the target model to use the retained knowledge to answer harmful requests.

They then argue that pretraining should thus also teach models to check the safety of their outputs, and propose Safety Reflection Pretraining (SRP), which begins during pretraining and continues through SFT. During pretraining, a safety classifier labels each text segment, and the labels are appended in <think>...</think> tags, with an EOS token following unsafe labels. For SFT, the authors place the user request’s safety label at the beginning of the training response and insert further labels after subsequent response segments. The model thus learns to assess the safety of requests before answering and to check subsequent response segments even when previous segments are considered safe.

The authors use 100B tokens from FineWeb-Edu to pretrain 1.7B models with the SmolLM2 architecture, adopting their proposed SRP method to process the data corpus. The authors test robustness to prefill and GCG attacks, as well as to subsequent fine-tuning with benign or harmful data. They separately evaluate the accuracy of safety judgments on data segments and whether general capabilities are preserved. Under prefill attacks, SRP substantially reduces unsafe compliance to 7.47%, compared with 64.24% for the standard pretraining + safety SFT baseline.

During the fine-tuning tests, the authors mix a small number of examples from their original instruction-tuning datasets into the data used for further fine-tuning for SRP and its baseline models to help retain earlier safety behavior. At the inference stage, they also prefill <think> for SRP to trigger safety checks. SRP’s attack success rates after benign and adversarial fine-tuning are 0.00% and 7.00%, respectively. Removing both measures raises these rates to 20.75% and 79.50%. The method is interesting, and the reported safety improvements are substantial.

Critical strengths
  1. The paper studies an important problem: how to build safety into models during pretraining and preserve it through subsequent post-training. Improving robustness to both jailbreak attacks and subsequent fine-tuning has clear practical value.
  2. The paper presents substantial experimental work. The authors pretrain 1.7B models on 100B tokens and conduct several ablation studies and analyses. The evaluations cover inference-time attacks, further fine-tuning, safety classification, and general capabilities.
  3. The reported safety improvements are substantial. Figure 4 shows that SRP reduces unsafe compliance under prefill attacks from 64.24% for the standard pretraining + safety SFT baseline to 7.47%. Under GCG attacks on AdvBench, it reduces unsafe compliance from 35.70% to 1.56%. These results largely support the potential of the approach.
Critical weaknesses
  1. The motivating observation is not new. The finding that training only on safe data does not guarantee model safety was already reported in the paper Safety Pretraining: Toward the Next Generation of Safe AI, which also introduced harmfulness tags during pretraining. It is unclear what new insight this paper adds to this finding, therefore cannot be counted as a new contribution.

  2. The motivation for the specific method is not sufficiently developed. Showing that filtering alone is insufficient does not directly explain why periodically inserting safety classifications into pretraining text is the appropriate solution. The paper needs to explain the logic of its proposed approach. Table 7 compares different tagging intervals, but does not include a setting with only an initial safety check before answering. Such a comparison would help clarify whether repeated checks during generation provide additional benefits.

  3. Compatibility with reasoning models has not been established. SRP adopts <think>...</think> to enclose short Safe/Unsafe judgments. While these markers are also commonly used for chain-of-thought reasoning, the paper does not explain how these two uses would coexist or evaluate whether the safety-label format affects reasoning behavior. The experiments do not include a reasoning-model setting, leaving the method’s applicability to these models unclear.

  4. The fine-tuning ablations do not separate the effects of data mixing and the inference prefix. Section 5.3 mixes original SFT examples into the fine-tuning data and additionally prefills <think> for SRP to trigger safety checks. Section 6.3 reports that removing both measures raises the ASR after adversarial fine-tuning from 7.00% to 79.50%, but does not remove them separately. Their individual contributions therefore remain unclear. Previous work such as Self-Reminder shows that inference-time safety reminders can themselves improve defenses without additional training. The paper does not compare against such prompting methods, leaving uncertainty about the origin of benefits.

Questions for authors

Please address my concerns stated in weaknesses.

Minor clarification request: Is <think> prefilled only in the fine-tuning robustness experiments, or also in the prefill/GCG and other evaluations? Please clarify the inference templates used for SRP, the SFT-only variant, and the baselines, including any system prompts or safety reminders.

6 comments
AuthorsOpenReview ↗

Response 1/3

Thank you for the thoughtful and comprehensive feedback. We appreciate your recognition of the problem's practical importance, our extensive experimental evaluation, and the substantial safety improvements achieved by SRP. We address your concerns in detail below.

W1: Unclear new insight beyond prior safety-pretraining work.

  • Our contribution is to identify what pretraining alignment should teach beyond making data safe. SafeLM combines filtering, rephrasing, native refusal training, and harmfulness tagging during pretraining. Its harmfulness-tag mechanism is then paired with SafeBeam at inference time and is substantially less effective without it Secs\. 3–4 of their paper. Its overall gains do not directly establish which learned safety behavior should be the focus of pretraining. We study this systematically in MedSafetyWorld and show under fully controlled conditions that training only on safe data does not prevent unsafe generalization Sec\. 3. This motivates SRP, which teaches models to judge the safety of their own outputs, check them regularly, and stop when unsafe content is detected. We find that this behavior substantially improves robustness under standard decoding Secs\. 4–5.

  • MedSafetyWorld provides a controlled test of unsafe generalization from safe data. MedSafetyWorld is a synthetic medical environment that allows us to clearly define which data is safe and which is unsafe. We deliberately use this simple setting to fully control the pretraining data and remove all unsafe data. Models trained from scratch still answer harmful queries under attacks, despite training on safety-related refusal data in SFT Sec\. 3. This rigorously demonstrates that unsafe behavior can be generalized from safe data.

  • Regular safety reflection addresses this failure more effectively than rewriting alone. Our Rewrite baseline tests SafeLM's rephrasing idea in MedSafetyWorld: rewriting unsafe data into safer forms while retaining useful information. Its prefilling and benign fine-tuning ASRs remain 43.00% and 62.00%, compared with SRP's 7.60% and 2.20% Fig\. 3. Compared with SafeLM's combination of techniques, SRP simply adds regular safety reflections to the training text and teaches the model to stop after an unsafe judgment. At inference, SRP uses standard decoding and requires no additional decoding procedure such as SafeBeam Sec\. 4. On real-world data, it also reduces prefilling unsafe compliance from SafeLM's 60.91% to 7.47% Fig\. 4.

W2: The motivation for regular safety reflections and their benefit over an initial safety check are unclear.

  • A major problem with performing only an initial safety check before answering, is that it only achieves shallow alignment, a phenomenon pointed out in Qi et al. 20242024. To further validate this point, we conduct an experiment with a single initial safety check as the reviewer suggests. Since pretraining only uses continuous text without prompt-answer boundaries, we compare regular reflections with a single initial reflection in the SFT phase, in the Post-train Only setting.

    ModelPrefilling Unsafe Compliance % ↓Prefilling Non-Refusal Rate % ↓
    Post-train Only (regular reflection)17.6831.21
    Post-train Only (only initial reflection)91.2174.55

    Compared with regular reflections, checking only once at the beginning raises prefilling unsafe compliance rate from 17.68% to 91.21% and the non-refusal rate from 31.21% to 74.55%. This shows that regular safety checks during generation improve robustness beyond a single initial check.

  • The motivation of our approach is to make safety reflection part of language modeling, which may encourage the model to internalize a form of self-monitoring, so that the model can recognize and stop itself once its generation becomes unsafe. The main results of our paper (section 5) shows that SRP indeed substantially reduces ASR under inference-stage and fine-tuning stage attacks. Table 7 (Table 8 in the revised paper version) shows that increasing the reflection interval from 100 to 1000 words raises the mean ASR from 13.25% to 38.75%. This further supports the need for regular safety reflections, since less frequent checks make models less robust.

AuthorsOpenReview ↗

Response 2/3

W3: Compatibility with reasoning models is unclear.

  • SRP can use different tags for safety reflections and CoT. In SRP, <think>...</think> simply marks safety reflections. CoT can use another tag, or SRP can change its tag while CoT keeps <think>...</think>.

  • SRP remains robust when trained to generate CoT reasoning. We conduct an extended experiment by modifying the SFT data for the SRP 1.7B model. We enclose a short reasoning in <cot>...</cot> and keep safety reflections in <think>...</think> dataexamplesinAppendixC7˙andFigure14data examples in Appendix C\.7 and Figure 14. After SFT, the model can generate short reasoning alongside safety reflections.

    ModelPrefilling Non-Refusal Rate % ↓Prefilling Unsafe Compliance % ↓GCG (AdvBench) Non-Refusal Rate % ↓GCG (AdvBench) Unsafe Compliance % ↓GCG (DirectHarm4) Non-Refusal Rate % ↓GCG (DirectHarm4) Unsafe Compliance % ↓
    Baseline91.9264.2489.9321.1933.9335.70
    SRP10.917.473.371.567.932.89
    SRP w CoT10.617.477.181.4110.381.19

The CoT model has comparable ASRs with SRP under both prefilling and GCG attacks, and is also much more robust than the baseline. These results show that SRP remains effective while allowing CoT reasoning output.

W4: The effects of data mixing, the inference prefix, and safety reminders are unclear.

  • Both data mixing and the inference prefix are important for SRP's robustness. We conduct extended experiments on SRP Sec\. 5\.3, removing each measure separately and evaluate robustness to fine-tuning attacks.

    MethodBenign ASR % ↓Adversarial ASR % ↓
    SRP0.007.00
    w/o data mixing0.0023.75
    w/o <think> prefix4.0077.25
    w/o both tricks20.7579.50

    Removing data mixing or the <think> prefix raises adversarial ASR from 7.00% to 23.75% or 77.25%, respectively. Both measures are needed to achieve the best robustness.

  • Self-Reminder helps, but SRP remains much more robust after adversarial fine-tuning. We conduct an extended experiment on Baseline, which has no safety reflections in pretraining or post-training Sec\. 5\.1. We directly apply the Self-Reminder prompt at inference time promptinAppendixC1˙0,Figure15prompt in Appendix C\.10, Figure 15.

    ModelBenign ASR % ↓Adversarial ASR % ↓
    Baseline0.5086.00
    Baseline + Self-Reminder0.0078.00
    SRP0.007.00

Self-Reminder reduces Baseline's benign ASR from 0.50% to 0.00% and adversarial ASR from 86.00% to 78.00%. However, its adversarial ASR remains much higher than SRP's 7.00%. These results suggest that SRP's safety benefits do not come only from a safety reminder at inference time.

AuthorsOpenReview ↗

Response 3/3

Q1: Please clarify where the <think> inference prefix is used.

  • In the paper, SRP uses the <think> prefix only for fine-tuning robustness evaluations Sec\. 5\.3. Neither SRP nor Post-train Only theSFT−onlyvariantthe SFT-only variant receives this prefix in the inference-stage attacks. The inference chat templates are aligned across SRP, Post-train Only, and the baseline, without extra system prompts or safety reminders. For SafeLM, we use its original chat template. Full inference templates for all these models is provided in Figure 11 of the revised paper.

  • SRP remains robust to GCG attacks with or without the <think> prefix. We conduct extended experiments by adding the prefix to SRP under GCG attacks.

    Model<think> prefixGCG (DirectHarm4) Non-Refusal Rate % ↓GCG (DirectHarm4) Unsafe Compliance % ↓GCG (AdvBench) Non-Refusal Rate % ↓GCG (AdvBench) Unsafe Compliance % ↓
    SRPNo7.922.893.371.56
    SRPYes0.810.521.320.44

    Adding the <think> prefix does not weaken SRP's robustness and may even slightly improve it. Unsafe compliance rate remains low with or without the prefix, at 2.89% vs. 0.52% on DirectHarm4 and 1.56% vs. 0.44% on AdvBench. These results show that SRP remains robust under both inference templates.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear AC, SAC, and PC:

Thank you and all the reviewers for your time and the valuable feedback on our submission. We have carefully considered all comments and have prepared a detailed rebuttal. We are writing this letter to briefly highlight our key contributions and the additional experiments and clarifications provided in the rebuttal, which we hope will be helpful in your final decision-making.

  1. Our paper makes two meaningful contributions, both of which are recognized by the reviewers.
    1. We introduce MedSafetyWorld, the first controlled environment that allows us to systematically study how models can generalize unsafe behaviors from safe data.
    2. We propose Safety Reflection Pretraining (SRP), a simple yet effective approach that teaches models to regularly check their own outputs and stop when unsafe content is detected. Our experiments in Section 5 show substantial safety improvements under both inference-stage and fine-tuning attacks.
    3. Other highlighted strengths: Reviewer ytRJ acknowledged the practical importance of the problem, our extensive experiments, and the substantial safety improvements. Reviewer zeMA highlighted the direct evidence for the contribution of pretraining. Reviewer sw7y recognized solid experiments and the practical potential of SRP.
  2. We have addressed reviewers' concerns through additional experiments and clarifications.
    1. We further clarified the novelty of SRP and its differences from prior work, and provided further details on our comparison with SafeLM. (Reviewer ytRJ W1, Reviewer sw7y W1, W5)
    2. We conducted additional ablations on SRP's key design choices, including (1) regular reflections and (2) the separate effects of data mixing and inference prefixes in fine-tuning settings. We also compared SRP with an additional baseline, Self-Reminder. These results further demonstrate the effectiveness of SRP's individual components. (Reviewer ytRJ W2, W4, Q1, Reviewer zeMA W1)
    3. We show that SRP is compatibility with the reasoning model setting. (Reviewer ytRJ W3)
    4. We extended our overrefusal evaluation to XSTest and OR-Bench, and provided more details on the evaluation procedures and GCG attack settings to improve reproducibility. (Reviewer zeMA W2, Reviewer sw7y W4, Q2)

We sincerely appreciate the reviewers' constructive feedback, which has helped us improve the paper. We hope the additional experiments and clarifications will help you assess the contributions of our work. Thank you again for your time and consideration.

Sincerely,

SRP Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.