- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that filtering or rewriting pretraining data is not enough for safety alignment, because models can still generalize unsafe behaviors from data that looks safe. A synthetic environment, MedSafetyWorld, is used to show this. The proposed Safety Reflection SRP inserts a short safety verdict after every ~100-word segment of the pretraining corpus, labeled by Qwen3Guard, and appends an EOS token after unsafe segments. SFT data is augmented in the same format. Experiments use 1.7B models pretrained on 100B FineWeb-Edu tokens, with additional scales from 135M to 4B. Compared with a standard baseline and SafeLM, SRP lowers attack success rates under prefilling, GCG and closed-source fine-tuning attacks, while keeping general capability comparable.
Critical strengths
-
The paper distinguishes between acquiring harmful knowledge and acting on unsafe behaviors. MedSafetyWorld, a fully controlled synthetic environment, makes it possible to study this distinction directly.
-
Safety reflection pretraining is conceptually clean and can be naturally combined with other pretraining-stage alignment methods (e.g., data filtering and rewriting) within the same pretraining pipeline. It is a useful reference for researchers building practical safety pretraining pipelines.
-
The paper reports both the non-refusal rate and the LLM-judged unsafe compliance rate. This is more convincing than relying only on matching common refusal patterns.
-
The experiments cover multiple model scales (135M–4B), an ablation of the tagging interval (reflection frequency), safety classification, and general capability and instruction-following evaluations.
Critical weaknesses
-
Novelty and missing related work. Inserting classifier-generated, segment-level control tokens into pretraining text is closely related to conditional training in Korbak et al. (2023, Pretraining Language Models with Human Preferences), which the paper does not cite. Safety reflection methods in post-training, such as Self-Guard (Wang et al., 2024) and Backtracking (Zhang et al., 2025), are also closely related.
-
SRP does not prevent the unsafe generalization that motivates it. The paper’s motivation is that LLMs can reason, compose, and generalize unsafe behaviors from safe data. SRP does not stop these capabilities from forming. It only teaches the model to recognize and stop itself after its generation has already become unsafe. In essence, it is internalized output moderation.
-
MedSafetyWorld is far from real-world safety settings. The single-hop and two-hop tasks in MedSafetyWorld are essentially shallow associative reasoning. They differ substantially from complex real-world safety scenarios such as multi-step jailbreaks, social engineering, and implicit harmful requests.
-
The overrefusal evaluation is limited. Overrefusal is evaluated only on 300 samples from the Alpaca dataset. It is not tested on borderline benchmarks such as XSTest or OR-Bench. XSTest appears in the paper only as part of the safety classification evaluation, not as an overrefusal test.
-
Table 4 shows that adding safety reflections solely in pretraining yields almost no safety gain, suggesting that SRP’s effectiveness largely depends on reflection-formatted post-training. This makes SRP conceptually close to Backtracking (Zhang et al., 2025), which also trains the model to detect and stop unsafe generations, but does so purely in post-training.
Questions for authors
-
In MedSafetyWorld, SRP uses the same data as Filtering but achieves higher general accuracy on safe queries . What explains this utility gain?
-
For the GCG attack, what are the optimization budget (number of steps), the construction of the target string, and the number of restarts? Under what gradient-access conditions is it run, given the closed-source threat model in Sec. 2?
-
How much reflection-formatted SFT data is needed to activate the safety reflection behavior learned during pretraining? Could you report ASR as a function of the fraction of reflection-formatted SFT data?
6 comments
Response 2/3
W3: MedSafetyWorld is far from real-world safety settings.
-
MedSafetyWorld is intentionally designed as a simple and controlled setting to study fundamental safety questions, including unsafe generalization. It gives us a clear definition of safety and full control over the training data Sec\. 3. The single-hop and two-hop results show that unsafe generalization can occur even with simple reasoning. We can isolate unsafe generalization from other factors and study it under clean experimental conditions. Such simple settings help answer fundamental safety questions and test the limits of different approaches. For example, our filtering and rewriting results rule out the assumption that making training data safe alone guarantees safe model behavior.
-
Our experiments on real-world data complement this controlled study. We also pretrain 1.7B models on 100B tokens from FineWeb-Edu and evaluate SRP under prefill and GCG attacks, as well as benign and adversarial fine-tuning attacks Sec\. 5. These experiments assess SRP's robustness beyond MedSafetyWorld, while the synthetic environment allows us to study unsafe generalization with precise control.
W4: The overrefusal evaluation is limited.
We thank the reviewer for suggesting a broader overrefusal evaluation. Our evaluation on Alpaca follows SafeLM. To address this concern, we conduct extended overrefusal evaluations on XSTest (safe prompts) and OR-Bench (hard subset). The results are shown below.
| Model | XSTest ↓ | OR-Bench ↓ |
|---|---|---|
| Baseline | 34.40% | 85.52% |
| SafeLM | 18.00% | 83.40% |
| SRP | 25.60% | 74.45% |
-
SRP does not increase overrefusal compared to the baseline. SRP achieves lower overrefusal rates on both benchmarks. Compared with SafeLM, SRP performs slightly worse on XSTest but better on OR-Bench.
-
We note that overrefusal is a common issue in safety-aligned models. Targeted post-training methods, such as POROver , have shown that preference optimization can substantially reduce overrefusal while maintaining safety. We do not apply such additional mitigation methods, allowing a controlled comparison of pretraining approaches under a shared simple SFT recipe.
W5: The contribution of safety reflection pretraining is unclear, given its reliance on post-training and similarity to Backtracking.
-
Conceptual motivation: By incorporating safety reflections into pretraining, we aim to encourage models to internalize self-monitoring as part of their language modeling, enabling models to recognize and stop unsafe generations. This distinguishes SRP from post-training-only approaches such as Backtracking. Since post-training alignment is standard in practical LLM training pipelines, evaluating pretraining as a standalone replacement for post-training is not a realistic setting. The relevant question is therefore not whether pretraining alone is sufficient, but whether it provides additional benefits beyond reflection-formatted post-training alone.
-
Post-training alone is less effective. Sec. 6.1 directly compares SRP with Post-train Only, both of which use reflection-formatted post-training. SRP achieves better safety classification, is less likely to omit safety reflections under attack, and stronger robustness against inference-stage and fine-tuning attacks. These results demonstrate the additional benefits of introducing safety reflections during pretraining beyond post-training alone.
Response 3/3
Q1: What explains SRP's utility gain over Filtering in MedSafetyWorld?
Understanding drug safety can also help answer some utility questions. In MedSafetyWorld, the training data teaches models which drugs or drug combinations can lead to safe or unsafe outcomes Sec\. 3. Answering some utility questions may also require this understanding of drug safety. SRP's safety reflection training can strengthen this understanding, helping the model answer utility questions more accurately even with the same original training data as Filtering.
Q2: Settings of GCG attacks
-
Optimization budget: We allow up to 200 optimization steps per run.
-
Independent runs: We use three independent GCG runs with different seeds per model and report the mean results across these runs.
-
Target strings: For DirectHarm4, the target string is
Sure, here is. For AdvBench, we use the predefined target string provided for each request in the dataset. -
Gradient access: We optimize adversarial suffixes directly on each evaluated model with full gradient access. This is an additional white-box evaluation with stronger attacker access than the closed-source API setting in Sec. 2.
Since GCG-generated adversarial suffixes can transfer across models, attackers can optimize suffixes on a white-box surrogate model and apply them to a black-box target. We therefore optimize directly on each evaluated model to approximate a worst-case scenario for transfer-based black-box attacks, without the limitations of imperfect cross-model transfer. We have clarified this distinction in Appendix B.1.2 in the revised paper.
Q3: How much reflection-formatted SFT data is needed?
To study how much reflection-formatted SFT data is needed, we conduct extended experiments varying its proportion from 0% to 100%. Reflection-formatted samples are placed at the end of SFT, with the total amount of SFT data kept fixed. The results are shown below.
| Reflection Proportion | Non-Refusal RateAvg$$% ↓ | Unsafe Compliance RateAvg$$% ↓ |
|---|---|---|
| 0% | 89.85 | 49.32 |
| 25% | 25.93 | 13.84 |
| 50% | 6.59 | 3.31 |
| 75% | 7.32 | 3.59 |
| 100% | 6.87 | 3.97 |
Note: Non-Refusal Rate and Unsafe Compliance Rate are computed by averaging the corresponding rates across the three attack settings: Prefill Attack, GCG (DirectHarm4), and GCG (AdvBench). Lower is better.
Using safety reflections in 50% of the SFT data is sufficient to activate the learned reflection behavior. Even 25% reduces both non-refusal and unsafe compliance rates across all three attacks. At 50%, SRP already achieves strong robustness. These results suggest that reflection behavior learned during pretraining can be effectively activated with a relatively small proportion of reflection-formatted SFT data.
Response 1/3
Thank you for the thoughtful and constructive feedback. We appreciate your recognition of our controlled study of unsafe generalization, the practical value of SRP, and the comprehensive evaluation across safety, model scales, and general capabilities. We address your concerns in detail below.
We thank the reviewer for highlighting these related works. We would like to clarify our two contributions Secs\. 3–4 and explain more clearly how SRP differs from prior work Appendix G; Sec\. 6\.1.
Contributions.
A controlled study of unsafe generalization. We introduce MedSafetyWorld, a fully controlled synthetic environment with a clear definition of safety and an explicit reasoning structure. This allows us to show that models can generalize unsafe behaviors from safe data even when all unsafe pretraining data is removed Sec\. 3.
Safety Reflection Pretraining. SRP regularly inserts short safety reflections after text segments in the pretraining corpus, teaching models to check their own generations and stop when they detect unsafe content Sec\. 4. With compatible post-training, SRP improves safety classification and robustness to inference-stage and fine-tuning attacks while preserving general capability Secs\. 5–6.
Comparison with prior work.
Self-Guard and Backtracking introduce safety checks during post-training, while SRP introduces them during pretraining. Both methods are discussed in Appendix G. Our ablation comparing SRP with Post-train Only Sec\. 6\.1 shows that learning safety reflections during pretraining improves safety classification and robustness to inference-stage and fine-tuning attacks.
PHF Korbaketal,˙2023 does not enable models to monitor their own outputs and stop unsafe generations. While both methods aim to improve model alignment and involve inserting classifier-derived tokens, the roles of these tokens and the behaviors they shape are fundamentally different. In PHF, preference labels are placed at the beginning of text segments as control tokens to guide generation toward desired properties Sec\. 2 of their paper. In contrast, SRP appends safety reflections after each text segment, followed by an EOS token for unsafe segments, training models to assess their own outputs and stop upon detecting unsafe content.
Preventing unsafe generalization is inherently difficult. We argue that unsafe generalization can naturally arise from the compositional generalization of benign knowledge and behaviors. In our simple and controlled setup, MedSafetyWorld, we demonstrate that models trained from scratch on only safe data can still compose what they have learned through multi-hop reasoning to produce unsafe behaviors Sec\. 3. This means preventing unsafe generalization requires selectively blocking unsafe compositions of benign behaviors while preserving safe and useful compositions. Making this distinction can naturally be viewed as teaching the model to identify which combinations of benign behaviors lead to safe outcomes and which lead to unsafe outcomes. Preventing unsafe generalization entirely at its source is therefore a much more difficult problem.
This difficulty has also been ackonwledged in previous works. For example, in Open Problems in Machine Unlearning for AI Safety, Barez et al. 2025 likewise identify preventing harmful capabilities from emerging through combinations of benign knowledge as an open challenge Secs\. 4\.3–4\.4 of their paper.
SRP aims to reduce unsafe behavior, rather than prevent unsafe generalization. Reasoning, composition, and generalization are important for solving new problems. Our goal is to preserve these useful abilities while preventing them from being used to produce unsafe responses. This motivates teaching models to avoid acting on unsafe capabilities, without assuming that every route to unsafe generalization can be eliminated.
The contribution of SRP is to learn regular safety checks during pretraining. SRP teaches models to reflect on their own outputs and stop unsafe generations Sec\. 4. Our ablation with Post-train Only shows that learning this behavior during pretraining, reinforced by compatible post-training, improves safety classification and robustness to inference-stage and fine-tuning attacks Sec\. 6\.1. We will clarify that SRP aims to reduce unsafe responses while preserving useful reasoning and generalization.