acceptodds
Under review as a conference paper at ICLR 2027

Beyond Refusal Rates: How SFT and RL Shape LLM Safety Discernment

Abstract

Safety alignment aims to make large language models helpful without facilitating harm. We investigate how supervised fine-tuning (SFT) and reinforcement learning (RL) shape safety alignment, using group relative policy optimization (GRPO) as a representative RL method. Starting from the same pretrained model and safety prompts, the resulting models achieve similar refusal rates on harmful requests and similar accuracy on math and reasoning benchmarks. Yet GRPO rejects fewer benign requests and more often returns to refusal after an attack forces its response to begin by agreeing to help. These differences concern safety discernment: the ability to answer benign requests and refuse harmful ones despite misleading context. To understand this contrast, we combine training ablations with internal analyses. Rewarding appropriate answers and refusals, rather than harmless content alone, helps GRPO avoid unnecessary refusals. We also find that reinforcing better responses while suppressing worse ones is sufficient to learn recovery after the forced opening. Internal interventions link this recovery to the continued influence of refusal-related information after the forced opening. Together, these findings connect safety discernment to specific training signals and reveal behavioral differences that refusal rates and task accuracy alone fail to capture.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM alignment, Safety, Robustness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_x1pMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper compares how SFT and GRPO teach models to refuse harmful requests while still answering benign ones. In the primary setting, models with similar ordinary refusal rates behave differently on benign requests and after their responses are forced to begin with a helpful phrase such as “I can help with that.” Training ablations examine which feedback produces these differences. In particular, Offline Signed Learning (OSL) increases the likelihood of better self-generated responses and decreases that of worse ones using a fixed response bank. The paper finds that this can produce substantial prefix recovery without prefix-specific training examples, and investigates associated activation changes.

Critical strengths
  1. The evaluation reveals safety differences hidden by ordinary refusal rates. The paper separately measures refusal of harmful requests, mistaken refusal of benign requests, and recovery after a forced helpful opening. Models that look similar on the first measure differ substantially on the others. This provides a useful explanation of why aggregate refusal and general-task accuracy are insufficient to characterize safety behavior.

  2. The training controls help identify a consequential learning signal. OSL and its controls start from the same checkpoint and use a shared bank of ordinary self-generated responses. Positive-only training and fitting the best response do not reproduce OSL’s strong recovery. These comparisons support the interesting finding that suppressing worse responses, alongside reinforcing better ones, can produce recovery without explicitly training on the tested prefixes.

Critical weaknesses
  1. Recovery improvements are not fully evaluated alongside retained capabilities. The primary Qwen comparison reports similar general-task accuracy, but the additional Llama experiment shows a substantial difference between SFT and GRPO: 47.57 versus 37.26. The central OSL comparison reports refusal and recovery outcomes without equally complete general-task evaluation. It is therefore unclear whether the strongest recovery gains preserve useful capabilities and how they compare at matched ordinary safety levels.

  2. The main recovery results need more independent evaluation. WildGuard judges generated responses during training and supplies much of the behavioral evaluation. This creates a possibility that the trained models become better aligned with that evaluator’s judgments without an equivalent improvement under independent assessment. The Llama Guard check on natural outputs is useful, but does not sufficiently validate the central OSL and forced-prefix results, where substantive harmful content should be assessed separately from refusal wording.

  3. The specific contribution of OSL needs clearer positioning against prior learning objectives. The paper shows that a fixed bank of responses with positive and negative coefficients can teach prefix recovery. Prefix vulnerability is already established [1], and the paper acknowledges the connection to GPG [2]. It should also discuss unlikelihood training [3] and CUT [4], which provide relevant precedents for discouraging undesirable outputs. A matched negative-feedback baseline would clarify whether recovery depends on OSL’s particular objective or reflects a broader effect of learning from negative responses.

  4. The tested prefixes cover a narrow family of perturbations. H5 forces a short helpful opening, and HC10 extends a closely related opening. These tests convincingly expose a failure of ordinary refusal, but do not establish recovery across substantially different wording or deeper forced continuations. The conclusions should distinguish recovery on the tested prefixes from more general robustness after an unsafe generation has begun.

  5. The relationship to RECAP [5] needs further clarification. Both works investigate how reward-based training enables models to recover from misleading prefixes and produce safe responses. The paper should explain more explicitly how its findings differ from and extend those of RECAP.

[1] Safety Alignment Should Be Made More Than Just a Few Tokens Deep. 2025.

[2] GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning. 2026.

[3] Neural Text Generation with Unlikelihood Training. 2020.

[4] Reasons to Reject? Aligning Language Models with Judgments. 2024.

[5] Large Reasoning Models Learn Better Alignment from Flawed Thinking. 2025.

Questions for authors

Please refer to my weakness.

Reviewer_iMnZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies safety alignment in Large Language Models (LLMs) and the differences between supervised fine-tuning (SFT) and reinforcement learning (RL). Their experimental motivation centers around the question: How do SFT and RL shape LLM safety discernment? To answer this question, they perform controlled experiments: starting from the same small-sized pretrained LLMs and safety prompts, they perform safety evaluations on the SFT- and RL-trained models. They find that SFT- and RL-trained models can achieve similar safety performance on math and reasoning tasks. The main difference is that GRPO-trained models reject fewer benign requests and are more robust to adversarial attacks that ask them to behave by agreeing to help. In their experiments, the main findings show that rewarding appropriate answers and refusals, rather than harmless content alone, helps GRPO avoid unnecessary refusals. The authors claim that their findings connect safety discernment to specific training signals and highlight the importance of training signals and behavioral differences.

Critical strengths

(1) Interesting and important research questions: The paper investigates LLM safety issues and the underlying mechanisms of SFT and RL in training LLMs. If proven generalizable, these findings are valuable for developing trustworthy LLMs.

(2) The paper is well written and easy to follow.

Critical weaknesses

(1) Limited evaluation. The experiments have significant limitations in terms of RL algorithms, model families, model sizes, and evaluation task types. For example, they only consider GRPO as the RL algorithm, while the central claim of this paper is targeted at the differences between SFT and RL. GRPO alone cannot represent the whole family of RL algorithms. In addition, they test only Qwen2.5 and Llama 3 series models, both released in 2024. Whether the conclusion generalizes to more recent models, such as the Qwen3.5 model family, remains unknown. Also, their tested model size is limited to 3B, which leaves the question of whether this conclusion generalizes to larger or smaller models. Their evaluation benchmarks also only contain single-turn tasks; whether the conclusion generalizes to multi-turn agentic tasks also remains unknown.

Questions for authors

(1) Can you further test DPO in your experiments as an intermediate algorithm between GRPO and SFT? How does it compare to other offline methods?

(2) How would the interpreted mechanism generalize to more recent models, for example, the Qwen3.5 series? How would the mechanism generalize to other model sizes? Could you also report the performance of larger models in the same model families, for example, Qwen2.5-7B and Llama3.1-8B?

(3) How would you compare your conclusion on the roles of SFT and RL and their effects with the findings revealed in the general context of LLM reasoning? For example, SFT memorizes; RL generalizes. [1]

Reference:

[1] SFT memorizes, RL generalizes: A comparative study of foundation model post-training. ICML 2025

Reviewer_zW5TOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how SFT and GRPO affect safety behavior beyond ordinary refusal rates. The primary comparison starts from the same pretrained model and training prompts and examines benign over-refusal and recovery after a forced helpful opening. The authors use training ablations to study the role of response-specific behavior feedback and introduce Offline Signed Learning (OSL), which reinforces higher-reward responses and suppresses lower-reward responses from a frozen collection of model-generated answers. OSL acquires substantial prefix recovery without training on attack prefixes. Activation interventions examine how refusal-related information introduced before the prefix influences the subsequent response.

Critical strengths

The paper is generally clear and well organized. The progression from behavioral differences to training ablations and internal interventions is easy to follow. Section 6.2 appropriately explains that the findings concern training signals, rather than guarantees associated with the names SFT or RL. The appendix provides useful implementation details and states the limits of the internal analysis.

The evaluation perspective has practical value. Ordinary harmful-request refusal, benign over-refusal, and recovery after a misleading opening capture different behaviors. The paper shows why they should be measured separately. The harmful-output results for the primary SFT and GRPO models in Table 3 also help connect prefix recovery to substantive safety outcomes.

Critical weaknesses

1. The main SFT–GRPO comparison does not fully separate the effects of the learning objective and response supervision. SFT uses one demonstration per prompt over two epochs, whereas GRPO samples eight responses per prompt over one epoch. The shared initialization and prompts are useful controls, but response diversity and training exposure still differ. Table 10 partly addresses checkpoint choice: at similar ordinary harmful-refusal rates, the average benign-refusal gap falls from 9.71 to 3.06 percentage points. This supports a smaller advantage, while showing that its size depends on the comparison point. Training curves under comparable response-token budgets would help clarify this effect.

2. The reported effects have not been tested across independent training runs. Appendix H states that each training arm has one trajectory. The large H5 recovery gap is promising, but the stability of the method ranking and the smaller benign-refusal gains remain uncertain. Variation across training and response-sampling seeds is unreported, leaving the repeatability of the central signed-learning result and the smaller benign-refusal gains uncertain.

**3. The safety and utility evaluation is incomplete for OSL. ** Its reported results primarily concern refusal rates. Independent harmful-output evaluation under natural and H5 conditions is absent. Table 7 provides such an independent check for natural SFT and GRPO outputs, but does not cover OSL. General-task accuracy and the quality of OSL's answers to benign requests are also unreported. The evidence therefore leaves open whether its recovery gains are accompanied by less harmful content and preserved answer quality. More broadly, the comparable-utility observation is limited to the primary Qwen comparison: Table 24 reports average accuracy of 47.57% for Llama SFT and 37.26% for Llama GRPO.

Questions for authors
  1. How do the benign-refusal and prefix-recovery gaps change when SFT and GRPO are compared across checkpoints under similar response-token budgets and ordinary harmful-refusal rates?

  2. Does OSL retain its recovery advantage when best-response SFT uses the same loss reduction and learning-rate schedule? How does positive-only OSL compare at a similar ordinary harmful-refusal rate?

  3. Can the authors report OSL's harmful-output rates under natural, H5, and HC10 conditions using an independent evaluator, together with its general-task accuracy and the quality of its answers to benign requests?

  4. Are the Table 4 results stable across independent training and rollout seeds? Please report the variation in recovery and benign-refusal rates.

  5. What explains the lower general-task accuracy of the Llama GRPO model in Table 24? Can the safety-discernment gains be retained at a checkpoint with comparable utility?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.