- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper compares how SFT and GRPO teach models to refuse harmful requests while still answering benign ones. In the primary setting, models with similar ordinary refusal rates behave differently on benign requests and after their responses are forced to begin with a helpful phrase such as “I can help with that.” Training ablations examine which feedback produces these differences. In particular, Offline Signed Learning (OSL) increases the likelihood of better self-generated responses and decreases that of worse ones using a fixed response bank. The paper finds that this can produce substantial prefix recovery without prefix-specific training examples, and investigates associated activation changes.
Critical strengths
-
The evaluation reveals safety differences hidden by ordinary refusal rates. The paper separately measures refusal of harmful requests, mistaken refusal of benign requests, and recovery after a forced helpful opening. Models that look similar on the first measure differ substantially on the others. This provides a useful explanation of why aggregate refusal and general-task accuracy are insufficient to characterize safety behavior.
-
The training controls help identify a consequential learning signal. OSL and its controls start from the same checkpoint and use a shared bank of ordinary self-generated responses. Positive-only training and fitting the best response do not reproduce OSL’s strong recovery. These comparisons support the interesting finding that suppressing worse responses, alongside reinforcing better ones, can produce recovery without explicitly training on the tested prefixes.
Critical weaknesses
-
Recovery improvements are not fully evaluated alongside retained capabilities. The primary Qwen comparison reports similar general-task accuracy, but the additional Llama experiment shows a substantial difference between SFT and GRPO: 47.57 versus 37.26. The central OSL comparison reports refusal and recovery outcomes without equally complete general-task evaluation. It is therefore unclear whether the strongest recovery gains preserve useful capabilities and how they compare at matched ordinary safety levels.
-
The main recovery results need more independent evaluation. WildGuard judges generated responses during training and supplies much of the behavioral evaluation. This creates a possibility that the trained models become better aligned with that evaluator’s judgments without an equivalent improvement under independent assessment. The Llama Guard check on natural outputs is useful, but does not sufficiently validate the central OSL and forced-prefix results, where substantive harmful content should be assessed separately from refusal wording.
-
The specific contribution of OSL needs clearer positioning against prior learning objectives. The paper shows that a fixed bank of responses with positive and negative coefficients can teach prefix recovery. Prefix vulnerability is already established [1], and the paper acknowledges the connection to GPG [2]. It should also discuss unlikelihood training [3] and CUT [4], which provide relevant precedents for discouraging undesirable outputs. A matched negative-feedback baseline would clarify whether recovery depends on OSL’s particular objective or reflects a broader effect of learning from negative responses.
-
The tested prefixes cover a narrow family of perturbations. H5 forces a short helpful opening, and HC10 extends a closely related opening. These tests convincingly expose a failure of ordinary refusal, but do not establish recovery across substantially different wording or deeper forced continuations. The conclusions should distinguish recovery on the tested prefixes from more general robustness after an unsafe generation has begun.
-
The relationship to RECAP [5] needs further clarification. Both works investigate how reward-based training enables models to recover from misleading prefixes and produce safe responses. The paper should explain more explicitly how its findings differ from and extend those of RECAP.
[1] Safety Alignment Should Be Made More Than Just a Few Tokens Deep. 2025.
[2] GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning. 2026.
[3] Neural Text Generation with Unlikelihood Training. 2020.
[4] Reasons to Reject? Aligning Language Models with Judgments. 2024.
[5] Large Reasoning Models Learn Better Alignment from Flawed Thinking. 2025.
Questions for authors
Please refer to my weakness.