- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies over-refusal in safety-aligned large reasoning models, describing high safety scores achieved through indiscriminate refusal as “refusal-shaped safety.” To address this issue, the authors propose Reason–Judge–Act (RJA), a reinforcement learning framework that combines harmful–benign prompt pairs, an explicit risk-judgement slot before answering, and rewards that penalize both unsafe responses and over-refusal. Across six backbones, RJA achieves the highest discernment score among the evaluated methods, while preserving average performance on the evaluated reasoning benchmarks.
Critical strengths
-
The findings shows explicit over-refusal in released safety-aligned models. Evaluating harmful requests alongside challenging benign requests provides a more informative assessment than harmful-request safety alone.
-
RJA improves the balance between safety and benign non-refusal across six backbones while preserving average performance on the evaluated reasoning benchmarks. The R1-8B ablations further support the benefits of the judgement slot and corrective turns over the underlying RL setup.
Critical weaknesses
- The problem formulation has limited novelty. Over-refusal and the need to balance safety with benign compliance are well discussed in the pervious works. The findings provide additional evidence for reasoning models rather than identifying a new problem.
- Training data are not controlled across methods. All baselines use released checkpoints, whereas RJA uses specifically constructed harmful–benign pairs. These pairs may themselves benefit performance on OR-Bench-Hard, making it difficult to attribute the gains to the proposed learning method.
Questions for authors
Q1. How does RJA compare with ThinkSafe or another strong baseline trained on the same harmful and benign prompts?
Q2. Does replacing the paired benign rewrites with independently sampled benign prompts retain the gains?
Q3. Do the judgement slot and corrective turns also improve Qwen3, where digit supervision is not used?
Thanks for the authors’ rebuttal and additional experiments. I appreciate it.
However, I remain concerned about the baseline selection and evaluation setting. ASCL relies on RAG and MCP services and trains the model to retrieve safety rubrics to decide whether to respond or refuse. While I appreciate the authors’ effort to reproduce ASCL, their statement that they “evaluate it like every other model, with the bare prompt...” suggests that ASCL’s required retrieval setup was not retained. Without RAG and rubric retrieval, ASCL’s core design is effectively disabled, making the comparison unfair. I also agree that using ASCL’s full evaluation pipeline while evaluating RJA with bare prompts would introduce another mismatch. Therefore, I do not think ASCL is a suitable RL baseline for RJA.
Overall, the experiments still mainly compare RJA with the selected SFT baselines. The additional ASCL result does not convincingly address my concern about RL baselines, given both the baseline choice and the evaluation setting. I hope the authors can reconsider the experimental organization and RL baseline selection, and provide fair comparisons demonstrating RJA’s advantages over other RL methods in safety–over-refusal balance and the contributions of its components. Therefore, I will maintain my score.