acceptodds
Under review as a conference paper at ICLR 2027

Safety Is Discernment, Not Refusal: Teaching Large Reasoning Models to Judge Before They Act

Abstract

Aligned large reasoning models achieve high safety rates on harmful-request benchmarks, yet they substantially over-refuse benign requests that resemble harmful ones. Among the released checkpoints we evaluate on DeepSeek-R1-Distill backbones, every one with a safety rate above 80% answers fewer than half as many of these requests as its base model. This *refusal-shaped safety*, earned by indiscriminate refusal, is invisible to harmful-request-only benchmarks. We therefore argue for evaluating safety as *discernment*: refusing harmful requests while answering benign ones. To teach discernment, we propose **Reason-Judge-Act (RJA)**, a multi-turn reinforcement learning framework. It trains on paired harmful and benign requests that share surface form but differ in intent, so surface cues alone cannot predict the correct behaviour. In each turn, the model reasons about the request, commits to a risk decision in a dedicated judgement slot, and acts on it. The reward penalises over-refusal as heavily as unsafe compliance, and corrective feedback lets the model revise its behaviour in later turns. Across six backbones from two model families, RJA achieves the highest discernment among the evaluated methods while preserving reasoning capability. Notably, with benign non-refusal largely preserved, it not only substantially raises safety on low-safety backbones but also consistently improves it on safer ones. Ablations show that both the corrective turns and the judgement slot contribute to discernment. These results suggest that models should learn to *judge before they act*, not merely to refuse. Data and checkpoints will be released upon acceptance, and code is available at https://anonymous.4open.science/r/RJA.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Reasoning Model, Safety Alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MhYxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies over-refusal in safety-aligned large reasoning models, describing high safety scores achieved through indiscriminate refusal as “refusal-shaped safety.” To address this issue, the authors propose Reason–Judge–Act (RJA), a reinforcement learning framework that combines harmful–benign prompt pairs, an explicit risk-judgement slot before answering, and rewards that penalize both unsafe responses and over-refusal. Across six backbones, RJA achieves the highest discernment score among the evaluated methods, while preserving average performance on the evaluated reasoning benchmarks.

Critical strengths
  • The findings shows explicit over-refusal in released safety-aligned models. Evaluating harmful requests alongside challenging benign requests provides a more informative assessment than harmful-request safety alone.

  • RJA improves the balance between safety and benign non-refusal across six backbones while preserving average performance on the evaluated reasoning benchmarks. The R1-8B ablations further support the benefits of the judgement slot and corrective turns over the underlying RL setup.

Critical weaknesses
  • The problem formulation has limited novelty. Over-refusal and the need to balance safety with benign compliance are well discussed in the pervious works. The findings provide additional evidence for reasoning models rather than identifying a new problem.
  • Training data are not controlled across methods. All baselines use released checkpoints, whereas RJA uses specifically constructed harmful–benign pairs. These pairs may themselves benefit performance on OR-Bench-Hard, making it difficult to attribute the gains to the proposed learning method.
Questions for authors

Q1. How does RJA compare with ThinkSafe or another strong baseline trained on the same harmful and benign prompts?

Q2. Does replacing the paired benign rewrites with independently sampled benign prompts retain the gains?

Q3. Do the judgement slot and corrective turns also improve Qwen3, where digit supervision is not used?

Reviewer_QgFvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies over-refusal in the safety alignment of large reasoning models. It frames the goal as discernment: refusing harmful requests while answering benign ones. The proposed method, RJA, combines paired harmful and benign training prompts, an explicit judgement slot, and a multi-turn RL procedure. The reward penalizes both unsafe responses and over-refusal, while corrective turns are introduced when the model gives a safe but non-refusing response to a harmful request. Across six backbones from two model families, RJA achieves the highest reported discernment score among the compared methods, while largely preserving general capabilities. The paper also includes ablations of the judgement slot and corrective-turn mechanism.

Critical strengths
  1. The paper focuses on an important problem. Over-refusal remains a practical limitation of safety alignment. Evaluating whether a model can distinguish harmful from benign requests is more informative than measuring safety alone, since a high safety score can be achieved by refusing too broadly.

  2. The experiments show a useful improvement in the balance between safety and over-refusal. The evaluation covers six backbones and RJA achieves the highest reported discernment score among the compared methods, with general capabilities are also largely preserved according to the reported average over five capability benchmarks.

  3. The ablation studies strengthen the empirical support for the method. The authors examine the contributions of the judgement slot and corrective-turn mechanism. The results provide some evidence that both components contribute to the method’s effectiveness, making the proposed design more convincing.

Critical weaknesses
  1. Missing comparisons with RL-based methods. RJA introduces a multi-turn RL mechanism and a tailored reward function, but all external baselines are SFT-based. It is therefore unclear how much of the improvement over these methods comes from the training data versus the method design. Comparing with RL baselines using the same training data would help clarify this.

  2. The selective multi-turn mechanism needs further investigation. RJA introduces corrective turns only for safe, non-refusing responses to harmful prompts, while other outcomes terminate the trajectory. This is a central feature of the proposed RL method, but the rationale for this selective design and its effects are not sufficiently explored. The existing ablation removes corrective turns entirely, but does not explain why correction should be restricted to this particular case.

  3. The over-refusal evaluation is limited to one benchmark. The results on OR-Bench-Hard1K are encouraging. Although the authors explain their choice of OR-Bench over XSTest, I would still like to see results on other benchmarks, such as FalseReject, to assess whether the improvements generalize.

Questions for authors
  1. Could the authors compare RJA with RL baselines, such as AlphaAlign, using the same training dataset under a comparable training budget, to better distinguish the contribution of the RL design from that of the data?

  2. Could the authors further investigate the multi-turn mechanism by extending corrective turns to other cases, such as over-refusal on benign prompts? Comparing these alternatives with the current design in terms of safety and benign compliance would help clarify the benefits of the selective correction mechanism.

  3. Could the authors report results on additional over-refusal benchmarks, such as FalseReject and XSTest, to provide stronger evidence that the improvements generalize?

1 comment

Thanks for the authors’ rebuttal and additional experiments. I appreciate it.

However, I remain concerned about the baseline selection and evaluation setting. ASCL relies on RAG and MCP services and trains the model to retrieve safety rubrics to decide whether to respond or refuse. While I appreciate the authors’ effort to reproduce ASCL, their statement that they “evaluate it like every other model, with the bare prompt...” suggests that ASCL’s required retrieval setup was not retained. Without RAG and rubric retrieval, ASCL’s core design is effectively disabled, making the comparison unfair. I also agree that using ASCL’s full evaluation pipeline while evaluating RJA with bare prompts would introduce another mismatch. Therefore, I do not think ASCL is a suitable RL baseline for RJA.

Overall, the experiments still mainly compare RJA with the selected SFT baselines. The additional ASCL result does not convincingly address my concern about RL baselines, given both the baseline choice and the evaluation setting. I hope the authors can reconsider the experimental organization and RL baseline selection, and provide fair comparisons demonstrating RJA’s advantages over other RL methods in safety–over-refusal balance and the contributions of its components. Therefore, I will maintain my score.

Reviewer_61uSOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that high safety rates in aligned reasoning models often reflect “refusal-shaped safety”: models refuse indiscriminately rather than discern harmful from benign intent. The authors define discernment as jointly refusing harmful requests and answering benign ones, and quantify it by the harmonic mean of safety and benign non-refusal. They diagnose several high-safety baselines as having low benign non-refusal and narrow refusal margins. They propose Reason–Judge–Act (RJA), a multi-turn RL framework using surface-form-matched harmful–benign pairs, a judgement slot before the answer, a five-outcome reward penalizing over-refusal as heavily as unsafe compliance, and corrective turns. Experiments on six backbones and seven SFT baselines show RJA achieves the highest discernment score while largely preserving reasoning capability.

Critical strengths

The problem is important and the diagnosis is convincing: the paper clearly quantifies refusal-shaped safety and shows that high safety can coexist with severe over-refusal. The discernment score is more deployment-relevant than safety rate alone, and the RJA design is coherent with informative ablations across six backbones. These strengths support acceptance because they reframe safety evaluation around intent discrimination rather than refusal count.

Critical weaknesses

The main weakness is evaluation robustness. Safety and benign non-refusal rely heavily on WildGuard and Qwen3Guard, yet the appendix reports agreement below 80% between them, which directly affects the reliability of the central conclusions. The paired benign data contain only 882 GPT-4o-generated counterparts, limiting scale and generalization. RJA is also weaker than SafePath and RealSafe under SR-Atk, and training-time over-refusal rises late in training while H declines, so checkpoint selection and attack robustness need stronger justification.

Questions for authors

Could you provide human validation or a third judge showing that the main conclusions hold when WildGuard and Qwen3Guard disagree? Is RJA’s weaker adversarial safety related to the paired data being mostly direct requests, and can you report failure types under attack? How is the final checkpoint selected when H declines late in training? Can RJA be compared directly with at least one RL-based safety-alignment baseline?

Comments on the paper

1 comment

Thanks for your rebuttal and the additional results.

However, I still have a concern regarding the design of RJA and its connection to the motivation of mitigating over-refusal.In the rebuttal, you state that the corrective turns are designed to improve safety and adding benign-over-refusal corrective would "push toward safety". But according to your explanation, i'm wondering would harmful-safe-non-refusal corrective turns instead lead to an increase in refusal? Therefore, I'm particularly interested in understanding the source of RJA’s superior over-refusal performance: does it come purely from the judgement slot, or from the training data distribution? It seems that the paper does not provide a sufficiently clear explanation of how the method’s design is specifically connected to its motivation.

BTW, RJA is a kind of RL-based safety alignment method. Although the rebuttal shows that it outperforms AlphaAlign on S&U, the other experiments in the paper and rebuttal still mainly compare against SFT-based methods, right? Based on my research experience in this area, RL-based methods often achieve better performance than SFT-based methods in improving safety while maintaining low over-refusal. So I think that the main comparisons for this paper should instead focus on RL baselines related to RJA, especially methods specifically designed to mitigate over-refusal (AlphaAlign is a relatively early baseline in this line of research).

This is also why I am particularly interested in how the corrective-turn design in RJA addresses over-refusal and what advantages it offers over other RL-based methods. I think this is a problem this paper need to figure out, so I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.