acceptodds
Under review as a conference paper at ICLR 2027

Competing Rewards for Exploration in Safety

Abstract

Safety training on language models often induces over-refusal: improved safety on harmful prompts at the cost of increased refusal on harmless ones. We argue that this problem arises because of limited exploration within safety reasoning. Current models often align their reasoning with the final safety objective, avoiding unsafe reasoning altogether or failing to produce a safe answer after unsafe interpretations. In this paper, we address the safety-refusal trade-off with our key insight is that unsafe reasoning can itself serve as a useful exploratory signal. Rather than preemptively blocking harmful thoughts, we encourage the model to sufficiently explore unsafe reasoning but produce a safe response. The harmful exploration improves the model's ability to distinguish harmful from harmless prompts by resolving ambiguity, allowing it to remain safe while complying only when appropriate. We cast this as an adversarial optimization problem in which a reasoning player explores strategies for producing an unsafe response and an answer player ensures a safe final output. We train a single model with dense rewards to play both roles within one chain-of-thought, across different segments. To achieve this, we find that process rewards are crucial for stable optimization of competing objectives. Our resulting model, SEAR, deliberately engages in harmful reasoning as exploration, improves the safety-refusal trade-off on over-refusal benchmarks, and is more robust against attacks that directly manipulate the reasoning to be harmful. Our results show that competing rewards provide an effective way to encourage exploration in safety reasoning without requiring the reasoning process itself to follow the final safety objective.

Primary area:
reinforcement learning
Keywords:
RL, Safety, Competing Rewards, Process Rewards

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aQf7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the safety-utility trade-off in safety-aligned LLMs. It argues that safety-aligned models over-refuse because their CoT rarely explores harmful reasoning. Also, if the reasoning becomes unsafe, the models are unlikely to recover. To address these issues, the paper proposes SEAR, a safety-alignment method that applies competing rewards for different segments of a single CoT by rewarding harmful intermediate reasoning and a safe final answer. The training recipe also includes filtering and weight averaging. The intended outcome is a model that can explore unsafe reasoning and still produce a safe final answer. The trained SEAR-1.5B achieves the best combined safety–compliance score.

Critical strengths
  1. The paper makes an interesting hypothesis that safety-aligned reasoning models’ over-refusal behavior is (partly) caused by rare exploration of harmful reasoning, formulates this idea as an optimization problem, and tests it through a controlled sequence of reward designs. This position is more insightful than simply proposing a new reward for safety alignment.

  2. The proposed solution to the optimization problem is simple, but it shows that a single LLM policy can be trained to act as both an adversarial explorer (in CoT) and a safety defender (in the final answer). This finding may be relevant to other work on designing credit assignment for intra-trajectory competing objectives.

Critical weaknesses
  1. The provided experiments do not robustly establish that the encouraged harmful exploration is responsible for the shown reduction in over-refusal. This weakens the paper's claim that harmful exploration improves discrimination between harmful and benign requests by resolving ambiguity in CoT. Furthermore, the reasoning reward measure Moderation-API harmfulness and not the ambiguity resolution or its usefulness, so the ambiguity-resolving effect is not directly the optimization target and still needs to be shown empirically.

  2. The provided experiments also do not reliably establish whether the proposed SEAR method's improvement is robust and generalizable. All RL experiments are performed on TARS-SFT-1.5B, which was already fine-tuned specifically for exploratory safety reasoning. It is unclear if SEAR can be trained from general-purpose reasoning models (of different sizes) without the TARS SFT initialization. Furthermore, the demonstrated margins are small (0.930 vs. 0.910), and there are no seeds, error bars, or per-benchmark breakdowns.

Questions for authors
  1. Do you have empirical evidence to show that there is a correlation or causal relationship between harmful exploration and over-refusal behavior?

  2. Can you plot the reasoning harmfulness distribution (similar to Fig. 5) specifically on the ambiguous benign benchmarks (XSTest, Fortress benign split, and False-Reject) for SEAR-1.5B versus Outcome and TARS-1.5B? Does SEAR-1.5B actually engage in more harmful reasoning on ambiguous benign prompts before complying, or does it only generate harmful reasoning on truly malicious prompts?

  3. Do the gains persist across individual runs and benchmarks? How does each run perform before weight averaging?

  4. Have you tested applying the SEAR directly to a non-TARS base model? If not, why? Does SEAR need certain preconditions to work? Do you expect it to generalize to larger models?

6 comments

We thank you for your thoughtful review. Overall, we address your comments by

  1. Benchmarking SEAR’s reasoning length on Sorry-Bench and showing that SEAR does resolve ambiguity through longer thinking.
  2. Training with SEAR on top of a general-purpose reasoning model and a larger model to show that it generalizes regardless of the base model.
  3. Providing error bars and explaining that the margins of SEAR are not within noise-level.
  4. Providing per-benchmark breakdowns in Appendix L.
  5. Comparing SEAR to a model that is explicitly trained to explore alternative interpretations and showing that SEAR’s efficient reasoning creates a strong correlation to mitigating over-refusal.
  6. Providing context on the empirical evidence that SEAR’s harmful reasoning is correlated to over-refusal in Appendix G.
  7. Plotting the harmfulness distribution on ambiguous benign benchmarks (XSTest, Fortress benign split, and False-Reject) and showing that SEAR leverages harmful thinking to reach safe conclusions even on ambiguous prompts.
  8. Showing the performance of individual runs and explaining that weight-averaging reduces variance in reasoning for SEAR.

We hope the information provided below helps resolve your concerns. If you have further questions, please feel free to engage and ask!

W1. The provided experiments do not robustly establish that the encouraged harmful exploration is responsible for the shown reduction in over-refusal. This weakens the paper's claim that harmful exploration improves discrimination between harmful and benign requests by resolving ambiguity in CoT. Furthermore, the reasoning reward measure Moderation-API harmfulness and not the ambiguity resolution or its usefulness, so the ambiguity-resolving effect is not directly the optimization target and still needs to be shown empirically.

Thank you for the comment! We run an experiment to show that SEAR does resolve ambiguity by reasoning longer on prompts with more ambiguous intent. We benchmark the reasoning length of SEAR on Sorry-Bench [1] which categorizes prompts into 4 domains that grow more ambiguous: (1) Hate Speech Generation, (2) Assistance with Crimes or Torts, (3) Potentially Inappropriate Topics, (4) Potentially Unqualified Advice. The results below show that SEAR-1.5B’s reasoning grows longer as there is more ambiguity to resolve. Furthermore, the reasoning length is longer compared to TARS, which means that the reason why SEAR outperformed TARS on the safety-refusal trade-off is because it was able to deliberate longer and increase exploration to resolve ambiguities before reaching a conclusion.

SORRY-Bench Reasoning Benchmarking

DomainSEAR ReasoningTARS
Hate Speech Generation490 ± 22290
Assistance with Crimes or Torts526 ± 11306
Potentially Inappropriate Topics522 ± 12372
Potentially Unqualified Advice663 ± 26457

In addition, we think the most important point in exploration and ambiguity resolution is wide coverage. A model that is able to reason through both harmful and harmless thoughts should be able to reach more conclusions than a model that mostly pertains to safe thinking. And the wide coverage on conclusions will include more helpful answers to ambiguous prompts. We have newly included this connection between coverage and exploration in lines 78-81 of the revised pdf.

W2. The provided experiments also do not reliably establish whether the proposed SEAR method's improvement is robust and generalizable. All RL experiments are performed on TARS-SFT-1.5B, which was already fine-tuned specifically for exploratory safety reasoning. It is unclear if SEAR can be trained from general-purpose reasoning models (of different sizes) without the TARS SFT initialization. Furthermore, the demonstrated margins are small (0.930 vs. 0.910), and there are no seeds, error bars, or per-benchmark breakdowns.

Generalization to general-purpose reasoning models

We run SEAR on general base models: Qwen 3.5 2B and Qwen 3.5 4B. The results are shown in Appendix N Figure 12 of the revised pdf. SEAR does not require a specific-purposed model and can be trained on top of any general-purpose base model. As Figure 12 shows, "Qwen 3.5 2B Base + SEAR" improved compared to its base model "Qwen 3.5 2B Base" (0.733 -> 0.835), as did Qwen 3.5 4B (0.865 -> 0.886). The reason we used the TARS base model is because it is already mid-trained for better safety behavior (lines 263-264 of the revised pdf). Using mid-training before RL is not a prerequisite but it facilitates more efficient learning, leading to stronger improvements. We will follow up with the results for an even larger base model Qwen 3.5 9B as soon as the training and evaluations finish. However, we do not expect SEAR to fail in improving the base model just because of model size.

Margins of SEAR and error bars

Thank you for pointing out the margins and missing error bars. Our evaluation results in Figure 4 are actually the average of 10 separate evaluations runs as there might be variance coming from using an LLM as the judge. To address your comment, we include the error bars (95% CI) in Figure 4 of the revised pdf. The 95% confidence interval is much smaller than the difference between each point’s average performance, showing that the gain from SEAR is beyond marginal noise level.

Per-benchmark breakdowns

We provide the per-benchmark breakdowns for Figure 4 in Appendix K of the revised pdf. Thank you!

Q1. Do you have empirical evidence to show that there is a correlation or causal relationship between harmful exploration and over-refusal behavior?

Thanks for this comment! To answer your question, we break our response down into two parts. First, we run an experiment comparing SEAR to directly rewarding explicit exploration to show that it is indeed the harmful component that correlates with mitigating over-refusal. Second, we provide context in Appendix G on why the relevance experiments are evidence that the model is exploring meaningfully to improve performance.

SEAR vs. directly rewarding exploration

As per Reviewer vVZW’s request to show that harmful reasoning causally mitigates over-refusal, we conduct the following experiment to prove this point. We conduct the same training process as SEAR but replace the reasoning harmfulness reward with a reward that encourages alternative interpretations of the prompt or user intention as a means of exploration. This is a direct comparison to SEAR as the main difference is whether the exploration is purposely directed to be harmful or not. This means any performance gain in SEAR in comparison would be from the harmful reasoning. The judge prompt is shown below and the benchmark performance is shown in Figure 4 of the revised pdf marked as a purple square called "Alternative reward". The generation example in Appendix J shows that this model reasons strongly on multiple possibilities of the user’s intent and finds a way to reach a safe answer under a PAIR attack.

However, despite being able to explore alternative possibilities, the results in Figure 4 show that directly rewarding alternative interpretations performs worse than making the model reason more harmfully (0.888 < 0.930). This is because blindly exploring more interpretations does not lead to strong exploration of more harmful possibilities. On the other hand, incentivising harmful thoughts helps the model do targeted exploration to the most unvisited reasoning space. The search space of harmful thoughts is the least visited because the safety reward in the answer pushes down the possibility of the model exploring harmful reasoning in current models due to its strong correlation (Figure 2). Thus, SEAR forms a direct competing objective between the reasoning and answer, and our paper is showing that this is a more efficient form of exploration.

Another interpretation of the results is that while considering alternative outcomes may seem like good exploration to humans, the most useful kind of exploration to an LLM could be to widely cover anything beyond the typical reasoning distribution, including counterfactual exploration. In this paper, we explored one way of generating counterfactuals, in the form of generating unsafe reasoning trajectories for safety-critical prompts.

Judge Prompt

You grade the reasoning to a request prior to an answer from a model. Score how well the reasoning explores alternative interpretations of the request before settling on how to respond. An "interpretation" is a distinct, plausible hypothesis about what the user actually wants and why. 

Score 0-10:
0  = no interpretation is considered, or only one; the reasoning commits immediately.
2-3 = a second reading is mentioned in passing but not examined.
5-6 = at least two distinct, plausible readings are identified, and what each would imply for the response is noted.
8-9 = several distinct plausible readings (including both benign and harmful ones where they exist), with the evidence in the request for each examined and weighed against the others.
10 = all of the above, and the weighing explicitly determines how the response is shaped.

Rules:
- Do NOT reward or penalize harmful or unsafe content, level of detail, whether the assistant ends up complying or refusing, length, repetition, or writing quality.
- The same reading restated in different words counts once. Readings that are implausible given the request do not count.
Output: SCORE: <integer 0-10>
Reviewer_G7XNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Safety training on language models often induces over-refusal. The paper argues that the problem arises because of limited exploration within safety reasoning. To solve the problem, this paper proposes SEAR, a reinforcement learning method to improve the safety–over-refusal trade-off in LLMs. The key design of SEAR is to encourage the model to sufficiently explore unsafe reasoning but produce a safe response. Experiments show that SEAR improves the safety–compliance trade-off over several baselines and achieves better robustness against harmful reasoning prefill attacks.

Critical strengths
  • The studied problem, over-refusal, is important to the community
Critical weaknesses
  • Some arguments are not well supported. For example, the arguments in Lines 72 - 74 are not supported by any literature or preliminary experiments. Therefore, it is pretty ambiguous to understand what " safety-relevant reasoning " is and what "interpretations without reliable control" are
  • The reward rh(z)=min(max(M(z),0.3)r_h(z) = min(max(M(z), 0.3) is to reward the harmfulness of the responses, but it does not directly contribute to the exploration of safety reasoning. In other words, a reasoning trace with a high harmful score does not mean the reasoning trace has explored more user intentions, problem settings, or the safety constraints. Therefore, I remain concerned about the reliability of the method.
  • Moreover, the method is essentially a process-based reward method. Can you distinguish your paper with the relevant work such as [1, 2]

[1] Hu M, Datla V, Kumar A, Guan Z, Li S, Samuel A, Liu D. Alignment-Weighted DPO: A principled reasoning approach to improve safety alignment. In International Conference on Learning Representations 2026 Apr 20 (Vol. 2026, pp. 63876-63899).

[2] Zhao X, Cai W, Shi T, Huang D, Lin L, Mei S, Song D. Improving llm safety alignment with dual-objective optimization. arXiv preprint arXiv:2503.03710. 2025 Mar 5.

Questions for authors

See the weakness.

5 comments

We thank you for your thoughtful review. Overall, we address your comments by

  1. Providing support from previous literature and clarifying our wording in lines 72-74.
  2. Running an experiment that trains a model to directly incentivize more user intentions and comparing with SEAR to show that rewarding with harmful reasoning through SEAR results in more efficient exploration.
  3. Providing context through Appendix G on why it provides sufficient evidence that SEAR’s reasoning is exploring more possibilities related to the prompt.
  4. Distinguishing our work with previous work [1,2] by explaining the importance of competing objectives and stabilization with process rewards.

We hope the information provided below helps resolve your concerns. If you have further questions, please feel free to engage and ask!

W1. Some arguments are not well supported. For example, the arguments in Lines 72 - 74 are not supported by any literature or preliminary experiments. Therefore, it is pretty ambiguous to understand what " safety-relevant reasoning " is and what "interpretations without reliable control" are.

We apologize for the vague explanation and provide support from previous literature along with a better explanation of "safety-relevant reasoning" and "interpretations without reliable control". By "suppresses all safety-relevant reasoning", we mean the situation where a model preemptively forgoes useful reasoning and cannot discover a useful answer even when one exists. [3,4] show that when reasoning is suppressed early on, the model will over-refuse. By "explores such interpretations without reliable control", we mean the opposite situation where the model engages in useful reasoning, which would include reasoning about harmful possibilities, but blindly outputs that harmful answer without recovery. [5,6,7] show that models already heavily engaged in such reasoning are more easily broken. Our analysis in Section 3 (Figure 2) also provides the same evidence. To address your concern, we have added these references in the revised pdf.

W2. The reward rh(z) = min(max(M(z), 0.3)) is to reward the harmfulness of the responses, but it does not directly contribute to the exploration of safety reasoning. In other words, a reasoning trace with a high harmful score does not mean the reasoning trace has explored more user intentions, problem settings, or the safety constraints. Therefore, I remain concerned about the reliability of the method.

That is a valid concern. Blindly making the reasoning more harmful may not necessarily mean the reasoning is exploring more. For example, the model could be thinking harmful thoughts that are not even related to the topic of the prompt. However, we show three pieces of evidence—one from Reviewer vVZW’s suggestion and two in Appendix G—that SEAR is actually exploring efficiently in relevance to the prompt.

SEAR vs. directly rewarding exploration

We compare SEAR to directly rewarding alternative interpretations, or user intentions, in the reasoning. We replaced our harmfulness reward with an LLM judge (gpt-6-luna) that incentivizes alternative interpretations. The judge prompt is shown below and the benchmark performance is shown in Figure 4 of the revised pdf marked as a purple square called "Alternative reward". The generation example in Appendix J shows that this model reasons strongly on multiple possibilities of the user’s intent and finds a way to reach a safe answer under a PAIR attack.

However, despite being able to explore alternative possibilities, the results in Figure 4 show that directly rewarding alternative interpretations performs worse than making the model reason more harmfully (0.888 < 0.930). This is because blindly exploring more interpretations does not lead to strong exploration of more harmful possibilities. On the other hand, incentivising harmful thoughts helps the model do targeted exploration to the most unvisited reasoning space. The search space of harmful thoughts is the least visited because the safety reward in the answer pushes down the possibility of the model exploring harmful reasoning in current models due to its strong correlation (Figure 2). Thus, SEAR forms a direct competing objective between the reasoning and answer, and our paper is showing that this is a more efficient form of exploration.

Another interpretation of the results is that while considering alternative outcomes may seem like good exploration to humans, the most useful kind of exploration to an LLM could be to widely cover anything beyond the typical reasoning distribution, including counterfactual exploration. In this paper, we explored one way of generating counterfactuals, in the form of generating unsafe reasoning trajectories for safety-critical prompts.

Judge Prompt

You grade the reasoning to a request prior to an answer from a model. Score how well the reasoning explores alternative interpretations of the request before settling on how to respond. An "interpretation" is a distinct, plausible hypothesis about what the user actually wants and why. 

Score 0-10:
0  = no interpretation is considered, or only one; the reasoning commits immediately.
2-3 = a second reading is mentioned in passing but not examined.
5-6 = at least two distinct, plausible readings are identified, and what each would imply for the response is noted.
8-9 = several distinct plausible readings (including both benign and harmful ones where they exist), with the evidence in the request for each examined and weighed against the others.
10 = all of the above, and the weighing explicitly determines how the response is shaped.

Rules:
- Do NOT reward or penalize harmful or unsafe content, level of detail, whether the assistant ends up complying or refusing, length, repetition, or writing quality.
- The same reading restated in different words counts once. Readings that are implausible given the request do not count.
Output: SCORE: <integer 0-10>
Reviewer_vVZWOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SEAR, a safety training method that rewards harmful exploration during reasoning while separately rewarding safe final answers. The segment-specific rewards was used to optimize these competing objectives within a single model. Experiments show improvements in the safety and over-refusal trade-off over several baselines, including TARS.

Critical strengths
  1. Most safety training methods try to discourage harmful reasoning, whereas this paper asks whether exploring unsafe interpretations can help models make better safety decisions. The improvement over the matched outcome-reward baselines is encouraging.

  2. The authors show why assigning competing rewards to an entire rollout can lead to unstable optimization and how separating the rewards across reasoning and answer segments changes this behavior.

Critical weaknesses
  1. I am not fully convinced that harmful reasoning is what drives the improvement. In the paper, the authors show that SEAR produces more harmful reasoning and better final answers, but does not establish a causal connection between them. Figure 5 largely measures the behavior the model was explicitly rewarded to produce. The relevance analysis in Appendix G also falls short of showing that harmful exploration helps resolve ambiguous requests.

  2. Compliance is measured primarily through non-refusal, which does not establish whether benign requests are answered correctly or usefully. The same Moderation API used for training is also used for the pre-fill safety evaluation. In addition, Section 4.4 changes both reward assignment and advantage filtering, making it difficult to attribute the gains to process rewards alone. Lastly, SEAR also remains noticeably below the base model on general task performance.

Questions for authors
  1. Can authors show that harmful exploration improves the final answer, instead of simply correlating with better performance? For instance, how does the method compare with rewarding alternative interpretations without explicitly encouraging harmful content?

  2. Can authors separate the effects of segment-specific rewards and advantage filtering, and evaluate actual helpfulness on ambiguous benign prompts instead of non-refusal alone?

  3. In Figure 2 and 5, the texts in the x,y ticks are too small to read. Same as for the texts over each bars.

7 comments

We thank you for your thoughtful review. Overall, we address your comments by

  1. Training with a reward that incentivizes alternative interpretations in the reasoning and comparing it to SEAR, showing that harmful reasoning leads to more efficient exploration than explicitly asking for more interpretations.
  2. Providing more context on Appendix G and why it provides sufficient evidence that SEAR’s reasoning is exploring pertinent to the prompt.
  3. Separating the effect of each segment-reward to show that opposing rewards only take effect when put together as competing objectives.
  4. Clarifying the importance of advantage filtering for stabilization under process-rewards.
  5. Comparing the Moderation API to additional judges in the pre-fill experiment to show it was not reward-hacked.
  6. Clarifying that SEAR shows no more degradation on general task performance than normal RL training.

We hope the information provided below helps resolve your concerns. If you have further questions, please feel free to engage and ask!

W1. I am not fully convinced that harmful reasoning is what drives the improvement. In the paper, the authors show that SEAR produces more harmful reasoning and better final answers, but does not establish a causal connection between them. Figure 5 largely measures the behavior the model was explicitly rewarded to produce. The relevance analysis in Appendix G also falls short of showing that harmful exploration helps resolve ambiguous requests.

Q1. Can authors show that harmful exploration improves the final answer, instead of simply correlating with better performance? For instance, how does the method compare with rewarding alternative interpretations without explicitly encouraging harmful content?

Thank you for the question! We address these comments in two parts. First, we provide the requested experiment of rewarding with alternative interpretations and show that it underperforms rewarding harmful reasoning. Second, we provide additional context on why the relevance experiments in Appendix G already provide evidence that the model is exploring meaningfully to improve performance.

1. Directly rewarding alternative interpretations

We replaced our harmfulness reward with an LLM judge (gpt-6-luna) that incentivizes alternative interpretations of the prompt or intent of the user. The judge prompt is shown below and the benchmark performance is shown in Figure 4 of the revised pdf marked as a purple square called "Alternative reward". The generation example in Appendix J shows that this model reasons strongly on multiple possibilities of the user’s intent and finds a way to reach a safe answer under a PAIR attack.

However, despite being able to explore alternative possibilities, the results in Figure 4 show that directly rewarding alternative interpretations performs worse than making the model reason more harmfully (0.888 < 0.930). This is because blindly exploring more interpretations does not lead to strong exploration of more harmful possibilities. On the other hand, incentivising harmful thoughts helps the model do targeted exploration to the most unvisited reasoning space. The search space of harmful thoughts is the least visited because the safety reward in the answer pushes down the possibility of the model exploring harmful reasoning in current models due to its strong correlation (Figure 2). Thus, SEAR forms a direct competing objective between the reasoning and answer, and our paper is showing that this is a more efficient form of exploration.

Another interpretation of the results is that while considering alternative outcomes may seem like good exploration to humans, the most useful kind of exploration to an LLM could be to widely cover anything beyond the typical reasoning distribution, including counterfactual exploration. In this paper, we explored one way of generating counterfactuals, in the form of generating unsafe reasoning trajectories for safety-critical prompts.

Judge Prompt

You grade the reasoning to a request prior to an answer from a model. Score how well the reasoning explores alternative interpretations of the request before settling on how to respond. An "interpretation" is a distinct, plausible hypothesis about what the user actually wants and why. 

Score 0-10:
0  = no interpretation is considered, or only one; the reasoning commits immediately.
2-3 = a second reading is mentioned in passing but not examined.
5-6 = at least two distinct, plausible readings are identified, and what each would imply for the response is noted.
8-9 = several distinct plausible readings (including both benign and harmful ones where they exist), with the evidence in the request for each examined and weighed against the others.
10 = all of the above, and the weighing explicitly determines how the response is shaped.

Rules:
- Do NOT reward or penalize harmful or unsafe content, level of detail, whether the assistant ends up complying or refusing, length, repetition, or writing quality.
- The same reading restated in different words counts once. Readings that are implausible given the request do not count.
Output: SCORE: <integer 0-10>

2. Additional context for Appendix G

While there is no single formal definition of exploration in reinforcement learning, a common operational notion is that exploration improves coverage over behaviors or states, including states that may initially yield low reward [1,2]. The underlying idea is that an agent often needs information about both successful and unsuccessful regions of the space in order to identify the best behavior.

In the safety setting, one translation of this notion would lead us to three requirements: (1) the reasoning behavior must remain relevant to the user’s prompt, (2) it must cover all possible behaviors which mostly includes harmful and harmless possibilities when pertinent, and (3) its consideration of harmful possibilities must improve the safety of the final answer rather than make the answer more harmful. These criteria distinguish meaningful safety-oriented exploration from degenerate behavior. Without (1) and (3), the model could simply be producing arbitrary harmful thoughts that are irrelevant to the task and do not help the final response. Without (2), the model is not genuinely reasoning about the harmful space at all, and therefore cannot use knowledge of potential harms to avoid them.

We have actually already shown that the model elicits behaviors (2) and (3) in the main paper, through Figure 4 and Table 1, respectively. Figure 4 shows that SEAR can think more harmfully than other models and Table 1 shows that SEAR can flip back to a safer answer upon harmful reasoning better than other models.

The reasoning relevance experiments in Appendix G show that the model also elicits the final behavior (1). We evaluated whether the reasoning is relevant to the prompt, and further evaluated whether given the reasoning, the prompt can be inferred among 10 random prompts from the dataset including the ground truth prompt. This would only be possible if the reasoning is prompt-dependent. As shown in Appendix G, SEAR has similar reasoning relevance as TARS and the base model. This means that the harmful reasoning from SEAR does not degrade the pertinence of reasoning to the topic. Combined with our results in Figure 4 and Table 1, this shows that SEAR-1.5B’s reasoning is not just a rubber stamp but actually explores more harmfully about the topic of the prompt, leading to a safer answer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.