acceptodds
Under review as a conference paper at ICLR 2027

Learning Safe Policies from Hard Counterfactual Negatives

Abstract

Safe reinforcement learning typically handles unsafe behavior by penalizing itduring training or preventing it from being executed. However, when an unsafe ac-tion offers competitive task value, these mechanisms do not explicitly require thepolicy to learn a safe alternative that is better. We introduce Hard-Negative Pol-icy Optimization (HNPO), a world-model-based method that uses such reward-competitive, near-boundary violations, or tempting failures, as hard negatives forpolicy learning. HNPO first identifies unsafe actions that remain credible un-der model uncertainty and behavioral support. A conservative hard-negative pro-poser then maintains access to informative negatives as the task policy becomessafer, learning from validated examples rather than directly optimizing throughworld-model gradients. To learn from these negatives, HNPO evaluates safe pol-icy actions and unsafe alternatives from the same latent state under a matchedcontinuation policy. Hard-Negative Policy Contrast constructs a utility referencefrom the validated negative group and optimizes safe policy actions to outperformthis reference by a margin, without treating proposer actions as on-policy sam-ples. HNPO leaves the backbone safe-RL objective and deployment-time safetymechanism unchanged. Across four Safety-Gymnasium continuous-control tasks,HNPO improves task reward over its backbone while maintaining low episodiccost, with larger gains on tasks exhibiting stronger reward–safety conflict. Abla-tions further demonstrate the complementary roles of credible negative selection,proposer-based coverage, and the negative-only reference.

Primary area:
reinforcement learning
Keywords:
Safe Reinforcement Learning, World Models, Counterfactual Learning, Hard Negative Mining, Policy Optimization, Model Uncertainty

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_NRV2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Hard-Negative Policy Optimization (HNPO), a world-model-based safe RL framework that treats reward-competitive, near-boundary unsafe actions as hard negatives for policy learning. HNPO introduces a credibility-aware mining process to identify near-boundary violations that are competitive and model-supported. Moreover, HNPO proposes a conservative negative proposer learned via weighted behavior cloning and validated elite distillation to sustain negative coverage as the policy becomes safer. On four tasks using USB-RL as the backbone, HNPO improves task reward while keeping episodic cost low.

Critical strengths
  1. Introduces the clear and useful notion of “tempting failures” and operationalizes them as hard negatives in a model-based safe RL pipeline.
  2. The credible negative mining integrates boundary checks, competitiveness, and behavioral support with uncertainty-aware robust evaluation—going beyond naive “unsafe” tagging.
Critical weaknesses
  1. Heavy reliance on the learned world model’s uncertainty calibration and short-horizon robustness; existing analyses suggest ensemble disagreement in latent imagination can decouple from long-horizon error, which could undermine credibility checks and boundary determinations.
  2. The counterfactual comparison varies only the first action under matched continuation, which may be insufficient when the crux of safe alternatives spans multi-step control.
  3. No formal guarantees or bounds on bias introduced by the negative-only anchor or on the stability of HNPC’s auxiliary gradients relative to the base objective.
  4. Compute and sample-efficiency overhead of negative mining, proposer training, and repeated robust evaluation are not quantified; it is unclear whether improvements come at substantial additional cost.
Questions for authors
  1. Does HNPO maintain or improve cumulative training-time safety (e.g., total number of violations) compared to USB-RL and SafeDreamer? Are costs consistently kept under budget across seeds, and how often are violations near-boundary vs. severe?
  2. How sensitive are results to the uncertainty coefficients, boundary margins, competitiveness thresholds, and the target margi? Can you provide sensitivity analyses or guidance for selecting them?
Reviewer_UF3vOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes "Hard-Negative Policy Optimization", an algorithm that leverages world-models to generate rollouts close to the safety boundary, which are used for training a safe behavior policy.

Critical strengths
  1. In high level the proposed approach is interesting: how can we propose simulated rollouts that are "at the boundary" of what's considered safe buy our current model to make the next deployment safe and more performant.
  2. Studying safe RL in the context of world-models is an important direction as it allows to evaluate and design policies completely in simulation, without ever needing to have unsafe interactions with the real world.
Critical weaknesses
  1. The reasoning behind the design of the algorithmic components can be made clearer. In addition many components are not very well grounded.
  2. The contribution is based on several heuristics. While this not necessarily a major problem, the empirical gains of the approach are modest, while they make HNPC more complex.
  3. The experiments are somewhat limited. While the paper compares its algorithm with several baselines, it uses only four environments. The ablations only on CarGoal1 (Figure 3) further demonstrate that the gains from the proposed method are relatively limited.
Questions for authors

Some minor comments:

  1. Some baselines that are used are not mentioned in the related works or cited.
  2. Line 75: worldmodel -> world model, line 38 implicit:when -> implicit: when.

Two improvements that I believe would significantly improve the paper:

  1. Is there a mathematical model that can motivate algorithm's design choices? It doesn't have to be a novel model or a complex one, but some sort of grounding for the approach would make the core claims more convincing.
  2. Since the method is built to tackle tasks where the unconstrained optimum is different than the constrained one (reward and costs at odds), some experiments that show this more precisely, and show that other methods fail on such tasks are important.
Reviewer_3DiNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how reward-competitive unsafe actions can provide useful supervision for learning better safe policies. To address this problem, the authors propose HNPO (Hard-Negative Policy Optimization), which identifies credible violations near the safety boundary and trains a dedicated proposer to maintain informative negative coverage. The method further introduces HNPC, which compares safe and unsafe actions from the same latent state under a shared continuation policy and encourages safe actions to outperform a negative-only utility reference. Experiments on four Safety-Gymnasium tasks report improved rewards over the USB-RL backbone while maintaining similarly low episodic costs. Ablations and evaluations without the safety planner further examine component contributions and improvements in the learned actor.

Critical strengths
  1. Clear and relevant motivation. The paper studies an interesting question: how reward-competitive unsafe actions can help the policy learn better safe alternatives. Focusing on informative near-boundary violations provides a concrete motivation for hard-negative learning.
  2. Careful method design. Matched-continuation evaluation makes the comparison between candidate actions more controlled. The proposer learns from validated examples without directly differentiating through world-model utility, and proposer actions are kept separate from task-policy samples during optimization.
  3. Promising empirical results and useful actor evaluation. The reported results show reward improvements over USB-RL while maintaining similarly low costs. Paired-seed comparisons and evaluations without the safety planner are useful for examining whether the improvements are reflected in the learned actor.
Critical weaknesses
  1. The presentation needs substantial improvement. The motivation and claimed benefits are repeated across sections, while several important design choices are insufficiently explained. The numerous filtering criteria, thresholds, and auxiliary quantities make the method difficult to follow without a complete algorithm and clearer explanations of how the components interact.
  2. The related-work discussion is insufficient. The discussion remains largely descriptive and does not adequately position HNPO relative to methods that learn from failures, construct counterfactual preferences, or use relative policy optimization. A more comprehensive comparison is needed to clarify which aspects constitute methodological contributions beyond combining existing techniques.
  3. The empirical improvements are not sufficiently convincing. Although HNPO reports higher rewards, the gains over USB-RL appear modest on several tasks, with overlapping confidence intervals in some comparisons. Paired statistical comparisons and quantitative summaries of the reward–cost trade-off would help establish whether the improvements are consistent and practically meaningful.
  4. The ablation study is too limited. Ablations are conducted only on CarGoal1 and primarily remove broad components. They do not sufficiently isolate the effects of the margin, uncertainty adjustment, behavioral-support filtering, or proposer training strategy. Broader evaluations and more targeted controls are needed to justify the individual design choices and establish whether the additional complexity is necessary.
Questions for authors
  1. What is the key methodological distinction between HNPO and existing approaches based on failure learning, counterfactual preferences, and relative policy optimization?
  2. Are the improvements over USB-RL statistically significant across paired seeds? Can the authors provide per-task quantitative comparisons of reward and cost to establish their practical significance?
  3. Do the ablation findings generalize beyond CarGoal1? What are the individual contributions of the margin, uncertainty adjustment, behavioral-support filtering, and proposer training strategy?
  4. Can the authors provide a complete algorithm with the relevant thresholds and update schedules, and clarify why the stated training budget is eight million steps while the learning curves end at two million?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.