- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Hard-Negative Policy Optimization (HNPO), a world-model-based safe RL framework that treats reward-competitive, near-boundary unsafe actions as hard negatives for policy learning. HNPO introduces a credibility-aware mining process to identify near-boundary violations that are competitive and model-supported. Moreover, HNPO proposes a conservative negative proposer learned via weighted behavior cloning and validated elite distillation to sustain negative coverage as the policy becomes safer. On four tasks using USB-RL as the backbone, HNPO improves task reward while keeping episodic cost low.
Critical strengths
- Introduces the clear and useful notion of “tempting failures” and operationalizes them as hard negatives in a model-based safe RL pipeline.
- The credible negative mining integrates boundary checks, competitiveness, and behavioral support with uncertainty-aware robust evaluation—going beyond naive “unsafe” tagging.
Critical weaknesses
- Heavy reliance on the learned world model’s uncertainty calibration and short-horizon robustness; existing analyses suggest ensemble disagreement in latent imagination can decouple from long-horizon error, which could undermine credibility checks and boundary determinations.
- The counterfactual comparison varies only the first action under matched continuation, which may be insufficient when the crux of safe alternatives spans multi-step control.
- No formal guarantees or bounds on bias introduced by the negative-only anchor or on the stability of HNPC’s auxiliary gradients relative to the base objective.
- Compute and sample-efficiency overhead of negative mining, proposer training, and repeated robust evaluation are not quantified; it is unclear whether improvements come at substantial additional cost.
Questions for authors
- Does HNPO maintain or improve cumulative training-time safety (e.g., total number of violations) compared to USB-RL and SafeDreamer? Are costs consistently kept under budget across seeds, and how often are violations near-boundary vs. severe?
- How sensitive are results to the uncertainty coefficients, boundary margins, competitiveness thresholds, and the target margi? Can you provide sensitivity analyses or guidance for selecting them?