- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies safe RL with hard (zero-cost) constraints and proposes SB-TRPO, a trust-region method in which each update only needs to achieve a fixed fraction of the largest cost reduction possible within the trust region, with the rest of the update used to improve reward. For the idealised update, the authors prove convergence to minimal cost and to the best reward among minimal-cost policies in finite MDPs. The practical algorithm approximates the update with an adaptive convex combination of the reward and cost TRPO steps, with a local improvement guarantee. The authors evaluate the method on Safety Gymnasium navigation and velocity tasks against ten baselines run with a zero cost limit, and report that SB-TRPO achieves a good balance between safety and reward.
Critical strengths
A conceptually simple generalisation of CPO with some theoretical support. The proposed update relaxes CPO's cost-reduction requirement through a single safety-bias parameter, and Example 1 gives a concrete illustration of how CPO's recovery step can lead to reward-suboptimal policies. The idealised update comes with an asymptotic optimality result in finite MDPs, and the practical algorithm is easy to implement and relatively cheap to run. However, the significance of these points is limited by the missing comparison with prior work on hard constraints.
Critical weaknesses
-
Missing related work and comparisons on hard-constrained safe RL. The paper presents the zero-cost setting as largely unexplored in model-free RL, but closely related formulations of hard or state-wise safety have been studied in prior model-free work, including SCPO [1], ASCPO [2], and reachability-based approaches such as RCRL and RESPO in [3]. These works target persistent or state-wise safety rather than the standard expected-cost formulation and are therefore highly relevant to the paper's motivation. However, none are discussed or included in the evaluation, which makes it difficult to assess the novelty of SB-TRPO relative to existing approaches designed specifically for stringent safety constraints. Another relevant recent baseline is sTRPO [4], a TRPO-based safe RL method with a safety-aware trust region.
-
The practical method does not reliably achieve the stated zero-cost objective. The paper is motivated by hard constraints requiring zero cost, but SB-TRPO does not reliably reach this target in the experiments. On the harder navigation tasks, safety probability remains substantially below 1, with noticeable variation across seeds, and the appendix notes that policies can return to infeasible behavior after previously reaching near-zero cost. This reveals a considerable gap between the idealized update, which is proven to converge to minimal cost, and the practical sampled algorithm that is actually evaluated.
[1] https://arxiv.org/pdf/2306.12594
[2] https://arxiv.org/pdf/2410.01212
[3] https://arxiv.org/pdf/2407.09645
[4] https://openreview.net/pdf?id=CZ7NIFcVEO
Questions for authors
- Could the authors compare SB-TRPO against recent methods designed for hard constraints with zero cost, such as SCPO, ASCPO, RCRL, and RESPO?
- Why are the main results based on statistics from the final 50 training episodes rather than independently evaluated frozen policies, given that the paper explicitly focuses on deployment-policy safety rather than safe exploration?
4 comments
Evaluation Protocol
Our evaluation protocol aims to robustly assess learned-policy performance. As described in Section 5.1, metrics are averaged over the last 50 episodes of training.
Evaluating a single frozen policy at the final checkpoint makes results sensitive to the arbitrary stopping point and transient training fluctuations. Averaging late-training performance instead provides a more robust measure of what an algorithm consistently achieves. This assesses learned policies rather than safety during exploration, aligning with our focus on deployment-policy safety.
This evaluation approach has also been adopted by influential RL works, including PPO [1] and FOCOPS [2].
We therefore consider late-training averages a meaningful and robust measure of learned-policy performance, particularly when evaluating both reward and safety.
References
[1] Schulman et al.: Proximal Policy Optimization Algorithms, 2017.
[2] Zhang, Vuong and Ross: First Order Constrained Optimization in Policy Space, NeurIPS 2020.
Practical Zero-Cost Satisfaction
We acknowledge that achieving exact zero cost remains challenging in practice. This limitation is explicitly recognised in our submission. The final paragraph of Section 6 notes that even methods designed to maintain feasibility, such as CPO, can fail to achieve near-perfect safety on harder tasks, despite often failing to achieve meaningful task completion (Table 1). We also discuss contributing factors, including sparse cost signals and finite-sample gradient noise.
We carefully distinguish theoretical guarantees from empirical performance. Our title frames SB-TRPO as a step towards safe RL with hard constraints, while the abstract attributes convergence guarantees to the idealised update and reports high safety alongside strong task performance, rather than guaranteed zero-cost satisfaction.
While reliably achieving both zero-cost satisfaction and high task performance remains an open challenge for model-free RL, we believe SB-TRPO represents meaningful progress, further supported by the additional comparisons above.
Thank you for the detailed response and additional experiments. However, I remain concerned about the reliability of the empirical comparisons and the insufficient analysis of the baseline results.
The reported performance of SCPO and ASCPO differs substantially from the results in their original papers, despite considerable similarities in the evaluation environments. In particular, the original papers report significantly better safety performance than CPO and TRPO-Lagrangian, whereas the results in Table 1 show the opposite trend. The authors have not sufficiently investigated or explained these discrepancies. Furthermore, the explanations provided for the poor performance of these baselines remain largely speculative, without sufficient supporting evidence. It is also unclear whether the baseline implementations and hyperparameter settings were faithfully reproduced.
I therefore maintain my original score.
We thank the reviewer for their follow-up. We would like to highlight several factors contributing to the observed performance of SCPO and ASCPO.
Differences in Evaluation Environments
Our experiments use the standard Safety Gymnasium benchmark of Ji et al. (2023), whereas SCPO and ASCPO evaluate on customised extensions of the older Safety Gym (Ray et al., 2019).
These differences can substantially affect task difficulty. For example, the original experiments include Goal tasks with 8 hazards, whereas Safety Gymnasium's Goal2 tasks contain 10 vases (which cause safety violations) in addition to 10 hazards, causing additional safety challenges. Furthermore, the original experiments predominantly consider Goal-navigation tasks rather than our more diverse Push, Circle and Velocity tasks.
Importantly, the original experiments also use a different cost signal. While both Safety Gym and Safety Gymnasium use binary violation costs by default, SCPO and ASCPO experiments employ graded hazard costs based on penetration depth (see SCPO, Appendix F.1, and ASCPO, Appendix B.1). These provide more informative feedback about the severity of violations, potentially facilitating cost-value estimation and safety learning.
Thus, the original papers demonstrate improvements over CPO and TRPO-Lagrangian on their customised benchmarks, rather than establishing that these improvements extend to the standard Safety Gymnasium benchmark of Ji et al. (2023).
Implementation and Hyperparameters
We used the authors' official SCPO and ASCPO implementations, adapting their environment interfaces for Safety Gymnasium while preserving their optimisation procedures and using the recommended hyperparameters.
Both methods use the same two-layer, 64-unit neural network architecture as SB-TRPO, with Tanh activations. We retained the original KL threshold of 0.02 and a zero-cost threshold. For ASCPO, we used and the recommended default values and , together with the authors' cost-critic subsampling and monotonic-descent techniques. All methods were trained for 20 million environment interactions per seed, matching SB-TRPO.
Interpretation of the Results
While the original experiments report strong safety performance on their customised environments, these improvements do not consistently carry over to the standard Safety Gymnasium benchmark of Ji et al. (2023).
As discussed in our revision, the maximum-cost formulations proposed by SCPO and ASCPO may provide a weaker supervision signal than the accumulated cost used by other methods, particularly with binary costs: once a violation occurs, subsequent violations do not increase the trajectory maximum, whereas accumulated cost continues to reflect additional violations, providing a signal that can be exploited to learn increasingly safe policies. The modified graded cost signals used in their original experiments may therefore be particularly favourable to these formulations, explaining their weaker performance on the standard Safety Gymnasium benchmark of Ji et al. (2023).
Our findings do not contradict the original results, but indicate that their reported improvements do not necessarily generalise to the standard Safety Gymnasium benchmark of Ji et al. (2023).
We thank the reviewer for their insightful comments. We have substantially extended the experiments and revised the manuscript to address these concerns.
Additional Comparisons with Methods Targeting Hard Safety Constraints
We have extended our evaluation with four additional baselines targeting stronger safety requirements: TRPO-Sauté, SCPO, ASCPO and RESPO, evaluated across all seven Safety Gymnasium tasks (extended Table 1).
Regarding RCRL, to the best of our knowledge, no public implementation of its relevant on-policy variant (RCO) is available. Instead, we evaluate RESPO, a more recent extension of reachability-based safe RL to stochastic settings that reports improved reward–safety trade-offs over RCRL.
We have also substantially revised the manuscript:
The additional experiments reveal several distinct limitations:
TRPO-Sauté achieves relatively high safety probabilities but often suffers from overconservatism, obtaining negative rewards on four of the five navigation tasks. With zero safety budget and binary costs, its state augmentation effectively tracks whether a violation has occurred, with a fixed penalty thereafter that may encourage overly conservative behaviour.
SCPO and ASCPO, despite targeting state-wise safety, achieve safety probabilities below 50% on six of the seven tasks. This may partly arise from their maximum trajectory-cost formulation: with binary costs, subsequent violations provide no additional learning signal once a trajectory becomes unsafe, potentially hindering learning when violations are frequent.
RESPO achieves competitive reward and safety on some tasks, particularly Car Push and Car Goal. However, safety probabilities remain below 50% on four tasks, reaching only 1.7% on Swimmer Velocity. One possible contributing factor is the difficulty of accurately learning the reachability estimator, which determines the balance between reward and cost optimisation.
Overall, these experiments reinforce our central finding: even methods explicitly designed for stronger safety requirements struggle to consistently achieve high safety alongside meaningful task performance. While SB-TRPO does not achieve perfect safety on all tasks, it offers a more consistently favourable reward–safety trade-off, supporting our approach of balancing progress towards safety with reward optimisation.