acceptodds
Under review as a conference paper at ICLR 2027

SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints

Abstract

In safety-critical domains, reinforcement learning (RL) systems must satisfy strict, zero-cost safety constraints while achieving meaningful task performance. Existing model-free methods can struggle to achieve high safety without substantially compromising task performance. We introduce *Safety-Biased Trust Region Policy Optimisation (SB-TRPO)*, a principled approach to RL with zero-cost constraints, which requires only a fixed fraction of the maximal cost reduction achievable within the trust region, thus retaining flexibility for reward optimisation. We show that the idealised update nevertheless converges to zero cost and maximal reward amongst zero-cost policies in finite MDPs. A practical gradient-based approximation provides local improvements in both safety and reward under suitable gradient alignment. Experiments on *Safety Gymnasium* demonstrate high safety alongside strong task performance.

Primary area:
reinforcement learning
Keywords:
Safe RL, hard constraints, Safety Gymnasium, trust region

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GTp9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies safe RL with hard (zero-cost) constraints and proposes SB-TRPO, a trust-region method in which each update only needs to achieve a fixed fraction of the largest cost reduction possible within the trust region, with the rest of the update used to improve reward. For the idealised update, the authors prove convergence to minimal cost and to the best reward among minimal-cost policies in finite MDPs. The practical algorithm approximates the update with an adaptive convex combination of the reward and cost TRPO steps, with a local improvement guarantee. The authors evaluate the method on Safety Gymnasium navigation and velocity tasks against ten baselines run with a zero cost limit, and report that SB-TRPO achieves a good balance between safety and reward.

Critical strengths

A conceptually simple generalisation of CPO with some theoretical support. The proposed update relaxes CPO's cost-reduction requirement through a single safety-bias parameter, and Example 1 gives a concrete illustration of how CPO's recovery step can lead to reward-suboptimal policies. The idealised update comes with an asymptotic optimality result in finite MDPs, and the practical algorithm is easy to implement and relatively cheap to run. However, the significance of these points is limited by the missing comparison with prior work on hard constraints.

Critical weaknesses
  1. Missing related work and comparisons on hard-constrained safe RL. The paper presents the zero-cost setting as largely unexplored in model-free RL, but closely related formulations of hard or state-wise safety have been studied in prior model-free work, including SCPO [1], ASCPO [2], and reachability-based approaches such as RCRL and RESPO in [3]. These works target persistent or state-wise safety rather than the standard expected-cost formulation and are therefore highly relevant to the paper's motivation. However, none are discussed or included in the evaluation, which makes it difficult to assess the novelty of SB-TRPO relative to existing approaches designed specifically for stringent safety constraints. Another relevant recent baseline is sTRPO [4], a TRPO-based safe RL method with a safety-aware trust region.

  2. The practical method does not reliably achieve the stated zero-cost objective. The paper is motivated by hard constraints requiring zero cost, but SB-TRPO does not reliably reach this target in the experiments. On the harder navigation tasks, safety probability remains substantially below 1, with noticeable variation across seeds, and the appendix notes that policies can return to infeasible behavior after previously reaching near-zero cost. This reveals a considerable gap between the idealized update, which is proven to converge to minimal cost, and the practical sampled algorithm that is actually evaluated.

[1] https://arxiv.org/pdf/2306.12594
[2] https://arxiv.org/pdf/2410.01212
[3] https://arxiv.org/pdf/2407.09645
[4] https://openreview.net/pdf?id=CZ7NIFcVEO

Questions for authors
  1. Could the authors compare SB-TRPO against recent methods designed for hard constraints with zero cost, such as SCPO, ASCPO, RCRL, and RESPO?
  2. Why are the main results based on statistics from the final 50 training episodes rather than independently evaluated frozen policies, given that the paper explicitly focuses on deployment-policy safety rather than safe exploration?
4 comments

We thank the reviewer for their insightful comments. We have substantially extended the experiments and revised the manuscript to address these concerns.

Additional Comparisons with Methods Targeting Hard Safety Constraints

We have extended our evaluation with four additional baselines targeting stronger safety requirements: TRPO-Sauté, SCPO, ASCPO and RESPO, evaluated across all seven Safety Gymnasium tasks (extended Table 1).

Regarding RCRL, to the best of our knowledge, no public implementation of its relevant on-policy variant (RCO) is available. Instead, we evaluate RESPO, a more recent extension of reachability-based safe RL to stochastic settings that reports improved reward–safety trade-offs over RCRL.

We have also substantially revised the manuscript:

  • Introduction (Section 1): Acknowledges prior work targeting stronger safety requirements and the remaining empirical challenges.
  • Related work (Section 2 and Appendix B): Discusses Sauté RL, SCPO, ASCPO, RCRL, RESPO and sTRPO, including their safety formulations and optimisation mechanisms, with extended details in Appendix B.
  • Empirical evaluation (Section 5): Presents the extended comparisons in Table 1.
  • Discussion (Section 6): Examines possible explanations for the observed limitations.

The additional experiments reveal several distinct limitations:

  • TRPO-Sauté achieves relatively high safety probabilities but often suffers from overconservatism, obtaining negative rewards on four of the five navigation tasks. With zero safety budget and binary costs, its state augmentation effectively tracks whether a violation has occurred, with a fixed penalty thereafter that may encourage overly conservative behaviour.

  • SCPO and ASCPO, despite targeting state-wise safety, achieve safety probabilities below 50% on six of the seven tasks. This may partly arise from their maximum trajectory-cost formulation: with binary costs, subsequent violations provide no additional learning signal once a trajectory becomes unsafe, potentially hindering learning when violations are frequent.

  • RESPO achieves competitive reward and safety on some tasks, particularly Car Push and Car Goal. However, safety probabilities remain below 50% on four tasks, reaching only 1.7% on Swimmer Velocity. One possible contributing factor is the difficulty of accurately learning the reachability estimator, which determines the balance between reward and cost optimisation.

Overall, these experiments reinforce our central finding: even methods explicitly designed for stronger safety requirements struggle to consistently achieve high safety alongside meaningful task performance. While SB-TRPO does not achieve perfect safety on all tasks, it offers a more consistently favourable reward–safety trade-off, supporting our approach of balancing progress towards safety with reward optimisation.

Evaluation Protocol

Our evaluation protocol aims to robustly assess learned-policy performance. As described in Section 5.1, metrics are averaged over the last 50 episodes of training.

Evaluating a single frozen policy at the final checkpoint makes results sensitive to the arbitrary stopping point and transient training fluctuations. Averaging late-training performance instead provides a more robust measure of what an algorithm consistently achieves. This assesses learned policies rather than safety during exploration, aligning with our focus on deployment-policy safety.

This evaluation approach has also been adopted by influential RL works, including PPO [1] and FOCOPS [2].

We therefore consider late-training averages a meaningful and robust measure of learned-policy performance, particularly when evaluating both reward and safety.

References

[1] Schulman et al.: Proximal Policy Optimization Algorithms, 2017.

[2] Zhang, Vuong and Ross: First Order Constrained Optimization in Policy Space, NeurIPS 2020.

Practical Zero-Cost Satisfaction

We acknowledge that achieving exact zero cost remains challenging in practice. This limitation is explicitly recognised in our submission. The final paragraph of Section 6 notes that even methods designed to maintain feasibility, such as CPO, can fail to achieve near-perfect safety on harder tasks, despite often failing to achieve meaningful task completion (Table 1). We also discuss contributing factors, including sparse cost signals and finite-sample gradient noise.

We carefully distinguish theoretical guarantees from empirical performance. Our title frames SB-TRPO as a step towards safe RL with hard constraints, while the abstract attributes convergence guarantees to the idealised update and reports high safety alongside strong task performance, rather than guaranteed zero-cost satisfaction.

While reliably achieving both zero-cost satisfaction and high task performance remains an open challenge for model-free RL, we believe SB-TRPO represents meaningful progress, further supported by the additional comparisons above.

Thank you for the detailed response and additional experiments. However, I remain concerned about the reliability of the empirical comparisons and the insufficient analysis of the baseline results.

The reported performance of SCPO and ASCPO differs substantially from the results in their original papers, despite considerable similarities in the evaluation environments. In particular, the original papers report significantly better safety performance than CPO and TRPO-Lagrangian, whereas the results in Table 1 show the opposite trend. The authors have not sufficiently investigated or explained these discrepancies. Furthermore, the explanations provided for the poor performance of these baselines remain largely speculative, without sufficient supporting evidence. It is also unclear whether the baseline implementations and hyperparameter settings were faithfully reproduced.

I therefore maintain my original score.

We thank the reviewer for their follow-up. We would like to highlight several factors contributing to the observed performance of SCPO and ASCPO.

Differences in Evaluation Environments

Our experiments use the standard Safety Gymnasium benchmark of Ji et al. (2023), whereas SCPO and ASCPO evaluate on customised extensions of the older Safety Gym (Ray et al., 2019).

These differences can substantially affect task difficulty. For example, the original experiments include Goal tasks with 8 hazards, whereas Safety Gymnasium's Goal2 tasks contain 10 vases (which cause safety violations) in addition to 10 hazards, causing additional safety challenges. Furthermore, the original experiments predominantly consider Goal-navigation tasks rather than our more diverse Push, Circle and Velocity tasks.

Importantly, the original experiments also use a different cost signal. While both Safety Gym and Safety Gymnasium use binary violation costs by default, SCPO and ASCPO experiments employ graded hazard costs based on penetration depth (see SCPO, Appendix F.1, and ASCPO, Appendix B.1). These provide more informative feedback about the severity of violations, potentially facilitating cost-value estimation and safety learning.

Thus, the original papers demonstrate improvements over CPO and TRPO-Lagrangian on their customised benchmarks, rather than establishing that these improvements extend to the standard Safety Gymnasium benchmark of Ji et al. (2023).

Implementation and Hyperparameters

We used the authors' official SCPO and ASCPO implementations, adapting their environment interfaces for Safety Gymnasium while preserving their optimisation procedures and using the recommended hyperparameters.

Both methods use the same two-layer, 64-unit neural network architecture as SB-TRPO, with Tanh activations. We retained the original KL threshold of 0.02 and a zero-cost threshold. For ASCPO, we used k=7k=7 and the recommended default values ω1=0.005\omega_1=0.005 and ω2=0.007\omega_2=0.007, together with the authors' cost-critic subsampling and monotonic-descent techniques. All methods were trained for 20 million environment interactions per seed, matching SB-TRPO.

Interpretation of the Results

While the original experiments report strong safety performance on their customised environments, these improvements do not consistently carry over to the standard Safety Gymnasium benchmark of Ji et al. (2023).

As discussed in our revision, the maximum-cost formulations proposed by SCPO and ASCPO may provide a weaker supervision signal than the accumulated cost used by other methods, particularly with binary costs: once a violation occurs, subsequent violations do not increase the trajectory maximum, whereas accumulated cost continues to reflect additional violations, providing a signal that can be exploited to learn increasingly safe policies. The modified graded cost signals used in their original experiments may therefore be particularly favourable to these formulations, explaining their weaker performance on the standard Safety Gymnasium benchmark of Ji et al. (2023).

Our findings do not contradict the original results, but indicate that their reported improvements do not necessarily generalise to the standard Safety Gymnasium benchmark of Ji et al. (2023).

Reviewer_rNTqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies reward maximization under a zero expected-cost constraint with nonnegative costs. It argues that maximizing immediate cost reduction during infeasible CPO updates can lead to safe but reward-suboptimal behavior. The proposed idealized update instead requires a fixed fraction of the maximum cost reduction achievable within the current trust region and maximizes reward subject to this requirement. For finite MDPs, Theorem 1 establishes asymptotic cost optimality and, for a safety bias strictly between zero and one, reward optimality among cost-optimal policies. The practical SB-TRPO algorithm approximates this update by combining reward-increasing and cost-decreasing natural-gradient steps, followed by a line search. Theorem 2 establishes local cost improvement and conditional reward improvement. Experiments on seven safety benchmarks compare the resulting reward and safety trade-offs with existing safe-RL methods.

Critical strengths
  1. The motivation identifies a consequential limitation of prioritizing maximal immediate cost reduction. A recovery update can suppress visits to a promising region instead of learning how to behave safely within it. Studying alternative update dynamics is worthwhile, and Example 1 makes this issue concrete rather than relying only on benchmark performance.

  2. Theorem 1 gives substantive support to the fractional cost-reduction principle: under exact optimization in a finite MDP, sacrificing some immediate cost reduction does not prevent asymptotic cost optimality and permits reward optimality among cost-optimal policies. This is stronger than merely showing that the proposed update is feasible or decreases cost locally.

Critical weaknesses
  1. The justification for the practical gradient combination is substantially weaker than the motivation for changing recovery dynamics. Combining the reward and cost directions is intuitive, but the paper does not establish when this particular approximation improves on pure recovery or on an exact solution of its own local subproblem, Update 3. If high reward favors an unsafe shortcut while safe task completion requires a qualitatively different route, combining the two local directions need not discover the safe route; it may simply slow abandonment of the shortcut. This is a concern about the practical algorithm, not a counterexample to Theorem 1. Theorem 2's reward guarantee requires the cost-reducing direction to be non-adverse to reward, leaving the conflicting-gradient case particularly relevant to this concern unresolved. The benchmark results also do not clearly separate better optimization from accepting more violations: on CarGoal, SB-TRPO improves safe reward but has a safe-episode probability of approximately 0.66, versus 0.82 for CPO. A comparison with an exact solver for Update 3, and controlled tasks with distinct safe and unsafe routes, would help determine whether the proposed approximation actually addresses the motivating failure mode.

  2. The convergence guarantee does not carry through to the implemented algorithm. Theorem 1 assumes exact optimization over policies using true returns, whereas SB-TRPO replaces returns with surrogates, restricts the local solution to a line segment between two directions, and applies backtracking. Approximate subproblem solutions are legitimate in trust-region methods, but retaining a KL constraint alone does not establish convergence: an appropriate sufficient-improvement or approximation-quality argument is still needed. Theorem 2 only establishes the existence of a sufficiently small step giving local improvement; it does not show convergence of repeated practical updates to zero cost, a constrained stationary point, or the optimum in Theorem 1. Moreover, Algorithm 1 accepts steps based on empirical KL and nonincrease of the cost surrogate, without checking reward improvement or preserving the original fractional cost-decrease requirement after backtracking. Appendix D verifies approximation accuracy on the same sampled surrogates but does not close this theoretical gap. This matters for the paper's hard-constraint framing, especially since several navigation tasks retain substantial violation probabilities at the end of training.

Questions for authors
  1. What evidence distinguishes the benefit of the fractional cost-reduction requirement from the effect of approximately solving Update 3 by a two-direction mixture? In particular, how does SB-TRPO compare with an exact solution of Update 3 under matched estimation and line-search settings, and what happens when safe and unsafe high-reward behaviors require distinct routes with conflicting local gradients? A theoretical characterization or a controlled experiment addressing this case could materially strengthen the case for the practical method.

  2. Which convergence properties, beyond one-step improvement under exact gradients, can be established for the actual sequence of SB-TRPO updates? Please clarify whether the mixture and line-search acceptance rule satisfy sufficient conditions for convergence of the relevant trust-region scheme, how the fractional decrease condition is affected by backtracking, and what additional assumptions would connect the practical algorithm to Theorem 1. If no such result is available, please delimit the convergence and hard-safety claims accordingly.

4 comments

We thank the reviewer for their careful assessment and constructive suggestions. We have conducted additional experiments and revised the manuscript to address both concerns, as detailed below.

Fractional cost reduction and the convex-combination approximation

We emphasise that the fractional cost-reduction requirement is the key ingredient of our approach. It avoids the overly aggressive recovery behaviour illustrated in Example 1, while Theorem 1 establishes asymptotic cost optimality and, for β∈(0,1)\beta\in(0,1), reward optimality among cost-optimal policies for the idealised Update 1. Both the exact solution of Update 3 and our convex combination approximate this idealised principle.

Our convex combination satisfies the prescribed linearised cost-reduction requirement while generally using less of the available KL budget, adaptively restricting reward-oriented updates without additional hyperparameters. Theorem 2 establishes local cost-improvement and conditional reward-improvement guarantees for this approximation.

Controlled experiment with conflicting gradients

To investigate whether the convex combination merely delays abandonment of unsafe behaviour, we have added Example 2 (Section 4.2 and Appendix C.2.1). This three-decision-state MDP offers a safe exit, an unsafe high-reward shortcut, and a distinct, longer route to a rewarding safe goal. We compute the discounted objectives and gradients analytically, avoiding gradient-estimation noise.

Despite conflicting reward and cost directions, both the convex combination and the exact solver learn the rewarding safe route, whereas CPO's recovery updates increasingly avoid the rewarding region and approach a reward-suboptimal policy. Thus, the convex combination can discover a qualitatively different safe route rather than merely slowing recovery.

Empirical comparison with the exact solver

We have additionally compared the convex combination with an exact solution of Update 3 restricted to the span of the computed conjugate-gradient directions, under matched optimisation settings (Section 5.3 and Appendix D.3.2, Table 6).

The exact solver achieves greater immediate predicted reward improvement, as expected. However, on PointPush, it only approaches the mixture's safety probability when increasing the safety bias to β=0.95\beta=0.95 (versus 0.750.75 for the mixture), but consequently achieves considerably lower reward. On CarCircle, both methods achieve near-perfect safety with comparable reward.

These results suggest that greater immediate reward improvement does not necessarily translate into a better long-term reward–safety trade-off. In particular, the mixture's adaptive restriction of reward-oriented updates can be beneficial, as demonstrated on PointPush.

Convergence guarantees and practical line-search behaviour

We agree that the convergence guarantees for the idealised update do not directly extend to the practical implementation. We clarify the scope of our theoretical results and provide additional evidence concerning the effect of backtracking.

Scope of theoretical guarantees. As distinguished in the abstract and introduction, Theorem 1 establishes asymptotic optimality for the idealised policy-space Update 1 in finite MDPs, whereas Theorem 2 establishes local improvement properties for the gradient-based approximation under exact gradients. Importantly, our practical convex combination preserves the central fractional cost-reduction requirement in the linearised problem, with strict local cost improvement and conditional reward improvement established by Theorem 2. We do not claim that these results establish asymptotic convergence of the sampled Algorithm 1.

Effect of backtracking. We agree that reducing the step size may weaken the original fractional cost-reduction requirement. To investigate its practical significance, we instrumented the original convex-combination implementation on Point Push and Car Circle, each trained for 20 million environment steps with β=0.75\beta=0.75.

All 2,000 proposed updates were accepted without backtracking, with no failed line searches. Whenever the estimated cost gradient was nonzero (999 and 154 updates, respectively), the achieved linearised cost-reduction fraction was 0.750.75, up to numerical precision. Furthermore, the sampled cost surrogate did not increase in any update, although the observed reductions were generally smaller than their linearised predictions. We have added these diagnostics to Appendix E (Table 8).

The line search does not explicitly enforce reward improvement, and we therefore do not claim monotonic reward improvement for the practical algorithm. Nevertheless, the sampled reward surrogate increased in 95.5% and 99.4% of updates on Point Push and Car Circle, respectively (Table 8).

Limitations. We acknowledge that neither the mixture nor the exact solution to (Update 3) establishes convergence of the practical policy sequence. Establishing such a result would require additional control of gradient-estimation and subproblem-approximation errors, sufficient progress in the true objectives, and assumptions on the expressiveness of the parameterised policy class. In particular, we do not claim asymptotic convergence or guaranteed zero-cost achievement for the sampled algorithm.

Manuscript revisions

We have revised Section 4.2 to clarify the relationship between the exact and approximate updates, added the controlled MDP in Example 2 (Appendix C.2.1), and extended the empirical ablations in Section 5.3 and Appendix D.3.2. We have also removed the previous numerical-instability motivation for the convex combination, instead clarifying its geometric properties and empirical behaviour. Finally, we have added line-search diagnostics to Appendix E (Table 8) and have explicitly clarified the limitations of the practical convergence guarantees in Section 6 (Limitations).

Thanks for the rebuttal. However, my main concern remains.

The additional three-state MDP is a useful example, but it does not resolve my concern about cases where the safe, high-reward behavior lies in a different policy mode from the unconstrained high-reward behavior. In such settings, the local reward and cost gradients may not guide the convex-combination update toward the safe high-reward solution. The example shows that the method can work in one constructed case, but does not establish that it works generally or characterize when it may fail.

Regarding the theoretical guarantees, a gap remains between the idealized asymptotic convergence result and the sampled practical algorithm: the latter has local improvement guarantees under exact gradients, but no established asymptotic convergence or guarantee of reaching zero cost. The line-search diagnostics are useful empirical evidence for two tasks, but do not close this theoretical gap.

We thank the reviewer for their follow-up.

Example 1 identifies a failure mode of CPO even for idealised updates, while Theorem 1 establishes asymptotic optimality for our proposed fractional cost-reduction principle. We believe these constitute valuable theoretical contributions.

We acknowledge that these guarantees do not directly extend to the sample-based implementation, as is also the case for TRPO and CPO. More general convergence guarantees for practical policy-gradient methods typically require strong assumptions on the MDP and policy class. We have clarified these limitations in the revised manuscript.

Reviewer_6iZXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies safe RL with hard constraints, where the cost threshold in a CMDP is set to 0. The authors argue that Lagrangian methods can lead to unsafe or overly conservative policies, while CPO-style methods can get stuck in the reward recovery phase. They propose SB-TRPO, which aims to achieve only a β\beta fraction of the largest cost decrease rather than reducing the cost to 0, while maximizing reward subject to this requirement. The authors prove convergence to the minimum cost and the highest reward among minimum-cost policies when β≤1\beta \le 1. On seven Safety Gymnasium tasks, SB-TRPO with β=0.75\beta = 0.75 achieves positive rewards with safety probabilities ranging from 0.66 to 0.99.

Critical strengths

Safe RL with hard constraints is important and practical.

The method is simple and inexpensive to implement on top of TRPO, and the experiments cover 11 baselines and 7 tasks.

Critical weaknesses

SB-TRPO balances reward and safety using a weight λ\lambda that is recalculated from each batch. If a batch has no safety violations, the cost gradient becomes zero, and the safety correction immediately disappears. This may explain why safety performance sometimes deteriorates again, as shown in Appendix C.2.2. Also, λ\lambda does not account for violations in previous batches. Even if violations continue, the method does not automatically increase its safety penalty. This may explain why safety performance stops improving on harder tasks.

The authors argue that the exact solution is numerically unstable, but this claim is not well supported. Lemma 8 shows that the exact solution only requires quantities that the algorithm already computes, along with solving a simple quadratic equation.

All ten baselines are designed for soft constraints and are simply run with the threshold set to 0. The claim that SB-TRPO is better suited to hard constraints therefore needs to be supported by a comparison with model-free methods designed for that setting, such as Sauté RL [1] and Reachability Constrained RL [2].

[1] Sootla, A., Cowen-Rivers, A.I., Jafferjee, T., Wang, Z., Mguni, D.H., Wang, J., and Bou Ammar, H. Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentation. Proceedings of the 39th International Conference on Machine Learning, 2022.

[2] Yu, D., Ma, H., Li, S. & Chen, J. Reachability Constrained Reinforcement Learning. Proceedings of the 39th International Conference on Machine Learning, 2022.

Questions for authors

Could the authors plot λ\lambda over the course of training?

What criterion was used to choose β=0.75\beta = 0.75?

How does the convex combination compare with the exact solution in Lemma 8 in terms of both performance and numerical stability?

4 comments

We thank the reviewer for their constructive comments. We have substantially extended the experiments and revised the manuscript to address these concerns.

Adaptive Safety Correction and Training Dynamics

We assume the reviewer is referring to the mixing coefficient μ\mu in Eq. (4).

Evolution of the mixing coefficient. In response to the reviewer's request, we have added Figure 10, showing the evolution of μ\mu during training on four representative tasks (three seeds each). Despite using the same fixed safety bias β=0.75\beta=0.75 throughout, μ\mu exhibits substantially different behaviour across environments: it remains around 0.75–0.79 on Point Push and Point Goal, decreases towards 0.1 on Car Circle, and fluctuates around 0.2–0.4 on Swimmer Velocity. This illustrates how SB-TRPO automatically adapts the balance between reward and cost reduction to the local gradient geometry, without task-specific tuning or accumulated penalties.

Absence of accumulated penalties. This is a deliberate design choice and one of the motivations for SB-TRPO. Under zero-cost constraints, standard Lagrangian multiplier updates can only accumulate penalties, making performance sensitive to the initial multiplier and its learning rate. If penalties grow too slowly, policies may remain unsafe; if they grow too quickly, optimisation may become excessively conservative. SB-TRPO instead requires a fixed fraction of the maximal locally achievable cost reduction, avoiding the need to accumulate penalties across updates.

Limitations of empirical safety. We agree that when a batch contains no observed violations, the estimated cost gradient may vanish, causing the update to prioritise reward. However, practical safety limitations are not unique to SB-TRPO: even methods such as CPO and C-TRPO, which often sacrifice substantial task performance to prioritise safety, fail to achieve near-perfect safety on several harder tasks. As discussed in Section 6, sparse cost signals and finite-sample gradient noise pose challenges shared across model-free methods.

Convex Combination versus Exact Solution

We agree that the previous numerical-instability justification for the convex combination was insufficiently supported and have removed it. We now clarify the geometric properties of the approximation and provide a direct empirical comparison.

Theoretical motivation. The key ingredient of SB-TRPO is the fractional cost-reduction requirement, which avoids the overly aggressive recovery behaviour illustrated in Example 1. Theorem 1 establishes asymptotic cost optimality and, for β∈(0,1)\beta\in(0,1), reward optimality amongst cost-optimal policies for the idealised Update 1.

Both the exact solution of Update 3 and our convex combination approximate this idealised principle. The convex combination satisfies the prescribed linearised cost-reduction requirement while generally using less of the available KL budget, restricting reward-oriented updates when the gradients conflict. Theorem 2 establishes local cost-improvement and conditional reward-improvement guarantees for this approximation.

Empirical comparison. We have compared the convex combination with an exact solution of Update 3 restricted to the span of the computed conjugate-gradient directions, under matched optimisation settings (Section 5.3 and Appendix D.3.2, Table 6).

The exact solver achieves greater immediate predicted reward improvement, as expected. However, on Point Push, it requires a substantially higher safety bias to approach the mixture's safety probability, but consequently achieves considerably lower reward. On Car Circle, both methods achieve near-perfect safety with comparable reward.

These results suggest that greater immediate reward improvement does not necessarily translate into a better long-term reward–safety trade-off. In particular, the mixture's adaptive restriction of reward-oriented updates can be beneficial, as demonstrated on Point Push.

We have also added a controlled MDP experiment (Example 2, Section 4.2 and Appendix C.2.1), using analytically computed objectives and gradients. Despite conflicting reward and cost directions, both the convex combination and exact solver discover the rewarding safe route, whereas CPO's recovery updates approach a reward-suboptimal policy.

Additional Comparisons with Methods Targeting Hard Safety Constraints

We have extended our evaluation with four additional baselines targeting stronger safety requirements: TRPO-Sauté, SCPO, ASCPO and RESPO, evaluated across all seven Safety Gymnasium tasks (extended Table 1).

Regarding RCRL, to the best of our knowledge, no public implementation of its relevant on-policy variant (RCO) is available. Instead, we evaluate RESPO, a more recent extension of reachability-based safe RL to stochastic settings.

The additional experiments reveal several distinct limitations:

  • TRPO-Sauté achieves relatively high safety probabilities but often suffers from overconservatism, obtaining negative rewards on four of the five navigation tasks. With zero safety budget and binary costs, its state augmentation effectively tracks whether a violation has occurred, with a fixed penalty thereafter that may encourage overly conservative behaviour.

  • SCPO and ASCPO, despite targeting state-wise safety, achieve safety probabilities below 50% on six of the seven tasks. Their maximum trajectory-cost formulation provides no additional signal for reducing the number of violations once a trajectory becomes unsafe, potentially hindering learning when violations are frequent.

  • RESPO achieves competitive reward and safety on some tasks, particularly Car Push and Car Goal. However, safety probabilities remain below 50% on four tasks, reaching only 1.7% on Swimmer Velocity. One possible contributing factor is the difficulty of accurately learning the reachability estimator used to balance reward and cost optimisation.

Overall, these experiments reinforce our central finding: even methods explicitly designed for stronger safety requirements struggle to consistently achieve high safety alongside meaningful task performance. While SB-TRPO does not achieve perfect safety on all tasks, it offers a more consistently favourable reward–safety trade-off.

Choice of Safety Bias β\beta

We selected β=0.75\beta=0.75 based on a lightweight grid search over β∈[0.6,0.9]\beta\in[0.6,0.9] on two tasks (Section 5.3 and Figure 12). The results show an approximately linear reward–safety trade-off, without a sharply distinguished optimal value. We therefore chose β=0.75\beta=0.75 as a moderate compromise between reward and safety.

Crucially, the same value is used across all seven benchmark tasks, without task-specific tuning. The ablation suggests that our approach is not overly sensitive to the precise choice of β\beta.

Manuscript Revisions

We have revised Sections 1–2 to discuss prior methods targeting stronger safety requirements; extended Table 1 with four additional baselines; clarified the exact and approximate updates in Section 4.2; added the controlled MDP example; extended the ablations in Section 5.3 and Appendix D.3.2; added the mixing-coefficient trajectories in Figure 10; and expanded Section 6 to discuss the remaining empirical challenges.

Thank you for the detailed response. The removal of the numerical-instability argument, the exact-solver comparison, and the four added hard-constraint baselines address my second and third concerns.

Two issues remain, and I encourage the authors to discuss them clearly in the paper. First, the new diagnostics suggest the adaptive behaviour plays a limited role. Substituting the closed-form steps into Eq. (4) gives μ=(ρ+β)/(1+ρ)\mu=(\rho+\beta)/(1+\rho), where ρ is the Fisher-metric cosine between the reward and cost gradients. On Car Circle, only 154 of 1,000 updates have a nonzero cost gradient, so most updates are pure reward steps, and the low average μ reflects whether a violation occurred rather than gradual adaptation. On Point Push and Point Goal, μ stays near β, so the update is effectively a fixed mixture. Second, the exact-solver results suggest that the gains come mainly from the convex combination’s shorter steps rather than from the principle supported by Theorem 1. A comparison with the exact solver under a matched KL budget would clarify this.

Given the expanded comparisons, I raise my score to 3.

Thank you for your constructive feedback and for reassessing our submission. The central idea of our approach is the fractional cost-reduction principle, motivated by Example 1 and supported by Theorem 1. Our convex-combination update provides one practical approximation of the idealised policy-space Update 1, and other gradient-based approximations are also possible.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Manuscript Revisions

We thank the reviewers for their constructive feedback. We briefly summarise our main findings, revisions and responses to the remaining concerns.

Additional baselines. We have substantially expanded our discussion of related work and evaluated TRPO-Sauté, SCPO, ASCPO and RESPO across all seven benchmark tasks (Table 1). Our results demonstrate that even methods explicitly designed for stronger safety requirements struggle to consistently achieve high safety alongside meaningful task performance, further supporting the favourable reward–safety trade-off achieved by SB-TRPO. We discuss potential explanations for the low safety achieved by SCPO and ASCPO, highlighting important differences between the standard Safety Gymnasium benchmark used in our experiments and the customised environments with graded cost signals used in their original papers. We also detail our use of their official implementations and recommended hyperparameters.

Theoretical contribution and scope. Example 1 identifies a failure mode of CPO even for idealised policy-space updates, whilst Theorem 1 establishes asymptotic optimality for our proposed fractional cost-reduction principle. Establishing convergence guarantees for practical, sample-based policy-gradient algorithms generally requires strong additional assumptions on the MDP and policy class. Our focus is instead on developing a theoretically grounded practical algorithm that directly addresses failure modes of existing methods.

Practical approximation. The central idea of our approach is the fractional cost-reduction principle, which admits different gradient-based approximations. Our additional controlled MDP experiment (Example 2) illustrates the benefits of this principle even when using our convex-combination approximation in a setting with conflicting reward and cost gradients. We have replaced the numerical-instability motivation for the convex combination with a discussion of its geometric properties and empirical behaviour, supported by additional exact-solver comparisons, mixing-coefficient trajectories and line-search diagnostics.

We believe that our theoretical findings and the substantially expanded empirical evaluation constitute a valuable contribution towards model-free reinforcement learning with hard safety constraints.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.