acceptodds
Under review as a conference paper at ICLR 2027

A Unified Bellman Operator for Safety-Critical Reinforcement Learning

Abstract

Reinforcement learning in safety-critical domains requires maximizing task performance while strictly adhering to safety constraints. Existing safe reinforcement learning paradigms *typically* force a trade-off: they either require *a priori* knowledge to provide strict safety guarantees (e.g., safety filters), or they enable joint learning but only satisfy safety constraints on average. In this work, we propose a novel Bellman operator that unifies performance and safety objectives into a *joint* value function. We show that temporal difference learning with the joint Bellman operator converges under a two-timescale stochastic approximation framework. On the fast timescale, the *safety value* of the learning joint policy is estimated, while the *joint value* is estimated on the slow timescale. Convergence is ensured by formulating the limiting dynamics as an *occupation-averaged* differential inclusion, and showing that it asymptotically converges to a set of limiting optimal safety-constrained task value functions. *Theoretically*, once converged, the resulting optimal policy maximizes task return while maintaining safety at all times. Empirical evaluations on continuous control tasks with *neural approximations* demonstrate stable convergence with near-zero safety violations at test time.

Primary area:
reinforcement learning
Keywords:
Safe Reinforcement Learning, Safety-Critical Domain, Deep Reinforcement Learning, Stochastic Approximation Theory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4Cj2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Bellman operator that unifies task value and safety value. In the tabular setting, the formulation is proven to converge to the optimal safety-constrained task value, given a sufficiently large margin scale. The framework is novel to my knowledge, yet the validation does not specifically back the optimality claim in continuous control environments (or at least its robustness under practical assumption violations).

Critical strengths
  1. The core idea is novel and conceptually clean, with substantial theoretical analysis.

  2. Substantial experimental results are included, including optimality-related tests in the tabular setting and practical tests with function approximations in MuJoCo environments.

Critical weaknesses
  1. The theory and the practical implementation have a notable mismatch. While the theoretical motivation is optimality, for various continuous tasks where the optimum lies near the constraint set boundary, optimality can require an intractably large KK. The paper thus does not discuss optimality in the practical continuous experiments, instead focusing on stable convergence, which is a distinct property compared to optimality and is not really justified by the theoretical framework. This appears to be unaddressable. However, I do not regard this as a major problem, since strict guarantees generally do not carry over with function approximation without additional system knowledge. I suggest improving the presentation of this mismatch by more explicitly stating it early in the paper.

  2. The practical KK is hand-tuned per environment, choosing ‘the maximum value of KK such that there are no safety violations / algorithmic instabilities’. The exact selection mechanism is not clear. Even with this tuning, convergence is not very stable for some environments (e.g., SafetyWalker2dVel-v1).

  3. No source code is provided with the submission. This does limit reproducibility at the time of review and affects my confidence score.

Questions for authors
  1. Figure 4, where the instability of large KK should be analyzed, does not show the standard deviation from different runs. I believe it is very important to see how much numerical instability large KK actually introduces, as optimality hinges on large KK.

  2. The authors could use the smallest gg among states in training that are certified safe to estimate a K~∗\tilde{K}^{\ast} (lower bound on K∗K^{\ast}). If even this K~∗\tilde{K}^{\ast} is way larger than the KK used in the experiments, the optimality claim might not be practically attainable.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 4Cj2

We thank the reviewer for their insightful comments and suggestions. The main concerns are (1) explicitly acknowledging the theory-practical mismatch early on in the paper, (2) adding variance bands in Fig. 4, (3) obtaining/estimating K~∗\tilde{K}^* based on the formula for g(s)g(s) provided, and (4) non-availability of source code. Together with the clarifications below, does this address the reviewer's concern?

 

W1. The theory and the practical implementation have a notable mismatch. While the theoretical motivation is optimality, for various continuous tasks where the optimum lies near the constraint set boundary, optimality can require an intractably large KK. The paper thus does not discuss optimality in the practical continuous experiments, instead focusing on stable convergence, which is a distinct property compared to optimality and is not really justified by the theoretical framework. This appears to be unaddressable. However, I do not regard this as a major problem, since strict guarantees generally do not carry over with function approximation without additional system knowledge. I suggest improving the presentation of this mismatch by more explicitly stating it early in the paper.

Thank you for the suggestion. We have clarified in the end of Section 1 that the theoretical analysis holds only for finite and tabular settings; with continuous control experiments, we intend to show the scalability and stable learning of our joint formulation for high-dimensional state-action spaces.

 

Figure 4, where the instability of large KK should be analyzed, does not show the standard deviation from different runs. I believe it is very important to see how much numerical instability large KK actually introduces, as optimality hinges on large KK.

Thank you for the suggestion. We have added the variance bands (±1\pm 1 std) to Figure 44 in Appendix C.44. We can generally see that, as KK increases, the variance band either stays the same or tends to slightly increase. We ran the experiments up to K=2000K=2000 as each of the KK values possesses convergent training performance; beyond this, we observed that not all environments possessed stable training.

 

The authors could use the smallest gg among states in training that are certified safe to estimate a K∗~\tilde{K^\ast} (lower bound on K∗K^\ast). If even this K∗~\tilde{K^\ast} is way larger than the KK used in the experiments, the optimality claim might not be practically attainable.

We would like to clarify that the calculation of K∗K^\ast only holds for finite tabular settings; for continuous spaces, this doesn't directly carry over. This is because there could exist an arbitrary number of states closer to the constraint boundary, and thus, from these states, Vπϕ≤gV^\phi_\pi\le g can be arbitrarily small and thus, δΩ=0\delta_\Omega = 0. Thus, we empirically study how the safety value truncation varies as KK increases in Appendix C.66. We collect transition data from sampling a batch every 20002000 steps and measure the number of active safety truncations in that batch. Over 5M training steps, we collect a total of 2,5002,500 data points, and average across the training seeds. We illustrate the results in Figure 77. It can be generally seen that the overall active safety value truncations for transitions during training reduces as KK increases (except SafetyHopperVel-v0, where the KK-increase converge close to similar values).

 

W3. No source code is provided with the submission. This does limit reproducibility at the time of review and affects my confidence score.

We understand the reviewer's concern around the source code availability; we will release the complete codebase upon acceptance. The currently revised appendix, to the best of our knowledge, supplies most of the required details and information pertaining to the empirical study and implementation. We'd be happy to answer further questions that the reviewer may have during the discussion period.

I thank the authors for the added experiments, which provide further insights into the practical continuous control experiments. I maintain my original recommendation.

Reviewer_6Yt8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies safety-critical reinforcement learning under adversarial disturbances, where the goal is to maximize task return while avoiding unsafe states at all times rather than only in expectation. The main idea is to unify task performance and safety at the value-function level: the paper defines a pure task value Q^ψ, a policy-conditioned safety value Q_π^φ derived from discounted Hamilton-Jacobi reachability, and a joint value Q in which task-value accrual is truncated by the safety value through a min operator. Because the maximin policy induced by Q is generally discontinuous, the authors analyze learning through a two-timescale stochastic approximation framework: the safety value is tracked on the fast timescale, while the joint value evolves on the slow timescale. They formulate the limiting dynamics using set-valued differential inclusions and occupation-measure averaging, prove forward invariance of a safe configuration set Q_cfg under suitable margin scaling K, and further show convergence toward optimal safety-constrained task values when K is sufficiently large and the safety margin is uniformly separated from the boundary of the viability kernel Ω. To scale the method to continuous control, the authors propose JointSAC, a neural approximation built on SAC-style actor-critic training with different learning rates for the safety and joint critics. Experiments on adversarial Gymnasium and SafetyGymnasium tasks report stable training, high task reward, and near-zero evaluation cost, while the paper also acknowledges that neural approximation breaks strict forward invariance and that K must be tuned carefully in practice.

Critical strengths
  1. The core contribution of the paper is clear: rather than simply weighting reward and cost, it constructs a joint Bellman operator with a safety truncation at the value-function level, so that task value is optimized under a safety upper bound; this is the foundation for the later two-timescale learning, safety certification, and optimality analysis.
  2. A specific theoretical point worth recognizing is Theorem 3 in Sec. 5 together with the supporting derivations in Appendix B.6: the paper translates “policy safety” into sign conditions on the coordinates of Q, namely the I/J index sets, the κ selection gap, and Q_cfg, and further proves that the safe configuration set is forward invariant and exponentially attracting; compared with only proving contraction of a single Bellman operator, this part more directly explains why “correct ranking of Q” leads to “kernel preservation of the greedy policy.”
  3. The organization of C_ε, dist(·, Q_cfg), Lemma 8 selection gap, Lemma 10 target bounds, and Corollary 3 violation contraction in Appendix B.6 is also relatively rigorous, giving a continuous derivation from the safe constraint set to contraction of the displacement in the limiting dynamics, so that the safety conclusion in Sec. 5 is supported by a fairly complete geometric proof chain rather than only by intuition.
  4. Another positive aspect is the optimal-stopping interpretation of the joint objective in Sec. 3.4 and Appendix B.3: Theorem 1/Theorem 7 formulate the process of “continuing to accrue task reward and then settling to a safety value upon entering the safety region” as a stopping-time objective, so that the min truncation in Eq. (3) is not merely an engineering trick but has an explicit optimal-stopping interpretation; the double-integrator comparison in Appendix C.1 also provides a useful low-dimensional sanity check.
Critical weaknesses
  1. The notation table in Sec. 2.1 and Appendix A.1 is incomplete: Eq. (1), Sec. 3.1, Appendices B.1/B.2, and Table 1 introduce or rely on objects such as Q^ψ, Q_π^φ, joint Q, Y, Π(q), and Q_cfg before a unified presentation; although these symbols are functionally distinct, the same type of object has multiple names—for example, the pure task value is denoted Q^ψ in the main text and Q_task in Appendix C.1, while the safety value is denoted Q_π^φ in the main text and Q_safe^* in Appendix C.1. A notation mapping would reduce the reading burden.
  2. The theory in Sec. 2.1, Sec. 5, Sec. 6, and Appendix B.6 relies on a finite/tabular setting, an exactly known or available viability kernel, kernel-preserving action sets, Assumption 3, δ_Ω > 0, and K ≥ K*; however, the empirical JointSAC implementation uses neural approximations, and the Limitations section itself acknowledges that strict forward invariance is violated. More importantly, K must be sufficiently large in theory, while the experiments also acknowledge that excessively large K can introduce training instability, weakening the “strictly safe and optimal” promise in the deep implementation.
  3. The empirical protocol in Sec. 7.2 and Appendices C.3/C.4 needs clearer reporting: the correspondence between training seeds, evaluation episodes, threshold d, and the cost definition should be made explicit; under a strict d = 0 threshold, a mean cost such as 0.03 ± 0.08 should not be directly described as satisfying the constraint; if cost is a per-step violation count, reporting only the mean is insufficient to establish near-zero violations; and LRSF is not cleanly separated from CMDP baselines.
Questions for authors
  1. Could you add a complete notation table in Sec. 2.1 or Appendix A.1 and define Q^ψ, Q_π^φ, joint Q, Y, Π(q), I/J, Q_cfg, and K at first use? If you keep Q_task and Q_safe^* in Appendix C.1, please also explain how these symbols correspond to the notation used in the main text, for example Q_task and Q^ψ for the task value, and Q_safe^* and Q_π^φ for the safety value.
  2. Could you clarify the relation between the discount factor γ and the annealed γ~ used in the experimental appendix? In particular, under which γ should the JointSAC safety certificate be interpreted, and when is it only approximating the strict viability-kernel semantics associated with γ → 1?
  3. Could you make the disturbance information structure explicit: does ν(s, a) mean that the disturbance observes the control action a before choosing d, and is this fully consistent with non-anticipative disturbance strategies in HJ reachability?
  4. In the experiments, are g(s), the viability kernel Ω, and A_Ω(s) known, exactly computed, discretized, or approximated by a critic? If they are not known objects, does the safety critic learn Ω_π or an approximation of Ω?
  5. Could you clarify the scaling notation: is β(·) intended as an extended class-K_e function, or is it effectively the linear scaling K g(s) used in the appendix? Please also explain how K is selected for each environment, and whether that selection criterion depends on evaluation-time violation outcomes.
  6. Could you report the evaluation metrics more explicitly, including the cost definition, threshold d, violation rate, maximum cost, 95% confidence intervals, training-time cost, and failed-rollout distributions for each environment? For mean costs that are close to but not exactly zero, please also explain how they should be interpreted under a strict threshold.
6 comments
AuthorsOpenReview ↗

Response to Reviewer 6Yt8

We thank the reviewer for their insightful comments and suggestions. The main concerns are (1) incomplete notation table for the paper, (2) clarification on γ\gamma and γ~\tilde{\gamma} used in experiments, (3) explicit details on the disturbance's information structure, (4) assumptions on how g(s),Ω,AΩg(s), \Omega, A_\Omega are obtained, (5) clarification on β(g)\beta(g) vs. Kg(s)K g(s), and (6) explicitly reporting evaluation metrics and interpreting near-zero mean costs. Together with the clarifications below, does this address the reviewer's concern?

 

Could you add a complete notation table in Sec. 2.1 or Appendix A.1 and define Q^ψ, Q_π^φ, joint Q, Y, Π(q), I/J, Q_cfg, and K at first use? If you keep Q_task and Q_safe^* in Appendix C.1, please also explain how these symbols correspond to the notation used in the main text, for example Q_task and Q^ψ for the task value, and Q_safe^* and Q_π^φ for the safety value.

Thank you for the suggestion. We have added a complete notation table in Appendix A.11 for additional clarity. We use QtaskQ_{\mathrm{task}} to refer to the general task-reward action-value function (safety-agnostic, general reinforcement learning), as described in Eq. (1). However, our joint action-value function also possesses the same task structure (albeit with the joint QQ-value nesting). Thus, we use QψQ^\psi only to denote this task component of our joint Bellman equation.

 

Could you clarify the relation between the discount factor γ and the annealed γ~ used in the experimental appendix? In particular, under which γ should the JointSAC safety certificate be interpreted, and when is it only approximating the strict viability-kernel semantics associated with γ → 1?

We would like to clarify that γ~\tilde{\gamma} is associated only with the updates for the safety value function used in JointSAC's Bellman equation (Eq. (2)). For the Joint QQ-value update (Eq. (3)), we don't anneal γ\gamma and set it to a constant value of 0.990.99. As training progresses, γ~\tilde{\gamma} gets annealed towards a value closer to one (for stability reasons), typically chosen to be 0.990.99 or 0.9990.999, as discussed in [1]; we anneal γ~\tilde{\gamma} towards γ=0.99\gamma = 0.99 for the first 11M steps, and keep it fixed for the rest of the training; this setup yielded a stable training performance. Thus, after 11M steps, the safety critic serves as a good approximation of the underlying viability kernel under the joint policy.

 

Could you make the disturbance information structure explicit: does ν(s, a) mean that the disturbance observes the control action a before choosing d, and is this fully consistent with non-anticipative disturbance strategies in HJ reachability?

We would like to clarify that the adversary ν(s,a)\nu(s, a) observes only the current controller's action along with the state, before choosing the disturbance value (note that the controller doesn't get this advantage). This setup follows the Stackelberg (leader-follower) strategy, where the disturbance player gets an instantaneous information advantage by observing the controller player's actions, and then decides the worst-case disturbance for the controller. This is indeed a non-anticipative strategy since ν\nu may depend on the current and past control inputs only (as discussed in [2, 3, 4]), and thus is fully consistent with the existing robust HJ-reachability literature.

 

In the experiments, are g(s), the viability kernel Ω, and A_Ω(s) known, exactly computed, discretized, or approximated by a critic? If they are not known objects, does the safety critic learn Ω_π or an approximation of Ω?

We only assume access to the margin function g(s)g(s). This is usually supplied as part of the operational design domain as defined in the HJ-reachability literature (e.g., [5]). Every other quantity - the viability kernel Ω\Omega, and the kernel-preserving actions AΩA_\Omega is not known apriori, and is learned during training; these quantities are approximated by the learned safety action-value function. Since the safety critic assesses the joint policy's safety, it learns Ωπ⊆Ω\Omega_\pi \subseteq \Omega; as illustrated in Appendix C.22, the safe set grows in the direction of exploration required by the task. In the limit of infinite exploration (which is also required by standard QQ-learning and HJ-reachability-based filtering approaches), Ωπ→Ω\Omega_\pi \rightarrow \Omega.

AuthorsOpenReview ↗

Response to Reviewer 6Yt8

Could you clarify the scaling notation: is β(·) intended as an extended class-K_e function, or is it effectively the linear scaling K g(s) used in the appendix? Please also explain how K is selected for each environment, and whether that selection criterion depends on evaluation-time violation outcomes.

We would like to clarify that the construction and convergence results of Sections 33-44 hold for any class-κe\kappa_e β\beta, since β∘g\beta \circ g has the same sign as gg; however, only the quantitative bounds of Sections 55-66 would change (we can potentially extend these bounds to general β\beta in future work). We take a running example of β(g(s))=Kg(s)\beta(g(s))=K g(s) throughout, as it is easy to intuit the slope of β\beta via KK (to illustrate the nature of safety-value truncation); it also simplifies the implementation of the algorithm (since it gives us a single hyperparameter knob to tune). Regarding the tuning of KK, we choose the appropriate value by ablating for various values of KK for each environment. As discussed in Appendix C.33 and C.44, a high enough KK value that allows for lower violations, high task performance, and good training stability is then finalized for each environment.

 

Could you report the evaluation metrics more explicitly, including the cost definition, threshold d, violation rate, maximum cost, 95% confidence intervals, training-time cost, and failed-rollout distributions for each environment? For mean costs that are close to but not exactly zero, please also explain how they should be interpreted under a strict threshold.

Thank you for the suggestion. As per your recommendation, we have included the cost definitions used for each environment explicitly in Table 22 of Appendix C.33. Additionally, we have reported the following metrics: violation rate (number of evaluation episodes that contained at least one constraint violation), maximum cost incurred, 9595\\% confidence intervals, and the distribution of failed rollouts across the seeds for each environment and method in Section C.77, Tables 44 and 55. Additionally, we also report the total violation counts per 1,5001,500 distinct evaluation episodes (where each of the 55 train seeds is evaluated for distinct 300300 evaluation seeds) in Table 1. We believe that this can better explain the near-zero cost values under a strict threshold. Additionally, to avoid confusion, we have reworded this to "significantly low safety violation rates", which is the percentage of total violations for 1,5001,500 distinct evaluation episodes. We can note from these results that JointSAC is able to maintain much lower safety violation rates (subject to neural approximations and noisy seeds) while maintaining high-enough task performance across the baselines.

 

[1] Fisac, Jaime F., et al. "Bridging Hamilton-Jacobi safety analysis and reinforcement learning." 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.

[2] Evans, Lawrence C., and Panagiotis E. Souganidis. "Differential games and representation formulas for solutions of Hamilton-Jacobi-Isaacs equations." Indiana University Mathematics Journal 33.5 (1984): 773-797.

[3] Fisac, Jaime F., et al. "A general safety framework for learning-based control in uncertain robotic systems." IEEE Transactions on Automatic Control 64.7 (2018): 2737-2752.

[4] Hsu, Kai-Chieh, Duy Phuong Nguyen, and Jaime Fernandez Fisac. "Isaacs: Iterative soft adversarial actor-critic for safety." Learning for Dynamics and Control Conference. PMLR, 2023.

[5] Hsu, Kai-Chieh, Haimin Hu, and Jaime F. Fisac. "The safety filter: A unified view of safety-critical control in autonomous systems." Annual Review of Control, Robotics, and Autonomous Systems 7 (2023).

AuthorsOpenReview ↗

Rebuttal Follow-Up

Dear Reviewer,

We have worked hard to incorporate the review feedback by running new experiments and revising the paper. Do the revisions and discussions above address your concerns? We would greatly appreciate your engagement.

Thanks!

The Authors

Reviewer_4fQEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a joint Bellman operator for safety-constrained task optimization under adversarial disturbances. The formulation uses a policy-dependent safety value to truncate task-value accumulation and feeds this safety evaluation back into policy improvement. To address the resulting coupling and discontinuities induced by greedy policy selection, the authors develop a two-timescale stochastic approximation analysis based on an averaged differential inclusion. They establish conditional safety and optimality results in the finite/tabular setting and introduce JointSAC as a neural approximation. Experiments include an exact tabular double-integrator example and adversarial continuous-control benchmarks, where the method generally achieves competitive returns with near-zero evaluation costs.

Critical strengths
  1. The paper addresses how to incorporate safety into task optimization so that the learned policy requires less reliance on separate runtime safety interventions. This is a relevant direction for safe reinforcement learning.

  2. The safety critic evaluates the policy induced by the joint value function, and this evaluation subsequently influences the joint Bellman update. The theoretical analysis explicitly addresses the resulting coupling, including discontinuities associated with changes in greedy policies, rather than assuming a fixed safety critic.

Critical weaknesses
  1. The introduction motivates a general approach to integrating safety and task performance, whereas the formulation and continuous-control experiments focus on adversarial disturbances. Although this setting is explicitly introduced, the paper could better explain its relationship to the motivating limitations and distinguish the contribution of the joint Bellman update from the role of adversarial training.

  2. Appendix C describes how conventional CMDP baselines are adapted into adversarial formulations and reports disturbance bounds and some training details. However, it does not clearly specify how evaluation adversaries are trained, selected, or assessed across methods. Equal disturbance bounds alone do not establish comparable attack effectiveness. The configurations and tuning procedures for the modified baselines are also insufficiently documented, which is particularly important given the poor performance of some baselines. Appendix C.1 provides a useful disturbance-free tabular comparison, but a matched continuous-control comparison with and without adversarial training would better clarify the source of the reported gains. A representative method specifically designed to jointly address task performance and robust safety would also strengthen the evaluation beyond LRSF and adversarial adaptations of conventional CMDP algorithms.

  3. Theorem 3 establishes forward invariance and attraction for the limiting dynamics within a specified neighborhood of the safe configuration set, under additional certification and margin-scale assumptions. Theorem 4 also assumes that the limiting set lies within this neighborhood. The paper does not establish that learning enters this region from arbitrary critic initialization. Consequently, the opening statement in Section 6—that forward invariance ensures the agent eventually learns a safe policy—appears insufficiently qualified. Appendix Remark 9 appropriately distinguishes the result from safety throughout training, but the main text should similarly clarify which conditions are established by learning and which remain assumptions.

  4. The caption of Fig1 states that trajectories must remain within (Ωψ\Omega^\psi) to maximize episodic return. Definition 1 partitions the state space according to task-continuation and safety values, while Section 3.4 provides an optimal-stopping interpretation of the resulting truncation. This explains when the joint objective switches to a safety payoff, but does not directly establish that remaining in (Ωψ\Omega^\psi) is necessary for maximizing the original episodic task return. The paper should distinguish these objectives or qualify the caption.

Questions for authors
  1. How are evaluation adversaries trained and selected? Are they the adversaries obtained during joint training, or are they separately optimized against frozen controllers? Please clarify the attack objectives, optimization budgets, restart and selection procedures, and the configurations and tuning of the modified baselines. Could a matched continuous-control comparison with and without adversarial training be provided?

  2. Beyond LRSF and the adversarially adapted CMDP baselines, could the authors include a representative method specifically designed to jointly optimize task performance and robust safety under a compatible disturbance model? If such a comparison is impractical, what differences in assumptions or objectives prevent it?

  3. What ensures that the learning process reaches the neighborhood required by Theorem 3, or that its limiting set satisfies the neighborhood condition in Theorem 4? If these remain assumptions, should the opening claim in Section 6 be explicitly qualified?

  4. Does the trajectory requirement in Figure 1 concern avoiding safety truncation, maximizing the optimal-stopping objective, or maximizing the original episodic task return? What establishes the claimed necessity? Separately, are boundary states with (g(s)=0) considered safe, and does the intended convention require any changes to the proofs?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 4fQE

We thank the reviewer for their insightful comments and suggestions. The main concerns are (1) additional details on adversary selection and non-adversarial experiments, (2) a potentially new representative method comparison, (3) insufficient qualification at the beginning of Section 6, and (4) clarification on the Figure 1 caption and how it relates to the joint objective function. Together with the clarifications below, does this address the reviewer's concern?

 

How are evaluation adversaries trained and selected? Are they the adversaries obtained during joint training, or are they separately optimized against frozen controllers? Please clarify the attack objectives, optimization budgets, restart and selection procedures, and the configurations and tuning of the modified baselines. Could a matched continuous-control comparison with and without adversarial training be provided?

Evaluation adversaries: Every method is evaluated against the adversary it was co-trained with; no adversary is re-optimized against a frozen controller. For JointSAC, the adversary observes the current state and the instantaneous control input (leader-follower Stackelberg setting) and then provides the disturbance that minimizes the joint action-value function. For Lagrangian baselines, we similarly train the adversary to observe the instantaneous control input, and then provide the disturbance that minimizes the constrained objective (i.e., maximizes −r+λ⋅c-r + \lambda\cdot c). Similarly, LRSF is also attacked by the disturbance policy of its two-player action-value function. Note that all of the adversaries apply forces at the same part of the robot body, with the same per-environment disturbance bound.

Baseline configuration: We use their well-tuned, stable hyperparameters from the omnisafe's reference configurations, with their best reported seeds (0,5,10,15,20)(0, 5, 10, 15, 20). We set the cost violation threshold d=0d=0 (hard-safety constraint); the disturbance is modeled to be part of the environment for CMDP baselines (where the adversary co-trains with the controller during training), and thus, the omnisafe implementations remain untouched. For LRSF, we use the two-player Stackelberg setting with τ\tau-GDA [1].

Non-adversarial training: Our framework can indeed be extended to only one-player setting (i.e., no adversarial disturbance). We provide the first set of results for JointSAC against the same set of baselines (without any adversaries) in Table 33 of Appendix C.5.15.1. From the table, we observe a similar conclusion: JointSAC is able to maintain lower violation rates while achieving higher task performance. LRSF remains safe by mostly not moving/moving very slowly, whereas the CMDP baselines trade safety for higher task rewards. Currently, we've run JointSAC on the same set of KK-values as for adversarial settings; we believe that the violation rates can be further reduced by fine-tuning KK values for each environment.

 

Beyond LRSF and the adversarially adapted CMDP baselines, could the authors include a representative method specifically designed to jointly optimize task performance and robust safety under a compatible disturbance model? If such a comparison is impractical, what differences in assumptions or objectives prevent it?

To the best of our knowledge, we aren't aware of a model-free framework that is designed to incorporate hard safety guarantees while jointly optimizing for task performance without any prior access to a safe set or safety-preserving forward-invariant action set. Our baselines therefore bracket the two families of relevant literature: robust Lagrangian methods (that provide joint optimization, with constraint satisfaction on average), and robust HJ-based filtering methods (capable of providing hard safety guarantees in a tabular setting, while being task-agnostic).

 

What ensures that the learning process reaches the neighborhood required by Theorem 3, or that its limiting set satisfies the neighborhood condition in Theorem 4? If these remain assumptions, should the opening claim in Section 6 be explicitly qualified?

Thank you for pointing this out and reading carefully. Indeed, Theorems 3 and 4 are local, and don't establish that learning from an arbitrary initialization reaches the assumed neighborhood. However, please note that this local structure is consistent with the existing theory in the literature. [2] establishes convergence to the attractor only when the limit set meets its basin, and claims of global attraction require additional assumptions [3]. The same applies to actor-critic approaches [4]. Thus, we have modified the opening claim in Section 6 appropriately to match the theorem's assumptions as per the reviewer's suggestion.

AuthorsOpenReview ↗

Response to Reviewer 4fQE

Does the trajectory requirement in Figure 1 concern avoiding safety truncation, maximizing the optimal-stopping objective, or maximizing the original episodic task return? What establishes the claimed necessity? Separately, are boundary states with (g(s)=0g(s)=0) considered safe, and does the intended convention require any changes to the proofs?

We would like to clarify that the Figure 1 caption refers to the joint (i.e., safety-constrained task) objective pertaining to Theorem 1, and this objective is what JointSAC optimizes. As explained in Section 3.43.4, the agent can continue accruing rewards as long as its trajectory remains inside Ωψ\Omega^\psi; when it first enters Ωϕ\Omega^\phi, the objective replaces the remaining return of the trajectory with the safety truncation value. Note that this doesn't end the agent's episode; the truncation only plays a role in calculating the objective, not as a terminal reward of sorts. Thus, if a policy π1\pi_1, starting from state s∈Ωψs \in \Omega^\psi, exits early, the value of that state under π1\pi_1 will be lower than a policy π2\pi_2 that exits Ωψ\Omega^\psi later (maximum, if it doesn't exceed at all). Hence, maximizing the joint objective favors policies that can keep the trajectories within Ωψ\Omega^\psi; for K≥K∗K \ge K^*, the task-optimal safe policy achieves maximum return while staying in Ωψ\Omega^\psi for all time.

Boundary states We would like to clarify that, for general MDPs, boundary states g(s)=0g(s)=0 cannot be assumed to be safe. The boundary states g(s)=0g(s)=0 are only part of the constraint set Fc:={s:g(s)≥0}\mathcal{F}^c := \{s:g(s) \ge 0\}, which is the complement of the failure set F\mathcal{F}; the states inside Fc\mathcal{F}^c are safe if and only if they are part of the safe set Ω⊆Fc\Omega\subseteq \mathcal{F}^c, which can only be determined after learning the safety/joint action value function. This is because there could be states inside the constraint set (that aren't currently failing), but there may not exist a policy that can keep the system from failing in the near future. This setting is consistent with the Hamilton-Jacobi reachability analysis, and all of our theoretical analysis is already built on this convention.

 

[1] Fiez, Tanner, and Lillian J. Ratliff. "Local convergence analysis of gradient descent ascent with finite timescale separation." International Conference on Learning Representations. 2021.

[2] Benaïm, Michel, Josef Hofbauer, and Sylvain Sorin. "Stochastic approximations and differential inclusions." SIAM Journal on Control and Optimization 44.1 (2005): 328-348.

[3] Borkar, Vivek S. "Stochastic approximation with two time scales." Systems & Control Letters 29.5 (1997): 291-294.

[4] Konda, Vijay R., and John N. Tsitsiklis. "On actor-critic algorithms." SIAM Journal on Control and Optimization 42.4 (2003): 1143-1166.

I thank the authors for providing clarifications but I still have following concerns:

  1. Appendix C.5.1 tightens the original safety constraints for the non-adversarial Gymnasium experiments, making the comparison unfair for isolating the effect of adversarial training. Could the authors provide a matched comparison under identical safety constraints, at least on representative environments such as Walker2d-v4 and Hopper-v4, where PPO-Lag already achieves relatively low costs?

2 Evaluating each method against its own co-trained adversary does not guarantee comparable attack effectiveness. Could the authors provide shared-adversary or cross-adversary evaluations on one or two representative environments?Additionally, relevant robust safe RL methods such as DRAC (TPAMI 2024) and SAFER (ICML 2023) are not discussed. Could the authors consider DRAC as a baseline or clarify the technical obstacles preventing such a comparison?

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal Summary (continued)

Reviewer 4Cj2

  1. Scope of guarantees: As per the reviewer’s suggestion, we have clarified at the end of Section 11 that our theoretical analysis holds only for finite tabular settings (similar to much of the state-of-the-art safe RL methods). With continuous control experiments, we intend to show the scalability and stable learning of our joint formulation for high-dimensional state-action spaces.
  2. Variance bands for Figure 44: To visualize the instability of large KK values, we add variance bands (±\pm 11 std) to Figure 44 in Appendix C.44, and illustrate the variation up to K=2000K=2000.
  3. Practically attainable KK values: We clarify that the calculations of K∗K^\ast only hold in tabular settings; for continuous spaces, δΩ(s)=0\delta_{\Omega}(s) = 0 as there are infinitely many states closer to the boundary. Thus, we empirically study how safety value truncation varies as KK increases, and provide results in Fig. 77, where we plot the percentage of active safety value truncations for different values of KK as training evolves. We generally observe that the overall truncations reduce as KK increases.
  4. Source code availability: We understand the reviewer’s concern and will release the complete codebase upon acceptance. We believe, to the best of our knowledge, that the currently revised appendix supplies most of the required details and information pertaining to the empirical study and implementation.

 

Kind regards.

The Authors

 

[1] Li, Zeyang, et al. "Safe reinforcement learning with dual robustness." IEEE Transactions on Pattern Analysis and Machine Intelligence 46.12 (2024): 10876-10890.

[2] Suttle, Wesley, et al. "Sampling-based safe reinforcement learning for nonlinear dynamical systems." International Conference on Artificial Intelligence and Statistics. PMLR, 2024.

AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair,

As the discussion period comes to a close, we would like to summarize the reviewers’ main concerns and how we addressed them

Reviewer 4fQE

  1. Cross-adversary evaluations: We provide cross-adversary evaluations of all the methods used in the paper in Table 88 of Appendix C.1010 on two representative environments: Walker2d-v4 and SafetyHopperVel-v1. Here, we evaluate the fully trained controller policy against other methods’ adversarial disturbances. We observe that JointSAC remains relatively robust to cross-adversaries in terms of maintaining task performance and lower safety violation rates.
  2. Non-adversarial evaluations: We clarify that our framework indeed extends to a general, single-player setting, and provide the results in Table 33 of Appendix C.5.15.1. We observe that JointSAC maintains lower safety violation rates while achieving higher task returns. While the safety gymnasium environments are evaluated on the default liveness margins, the margins in standard Gymnasium environments were initially tightened to serve as strict safety specifications. However, to address the reviewer’s unfair comparison concern, we revert back to the default liveness margin specification on two representative environments: Walker2d-v4 and Hopper-v4. We provide results in Table 77 of Appendix C.99, and observe that JointSAC, along with LRSF, achieves zero safety constraint violations.
  3. DRAC baseline: We thank the reviewer for the references, and have included them in Section 11, where we have justified our relative positioning against them. Additionally, we include DRAC as a baseline, and implement it from scratch as we couldn’t find a publicly available repository/hyperparameter list. To the best of our efforts, we tune it, reproduce the original training curves in DRAC [1]’s Figure 22 (on the Walker environment), and then include it as a baseline candidate; we provide the results in Table 66 and Table 88 (cross-adversaries) of Appendix C.88. We observe that DRAC achieves comparable task performance to JointSAC, albeit with a slightly higher safety-violations count.
  4. Assumptions clarification in Section 6: We thank the reviewer for pointing this out; we have updated the opening claim in Section 6 appropriately to match our theorem’s local assumptions (which is in line with the established literature theory) as per the reviewer’s suggestion.

 

Reviewer 6Yt8

  1. Complete notation table: As per the reviewer’s suggestion, we have added a complete notation table in Appendix A.11 and have clarified the notation’s meaning and where they were initially defined.
  2. Explicit evaluation metrics: Following the reviewer’s suggestion, we have reported violation rates, maximum cost incurred, 9595\\% confidence intervals, and the distribution of failed rollouts across the seeds for each environment and method in Section C.77, Tables 44 and 55. For brevity and the sake of clarity, we have reworded “near-zero cost” to “significantly lower violation rates” throughout the paper; we believe that this aids in clearly distinguishing safety failures against “near-zero costs”. Finally, we have added the cost definitions used for each environment explicitly in Table 22 of Appendix C.33.
  3. Scope of guarantees: We clarify that our hard-safety guarantees hold only in tabular settings (similar to much of the state-of-the-art safe RL methods) early on in the paper, specifically in Section 11, and have already acknowledged this originally in the limitations section. Moreover, we also clarify that the hard-mask baseline, along with some existing literature such as Suttle et. al. 20242024 [2], uses privileged information about the exact kernel-preserving action states to hard-guarantee safety; on the other hand, our framework does not assume access to any model-based information, and jointly learns task and safety values solely via interactions in a fully model-free setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.