- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how reinforcement learning agents can earn high rewards while keeping the largest cost during a task below a safety limit, even when the environment changes. A surrogate objective is used instead of Lagrangian optimisation and provides conditions for safety during training and convergence to a near-optimal policy. The paper adopts a reachability-based formulation on the original state space and analyses a surrogate that takes the maximum of scaled negative reward and constraint violation. For exact proximal updates in finite MDPs, the paper establishes approximate safety throughout optimisation and a stationarity guarantee, with global near-optimality under additional nondegeneracy and coverage assumptions. Experiments on continuous-control tasks suggest that the method balances reward and safety under changes in the environment’s dynamics.
Critical strengths
-
The paper addresses an important safety objective: controlling large individual costs under dynamics mismatch. This complements cumulative-cost formulations, which can obscure isolated severe violations.
-
The distinction between expected trajectory peak cost and the reachability-based value is useful. The augmented CMDP construction clarifies when additive constrained optimization applies, while Remark 1 acknowledges that the original-state formulation generally represents a different constraint.
-
The convergence analysis provides a meaningful contribution beyond the surrogate’s solution-quality guarantee. Representing the objective as a maximum of smooth branch values enables weak-convexity analysis and an approximate-safety bound for intermediate proximal iterates.
-
The experiments examine several relevant aspects, including multiple constraints, feasible initialization, sensitivity to the surrogate parameter, and dynamics perturbations. Evaluating observed trajectory peaks rather than relying exclusively on critic predictions is also appropriate.
-
The appendix explicitly discusses limitations of the theory, including the nondegeneracy condition and the unresolved analysis at points where reward and constraint branches meet. This helps readers assess the scope of the guarantees.
-
The method handles multiple safety constraints without learning a separate Lagrange multiplier for each one, offering a simpler way to manage competing constraints.
Critical weaknesses
-
Theorem 2 establishes an approximate-safety bound from a feasible initialization. However, the contributions and conclusion claim that violation never rises above its initial level. Since the theorem permits positive violation up to even when initial violation is zero, this stronger claim is not established by the provided proof.
-
The global optimality guarantee in Theorem 3 requires matching gradients across all active branches and a finite coverage coefficient. As acknowledged in Remark 8, surrogate minimisers may lie where the reward and constraint branches meet, where the gradient condition can fail. Since the analysis does not establish that the selected iterate satisfies this condition, the global convergence rates remain conditional and may not cover important solutions.
-
SO and RP-CRL are trained under different conditions: RP-CRL also sees changes in the environment during training. Therefore, it is unclear whether its better performance comes from the IPM critic or from having trained with these changes.
-
The experiments include several relevant baselines and studies of the surrogate parameter and policy initialization. However, the number of seeds and matched training budgets are unclear, and computational overhead is not reported. The robustness comparison also changes the training environment alongside the IPM critic, making it difficult to identify which component produces the improvement.
Questions for authors
-
Can the nondegeneracy condition be verified for the returned policy? What guarantee remains when the selected iterate lies at a reward–constraint kink with different active gradients?
-
How does the one-step peak TD advantage, sampled under ordinary trajectories, account for the killed-chain occupancy appearing in the theoretical gradient?
-
Were SO and RP-CRL trained with identical dynamics perturbations? If not, how should their comparison be interpreted?
-
Please clarify the implemented branch-selection rule, the number of independent seeds, training budgets, and the definition of the reported violation rate.
Also see the weaknesses section. The paper offers a useful conditional convergence analysis for a relevant surrogate, but its strongest safety and global optimality claims need narrower wording. My main reservations concern the scope of the guarantees, their connection to the practical actor update, and the interpretation of the experimental comparisons.