- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Chance-CPO, a policy optimization method designed to limit the probability that cumulative cost reaches or exceeds a prescribed threshold. The method uses a statistical confidence bound on the violation probability to construct an expected-cost surrogate constraint, which is adjusted as the policy changes. This formulation enables policy updates based on reward and cost advantage estimates. The authors evaluate Chance-CPO on Safety Gym tasks and compare its reward and safety performance with constrained PPO and QCPO.
Critical strengths
- The paper addresses a clearly defined and practically important problem in safe reinforcement learning.
- The proposed method offers an interesting way to address chance constraints by combining a statistical bound on the violation probability with an expectation-based constraint that can be used in policy optimization.
Critical weaknesses
-
The finite-sample guarantee is not established for the implemented constraint. Proposition 1 and Eq. (7) rely on the true expected cost , whereas Eq. (12b) and (13) replace it with the sample mean . This estimate can underestimate , and the Clopper–Pearson bound on the violation probability does not control that estimation error. Consequently, the stated safety guarantee does not directly apply to the practical update. The authors should provide a finite-sample upper bound for and account for its error probability, or clearly limit the guarantee to the idealized formulation.
-
The authors compare with all baselines only at , and do not compare at , , or even lower values. This permits up to 30% of trajectories to exceed the specified cumulative-cost threshold, which is a relatively permissive target if such exceedances are regarded as unsafe. It is therefore unclear whether the reported advantage over the baseline persists at or .
Questions for authors
The experiments use a confidence level of . How does Chance-CPO perform at higher confidence levels, such as 0.95 or higher?