acceptodds
Withdrawn from ICLR 2027

Confidence-aware chance-constrained policy optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In many real-world applications, reinforcement learning (RL) agents are expected to balance reward maximization and safety considerations. Constrained RL can address this issue by maximizing expected cumulative return while satisfying safety-related constraints. While RL algorithms with expected cumulative cost constraints have been extensively investigated, these expectation-based formulations provide a limited characterization of risk and do not directly capture the probability of safety violations. In this work, we propose a confidence-aware chance-constrained policy optimization (Chance-CPO) method that keeps the probability of constraint violation below a prescribed threshold at a user-specified confidence level. The proposed approach establishes a connection between expectation-based constraints and chance constraints, and uses expectation-related gradients to facilitate the satisfaction of chance constraints. By reducing an upper bound on the violation probability, Chance-CPO accounts for both the expected cumulative cost and the probability of unsafe events. We demonstrate the effectiveness of the proposed method on simulated robot tasks for which safety constraints are important.

Primary area:
reinforcement learning
Keywords:
Constrained Markov decision process, safe reinforcement learning, chance constraints

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QHxZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Chance-CPO, a policy optimization method designed to limit the probability that cumulative cost reaches or exceeds a prescribed threshold. The method uses a statistical confidence bound on the violation probability to construct an expected-cost surrogate constraint, which is adjusted as the policy changes. This formulation enables policy updates based on reward and cost advantage estimates. The authors evaluate Chance-CPO on Safety Gym tasks and compare its reward and safety performance with constrained PPO and QCPO.

Critical strengths
  1. The paper addresses a clearly defined and practically important problem in safe reinforcement learning.
  2. The proposed method offers an interesting way to address chance constraints by combining a statistical bound on the violation probability with an expectation-based constraint that can be used in policy optimization.
Critical weaknesses
  1. The finite-sample guarantee is not established for the implemented constraint. Proposition 1 and Eq. (7) rely on the true expected cost ηc\eta_c, whereas Eq. (12b) and (13) replace it with the sample mean EDk[Jc]\mathbb{E}_{D_k}[J_c]. This estimate can underestimate ηc\eta_c, and the Clopper–Pearson bound on the violation probability does not control that estimation error. Consequently, the stated 1−β1-\beta safety guarantee does not directly apply to the practical update. The authors should provide a finite-sample upper bound for ηc\eta_c and account for its error probability, or clearly limit the guarantee to the idealized formulation.

  2. The authors compare with all baselines only at α=0.3\alpha=0.3, and do not compare at α=0.2\alpha=0.2, 0.10.1, or even lower values. This permits up to 30% of trajectories to exceed the specified cumulative-cost threshold, which is a relatively permissive target if such exceedances are regarded as unsafe. It is therefore unclear whether the reported advantage over the baseline persists at α=0.1\alpha=0.1 or 0.20.2.

Questions for authors

The experiments use a confidence level of 1−β=0.81-\beta=0.8. How does Chance-CPO perform at higher confidence levels, such as 0.95 or higher?

Reviewer_5rjpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies constrained RL with a chance constraint on the discounted cumulative cost, required to hold at a user-specified confidence level. The paper proposes Chance-CPO, which combines Markov and Clopper–Pearson bounds to translate a trajectory-level chance constraint into a scaled expected-cost constraint. An augmented MDP supports policy-shift correction, while a practical PPO-based implementation is evaluated on six Safety Gym tasks.

Critical strengths
  • The distinction between expected cost, violation probability, and statistical confidence is important and clearly motivated.
  • The experiments include five seeds, post-training evaluation, additional environments, and ablations of both confidence and policy-shift corrections. The reported reductions in empirical violation rates are encouraging.
Critical weaknesses
  • The connection between finite trajectories and the target violation event is incomplete. Appendix C.2 establishes equality of expected costs under geometric truncation, not equality of exceedance probabilities. For example, let c0=0,c1=1c_0=0,c_1=1, with subsequent costs zero, γ=1/2\gamma=1/2, and d=1/4d=1/4. The infinite discounted cost is always 1/21/2, so violation probability is one. With Pr⁡(T=t)=2−t−1\Pr(T=t)=2^{-t-1}, the truncated sum exceeds dd only when T≥1T\ge1, with probability 1/21/2. Thus, unbiased cost estimation does not justify the binomial model for nkn_k. The actual counting procedure needs clarification; this example does not establish what the code implements (Section 5; Appendices A.1 and C.2).
  • The policy-transfer bound requires an additional boundedness argument. The auxiliary cost contains wt=γ−tw_t=\gamma^{-t}, whereas Eq. 11 uses a maximum auxiliary advantage over augmented states. Bounded original costs do not establish that this maximum is finite. Without an appropriate assumption or alternative bound, the transfer correction may become vacuous. This does not invalidate the Markov fallback, but it undermines the claimed reduction in conservatism under policy updates (Eqs. 8–11).
Questions for authors

Please refer to the weaknesses part.

Reviewer_rUbMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Chance-CPO for RL with chance constraints. The method uses a Clopper–Pearson confidence bound to adapt an expected-cost constraint and introduces an auxiliary MDP to account for changes in violation probability during policy updates. A PPO-based implementation is evaluated on six Safety Gym tasks.

Critical strengths
  • The paper addresses a useful problem that accounts for both finite-sample uncertainty and policy changes when enforcing chance constraints.

  • The distinction between expected cost and violation probability is well motivated.

  • The experiments cover several robot morphologies and include multiple seeds and ablations.

Critical weaknesses
  • My main concern is whether the policy-shift bound applies to the proposed auxiliary MDP. Its auxiliary cost grows as γ−t\gamma^{-t}, so the maximum advantage appearing in the policy-shift bound need not be finite, even when the original costs are bounded. I could not find assumptions or an argument ensuring that Eq. (11) provides a finite, useful correction. This matters because the confidence guarantee is a central contribution.

  • The practical algorithm also departs from the formulation in ways that are not sufficiently justified. For instance, Eq. (15a) ignores the gradient through the adaptive scaling parameter, although that parameter depends on the policy. Also, reducing expected cost can increase violation probability, so it is unclear why this update should enforce the intended chance constraint.

Questions for authors

Please help resolve the concerns in the weakness part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.