acceptodds
Under review as a conference paper at ICLR 2027

Cantelli Constrained Policy Optimization

Abstract

We introduce Canary, a risk-averse method designed to optimize Value-at-Risk (VaR) constrained reinforcement learning (RL) problems. We employ Cantelli's inequality to obtain a tractable, conservative and smooth bound on the VaR constraint based on the first two moments of the cost return. This yields a constraint estimator that remains stable with tight violation thresholds in dense cost regimes. Extending the trust-region framework of the Constrained Policy Optimization (CPO) method, we further provide worst-case bounds for both policy improvement and constraint violation during the training process. Empirically during training, Canary is the only method that reliably satisfies the VaR constraint in every environment tested.

Primary area:
reinforcement learning
Keywords:
Safe reinforcement learning, chance constraints, Value-at-Risk, Cantelli's inequality

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_brnSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Canary, an on-policy reinforcement learning method for maximizing expected reward subject to a Value-at-Risk constraint on cumulative cost. The method uses Cantelli’s inequality to construct a conservative safety approximation based on the mean and variance of the cost return. Through state and cost augmentation, the second cost moment is expressed as an expected return and incorporated into the CPO trust-region framework. The paper derives worst-case constraint violation bounds in the discounted setting and analyzes the relative variance of constraint estimators in dense-cost regimes. Experiments on EcoAnt, PointGoal, and IcyLake report lower average violation probabilities and smaller fluctuations than the evaluated baselines, with a trade-off in reward.

Critical strengths

1 The method has a clear design and practical value. It uses the first two cost moments to construct a conservative approximation to the VaR constraint. State augmentation then allows the second moment to be incorporated into the CPO framework. This provides an implementable approach to tail-risk control without assuming a Gaussian cost distribution.

2 The main experiments provide useful empirical evidence. The authors use 30 seeds across three environments and report violation rates, their fluctuations, and rewards. Canary achieves lower average violation rates and smaller fluctuations on these tasks. The paper also reports the associated reward cost, which helps explain the trade-off.

Critical weaknesses

The theoretical result does not fully cover the experimental setting. Theorem 4.1 assumes γc<1, but all experiments use γc=1. No corresponding finite-horizon result is provided, so the theorem cannot directly establish a training-time violation bound for these experiments. Also, the safe-seed percentage in Table 1 is based on the average violation rate over the second half of training. It does not show that the constraint holds at each training stage. This limits the evidence for training-time safety.

The solver’s treatment of nearly parallel constraints needs further explanation. Appendix C.1 drops the mean constraint when the two constraint gradients are nearly parallel. Parallel gradients do not imply redundancy, since the constraint offsets and their strictness can differ. Under the stated rule, the stricter mean constraint could be removed. It remains unclear how the final feasibility check handles this case and how a rejected candidate is replaced with a valid update.

Questions for authors

How does Theorem 4.1 apply to the experimental setting with γc=1? Can you provide a finite-horizon counterpart and state its dependence on episode length? If the current theorem does not cover this setting, please clarify which training-time safety claims remain theoretically supported.

What conditions justify dropping the mean constraint? How does the implementation handle gradients that point in the same direction when the mean constraint is stricter? After dropping a constraint, does the solver still check both original constraints and search for another feasible update if the candidate violates the mean constraint?

Can you report the proportions of nominal updates, Cantelli restoration steps, and mean restoration steps from the training logs, together with violation rates at different training stages? These statistics would be especially useful for the EcoAnt and PointGoal runs that converge with cB>0.

How does Canary’s reward compare with CPO and CVaR-CPO at similar empirical violation rates? Can existing results or an additional comparison help distinguish the benefit of more stable estimation from the benefit of choosing a more conservative policy?

2 comments
AuthorsOpenReview ↗

Response to brnS

Thank you for the specific questions.

Q1: Theorem 4.1 at γc=1\gamma_c=1, and safety during training. With the finite-horizon counterpart (General Response B), the training-time statements of Section 4 hold in the experimental setting: the per-update bound for nominal updates, and the mean-cost bound μ(πk+1)≤ℓ+KTαC\mu(\pi_{k+1})\le\ell+K_T\alpha^C for Tiers a–b (Corollary 4.2). As in CPO, both are for the idealized update, so we also report violation at each training stage. At ε=0.02\varepsilon=0.02, the table reports each training quartile's mean violation rate over 30 seeds and the share of seeds whose rate is at most 0.02. EcoAnt (common event) and the IcyLake baselines use new runs, and the other rows use submitted runs.

EnvMethodQ1Q2Q3Q4
EcoAntCanary.002 (100%).002 (100%).002 (100%).003 (100%)
EcoAntCPO.010 (100%).029 (0%).032 (0%).027 (0%)
EcoAntCVaR-CPO.014 (100%).037 (0%).042 (0%).057 (0%)
PointGoalCanary.094 (0%).013 (80%).013 (80%).008 (97%)
PointGoalCPO.185 (0%).080 (0%).072 (0%).047 (0%)
PointGoalCVaR-CPO.040 (7%).017 (77%).017 (67%).013 (90%)
IcyLakeCanary.147 (0%).012 (87%).016 (90%).016 (83%)
IcyLakeCPO.146 (0%).047 (0%).041 (7%).037 (17%)
IcyLakeCVaR-CPO.102 (0%).024 (17%).019 (60%).019 (70%)

On PointGoal and IcyLake, every method violates in the first quartile, while learning the task. From the second quartile on, Canary meets the threshold on 80–100% of seeds, against 0–17% for CPO and 0–90% for CVaR-CPO.

Q2: nearly parallel constraints. The Appendix C.1 sentence was imprecise, and we will correct it. The implementation never drops the mean constraint. If SS is near-singular, the solver discards only the candidate with both constraints active. It keeps a single-constraint candidate only if it also satisfies the other linearized constraint. If no candidate is valid, it takes the Tier b direction. The line search then checks the mean surrogate in every tier. If the mean constraint holds, the mean surrogate may use the available slack. If the mean constraint is violated, the mean surrogate must not increase. The line search applies the same test to the Cantelli surrogate in Tier a, requires no increase in Tier b, and omits that check in Tier c.

Q3: tier proportions. Second-half fractions are in our reply to 141E (Q1). By quartile, Tier b rises from 0.61 to 0.80 on EcoAnt and stays near 0.80 on PointGoal. On IcyLake, Tier a rises from 0.26 to 0.64. Tier c accounts for less than 0.2% of second-half updates.

Q4. General Response (A) gives this comparison.

Thank you for the detailed responses. The finite-horizon extension addresses my concern about the theoretical setting. The clarification that the implementation retains the mean constraint also addresses my concern about nearly parallel constraints. The comparisons at similar empirical violation rates strengthen the evidence for a useful safety–reward trade-off, particularly on EcoAnt and PointGoal.

Overall, the responses have sufficiently addressed my main concerns, and I am updating my rating to 3 (weak accept).

Reviewer_K299OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Canary, a policy optimization method for chance-constrained RL. It uses Cantelli’s inequality to replace the chance constraint with conditions on the first two cost moments, and develops a trust-region update with bounds on surrogate constraint violations. The paper also analyzes estimator variance and evaluates the method on three environments.

Critical strengths
  • The moment-based formulation provides a useful alternative to estimating rare violations directly. The augmented-cost construction makes the second moment compatible with the trust-region framework.

  • The experiments are comprehensive on both violation rates and variability, providing useful evidence about the method’s practical behavior.

Critical weaknesses
  • The stability claim needs a more careful qualification. Section 5 defines stability for a fixed policy as the violation threshold decreases, but the indicator estimator’s variance does not diverge under that limit when its violation probability remains fixed. The divergence argument instead considers policies whose violation probability decreases with the threshold. The analysis may support this narrower comparison, but it does not establish the broader claim as currently stated.
  • There is gap between training-time bound and experiments as Theorem 4.1 requires a cost discount below one, while exps use undiscounted costs.
Questions for authors
  • Can the stability statement be reformulated to specify which quantities remain fixed and which change as the threshold decreases?
  • Does a finite-horizon version of Theorem 4.1 cover the experimental setup?
2 comments
AuthorsOpenReview ↗

Response to K299

Thank you for the review.

Q1: stability statement. We will reformulate it as you suggest: with the policy and batch size NN fixed and the cost limit tracking the threshold, the indicator and CVaR relative variances grow as 1/(εN)1/(\varepsilon N), while Cantelli's stays Θ(1/(fN))\Theta(1/(fN)), where ff is the share of trajectories with non-zero cost. Our reply to 141E (Q2) gives the conditions and an example cost family.

Q2: finite-horizon Theorem 4.1. Yes (General Response B), although it is standard in the literature to pair discounted infinite horizon theory with undiscounted finite-horizon experiments.

Thanks the author for answering my questions. Please also revise the paper according to the weakness part. I will keep my original positive score of 3 for now.

Reviewer_141EOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies reinforcement learning with a value-at-risk (VaR) constraint, where the objective is to maximize expected reward subject to

Pr⁡(C(τ)>ℓ)≤ε.\Pr(C(\tau)>\ell)\leq\varepsilon.

The authors propose Canary, which uses Cantelli’s inequality to construct a smooth, conservative, and distribution-free surrogate for the VaR constraint based on the mean and variance of the cost return.

To incorporate the second moment into trust-region policy optimization, the paper decomposes the squared cost return into a discounted sum of local augmented costs and augments the state with the accumulated cost and discount factor. Building on CPO, the resulting method imposes both the Cantelli constraint and a mean-cost constraint, provides a worst-case bound on surrogate-constraint violations for nominal updates, and introduces a recovery mechanism for infeasible updates.

The paper also analyzes the relative variance of the indicator, CVaR, and Cantelli constraint estimators, relating their effective sample sizes to pNpN, εN\varepsilon N, and fNfN, respectively. This analysis is intended to explain their behavior as the violation threshold becomes stringent. Experiments in three environments with 30 random seeds show that Canary generally achieves lower average violation probabilities and less variation in the latter part of training, albeit with some loss of reward. The authors also decompose the observed conservatism into inequality slack and optimizer slack.

Critical strengths

1. The method is well motivated and leads to a complete, implementable algorithm

Compared with an indicator estimator that relies only on rare violation events, using the first two moments of the cost return can leverage information from a larger fraction of trajectories. The augmented-cost construction in Equations (17)–(20) allows the second moment to be incorporated into value estimation and trust-region updates. Retaining the mean-cost constraint also correctly handles the condition required by Cantelli’s inequality.

The contribution therefore goes beyond simply applying Cantelli’s inequality: it combines the augmented-cost formulation, the two-constraint update, the recovery mechanism, and an analysis of estimation stability. Together, these components provide a simple and interpretable approach to safe policy optimization at low violation probabilities.

2. The experiments cover relevant baselines and report safety, variability, and the cost of conservatism

The main experiments use 30 random seeds and compare against indicator-CPO, CPPO, and CVaR-CPO. The authors report violation probabilities in the latter part of training, their variability, the fraction of seeds satisfying the threshold, and reward returns. The appendix also describes baseline tuning and analyzes the bias of the practical estimators. Thus, the evaluation does not focus solely on average safety while omitting the cost of conservatism.

The distinction between inequality slack and optimizer slack in Figure 3 is also valuable. Overall, the experiments provide reasonably consistent preliminary evidence that Canary can achieve stable, low violation probabilities in the latter part of training.

Critical weaknesses

1. The scope of the training-time theory and stability analysis does not yet adequately cover the main experiments and their interpretation

Theorem 4.1 applies to nominal updates satisfying OP (25), and its bound contains a factor of (1−γc)−2(1-\gamma_c)^{-2}, whereas the main experiments use γc=1\gamma_c=1. Finite episodes ensure that the cost moments exist, but they do not directly make this discounted, infinite-horizon approximation bound applicable to the experiments. The theorem also does not fully account for recovery updates or for estimation errors in the advantages and cost moments.

The low violation probabilities observed late in training are therefore useful empirical results, but they cannot be directly interpreted as training-time safety guaranteed by this theorem. Likewise, the interpretation in Section 6.3 of the constraint excess in EcoAnt and PointGoal as the regime bounded by Theorem 4.1 requires additional justification.

The stability analysis raises a related conceptual issue. The main text defines stability for a fixed policy as ε→0\varepsilon\to0, but Appendix B.1 uses p=cεp=c\varepsilon, with cc held fixed, to obtain the divergence of the indicator estimator. For fixed p>0p>0,

$ \kappa_{\mathrm{Ind}}^2 =\frac{p(1-p)}{N(p-\varepsilon)^2} \longrightarrow \frac{1-p}{Np}. $

It is therefore important to clarify whether the analysis concerns a fixed policy or a sequence of policies that changes with the threshold while maintaining proportional slack.

In addition, the uniform proportional-slack condition in Assumption 5.4 may exclude natural sequences of Cantelli-feasible policies. For example, with fixed μ=aℓ\mu=a\ell, 0<a<10<a<1, BB can remain bounded, while the numerator of the signal share in Equation (83) contains −μ2/ε-\mu^2/\varepsilon and can grow as ε→0\varepsilon\to0. This is not a direct counterexample to the conditional proposition, but it suggests that its scope needs to be explained more carefully.

2. The experiments do not yet fully separate the benefits of stable estimation from conservatism and implementation-specific estimation choices

Canary’s main empirical advantage is its lower violation probability and variability late in training, but this comes with reduced reward. On PointGoal, Canary and CVaR-CPO have the same fraction of safe seeds, 87%, while their average returns are 4.0±0.64.0\pm0.6 and 6.4±2.06.4\pm2.0, respectively. The current results therefore do not fully establish whether Canary retains an optimization advantage when methods are compared at similar realized violation probabilities.

The supplementary implementation also raises two questions directly related to the algorithm and the comparison protocol.

First, in EcoAnt, the raw cost is computed from the action output by the policy, whereas battery depletion in the indicator environment is driven by the noisy action. If these signals are used directly for optimization and evaluation, the corresponding VaR events are not exactly the same. The authors should clarify whether the main table recomputes violation probabilities for all methods using a common raw-cost definition. Otherwise, part of the EcoAnt comparison could reflect differences in the constrained event rather than differences between the methods.

Second, the continuous implementation of Canary uses Monte Carlo moment estimates pooled across rollouts, whereas the algorithm in the paper primarily describes estimates computed from the current policy batch. Pooling mixes data from different historical policies and may introduce additional constraint smoothing and lag. Its relationship to the estimator and stability analysis in the paper should be explained, and the respective contributions of the Cantelli constraint and historical pooling should be distinguished.

These issues do not by themselves demonstrate that the experimental results are incorrect, but they currently limit my ability to assess the source of the empirical gains and the validity of the baseline comparisons.

Questions for authors
  1. Connection between theory and the experimental setting: Can the authors provide an applicable surrogate-error bound for the finite-horizon, undiscounted setting with γc=1\gamma_c=1 used in the experiments? What fraction of updates falls into Tiers a, b, and c in each environment? If this connection cannot currently be established, could the training-time theoretical contribution be explicitly restricted to the discounted setting with exact estimates and nominal updates?

  2. Asymptotic framework for the stability analysis: As ε→0\varepsilon\to0, which quantities are held fixed and which are allowed to change? Please clarify how p=cεp=c\varepsilon is consistent with the fixed-policy definition, and describe a nontrivial family of policies and cost distributions for which Assumptions 5.2–5.4 hold near the feasibility boundary. Does Figure 1 use the same frozen policy and independent repeated batches, and how are the population gradient and the denominator of the relative variance estimated? How should the results be interpreted for IcyLake, whose discrete cost-return distribution does not satisfy the no-atom assumption at the quantile?

  3. Conservatism versus optimization benefits: Can the authors compare Canary and CVaR-CPO at similar realized violation probabilities, or provide safety–reward Pareto curves? This would help distinguish optimization benefits arising from stable estimation from the effects of simply imposing a more conservative constraint.

  4. Consistency of the implementation and evaluation protocol: In EcoAnt, are the final violation probabilities for all methods computed using the same raw-cost definition, or do different methods use violation events based on policy actions and noisy actions, respectively? In addition, is cross-rollout moment pooling used in the main experiments? If so, please explain its relationship to the current-policy estimator described in the paper and provide an ablation without pooling to separate the stability benefits of the Cantelli constraint from those of historical pooling.

2 comments
AuthorsOpenReview ↗

Response to 141E

Thank you for the detailed review.

Q1: theory and tiers. General Response (B) gives the bound for the finite-horizon, undiscounted setting, although it is standard to pair discounted theory with undiscounted experiments. Tier fractions in the second half of training (30 seeds, ε=0.02\varepsilon=0.02):

EnvTier a (nominal)Tier b (Cantelli restoration)Tier c (mean restoration)
EcoAnt0.220.780.00
PointGoal0.190.810.00
IcyLake0.610.390.00

We adopt your scoping: where Tier b dominates (EcoAnt, PointGoal), VaR satisfaction is an empirical result. The conservatism of the Cantelli constraint (Section 4.1) leaves a margin there. We will revise Section 6.3.

Q2: asymptotic framework. We will state it explicitly. The policy, its cost-return distribution, the non-zero-cost share f=Pr⁡(C>0)f=\Pr(C>0) and the batch size NN are fixed. The cost limit tracks the threshold, ℓε=VaRε(π)\ell_\varepsilon=\mathrm{VaR}_\varepsilon(\pi), so the violation probability pp equals ε\varepsilon. The indicator constraint value is zero there, so we measure each relative variance against the sampled signal. It is (1−ε)/(εN)(1-\varepsilon)/(\varepsilon N) for the indicator and Θ(1/(εN))\Theta(1/(\varepsilon N)) for CVaR under Assumption 5.2. For Cantelli, it is Θ(1/(fN))\Theta(1/(fN)) if σ2≠μ2\sigma^2\ne\mu^2, ∇σ2≠0\nabla\sigma^2\ne0, the cost and score-weighted cost have finite fourth moments, and the combined signal is non-degenerate on {C>0}\lbrace C>0\rbrace. A zero-inflated lognormal family meets these conditions with constants independent of ε\varepsilon: C=a eu+ξC=a\thinspace e^{u+\xi}, with risk action a∼Bern(f)a\sim\mathrm{Bern}(f), Gaussian action uu and Gaussian noise ξ\xi. In Monte Carlo down to ε=10−5\varepsilon=10^{-5}, the Cantelli relative variance stays flat, and the indicator and CVaR relative variances grow as 1/ε1/\varepsilon. We will revise Proposition 5.5 and Appendix B accordingly.

Proportional slack. We agree with your reading of Eq. (83). Proposition 5.5 does not need a strictly Cantelli-feasible sequence, because it compares the estimators at one policy on the path above. A new remark will state this.

Figure 1. Yes. We use 30 frozen, randomly initialized policies and 24 independent batches of 256 trajectories per policy. ℓε\ell_\varepsilon is the empirical (1−ε)(1-\varepsilon)-quantile of each policy's pooled trajectories. The value relative variance is the batch variance over the squared batch mean, of the sampled signal for the indicator and CVaR and of the plug-in constraint for Cantelli. The gradient relative variance is 1B∑b∥g^b−gˉ∥2/∥gˉ∥2\frac1B\sum_b\Vert \hat g_b-\bar g\Vert^2/\Vert \bar g\Vert^2, where B=24B=24, g^b\hat g_b is the gradient estimate of batch bb, and the mean gˉ\bar g estimates the population gradient. Batches share the pooled quantile, pooled mean and cost critics. The environment is an IcyLake variant with Gaussian per-step cost, so the cost return has no atoms. For integer-cost IcyLake, the Cantelli result does not need that assumption, and the indicator and CVaR relative variances still satisfy κ2≥(1−ε)/(εN)\kappa^2\ge(1-\varepsilon)/(\varepsilon N) if the cost exceeds the quantile with positive probability. We will add these details to the caption and appendix.

Q3. General Response (A) gives this comparison.

Q4: EcoAnt event. Thank you for catching this. In the submission, CPO's indicator was trained and scored on battery depletion, which uses the noisy executed actions. The other methods were trained and scored on the cost of the commanded actions, as output by the policy. We retrained CPO with its indicator on the same cost, so every method now has the same constraint and is scored on it (30 seeds). The retrained CPO's violation probability is .028 (submitted: .030), and none of its 30 seeds meets the 0.02 threshold. The ranking is unchanged and the revised Table 1 reports these results.

Pooling. Yes. On EcoAnt and PointGoal, Canary's moments use an exponentially weighted pool of completed episodes (decay 0.9, about 10 rollouts). IcyLake uses the current batch. CVaR-CPO also pools completed episodes, in a 256-episode buffer for its CVaR offset. Canary's discounted episode counts average about 250 on EcoAnt and 25 on PointGoal late in training. Without pooling, Canary stays safe on every EcoAnt seed (P^\hat P .005, J^\hat J 100). On PointGoal, the rollout structure requires the pool: 87.5% of rollouts complete no episode, so most updates have no current-batch moment estimate (P^\hat P .51 without the pool). Section 5 analyzes the current-policy estimator, and pooling adds samples from recent policies to it. We will describe the pool in Algorithm 1.

Thank you for the thorough responses. The matched-violation comparison and the finite-horizon, undiscounted theory address my main concerns about conservatism and the theory–experiment connection. The clarified asymptotic setup, corrected EcoAnt comparison, and pooling ablations are also helpful. These clarifications strengthen the paper.

Reviewer_ryZYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Canary, an on-policy RL method for problems with a Value-at-Risk (VaR) constraint on cumulative cost, which keeps the probability that cost exceeds a limit below a small threshold. Canary replaces the VaR constraint with a bound from Cantelli's one-sided Chebyshev inequality: a distribution-free safe approximation that depends only on the first two moments (mean and variance) of the cost return and is more conservative than the usual CVaR constraint. To fit it into the CPO framework, this work rewrites the cost return's second moment as the return of an augmented per-step cost on a state augmented with the cumulative cost, yielding a closed-form two-constraint trust-region update. Theoretically, they extend CPO's worst-case per-update violation bound to the new constraints and, under the stated assumptions in dense cost regimes, show that the relative variance of the Cantelli estimator stays bounded as the threshold shrinks, unlike the indicator and CVaR estimators. Empirically, on three environments at low threshold 0.02, Canary achieves the lowest converged violation probability.

Critical strengths

The moment-based construction provides a useful alternative to rare event estimation of tail constraints: it draws signal from every trajectory rather than relying on rare failures. In the experiments, Canary was the method that achieves the lowest converged violation rate.

Critical weaknesses
  1. Canary's main empirical advantage, the highest rate of safe seeds, is difficult to attribute to the method itself. The Cantelli constraint is stricter than both the VaR and CVaR constraints and allows fewer policies (Section 4.1). Running all methods with the same target threshold of 0.02 therefore favours Canary from the start, because Canary in effect aims for a much lower violation probability. Also, Canary achieves lower reward than CVaR-CPO on PointGoal and IcyLake (Table 1). Canary's higher safe rate could come from two sources: (1) its moment-based estimator is more stable, or (2) its constraint is simply stricter. The current experiments cannot distinguish them. One way to separate them is to make the baselines equally strict, for example by running CPO and CVaR-CPO with a smaller threshold until their violation probabilities match Canary's, and then comparing rewards. If Canary still earns more reward at the same violation level, the stable estimator is likely the source of the gain. If not, the gain mainly reflects the stricter constraint.

  2. The theoretical results also do not explain why Canary is safe in the experiments. The paper's safety argument is that a policy satisfying the Cantelli constraint also satisfies the VaR constraint (Section 4.1). However, Section 6.3 reports that every EcoAnt and PointGoal seed converges with a positive Cantelli constraint value, which means the final policies do not satisfy the Cantelli constraint. So the observed safety in these two environments does not follow from the paper's argument. Rather, it reflects that the Cantelli bound happens to be loose for these cost distributions. Also, the bound in Theorem 4.1 becomes vacuous as the cost discount approaches one, but all experiments use undiscounted costs (Section 6.2). Thus, the theorem does not support the reported runs. Since the Cantelli form of a chance constraint is already known (El Ghaoui et al., 2003) and Theorem 4.1 is a modest extension of CPO's bound, the contribution of this paper relies mainly on the experiments, which makes the comparison problem above decisive.

Questions for authors
  1. Unlike CVaR, VaR ignores how much the cost exceeds the limit, and CVaR is the more common choice in safe RL partly for this reason. In which applications is VaR the more appropriate choice? Since every Cantelli-feasible policy also satisfies the CVaR constraint, would Canary be more accurately described as a stable but more conservative alternative to CVaR constraints?
  2. Do the policies and critics of all methods observe the accumulated cost, or only Canary's (Section 4.2, Algorithm 1)? The indicator cost used by CPO also depends on the accumulated cost, so a policy that cannot observe it may be at a disadvantage. If only Canary receives it, the Canary's better safe behavior may come from extra information rather than from the constraint, and it would be great to see the baseline experiments with the same augmented state.
  3. Please compare the methods at matched strictness, for example by running CPO and CVaR-CPO with smaller thresholds until their violation probabilities match Canary's, and reporting the reward at each level. If the baselines reach similar safety with higher reward, the main contribution reduces to choosing a stricter constraint.
  4. Section 6.3 reports that every EcoAnt and PointGoal seed converges with a positive Cantelli constraint value, so the final policies violate the constraint Canary is designed to enforce. Why does this happen?
  5. Can Theorem 4.1 be stated for the finite-horizon, undiscounted setting used in all experiments?
  6. The paper targets small thresholds, but the main experiments use only 0.02. It would be great to see the results at smaller thresholds (e.g., 0.005, 0.001) together with the reward.
1 comment
AuthorsOpenReview ↗

Response to ryZY

Thank you for the thorough review. We ran the comparisons you proposed. W is a response to a weakness and Q a response to a question.

W1 / Q3: stricter constraint or more stable estimator? Canary's constraint is stricter by design, and at a common ε\varepsilon it trades reward for safety. Your test compares the methods at matched violation probability instead, where the VaR-constrained objective prefers the policy with higher reward. There, Canary dominates on EcoAnt and PointGoal, and achieves comparable reward on IcyLake (General Response A), so the stricter constraint alone does not explain the gain. Q2 adds the augmented-state control, and Canary's violation also varies least in the second half of training (Table 1, σ^\hat\sigma), consistent with a more stable moment-based estimator.

W2 / Q4: why cB>0c_B>0 at convergence? As Section 4.5 states, no update tier guarantees constraint satisfaction: like CPO's, Canary's trust-region update bounds violation rather than enforcing the constraint exactly, so a small positive cBc_B at convergence is expected. On EcoAnt and PointGoal, most second-half updates are Cantelli restorations (Tier b, 78% and 81%). CPO behaves the same way: it exceeds its target in all three environments (P^\hat P = .029, .060 and .039 at ε=0.02\varepsilon=0.02), and its excess is on the violation probability itself. Canary's excess is instead on the Cantelli bound, which is conservative by design (Section 4.1): it is attained only by a worst-case two-point distribution, so a small excess over the Cantelli boundary can still leave a margin to the VaR constraint. We will revise Section 6.3 to state that converged VaR satisfaction is shown empirically.

Q5. Yes. General Response (B) gives the finite-horizon theorem, although it is standard practice to pair undiscounted experiments with discounted theory in the literature

Novelty. El Ghaoui et al. (2003), which the paper cites, develop moment-based worst-case VaR optimization for static portfolios. In RL, the second moment of a return is not an expected return of the original MDP, so trust-region policy optimization does not apply to it directly. Our contribution makes it apply: the augmented-cost decomposition of E[C2]\mathbb E[C^2], the closed-form two-constraint update with recovery tiers, the stability analysis and the finite-horizon bound.

Q1: when is VaR the right constraint? VaR fits tasks where failure is binary and the size of an exceedance does not matter: a margin call, a depleted battery, a collision, or a regulatory limit on failure probability. CVaR fits tasks where larger exceedances are worse. Your description is accurate for the constraint sets, and we will add it to the introduction.

Q2: augmented state. In the submitted runs, only Canary's networks received the accumulated cost and the discount power, (yt,γct)(y_t,\gamma_c^t). On EcoAnt, the environment observation already contains the battery level. We reran CPO and CVaR-CPO with the augmented state (ε=0.02\varepsilon=0.02, same codebase and seeds):

EnvMethodEnv. obs. P^\hat P / J^\hat J / seeds+ (yt,γct)(y_t,\gamma_c^t) P^\hat P / J^\hat J / seeds
IcyLakeCPO.039 / .96 / 2/30.029 / .96 / 4/30
IcyLakeCVaR-CPO.019 / .95 / 20/30.018 / .95 / 20/30
PointGoalCPO.060 / 12.1 / 0/30.071 / 24.9 / 1/30
PointGoalCVaR-CPO.015 / 6.4 / 26/30.018 / 8.3 / 21/30

Overall the result is mixed and changes little. Canary's safety advantage therefore does not come from the augmented state.

Q6: smaller thresholds. Results at ε=0.005\varepsilon=0.005 (P^\hat P / J^\hat J / seeds with P^≤0.005\hat P\le0.005; 30 seeds):

EnvCanaryCPOCVaR-CPO
EcoAnt.001 / 37 / 30/30.015 / 173 / 0/30.051 / 116 / 0/30
IcyLake.0016 / .46 / 27/30.040 / .96 / 0/30.022 / .95 / 0/30
PointGoal.007 / 0.4 / 15/30.036 / 5.7 / 0/30.010 / 1.3 / 6/30

Only Canary meets ε=0.005\varepsilon=0.005 on any EcoAnt or IcyLake seed. On PointGoal, no method's mean violation reaches 0.005 within the training budget; Canary comes closest (.007, 15 of 30 seeds).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Shared General Response

We thank the reviewers for their careful and constructive reviews. We will respond individually, but shared points of interest for all are:

  • At similar violation probability, Canary achieves stronger reward accumulation than CPO and CVaR-CPO on EcoAnt and PointGoal, and comparable reward on IcyLake (A, below).
  • At target violation threshold ε=0.005\varepsilon=0.005, only Canary satisfies the constraint on EcoAnt and IcyLake, and comes closest to satisfying on PointGoal.
  • Canary responds best to changes in violation threshold at small values. For example, on EcoAnt and IcyLake, tightening ε\varepsilon from 0.02 to 0.005 lowers Canary's violation probability (.002 → .001, .016 → .002) but not CVaR-CPO's (.048 → .051, .019 → .022) (A).
  • Canary's safety advantage persists with augmented-state baselines (reply to ryZY), a common EcoAnt event, and no EcoAnt pooling (reply to 141E).
  • Theorem 4.1 now covers the undiscounted, finite-horizon setting of the experiments, although pairing the undiscounted finite-horizon setting in experimental results with the discounted infinite-horizon setting in the theory is standard practice in the literature (B).

(A) Safety–reward comparison at matched violation. At each environment's cost limit, we fix Canary at ε=0.1\varepsilon=0.1 and report, for each baseline, the threshold whose violation P^\hat P is closest to Canary's, or the strictest tested (ε=0.005\varepsilon=0.005) if none reaches it. Values are second-half-of-training means ± standard error over 30 seeds.

EnvMethodε\varepsilonP^\hat PJ^\hat J
EcoAntCanary0.1.0127 ± .0009184 ± 7
CPO0.005.0133 ± .0004112 ± 5
CVaR-CPO0.005.039 ± .003116 ± 6
PointGoalCanary0.1.023 ± .00126.0 ± 1.7
CPO0.005.036 ± .0025.7 ± 0.9
CVaR-CPO0.05.026 ± .00112.9 ± 2.5
IcyLakeCanary0.1.018 ± .001.92 ± .02
CPO0.005.040 ± .003.962 ± .003
CVaR-CPO0.02.019 ± .002.949 ± .002

(B) Theory for undiscounted, finite-horizon costs. Pairing discounted theory with undiscounted episodic costs is standard practice. CPO's official code (safety_discount=1), Safety Gym ("finite horizon undiscounted … cumulative cost formulations"), PCPO and Sauté all use it. For non-negative per-step costs ctc_t and cost discount γc≤1\gamma_c\le1, the undiscounted constraint is also the stricter one, because ∑tγctct≤∑tct\sum_t\gamma_c^tc_t\le\sum_tc_t. Nonetheless, for completeness and to respond to the direct questions from reviewers, we now also prove Theorem 4.1 for γc=1\gamma_c=1 and horizon TT, the setting of every experiment, and we will add it as an appendix. The proof replaces the advantages of Theorem 4.1 with time-indexed advantages AtgA_t^g for each cost signal gg, with αg=max⁡t<Tmax⁡x∣Ea∼π[Atg(x,a)]∣\alpha^g=\max_{t<T}\max_x|\mathbb E_{a\sim\pi}[A_t^g(x,a)]|. The trust region bounds the time-averaged KL divergence, DˉKLT≤δ\bar D^T_{\mathrm{KL}}\le\delta. Theorem 4.1 and Corollary 4.2 then hold with MM replaced by MT=∣μ(πk)∣+TαCM_T=|\mu(\pi_k)|+T\alpha^C and KK replaced by

KT=min⁡{2(T−1),2δT2(T−1)(2T−1)/6}≤2δ T2/3.K_T=\min\Big\lbrace 2(T-1),\quad \sqrt{2\delta}\sqrt{T^2(T-1)(2T-1)/6}\Big\rbrace\le\sqrt{2\delta}\thinspace T^2/\sqrt3.

The square-root term is asymptotically 1/31/\sqrt3 of the discounted constant at the matching effective horizon 1/(1−γc)=T1/(1-\gamma_c)=T. For the KL direction that TRPO uses, the constant is (T−1)min⁡{2,2Tδ}=O(T3/2δ)(T-1)\min\{2,\sqrt{2T\delta}\}=O(T^{3/2}\sqrt\delta), and for small TδT\delta an explicit construction attains this order. As with CPO's bound, these are worst-case results for the idealized update with exact advantages and moments.

AuthorsOpenReview ↗

Rebuttal Revision

Revised paper and supplementary uploaded. Changes are highlighted in blue. The main ones are:

  • Section 6.3 and Table 2 (new): reward at matched violation probability. At similar or lower violation probability, Canary attains higher reward than CPO and CVaR-CPO on EcoAnt and PointGoal, and comparable reward on IcyLake (ryZY W1/Q3, 141E Q3, brnS Q4).
  • Table 1, EcoAnt: CPO is retrained with its indicator on the commanded-action cost, so every method trains on and is scored on the same constraint (141E Q4).
  • Appendix A.7 (new): Theorem 4.1 and Corollary 4.2 for the undiscounted, finite-horizon setting of the experiments (ryZY Q5, 141E Q1, brnS Q1).
  • Section 5 and Appendix B: Proposition 5.5 states its fixed-policy setting explicitly. The assumption needed only for the CVaR result moves to Assumption B.1 (141E Q2).
  • Appendix C: the cross-rollout moment pool, and the solver's handling of nearly parallel constraint gradients in C.1 (141E Q4, brnS W2/Q2).
  • Appendix D.2 (new): the protocol behind Figure 1 (141E Q2).

The supplementary code regenerates every result in our replies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.