- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Canary, an on-policy reinforcement learning method for maximizing expected reward subject to a Value-at-Risk constraint on cumulative cost. The method uses Cantelli’s inequality to construct a conservative safety approximation based on the mean and variance of the cost return. Through state and cost augmentation, the second cost moment is expressed as an expected return and incorporated into the CPO trust-region framework. The paper derives worst-case constraint violation bounds in the discounted setting and analyzes the relative variance of constraint estimators in dense-cost regimes. Experiments on EcoAnt, PointGoal, and IcyLake report lower average violation probabilities and smaller fluctuations than the evaluated baselines, with a trade-off in reward.
Critical strengths
1 The method has a clear design and practical value. It uses the first two cost moments to construct a conservative approximation to the VaR constraint. State augmentation then allows the second moment to be incorporated into the CPO framework. This provides an implementable approach to tail-risk control without assuming a Gaussian cost distribution.
2 The main experiments provide useful empirical evidence. The authors use 30 seeds across three environments and report violation rates, their fluctuations, and rewards. Canary achieves lower average violation rates and smaller fluctuations on these tasks. The paper also reports the associated reward cost, which helps explain the trade-off.
Critical weaknesses
The theoretical result does not fully cover the experimental setting. Theorem 4.1 assumes γc<1, but all experiments use γc=1. No corresponding finite-horizon result is provided, so the theorem cannot directly establish a training-time violation bound for these experiments. Also, the safe-seed percentage in Table 1 is based on the average violation rate over the second half of training. It does not show that the constraint holds at each training stage. This limits the evidence for training-time safety.
The solver’s treatment of nearly parallel constraints needs further explanation. Appendix C.1 drops the mean constraint when the two constraint gradients are nearly parallel. Parallel gradients do not imply redundancy, since the constraint offsets and their strictness can differ. Under the stated rule, the stricter mean constraint could be removed. It remains unclear how the final feasibility check handles this case and how a rejected candidate is replaced with a valid update.
Questions for authors
How does Theorem 4.1 apply to the experimental setting with γc=1? Can you provide a finite-horizon counterpart and state its dependence on episode length? If the current theorem does not cover this setting, please clarify which training-time safety claims remain theoretically supported.
What conditions justify dropping the mean constraint? How does the implementation handle gradients that point in the same direction when the mean constraint is stricter? After dropping a constraint, does the solver still check both original constraints and search for another feasible update if the candidate violates the mean constraint?
Can you report the proportions of nominal updates, Cantelli restoration steps, and mean restoration steps from the training logs, together with violation rates at different training stages? These statistics would be especially useful for the EcoAnt and PointGoal runs that converge with cB>0.
How does Canary’s reward compare with CPO and CVaR-CPO at similar empirical violation rates? Can existing results or an additional comparison help distinguish the benefit of more stable estimation from the benefit of choosing a more conservative policy?
2 comments
Thank you for the detailed responses. The finite-horizon extension addresses my concern about the theoretical setting. The clarification that the implementation retains the mean constraint also addresses my concern about nearly parallel constraints. The comparisons at similar empirical violation rates strengthen the evidence for a useful safety–reward trade-off, particularly on EcoAnt and PointGoal.
Overall, the responses have sufficiently addressed my main concerns, and I am updating my rating to 3 (weak accept).
Response to brnS
Thank you for the specific questions.
Q1: Theorem 4.1 at γc=1, and safety during training. With the finite-horizon counterpart (General Response B), the training-time statements of Section 4 hold in the experimental setting: the per-update bound for nominal updates, and the mean-cost bound μ(πk+1)≤ℓ+KTαC for Tiers a–b (Corollary 4.2). As in CPO, both are for the idealized update, so we also report violation at each training stage. At ε=0.02, the table reports each training quartile's mean violation rate over 30 seeds and the share of seeds whose rate is at most 0.02. EcoAnt (common event) and the IcyLake baselines use new runs, and the other rows use submitted runs.
On PointGoal and IcyLake, every method violates in the first quartile, while learning the task. From the second quartile on, Canary meets the threshold on 80–100% of seeds, against 0–17% for CPO and 0–90% for CVaR-CPO.
Q2: nearly parallel constraints. The Appendix C.1 sentence was imprecise, and we will correct it. The implementation never drops the mean constraint. If S is near-singular, the solver discards only the candidate with both constraints active. It keeps a single-constraint candidate only if it also satisfies the other linearized constraint. If no candidate is valid, it takes the Tier b direction. The line search then checks the mean surrogate in every tier. If the mean constraint holds, the mean surrogate may use the available slack. If the mean constraint is violated, the mean surrogate must not increase. The line search applies the same test to the Cantelli surrogate in Tier a, requires no increase in Tier b, and omits that check in Tier c.
Q3: tier proportions. Second-half fractions are in our reply to 141E (Q1). By quartile, Tier b rises from 0.61 to 0.80 on EcoAnt and stays near 0.80 on PointGoal. On IcyLake, Tier a rises from 0.26 to 0.64. Tier c accounts for less than 0.2% of second-half updates.
Q4. General Response (A) gives this comparison.