- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work investigates CVaR-constrained reinforcement learning in the PAC setting. The authors propose a three-phase algorithm, CVaR-PDS. Phase I performs primal-dual learning for a collection of fixed-VaR subproblems; Phase II employs reward-free exploration to learn a global empirical model; and Phase III uses this model for post-hoc policy evaluation and selection. This design requires strong primal-dual guarantees only for the unknown fixed-VaR subproblem corresponding to the VaR of the optimal policy on the discretized MDP, rather than for all subproblems. The authors further establish both upper and lower bounds on the sample complexity of CVaR-constrained RL.
Critical strengths
This paper provides a rigorous theoretical analysis. Its results cover key aspects of the problem, with detailed proofs of suboptimality and constraint-violation transfer, as well as upper and lower bounds on sample complexity. Furthermore, the careful design of the three-stage algorithm avoids requiring Slater’s condition for every VaR subproblem; the condition is needed only for the subproblem associated with the VaR of an optimal policy in the discretized MDP.
Critical weaknesses
Although the paper’s main contributions are theoretical, the experimental section is rather limited, consisting of a single small-scale instance and a brief discussion. More importantly, even for this instance, the experimental implementation departs from the theoretically prescribed parameters and stopping rule. The authors explicitly state in the appendix that “This fixed segment in Phase II is an engineering budget rather than the theoretical stopping time ” and that “The displayed , learning rates, , and bonus scale are frozen empirical choices. They are not the theorem’s parameter schedule. The frozen empirical bonus profile also differs from the bonuses in Algorithms 1 and 2.” Consequently, the experimental implementation is not directly covered by the PAC guarantee in Theorem 6.1, leaving a gap between the theoretical analysis and the empirical evaluation. These deviations, even on a small-scale instance, raise questions about the practical feasibility of the algorithm under its theoretically prescribed settings.
More detailed reporting of the theoretically required versus actual sample budgets, particularly the number of episodes needed to satisfy the Phase II stopping criterion, would help clarify this gap. Results on additional benchmarks and sensitivity analyses of key hyperparameters would also strengthen the empirical evaluation and help assess the method’s practical applicability.
Questions for authors
-
Could the authors explain why the experiments use a fixed Phase II budget and empirical choices for , the learning rates, , and the bonus scale and profile, rather than the stopping rule and settings prescribed by the theoretical analysis? Are the theoretically prescribed settings overly conservative in practice, even for the small-scale instance evaluated in the paper?
-
Although the paper avoids requiring Slater’s condition for every VaR subproblem, Assumption 5.2 still needs further explanation. Strict CVaR feasibility does not appear to imply that Assumption 5.2 necessarily holds. Could the authors provide further explanation of Assumption 5.2 and explain how to determine a valid positive value of required by Algorithm 1 when is unknown?
5 comments
Rebuttal to Reviewer 6rHP (Part 2/2)
Additional results on a larger MDP: We also evaluated all algorithms on a larger MDP with continuous risk costs under a 550000-episode cap. In this experiment, , , , , , and . We conduct 10 independent runs for each algorithm, and report the results as mean standard deviation over 10 runs:
| Algorithm | Suboptimality | CVaR violation | Joint success |
|---|---|---|---|
| CVaR-CP (ours) | |||
| PG-CVaR | |||
| CVaR-CPO | |||
| OptPrimalDual-CMDP |
Across all 10 runs, CVaR-CP achieves both -feasibility and -optimality, whereas PG-CVaR and CVaR-CPO attain -feasibility but miss the -optimality target.
Sensitivity of key parameters: Following the reviewer's suggestion, we varied the accuracy parameter and the risk level separately to validate the practical performance of our algorithm. We report the results as mean standard deviation over 3 independent runs:
| ε | α | Suboptimality | CVaR violation | Joint success |
|---|---|---|---|---|
Across all settings, CVaR-CP achieves 3/3 joint success. As varies from to , the CVaR violation of CVaR-CP steadily decreases from to . In addition, as ranges from to stricter , CVaR-CP consistently satisfies both -optimality and -feasibility.
Thank you again for your time. If our response addresses your concerns satisfactorily, we would like to kindly ask you to consider raising your rating. Certainly, we are more than happy to answer any follow-up questions.
References:
- Chow et al. Risk-Constrained Reinforcement Learning with Percentile Risk Criteria. JMLR, 2018.
- Zhang et al. CVaR-Constrained Policy Optimization for Safe Reinforcement Learning. IEEE TNNLS, 2025.
- Efroni et al. Exploration-Exploitation in Constrained MDPs. arXiv:2003.02189, 2020.
Dear Reviewer 6rHP, we were wondering if our response resolved your concerns?
Dear Reviewer 6rHP,
As the discussion period (authors + reviewers + AC) is nearing its deadline, we are eager to learn whether our responses have addressed your concerns. If you have any further questions, we would be more than happy to answer them.
Thank you!
Best regards,
Authors
Rebuttal to Reviewer 6rHP (Part 1/2)
Thank you for your helpful comments! We will incorporate the simplified algorithm without requiring Assumption 5.2 (Slater's condition) or the Slater margin ζξ, new experiments, and clarifications presented below into our revision.
Q2 (Simplified algorithm and removal of Slater's condition and the Slater margin ζξ): Thank you for your insightful question, which prompted us to improve our algorithm and remove unnecessary assumptions.
Our algorithm can be simplified into a two-phase framework, i.e., Reward-Free Exploration (RFE) followed by planning (referred to as CVaR-CP, CVaR-Constrained Planning). Specifically, CVaR-CP reparameterizes the augmented state by (νi−yh)+, where νi is the VaR candidate and yh is the cumulative risk cost prior to step h. This allows the same Bellman recursion to be used for all VaR candidates, and thus for a given Lagrange multiplier, a single shared backward dynamic programming evaluates the optimal policies for all VaR candidates.
Since CVaR-CP drops the original online primal-dual learning phase (Phase I) and performs optimal policy planning for all VaR candidates together, it achieves an improved sample complexity of O(αε2S2AH2+αε3SAH3). In particular, by allocating a small fraction of ε as the planning slack parameter, the Lagrange multiplier can be upper bounded by O(εH) without relying on strong duality, and thus CVaR-CP does not require Slater's condition or the Slater margin ζξ.
W1, W2, Q1 (Experiments, stopping rule, and sensitivity): We evaluated the simplified algorithm CVaR-CP against the baselines on the MDP described in Appendix A.1 under the same 400000-episode cap. We implemented our algorithm according to the theoretically guaranteed procedure, including the RFE stopping rule, to show its practical performance. Each algorithm was evaluated over 10 independent runs. Joint success means that both suboptimality and CVaR violation are at most ε=0.1. The results are reported as mean ± standard deviation over 10 runs:
CVaR-CP succeeds in all 10 runs using less running time than all baselines. CVaR-CPO and PG-CVaR pass the ε-feasibility test in 10 runs and 4 runs, respectively, but they fail to achieve joint success due to high suboptimality. OptPrimalDual-CMDP does not attain ε-feasibility in all runs, since it fails to handle the risk constraint.