- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes JRCPO, a constrained reinforcement learning framework that assesses risk using the joint distribution of reward and cost returns. It employs a joint Gaussian critic to compute the conditional value-at-risk (CVaR) of a scalarized reward–cost objective in closed form. Experiments on several multi-cost tasks evaluate the method’s effectiveness in balancing reward maximization and cost control.
Critical strengths
The paper addresses an important, interesting, and meaningful problem. The idea of evaluating risk along the reward–cost trade-off direction and capturing dependencies through a joint covariance model is appealing. The experiments cover a broad scope, are comprehensive in comparison, and include detailed findings, along with extensive ablation studies.
Critical weaknesses
The problem formulation does not appear to support the paper’s central claim of reducing unnecessary conservatism through joint risk constraints. The authors state that they "formulate a joint-risk constraint problem, which focuses risk evaluation on realizable events to avoid unnecessary conservatism". However, under the stated definition of , CVaR duality implies that the cost constraints in Eq. (8) are equivalent to the componentwise CVaR constraints in Eq. (7). Please see Question 1 for details.
Questions for authors
-
The cost constraints in Eq. (8) appear to be equivalent to those in Eq. (7), and Eq. (8) does not seem to eliminate conservatism. My understanding is that, for any fixed policy , applying CVaR duality on the joint return probability space gives Consequently, requiring each cost inequality in Eq. (8) to hold for every appears equivalent to retaining all the componentwise CVaR constraints in Eq. (7). Although is defined using the joint distribution, different cost components may attain their worst-case expectations under different measures in ; these constraints do not require the components’ worst-case outcomes to occur simultaneously. Is this interpretation correct? From this perspective, does Eq. (8) fail to reduce conservatism?
-
Could the authors provide a derivation of the multiplier update in Eq. (18)?
6 comments
Rebuttal Cont. (Example)
Example
Consider a one-step CMDP with a deterministic policy and the action . The reward and costs are
where . The cost limits are . The marginal distribution of each cost does not depend on . For both Lagrangians, we maximize over for each and then minimize the result over . We use for .
Lagrangian of Eq. 7. Since , the Lagrangian is
The solution is and . The Lagrangian treats the costs one at a time, so this solution does not depend on .
Eq. 10 with . Since , its CVaR is . Eq. 10 equals the Lagrangian of Eq. 7 and gives the same solution.
Eq. 10 with . Since , Eq. 10 is
Eq. 10 is concave in . When , the maximum is on the ridge . The absolute value term is on this ridge. Maximizing on the ridge gives the solution and the maximum value :
With , . Equality holds only at . The solution is . Both mean costs equal the cost limit . The combined cost equals for every . The mean return is , which is higher than that of Eq. 7.
At , the two costs cannot take their worst values at the same time. Eq. 10 uses the joint distribution of the costs and gives a less conservative solution.
Q2. Derivation of the multiplier update in Eq. (18).
We give the derivation of Eq. 18 below.
While writing this derivation, we noticed that the joint risk in the submitted manuscript was missing the factor in Eq. 13 and 17 as well as the factor in Eq. 18. We have corrected these three equations in the revised manuscript. The submitted code is already implemented with the corrected equations, so all reported results remain unchanged.
The dual update takes the gradient of the actor loss without the factor since it does not change the policy that minimizes Eq. 13 [Stooke]. We do not backpropagate through the joint risk, and with for , projected gradient ascent gives
\boldsymbol{\lambda}_{t+1} &= \left[ \boldsymbol{\lambda}_t + \eta_\lambda \nabla_{\boldsymbol{\lambda}} \left( \left(1 + \\|\boldsymbol{\lambda}\\|_1\right) \mathcal{L}(\pi; \boldsymbol{\lambda}) \right) \right]_+ \\\\ &= \left[ \boldsymbol{\lambda}_t + \eta_\lambda \nabla_{\boldsymbol{\lambda}} \left( -\mu^{\pi}_{R} + \boldsymbol{\lambda}^{T}\left(\boldsymbol{\mu}^{\pi}_{C} - \mathbf{d} + \frac{\kappa_{\alpha} \sqrt{\bar{\boldsymbol{\lambda}}_t^{T} \Sigma \bar{\boldsymbol{\lambda}}_t}}{\\|\bar{\boldsymbol{\lambda}}_t\\|_1}\mathbf{1}\right) \right) \right]_+ \\\\ &= \left[ \boldsymbol{\lambda}_t + \eta_\lambda \left( \boldsymbol{\mu}^{\pi}_{C} - \mathbf{d} + \frac{\kappa_{\alpha} \sqrt{\bar{\boldsymbol{\lambda}}_t^{T} \Sigma \bar{\boldsymbol{\lambda}}_t}}{\\|\bar{\boldsymbol{\lambda}}_t\\|_1}\mathbf{1} \right) \right]_+. \end{aligned}$$ Applying the same derivation to the normalized joint return $\bar{\mathbf{Z}}^\pi$, and estimating $\bar{\boldsymbol{\mu}}^{\pi}_{C}$ and $\bar{\Sigma}$ by $\widehat{\boldsymbol{\mu}}^{\pi}_C$ and $\widehat{\Sigma}$ from the joint return samples in the sliding window $W$, gives Eq. 18: $$\begin{aligned} \boldsymbol{\lambda}_{t+1} &= \left[ \boldsymbol{\lambda}_t + \eta_\lambda \left( \bar{\boldsymbol{\mu}}^{\pi}_{C} - \mathbf{1} + \frac{\kappa_{\alpha} \sqrt{\bar{\boldsymbol{\lambda}}_t^{T} \bar{\Sigma} \bar{\boldsymbol{\lambda}}_t}}{\\|\bar{\boldsymbol{\lambda}}_t\\|_1}\mathbf{1} \right) \right]_+ \\\\ &\approx \left[ \boldsymbol{\lambda}_t + \eta_\lambda \left( \widehat{\boldsymbol{\mu}}^{\pi}_C - \mathbf{1} + \frac{\kappa_{\alpha} \sqrt{\bar{\boldsymbol{\lambda}}_t^{T} \widehat{\Sigma} \bar{\boldsymbol{\lambda}}_t}}{\\|\bar{\boldsymbol{\lambda}}_t\\|_1}\mathbf{1} \right) \right]_+. \end{aligned}$$ We added the derivations of Eq. 13 and 18 to Appendix F.1 and F.2, respectively. ### **References** [Stooke] Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by PID Lagrangian methods. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9133–9143. PMLR
We greatly appreciate the reviewer's detailed and thoughtful feedback.
The reviewer's derivation holds for each cost constraint. The interpretation considers the constraints of Eq. 8 one at a time. JRCPO optimizes the Lagrangian of Eq. 8 (Eq. 9 and 10). This Lagrangian reduces the conservatism when the costs cannot take their worst values at the same time.
Constraints and Lagrangians of Eq. 7 and Eq. 8
By the coherent duality (Theorem 1), CVaRα(ZC(i)π)=supQ∈TEQ[ZC(i)π] for each cost. Thus, the condition "for all Q∈T" in Eq. 8 is equivalent to the CVaR constraint of each cost in Eq. 7. The Lagrangian of Eq. 7 is μRπ−∑i=1nλi(CVaRα(ZC(i)π)−di). Its cost term is ∑i=1nλisupQi∈TEQi[ZC(i)π]. Each cost takes its worst case under a different measure Qi. This is the case the reviewer described.
The Lagrangian of Eq. 8 is Eq. 9. Taking the infimum over T gives Eq. 10:
infQ∈TEQ[ZRπ−∑i=1nλi(ZC(i)π−di)]=−CVaRα(−ZRπ+λT(ZCπ−d)).
A single Q applies to all costs combined by λ. In Eq. 10, the costs cannot take their worst cases under different measures.
In the cost term, Eq. 7 takes the sum of the worst cases, and Eq. 10 takes the worst case of the sum. By the subadditivity of CVaR,
∑i=1nλiCVaRα(ZC(i)π)≥CVaRα(λTZCπ).
Equality holds only when the worst cases of all costs occur on the same event. Otherwise, Eq. 7 penalizes an unrealizable event in which all costs take their worst values at the same time.