acceptodds
Under review as a conference paper at ICLR 2027

Joint-risk Aware Constrained Risk-sensitive Reinforcement Learning

Abstract

Maximizing rewards under various cost constraints is a frequent requirement when applying reinforcement learning (RL) to domains such as robotics and energy systems. Since expected-cost constraints alone cannot prevent rare but severe violations, constrained risk-sensitive RL is introduced to address this limitation through constraints on the risk of cost returns. However, because these constraints assess each cost independently using marginal distributions, they enforce overly conservative trade-offs by penalizing physically impossible events. We propose JRCPO, which preserves expected-cost constraints while replacing disjoint marginal risk constraints into a joint risk constraint over the reward-cost distribution. Utilizing the coherent duality of the Conditional Value at Risk (CVaR), JRCPO evaluates the CVaR of a scalarized objective along the Lagrange multiplier-directed trade-off path. This approach captures the joint distribution and considers only physically realizable events, resulting in an efficient reward-cost trade-off. Consequently, we demonstrate via evaluations on multi-cost tasks that the proposed method achieves higher reward returns than methods with cost-wise independent marginal risk constraints, and lower cost returns than expectation-based methods.

Primary area:
reinforcement learning
Keywords:
constrained reinforcement learning, risk-sensitive reinforcement learning, joint reward-cost distribution, safe reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_SWyKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes JRCPO, a constrained reinforcement learning framework that assesses risk using the joint distribution of reward and cost returns. It employs a joint Gaussian critic to compute the conditional value-at-risk (CVaR) of a scalarized reward–cost objective in closed form. Experiments on several multi-cost tasks evaluate the method’s effectiveness in balancing reward maximization and cost control.

Critical strengths

The paper addresses an important, interesting, and meaningful problem. The idea of evaluating risk along the reward–cost trade-off direction and capturing dependencies through a joint covariance model is appealing. The experiments cover a broad scope, are comprehensive in comparison, and include detailed findings, along with extensive ablation studies.

Critical weaknesses

The problem formulation does not appear to support the paper’s central claim of reducing unnecessary conservatism through joint risk constraints. The authors state that they "formulate a joint-risk constraint problem, which focuses risk evaluation on realizable events to avoid unnecessary conservatism". However, under the stated definition of T\mathcal T, CVaR duality implies that the cost constraints in Eq. (8) are equivalent to the componentwise CVaR constraints in Eq. (7). Please see Question 1 for details.

Questions for authors
  1. The cost constraints in Eq. (8) appear to be equivalent to those in Eq. (7), and Eq. (8) does not seem to eliminate conservatism. My understanding is that, for any fixed policy π\pi, applying CVaR duality on the joint return probability space gives sup⁡Q∈TEQ[ZC(i)π−di]=CVaRα(ZC(i)π)−di.\sup_{\mathbb Q\in\mathcal T}\mathbb E_{\mathbb Q}\left[Z^\pi_{C^{(i)}}-d_i\right]=\text{CVaR}_{\alpha} (Z^\pi_{C^{(i)}})-d_i. Consequently, requiring each cost inequality in Eq. (8) to hold for every Q∈T\mathbb Q\in\mathcal T appears equivalent to retaining all the componentwise CVaR constraints in Eq. (7). Although T\mathcal T is defined using the joint distribution, different cost components may attain their worst-case expectations under different measures in T\mathcal T; these constraints do not require the components’ worst-case outcomes to occur simultaneously. Is this interpretation correct? From this perspective, does Eq. (8) fail to reduce conservatism?

  2. Could the authors provide a derivation of the multiplier update in Eq. (18)?

6 comments

We greatly appreciate the reviewer's detailed and thoughtful feedback.

W1, Q1. Interpretation and conservatism of Eq. (8).

The reviewer's derivation holds for each cost constraint. The interpretation considers the constraints of Eq. 8 one at a time. JRCPO optimizes the Lagrangian of Eq. 8 (Eq. 9 and 10). This Lagrangian reduces the conservatism when the costs cannot take their worst values at the same time.

Constraints and Lagrangians of Eq. 7 and Eq. 8

By the coherent duality (Theorem 1), CVaRα(ZC(i)π)=sup⁡Q∈TEQ[ZC(i)π]\text{CVaR}_{\alpha}(Z^{\pi}_{C^{(i)}}) = \sup_{\mathbb{Q} \in \mathcal{T}} \mathbb{E}_{\mathbb{Q}}[Z^{\pi}_{C^{(i)}}] for each cost. Thus, the condition "for all Q∈T\mathbb{Q} \in \mathcal{T}" in Eq. 8 is equivalent to the CVaR constraint of each cost in Eq. 7. The Lagrangian of Eq. 7 is μRπ−∑i=1nλi(CVaRα(ZC(i)π)−di)\mu^{\pi}_{R} - \sum_{i=1}^{n} \lambda_i \big(\text{CVaR}_{\alpha}(Z^{\pi}_{C^{(i)}}) - d_i\big). Its cost term is ∑i=1nλisup⁡Qi∈TEQi[ZC(i)π]\sum_{i=1}^{n} \lambda_i \sup_{\mathbb{Q}_i \in \mathcal{T}} \mathbb{E}_{\mathbb{Q}_i}[Z^{\pi}_{C^{(i)}}]. Each cost takes its worst case under a different measure Qi\mathbb{Q}_i. This is the case the reviewer described.

The Lagrangian of Eq. 8 is Eq. 9. Taking the infimum over T\mathcal{T} gives Eq. 10:

inf⁡Q∈TEQ[ZRπ−∑i=1nλi(ZC(i)π−di)]=−CVaRα(−ZRπ+λT(ZCπ−d)).\inf_{\mathbb{Q} \in \mathcal{T}} \mathbb{E}_{\mathbb{Q}}\Big[Z^{\pi}_{R} - \sum_{i=1}^{n} \lambda_i \big(Z^{\pi}_{C^{(i)}} - d_i\big)\Big] = -\text{CVaR}_{\alpha}\big(-Z^{\pi}_{R} + \boldsymbol{\lambda}^{T}(\mathbf{Z}^{\pi}_{C} - \mathbf{d})\big).

A single Q\mathbb{Q} applies to all costs combined by λ\boldsymbol{\lambda}. In Eq. 10, the costs cannot take their worst cases under different measures.

In the cost term, Eq. 7 takes the sum of the worst cases, and Eq. 10 takes the worst case of the sum. By the subadditivity of CVaR,

∑i=1nλiCVaRα(ZC(i)π)≥CVaRα(λTZCπ).\sum_{i=1}^{n} \lambda_i \text{CVaR}_{\alpha}(Z^{\pi}_{C^{(i)}}) \ge \text{CVaR}_{\alpha}(\boldsymbol{\lambda}^{T}\mathbf{Z}^{\pi}_{C}).

Equality holds only when the worst cases of all costs occur on the same event. Otherwise, Eq. 7 penalizes an unrealizable event in which all costs take their worst values at the same time.

AuthorsOpenReview ↗

Rebuttal Cont. (Example)

Example

Consider a one-step CMDP with a deterministic policy and the action θ=(θ1,θ2)∈[−1,1]2\theta = (\theta_1, \theta_2) \in [-1, 1]^2. The reward and costs are

R=θ1+θ2,Ci=(θi+1)2(ei+1),e1∼U[−1,1],e2=ρe1,R = \theta_1 + \theta_2, \qquad C_i = (\theta_i + 1)^2 (e_i + 1), \qquad e_1 \sim U[-1, 1], \qquad e_2 = \rho e_1,

where ρ∈−1,1\rho \in \\{-1, 1\\}. The cost limits are d1=d2=1d_1 = d_2 = 1. The marginal distribution of each cost does not depend on ρ\rho. For both Lagrangians, we maximize over θ\theta for each λ\boldsymbol{\lambda} and then minimize the result over λ\boldsymbol{\lambda}. We use CVaRα(a+be1)=a+(1−α)∣b∣\text{CVaR}_{\alpha}(a + b e_1) = a + (1-\alpha)\lvert b \rvert for e1∼U[−1,1]e_1 \sim U[-1, 1].

Lagrangian of Eq. 7. Since CVaRα(Ci)=(2−α)(θi+1)2\text{CVaR}_{\alpha}(C_i) = (2-\alpha)(\theta_i+1)^2, the Lagrangian is

θ1+θ2−∑i=12λi((2−α)(θi+1)2−1).\theta_1 + \theta_2 - \sum_{i=1}^{2} \lambda_i \big((2-\alpha)(\theta_i+1)^2 - 1\big).

The solution is θi=1/2−α−1\theta_i = 1/\sqrt{2-\alpha} - 1 and λi=1/(22−α)\lambda_i = 1/(2\sqrt{2-\alpha}). The Lagrangian treats the costs one at a time, so this solution does not depend on ρ\rho.

Eq. 10 with ρ=1\rho = 1. Since λ1C1+λ2C2=(λ1(θ1+1)2+λ2(θ2+1)2)(1+e1)\lambda_1 C_1 + \lambda_2 C_2 = \big(\lambda_1(\theta_1+1)^2 + \lambda_2(\theta_2+1)^2\big)(1 + e_1), its CVaR is (2−α)(λ1(θ1+1)2+λ2(θ2+1)2)(2-\alpha)\big(\lambda_1(\theta_1+1)^2 + \lambda_2(\theta_2+1)^2\big). Eq. 10 equals the Lagrangian of Eq. 7 and gives the same solution.

Eq. 10 with ρ=−1\rho = -1. Since λ1C1+λ2C2=λ1(θ1+1)2+λ2(θ2+1)2+(λ1(θ1+1)2−λ2(θ2+1)2)e1\lambda_1 C_1 + \lambda_2 C_2 = \lambda_1(\theta_1+1)^2 + \lambda_2(\theta_2+1)^2 + \big(\lambda_1(\theta_1+1)^2 - \lambda_2(\theta_2+1)^2\big)e_1, Eq. 10 is

θ1+θ2+λ1+λ2−[λ1(θ1+1)2+λ2(θ2+1)2+(1−α)∣λ1(θ1+1)2−λ2(θ2+1)2∣].\theta_1 + \theta_2 + \lambda_1 + \lambda_2 - \Big[\lambda_1(\theta_1+1)^2 + \lambda_2(\theta_2+1)^2 + (1-\alpha)\big\lvert \lambda_1(\theta_1+1)^2 - \lambda_2(\theta_2+1)^2 \big\rvert\Big].

Eq. 10 is concave in θ\theta. When (α/(2−α))2≤λ1/λ2≤((2−α)/α)2\big(\alpha/(2-\alpha)\big)^2 \le \lambda_1/\lambda_2 \le \big((2-\alpha)/\alpha\big)^2, the maximum is on the ridge λ1(θ1+1)2=λ2(θ2+1)2\lambda_1(\theta_1+1)^2 = \lambda_2(\theta_2+1)^2. The absolute value term is 00 on this ridge. Maximizing on the ridge gives the solution and the maximum value g(λ)g(\boldsymbol{\lambda}):

θ1=λ1+λ24λ1λ2−1,θ2=λ1+λ24λ1λ2−1,g(λ)=18(1λ1+1λ2)2+λ1+λ2−2.\theta_1 = \frac{\sqrt{\lambda_1} + \sqrt{\lambda_2}}{4\lambda_1\sqrt{\lambda_2}} - 1, \qquad \theta_2 = \frac{\sqrt{\lambda_1} + \sqrt{\lambda_2}}{4\sqrt{\lambda_1}\lambda_2} - 1, \qquad g(\boldsymbol{\lambda}) = \frac{1}{8}\left(\frac{1}{\sqrt{\lambda_1}} + \frac{1}{\sqrt{\lambda_2}}\right)^2 + \lambda_1 + \lambda_2 - 2.

With s=λ1+λ2s = \lambda_1 + \lambda_2, g(λ)≥1/s+s−2≥0g(\boldsymbol{\lambda}) \ge 1/s + s - 2 \ge 0. Equality holds only at λ∗=(1/2,1/2)\boldsymbol{\lambda}^{\ast} = (1/2, 1/2). The solution is θ∗=(0,0)\theta^{\ast} = (0, 0). Both mean costs equal the cost limit 11. The combined cost 12C1+12C2\frac{1}{2}C_1 + \frac{1}{2}C_2 equals 11 for every e1e_1. The mean return is 00, which is higher than that of Eq. 7.

At ρ=−1\rho = -1, the two costs cannot take their worst values at the same time. Eq. 10 uses the joint distribution of the costs and gives a less conservative solution.

Q2. Derivation of the multiplier update in Eq. (18).

We give the derivation of Eq. 18 below.

While writing this derivation, we noticed that the joint risk in the submitted manuscript was missing the factor ∣λ∣1/∣λˉ∣1\\|\boldsymbol{\lambda}\\|_1/\\|\bar{\boldsymbol{\lambda}}\\|_1 in Eq. 13 and 17 as well as the factor 1/∣λˉt∣11 / \\|\bar{\boldsymbol{\lambda}}_t\\|_1 in Eq. 18. We have corrected these three equations in the revised manuscript. The submitted code is already implemented with the corrected equations, so all reported results remain unchanged.

The dual update takes the gradient of the actor loss without the factor 1/(1+∣λ∣1)1/(1+\\|\boldsymbol{\lambda}\\|_1) since it does not change the policy that minimizes Eq. 13 [Stooke]. We do not backpropagate through the joint risk, and with λT1=∣λ∣1\boldsymbol{\lambda}^{T}\mathbf{1} = \\|\boldsymbol{\lambda}\\|_1 for λ⪰0\boldsymbol{\lambda} \succeq \mathbf{0}, projected gradient ascent gives

\boldsymbol{\lambda}_{t+1} &= \left[ \boldsymbol{\lambda}_t + \eta_\lambda \nabla_{\boldsymbol{\lambda}} \left( \left(1 + \\|\boldsymbol{\lambda}\\|_1\right) \mathcal{L}(\pi; \boldsymbol{\lambda}) \right) \right]_+ \\\\ &= \left[ \boldsymbol{\lambda}_t + \eta_\lambda \nabla_{\boldsymbol{\lambda}} \left( -\mu^{\pi}_{R} + \boldsymbol{\lambda}^{T}\left(\boldsymbol{\mu}^{\pi}_{C} - \mathbf{d} + \frac{\kappa_{\alpha} \sqrt{\bar{\boldsymbol{\lambda}}_t^{T} \Sigma \bar{\boldsymbol{\lambda}}_t}}{\\|\bar{\boldsymbol{\lambda}}_t\\|_1}\mathbf{1}\right) \right) \right]_+ \\\\ &= \left[ \boldsymbol{\lambda}_t + \eta_\lambda \left( \boldsymbol{\mu}^{\pi}_{C} - \mathbf{d} + \frac{\kappa_{\alpha} \sqrt{\bar{\boldsymbol{\lambda}}_t^{T} \Sigma \bar{\boldsymbol{\lambda}}_t}}{\\|\bar{\boldsymbol{\lambda}}_t\\|_1}\mathbf{1} \right) \right]_+. \end{aligned}$$ Applying the same derivation to the normalized joint return $\bar{\mathbf{Z}}^\pi$, and estimating $\bar{\boldsymbol{\mu}}^{\pi}_{C}$ and $\bar{\Sigma}$ by $\widehat{\boldsymbol{\mu}}^{\pi}_C$ and $\widehat{\Sigma}$ from the joint return samples in the sliding window $W$, gives Eq. 18: $$\begin{aligned} \boldsymbol{\lambda}_{t+1} &= \left[ \boldsymbol{\lambda}_t + \eta_\lambda \left( \bar{\boldsymbol{\mu}}^{\pi}_{C} - \mathbf{1} + \frac{\kappa_{\alpha} \sqrt{\bar{\boldsymbol{\lambda}}_t^{T} \bar{\Sigma} \bar{\boldsymbol{\lambda}}_t}}{\\|\bar{\boldsymbol{\lambda}}_t\\|_1}\mathbf{1} \right) \right]_+ \\\\ &\approx \left[ \boldsymbol{\lambda}_t + \eta_\lambda \left( \widehat{\boldsymbol{\mu}}^{\pi}_C - \mathbf{1} + \frac{\kappa_{\alpha} \sqrt{\bar{\boldsymbol{\lambda}}_t^{T} \widehat{\Sigma} \bar{\boldsymbol{\lambda}}_t}}{\\|\bar{\boldsymbol{\lambda}}_t\\|_1}\mathbf{1} \right) \right]_+. \end{aligned}$$ We added the derivations of Eq. 13 and 18 to Appendix F.1 and F.2, respectively. ### **References** [Stooke] Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by PID Lagrangian methods. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9133–9143. PMLR
Reviewer_t4NyOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper considers constrained risk-sensitive RL under multiple risk constraints. The motivation of the study is that existing work assesses each cost independently using marginal distributions. To address this, this paper proposes Joint Risk Constrained Policy Optimization (JRCPO), which preserves expected-cost constraints while replacing disjoint marginal risk constraints into a joint risk constraint over the reward-cost distribution. The paper further proposes an actor-critic algorithm that evaluates this joint risk tractably under a Gaussian assumption.

Critical strengths
  1. The paper is clearly written, the problem is clearly motivated.
  2. The use of Gaussian approximation method to approximate the return to reduce the computational complexity is helpful.
Critical weaknesses
  1. The theoretical contribution of the paper is limited. The Gaussian approximation is borrowed from (Srikant, 2025). The paper does not establish how this approximation impacts the learned policy.
  2. The conversion from individual uncertainty set associated with individual risk constraint to the joint uncertainty set in the joint distribution is not clearly established.
Questions for authors
  1. How is the joint uncertainty set in the first paragraph of Section 3.3 established?
  2. What is the impact of the Gaussian approximation on the learned policy?
4 comments

We greatly appreciate the reviewer's feedback.

W1. The theoretical contribution of the paper is limited. The Gaussian approximation is borrowed from (Srikant, 2025). The paper does not establish how this approximation impacts the learned policy.

We added two analyses of the Gaussian approximation to the revised manuscript: one for MDPs where it holds (Appendix D.1) and one for MDPs where it does not (Appendix D.2).

MDPs where the Gaussian approximation holds

Theorem 2 uses the Markov chain CLT of Srikant (2025) under two conditions, ergodicity and γ↑1\gamma \uparrow 1. In an ergodic MDP, every stationary policy induces a Markov chain that satisfies this CLT, so the partial sums of rewards and costs converge to a Gaussian distribution. The discounted return is a weighted sum of these partial sums, and it also converges to a Gaussian distribution as γ↑1\gamma \uparrow 1 by the functional CLT for Markov chains (Meyn & Tweedie, 1993). Thus, whether the approximation holds depends on the MDP. We added this derivation to Appendix D.1.

MDPs where the Gaussian approximation does not hold

In these MDPs, the Gaussian critic still estimates the mean and covariance of the joint return. The Gaussian negative log-likelihood is minimized at the true mean and covariance, even when the return is not Gaussian. The joint risk computed from them is a second-order approximation of the risk. We added this analysis to Appendix D.2 and discuss its impact on the learned policy in our response to Q2.

MO-LunarLander is non-ergodic, so the Gaussian approximation may not hold there. JRCPO still achieved strong empirical results on this task.

Q2. What is the impact of the Gaussian approximation on the learned policy?

The Gaussian approximation affects the learned policy through the joint risk. We describe its effect on the policy update, its approximation error, and its effect on the critic.

Policy update

Under the Gaussian approximation, the joint risk is καλˉTΣλˉ\kappa_{\alpha}\sqrt{\bar{\boldsymbol{\lambda}}^{T}\Sigma\bar{\boldsymbol{\lambda}}} (Eq. 12), the standard deviation of the projected joint return scaled by κα\kappa_{\alpha}. Its gradient has the same direction as the gradient of the variance λˉTΣλˉ\bar{\boldsymbol{\lambda}}^{T}\Sigma\bar{\boldsymbol{\lambda}}. Thus, the actor loss in Eq. 17 updates the policy in the same direction as policy gradients for variance-related risk criteria [Zhang]. The learned policy reduces the spread of the joint return along the multiplier direction.

Approximation error

For any distribution of Z\mathbf{Z} with mean μ\boldsymbol{\mu} and covariance Σ\Sigma, CVaR has the following sharp upper bound [Bernard]:

CVaRα(λTZ)≤λTμ+1−ααλTΣλ.\text{CVaR}_{\alpha}(\boldsymbol{\lambda}^{T}\mathbf{Z}) \le \boldsymbol{\lambda}^{T}\boldsymbol{\mu} + \sqrt{\frac{1-\alpha}{\alpha}}\sqrt{\boldsymbol{\lambda}^{T}\Sigma\boldsymbol{\lambda}}.

The Gaussian approximation uses κα\kappa_{\alpha} in place of (1−α)/α\sqrt{(1-\alpha)/\alpha}, so it underestimates CVaRα(λˉTZ)\text{CVaR}_{\alpha}(\bar{\boldsymbol{\lambda}}^{T}\mathbf{Z}) by at most ((1−α)/α−κα)λˉTΣλˉ\big(\sqrt{(1-\alpha)/\alpha}-\kappa_{\alpha}\big)\sqrt{\bar{\boldsymbol{\lambda}}^{T}\Sigma\bar{\boldsymbol{\lambda}}}. At α=0.1\alpha=0.1, this is 1.2451.245 times the standard deviation of the projected joint return. The gap becomes smaller as the policy reduces this standard deviation. The same holds for the Lagrange multipliers, because the dual update in Eq. 18 uses the same joint risk. We added this analysis to Appendix D.2.

Critic

The joint risk also keeps the critic stable. The Gaussian critic is trained by minimizing the Gaussian negative log-likelihood, which is equivalent to minimizing the KL divergence from the Bellman target. The distributional Bellman operator is not a contraction in the KL divergence [Bellemare]. Without the joint risk, the actor prefers actions with an overestimated reward and underestimated costs, and the Bellman residual of the critic can keep growing. The joint risk grows with this residual along the multiplier direction, so it penalizes these actions (Appendix G).

AuthorsOpenReview ↗

Rebuttal Cont.

W2, Q1. Joint uncertainty set in Section 3.3.

The uncertainty set extends the one-dimensional case to the nn-dimensional setting.

Consider the coherent duality of the CVaRα\text{CVaR}_ {\alpha} of a cost CC whose associated probability measure is P\mathbb{P}, CVaRα(C)=sup⁡Q∈TEQ[C], T=Q≪P∣dQdP≤1α.\text{CVaR}_ {\alpha}(C) = \sup_ {\mathbb{Q} \in \mathcal{T}} \mathbb{E}_ {\mathbb{Q}}[C],~ \mathcal{T} = \\{ \mathbb{Q} \ll \mathbb{P} \mid \frac{d \mathbb{Q}}{d\mathbb{P}} \le \frac{1}{\alpha} \\}. One may interpret it as a worst-case weighted average of CC, whose maximal weight is constrained by 1α\frac{1}{\alpha}. Note that the weight identifies the risk measure CVaRα\text{CVaR}_ {\alpha}.

Similarly, we can construct a set on a cost random vector C=(C1,…,Cn)\mathbf{C}= (C_ 1, \dots, C_ n) and its associated probability measure P′\mathbb{P}' as T(C)=Q≪P′∣dQdP′≤1α.\mathcal{T}(\mathbf{C}) = \\{\mathbb{Q} \ll \mathbb{P}' \mid \frac{d \mathbb{Q} }{ d \mathbb{P}'} \le \frac{1}{\alpha} \\}. This is also a set of weight functions.

Further, we can compare the inclusion relation of the uncertainty sets.

Let proji\text{proj}_ {i} be the projection operator. The marginal uncertainty set is Ti=Qi≪Pi∣dQidPi≤1α\mathcal{T}_ {i} = \\{ \mathbb{Q}_ {i} \ll \mathbb{P}_ {i} \mid \frac{d\mathbb{Q}_ {i}}{d \mathbb{P}_ {i}} \le \frac{1}{\alpha} \\}, where Pi=P′∘proji−1\mathbb{P}_ {i} = \mathbb{P}' \circ \text{proj}^{-1}_ {i}. We can observe T(C)⊆⋂i=1nQ≪P′∣Q∘proji−1∈Ti.\mathcal{T}(\mathbf{C}) \subseteq \bigcap_ {i=1}^{n} \\{ \mathbb{Q} \ll \mathbb{P}' \mid \mathbb{Q} \circ \text{proj}^{-1}_ {i}\in \mathcal{T}_ i \\}.

The inclusion is strict whenever there exists a measurable set AA with P′(A)∈(0,α)\mathbb{P}'(A) \in (0, \alpha) satisfying P′(A∣Ci)≤P′(A)α\mathbb{P}'(A \mid C_ i) \le \frac{\mathbb{P}'(A)}{\alpha} P′\mathbb{P}'-a.s. for all i∈1,…,ni \in \\{1, \dots, n\\}.

Our construction is a special instance of zonoid trimming [Cascos & Molchanov, 2007]. We added this to Appendix E.

References

[Zhang] Zhang, Shangtong, Bo Liu, and Shimon Whiteson. "Mean-variance policy iteration for risk-averse reinforcement learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021.

[Bellemare] Bellemare, Marc G., Will Dabney, and Rémi Munos. "A distributional perspective on reinforcement learning." International conference on machine learning. Pmlr, 2017.

[Cascos] Cascos, Ignacio, and Ilya Molchanov. "Multivariate risks and depth-trimmed regions." Finance and Stochastics 11.3 (2007): 373-397.

Thanks for the clarifications, which are helpful. I raise my rating to 3.

Thank you for your response and for considering our clarifications. We sincerely appreciate your time and valuable feedback, and we remain happy to address any further questions or concerns.

Reviewer_fDayOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors study constrained risk-sensitive RL with multiple costs. They argue that existing methods constrain the risk of each cost separately through its marginal distribution, which implicitly hedges against all costs reaching their worst outcomes at the same time and therefore leads to overly conservative policies. They propose JRCPO, which keeps the expected cost constraints but replaces the separate risk constraints with a single joint risk. This joint risk is evaluated on the reward and costs combined along the direction of the Lagrange multipliers, using the dual representation of the conditional value at risk. They approximate the joint return by a Gaussian distribution, so that the risk has a closed form, and they update the multipliers from a sliding window of empirical episode returns. Experiments on various multi-cost control tasks compare JRCPO with expected-cost and marginal risk-sensitive baselines.

Critical strengths
  • The motivation is clear and convincing. Constraining each cost through its own marginal risk implicitly protects against all costs being at their worst simultaneously, and the authors illustrate well with Figure 1 and the legged robot example why this can be unnecessarily conservative when costs are negatively correlated.
  • The experiments also confirm that the marginal risk-sensitive baseline SRCPO tends to leave the cost limits unused and to sacrifice reward, which supports this motivation.
  • The empirical evaluation is broad and the gains are consistent. The authors compare against various baselines on various multi-cost tasks. Across these tasks, JRCPO satisfies the mean cost limits and achieves higher returns than SRCPO, and Figure 3 shows clearly how its episodes concentrate closer to the cost limits with less spread along the multiplier direction. Although the learning curves in Figure 5 show occasional violations of the cost limits, the reward and cost trade-off in Figure 4 indicates that JRCPO consistently reaches a better operating point than the baselines.
Critical weaknesses
  1. My main concern with this paper is that “the Gaussian assumption” is strong, and it makes the theoretical contribution much narrower than the title and the introduction suggest. The closed form in Theorem 3 holds exactly only when the joint return is Gaussian, and in that case it is a standard result, so the joint risk reduces to the mean plus a multiple of the standard deviation along the multiplier direction. The justification for the assumption, Theorem 2, is attributed to Srikant (2025), but I could not find in that work any statement that the return distribution can be approximated by a Gaussian and I believe the cited result appears to have been misinterpreted. If the return distribution were Gaussian in general, the entire field of distributional RL would simply reduce to estimating the mean and variance of the return. As a result, the title and the abstract overstate the generality of the method. I would be willing to raise my score depending on the authors' explanation of this point.

  2. The paper is not yet sufficiently polished.

  • Section 4.3 lacks a clear structure, and it is hard to follow how the sliding window is built and used in the multiplier update.
  • The entropy bonus mentioned in Section 4.2 does not appear in the actor loss of Eq. 17. In Section 5.3, the dispersion is discussed before it is defined in Eq. 19, and it is not stated whether a smaller dispersion is desirable or not.

Minor issues

  • Line 017, 044, 068, 177: "physically impossible" →\rightarrow "infeasible" or "not realizable" would describe these events more precisely.
  • Eq. 10: the underbraces repeat definitions already given in the text and can be removed.
  • Tables 1 to 3: the value after the slash is not defined in the captions.
Questions for authors
  1. Could the authors point to the specific statement in Srikant (2025) that supports Theorem 2? Have the authors considered treating Eq. 12 as a second-order approximation of the risk, based only on the mean and covariance of the joint return, instead of relying on a Gaussian assumption?
  2. How is the sliding window constructed and updated, and how is the entropy bonus incorporated into the actor loss in Eq. 17?
4 comments

We greatly appreciate the reviewer's detailed and careful feedback.

W1, Q1. The Gaussian assumption is too strong.

Section 3 and 4 of Srikant address the Central Limit Theorem (CLT) for Markov chains (Theorems 2 and 3), and building on these results, under specific conditions (γ↑1\gamma \uparrow 1 and ergodicity), an MDP admits a proper Gaussian approximation. Because there is a slight difference in the exact assumptions, we supplemented these details as follows. Our reason for citing Srikant was its general formulation of the Markov chain CLT and its direct connection to RL (TD-learning).

Specifically, once a policy is fixed, the associated rewards and costs are realized as a Markov chain. For the vector rt\mathbf{r}_t of the reward and costs at step tt with stationary mean rˉ\bar{\mathbf{r}}, the Markov chain CLT of Srikant (2025) gives 1n∑t=0n−1(rt−rˉ)→dN(0,Σ∞)as n→∞.\frac{1}{\sqrt{n}}\sum_{t=0}^{n-1}(\mathbf{r}_t-\bar{\mathbf{r}}) \xrightarrow{d} \mathcal{N}(\mathbf{0},\boldsymbol{\Sigma}_{\infty}) \quad \text{as } n\to\infty.

By summation by parts, the discounted return is a weighted sum of these partial sums,

ZRπ−rˉ1−γ=(1−γ)∑n=1∞γn−1∑t=0n−1(rt−rˉ),\mathbf{Z}^{\pi}_{R}-\frac{\bar{\mathbf{r}}}{1-\gamma} = (1-\gamma)\sum_{n=1}^{\infty}\gamma^{n-1}\sum_{t=0}^{n-1}(\mathbf{r}_t-\bar{\mathbf{r}}),

and its weights are concentrated on horizons of order (1−γ)−1(1-\gamma)^{-1}, which grow as γ↑1\gamma \uparrow 1. Applying the functional form of this CLT to each projection of rt\mathbf{r}_t and combining the projections by the Cramér–Wold Theorem [Cramér] gives

1−γ2(ZRπ−rˉ1−γ)→dN(0,Σ∞)as γ↑1.\sqrt{1-\gamma^{2}}\left(\mathbf{Z}^{\pi}_{R}-\frac{\bar{\mathbf{r}}}{1-\gamma}\right) \xrightarrow{d} \mathcal{N}(\mathbf{0},\boldsymbol{\Sigma}_{\infty}) \quad \text{as } \gamma\uparrow 1.

That is, the value function (return) converges in distribution to a normal distribution as γ↑1\gamma \uparrow 1 (to satisfy this condition, we used γ=0.999\gamma=0.999 in our experiments). We added this derivation to Appendix D.1. Please note that we do not claim that a Gaussian approximation holds across all environments. As already stated in our paper, conditions such as ergodicity are required.

Regarding the second-order approximation suggested by the reviewer, the mean and covariance of the joint return already determine an upper bound of the risk. For any distribution of Z\mathbf{Z} with mean μ\boldsymbol{\mu} and covariance Σ\Sigma, the projection λTZ\boldsymbol{\lambda}^{T}\mathbf{Z} has mean λTμ\boldsymbol{\lambda}^{T}\boldsymbol{\mu} and variance λTΣλ\boldsymbol{\lambda}^{T}\Sigma\boldsymbol{\lambda}, and the sharp upper bound of CVaR given only this mean and variance [Bernard] gives

CVaRα(λTZ)≤λTμ+1−ααλTΣλ.\text{CVaR}_{\alpha}(\boldsymbol{\lambda}^{T}\mathbf{Z}) \le \boldsymbol{\lambda}^{T}\boldsymbol{\mu} + \sqrt{\frac{1-\alpha}{\alpha}}\sqrt{\boldsymbol{\lambda}^{T}\Sigma\boldsymbol{\lambda}}.

This bound has the same form as Eq. 12 with κα\kappa_{\alpha} replaced by (1−α)/α\sqrt{(1-\alpha)/\alpha} (which is 33 and κα=1.755\kappa_{\alpha}=1.755 at α=0.1\alpha=0.1). Therefore, under the second-order approximation, the actor loss and the dual update keep the same form, and the joint risk remains proportional to λˉTΣλˉ\sqrt{\bar{\boldsymbol{\lambda}}^{T}\Sigma\bar{\boldsymbol{\lambda}}}. However, this bound is attained by a two-point distribution and is overly conservative as a model of the return. Bernard et al. [Bernard] tighten this bound by restricting it to the distributions close to a reference distribution. In this view, the Gaussian distribution is our reference distribution, supported by the Markov chain CLT under the conditions above. When these conditions do not hold, the Gaussian approximation underestimates CVaRα(λˉTZ)\text{CVaR}_{\alpha}(\bar{\boldsymbol{\lambda}}^{T}\mathbf{Z}) by at most ((1−α)/α−κα)λˉTΣλˉ\big(\sqrt{(1-\alpha)/\alpha}-\kappa_{\alpha}\big)\sqrt{\bar{\boldsymbol{\lambda}}^{T}\Sigma\bar{\boldsymbol{\lambda}}}, and this gap shrinks with the standard deviation of the projected joint return, which the joint risk in the actor loss penalizes.

Moreover, our method achieved strong empirical results in non-ergodic environments like (MO-)LunarLander, demonstrating that the Gaussian approximation works well in practice. As the reviewer pointed out, this is because the formulation effectively serves as a second-order approximation, and we added this discussion to Appendix D.2.

AuthorsOpenReview ↗

Rebuttal Cont.

W2, Q2. The sliding window is not clearly described, the entropy bonus is missing in Eq. 17, and the dispersion is not clearly discussed.

We revised Section 4.3 (sliding window), Eq. 17 (entropy bonus), and Section 5.3 (dispersion) as follows.

Sliding window

We revised Section 4.3 and Algorithm 1 to describe how the sliding window WW is constructed and updated, and stated the window size ∣W∣|W| and the update interval uu in Table 6. We also added the derivation of Eq. 18 to Appendix F.2.

For each completed episode, we add the discounted, normalized joint return sample Zsampleπ\mathbf{Z}^{\pi}_{\text{sample}} to the sliding window WW and keep the most recent ∣W∣|W| samples. Once WW is full, we update the Lagrange multipliers λ\boldsymbol{\lambda} every uu completed episodes by Eq. 18, using μ^Cπ\widehat{\boldsymbol{\mu}}^{\pi}_C and Σ^\widehat{\Sigma} computed from the joint return samples in WW. We set ∣W∣=100|W|=100 and u=10u=10 for all tasks.

Entropy bonus

The entropy bonus is only included in the reward return, as in WCSAC. Eq. 17 now contains it as the term βsRlog⁡π(a∣s)\frac{\beta}{s_R}\log\pi(a\mid s), where β\beta is the entropy coefficient. We had left this term out of Eq. 17 for simplicity, and added it following the reviewer's feedback.

L(π;λ)=11+∣λ∣1(−μˉR,ψ+βsRlog⁡π(a∣s)+⋯ )\mathcal{L}(\pi; \boldsymbol{\lambda})=\frac{1}{1 + \\|\boldsymbol{\lambda}\\|_1}\left(-\bar{\mu}_{R, \psi} + \frac{\beta}{s_R}\log\pi(a\mid s) + \cdots\right)

Dispersion

As the reviewer pointed out, Section 5.3 discussed the dispersion before defining it. We moved the definition of dispersion (Eq. 19) before its first use and stated that a smaller dispersion is desirable. At the same mean, a smaller dispersion gives a smaller CVaR, and adverse episodes stay closer to the mean.

Minor. physically impossible, underbraces in Eq. 10, and the value after the slash in Tables 1 to 3.

We replaced "physically impossible" with "unrealizable" at lines (e.g. 017, 044, etc.) and inside Figure 1. Following the reviewer's feedback, we removed the underbraces in Eq. 10. The value after slash for Table 1 to 3 is the CVaR$_{0.1}$ over evaluation episodes, which was described only in Appendix A.3. We stated it in Section 5.1 and in the captions of Table 1 (Table 2 and 3 follow Table 1).

References

[Cramér] Harald Cram´er and Herman Wold. Some theorems on distribution functions. Journal of the London Mathematical Society, 1(4):290–294, 1936

[Bernard] Carole Bernard, Silvana M. Pesenti, and Steven Vanduffel. Robust distortion risk measures. Mathematical Finance, 34(3):774–818, 2024. doi: 10.1111/mafi.12414

Thank you for your detailed response and for adding the derivation to Appendix D.1. However, my concern about the Gaussian assumption remains, because the result of Srikant (2025) and the return distribution modeled in your paper describe different kinds of uncertainty.

The results in Srikant (2025) describe the uncertainty of an estimate. Its CLTs show that the estimation error is asymptotically Gaussian and vanishes as the sample size increases. In contrast, distributional RL captures the intrinsic (aleatoric) randomness of the return itself from each state–action pair. This randomness does not vanish with more data, and its shape reflects which outcomes the agent may reach, such as landing or crashing, or violating a constraint or not. And such outcomes naturally produce a mixture, which is far from Gaussian. Therefore, a central limit theorem for an estimation error does not imply that the return distribution is Gaussian.

I appreciate the discussion of the second-order approximation. If your method is best understood as relying only on the mean and covariance of the joint return, I would suggest stating this explicitly in the main text, and making clear in the title and abstract that the method relies on this approximation rather than on the full return distribution. At this point, while I appreciate the core idea and methodology, I am convinced that the result of Srikant (2025) does not justify the Gaussian assumption on the return distribution in distributional RL, so I will keep my score.

We thank the reviewer for the follow-up comment.

About CLT

We respectfully disagree. The Central Limit Theorem characterizes the asymptotic distribution of the sum or average of random variables, NOT the point estimation of an underlying probability, which is the domain of the Law of Large Numbers.

Consider an infinite coin-tossing MDP whose immediate reward is −1-1 or 11 with uniform probability 12\frac{1}{2}. By the standard CLT, the normalized cumulative sum of rewards over TT steps, 1T∑t=1TRt\frac{1}{\sqrt{T}}\sum_{t=1}^T R_t, tends in distribution (in the weak sense) to the standard Gaussian N(0,1)\mathcal{N}(0, 1) as T→∞T \to \infty. This is consistent with the Markov chain CLT.

Contrary to the reviewer's comment about vanishing estimation error, no matter how many paths we sample, each rollout yields only a single realized path, and we cannot predict the exact sequence of immediate rewards (the realized return). Even as the return distribution approaches N(0,1)\mathcal{N}(0, 1), this path-wise variation remains as non-vanishing aleatoric uncertainty, corresponding directly to UnU_n in Theorem 2 of Srikant (2025), which converges to a normal distribution via dW(Un,Σ∞1/2Z)≤C(1−β)2n−β/2d_{\mathcal{W}}(U_n, \Sigma^{1/2}_{\infty} Z) \le \frac{C}{(1-\beta)^{2}} n^{-\beta/2}, just as in other distributional RL methods (e.g., IQN).

In contrast, the sample mean of rewards converges to 00 via the Law of Large Numbers; i.e., epistemic uncertainty (from a lack of experience) vanishes.

The 'landing or crashing' scenario cited by the reviewer is an absorbing MDP, which violates the ergodic mixing conditions required for the Markov CLT to hold.

Additionally, Srikant's (2025) Gaussian estimation error result for TD learning (Theorem 4) has nothing to do with our justification for approximating the return distribution.

About Second-order Approximation

On line 252 of the revised manuscript, we explicitly stated: "When the conditions of Theorem 2 do not hold, Theorem 3 remains a second-order approximation of the risk."

Reviewer_px35OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present a Joint Risk Constrained Policy Optimization (JRCPO) algorithm that minimizes the multivariate conditional value-at-risk (CvaR) that can be effective in real-world situations when RL needs to be applied under various cost constraints such as robotics and self-driving. Instead of imposing separate marginal risk constraints—which can combine adverse outcomes that cannot occur together—it evaluates CVaR of a scalarized reward-cost direction induced by Lagrange multipliers. The paper is well written and easy to follow. The authors provide empirical evaluation of their work.

Critical strengths
  • The paper identifies a meaningful issue with separate marginal risk constraints.
  • The proposed joint critic uses covariance information and yields a tractable closed-form Gaussian CVaR objective along the current reward-cost tradeoff direction.
  • The proposed method preserves mean-cost constraints while adding a risk-sensitive policy objective.
  • Empirical evaluation provides insights on expected-cost and marginal-risk baselines.
  • The ablations test components of the joint-risk formulation and reward normalization.
  • The paper reports limitations of Gaussian modeling and reward-scale sensitivity.
Critical weaknesses

None.

Questions for authors
  • Since there are numerical failures in some ablations, how stable are the results across seeds and reward-scale choices?
5 comments

We greatly appreciate the reviewer's positive feedback.

Q1. Since there are numerical failures in some ablations, how stable are the results across seeds and reward-scale choices?

The numerical failures in the ablation studies show that the joint risk also keeps the critic stable. JRCPO itself had no numerical failure in any of its 60 training runs across seeds and reward scale (Table 1 to 3), and its results are stable across seeds and not overly sensitive to the reward scale sRs_R.

Numerical failures in the ablation studies

All numerical failures in ablation studies occurred in the critic of two variants that remove all or part of the joint risk:

  1. Risk-neutral: 3 seeds on MO-LunarLander, between 3.6M and 4.8M of 5M training steps. This variant removes the joint risk from the actor loss and dual updates.
  2. Diagonal: 1 seed on HalfCheetahVelocity, at 9.5M of 10M training steps. This variant sets the off-diagonal entries of Σˉ\bar{\Sigma} to 00.

The joint risk keeps the critic stable through the critic covariance Σˉψ\bar{\Sigma}_\psi, which contains the mean Bellman residual ε\boldsymbol{\varepsilon}. Applying Eq. 14 to the critic gives

∇μˉψLNLL=Σˉψ−1ε,∇ΣˉψLNLL=0;⟹;Σˉψ=B+εεT.\nabla_{\bar{\boldsymbol{\mu}}_\psi}\mathcal{L}_{\mathrm{NLL}} = \bar{\Sigma}_\psi^{-1}\boldsymbol{\varepsilon}, \qquad \nabla_{\bar{\Sigma}_\psi}\mathcal{L}_{\mathrm{NLL}} = 0 \\;\Longrightarrow\\; \bar{\Sigma}_\psi = B + \boldsymbol{\varepsilon}\boldsymbol{\varepsilon}^{T}.

The mean gradient pulls μˉψ\bar{\boldsymbol{\mu}}_\psi toward b\boldsymbol{b}, but b\boldsymbol{b} is computed from μˉψ\bar{\boldsymbol{\mu}}_\psi at the next state, so fitting the mean propagates the residual to earlier states over the long effective horizon (Section 4.2 of [Fujimoto]). Through the covariance gradient, Σˉψ\bar{\Sigma}_\psi absorbs the remaining residual, which in turn decreases the mean gradient [Seitzer]. Computed from this covariance, the joint risk is proportional to λˉTΣˉψλˉ\sqrt{\bar{\boldsymbol{\lambda}}^{T}\bar{\Sigma}_\psi\bar{\boldsymbol{\lambda}}}, which grows with the residual along the multiplier direction,

λˉTΣˉψλˉ=λˉTBλˉ+(λˉTε)2≥∣λˉTε∣.\sqrt{\bar{\boldsymbol{\lambda}}^{T}\bar{\Sigma}_\psi\bar{\boldsymbol{\lambda}}} = \sqrt{\bar{\boldsymbol{\lambda}}^{T}B\bar{\boldsymbol{\lambda}} + (\bar{\boldsymbol{\lambda}}^{T}\boldsymbol{\varepsilon})^{2}} \geq \lvert\bar{\boldsymbol{\lambda}}^{T}\boldsymbol{\varepsilon}\rvert.

Since the actor loss in Eq. 17 contains λˉTμˉψ=λˉTb+λˉTε\bar{\boldsymbol{\lambda}}^{T}\bar{\boldsymbol{\mu}}_\psi = \bar{\boldsymbol{\lambda}}^{T}\boldsymbol{b} + \bar{\boldsymbol{\lambda}}^{T}\boldsymbol{\varepsilon}, the actor prefers actions with λˉTε<0\bar{\boldsymbol{\lambda}}^{T}\boldsymbol{\varepsilon} < 0, for which the joint risk is also larger. In Risk-neutral, the residual can keep growing until a numerical failure occurs. Diagonal keeps only the diagonal part of

(λˉTε)2=λˉTdiag⁡(εεT)λˉ⏟diagonal+λˉT(εεT−diag⁡(εεT))λˉ⏟off-diagonal(\bar{\boldsymbol{\lambda}}^{T}\boldsymbol{\varepsilon})^{2} = \underbrace{\bar{\boldsymbol{\lambda}}^{T}\operatorname{diag}(\boldsymbol{\varepsilon}\boldsymbol{\varepsilon}^{T})\bar{\boldsymbol{\lambda}}}_{\text{diagonal}} + \underbrace{\bar{\boldsymbol{\lambda}}^{T}\big(\boldsymbol{\varepsilon}\boldsymbol{\varepsilon}^{T} - \operatorname{diag}(\boldsymbol{\varepsilon}\boldsymbol{\varepsilon}^{T})\big)\bar{\boldsymbol{\lambda}}}_{\text{off-diagonal}}

and misses the off-diagonal part, which is positive for an overestimated reward and underestimated costs (ε<0\boldsymbol{\varepsilon} < \mathbf{0}). We added this analysis to Appendix G.

AuthorsOpenReview ↗

Rebuttal Cont.

Stability across seeds

The results of JRCPO are stable across seeds, showing variation comparable to that of the baselines. The table below reports the standard deviations across 5 seeds in Table 1 and 2, normalized by Eq. 15 and averaged across cost components. We compare JRCPO with PPO-Lag, SDAC, and SRCPO, the baselines that satisfy all mean-cost limits in at least 3 of the 6 tasks (for EV2Gym, we consider Δmax⁡=4\Delta_{\max}=4 case). JRCPO's standard deviations are on a similar scale to those of the baselines for both reward and costs.

|Return|Cost
Task|PPO-LagSDACSRCPOJRCPO|PPO-LagSDACSRCPOJRCPO
PointCircle1|2.703.740.7810.213|0.1232.910.1400.108
CarGoal2|10.89.0013.76.36|0.1480.09680.1090.162
HalfCheetahVelocity|0.01383.690.4230.0413|0.1081.790.1320.134
HopperVelocity|7.103.623.993.77|0.4250.2470.3360.637
MO-LunarLander|1.610.8900.3900.154|0.1620.1210.08530.0741
EV2Gym|0.004500.02520.02290.0346|0.03430.1000.04690.0545

Reward scale

No numerical failure occurred when changing the reward scale sRs_R, and JRCPO is not overly sensitive to it. The reward scale sRs_R is a hyperparameter that sets the reward-cost trade-off in Eq. 17, and we set it by the fixed rule in Appendix A.2 without tuning.

In Eq. 13, the joint risk is proportional to the standard deviation of the projected joint return. Therefore, we set sRs_R to the standard deviation of the reward return, computed from reward-only PPO for simplicity. Since these policies ignore the costs, their return distribution can differ from that of JRCPO policies, and a better sRs_R may exist for each task.

Table 3 compares three other choices of sRs_R on MO-LunarLander and HalfCheetahVelocity. With Half (sR/2s_R/2) and Double (2sR2 s_R), the violation ratio is below 50% on both tasks. Unscaled (sR=1s_R = 1) represents an extreme choice, since the rule gives sR=30.37s_R = 30.37 and 217.72217.72 on the two tasks, respectively. It violates the mean-cost limit, but no numerical failure occurs.

References

[Seitzer] Maximilian Seitzer, Arash Tavakoli, Dimitrije Antic, and Georg Martius. On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=aPOpXlnV1T.

[Fujimoto] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1587–1596. PMLR, 2018. URL https://proceedings.mlr.press/v80/fujimoto18a.html

Thank you again for your positive review. As the discussion period is closing soon, we wanted to check whether our response addressed your question regarding stability across seeds and reward scales. Please let us know if anything remains unclear.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all reviewers for their constructive feedback. We have revised our manuscript to incorporate their suggestions, and the updated version has been uploaded. We hope the reviewers find that these updates satisfactorily address all remaining concerns. A detailed summary of the revisions is provided below:


Justification of the Gaussian approximation and its impact on learning (px35, fDay, t4Ny)

  • We added the derivation of the Gaussian approximation from the Markov chain CLT (Appendix D.1).
  • We added the second-order approximation of the risk, using only the mean and covariance of the joint return (Appendix D.2).
  • We stated that the closed-form Gaussian CVaR remains a second-order approximation when the Gaussian approximation does not hold (Section 4.1).
  • We added an explanation of how the joint risk keeps the critic stable (Appendix G).

Joint uncertainty set and conservatism (t4Ny, SWyK)

  • We added the construction of the uncertainty set T\mathcal{T} from the coherent duality and its inclusion relation with the marginal uncertainty sets (Appendix E).
  • We added a toy CMDP experiment in which only the correlation between two costs changes (Appendix C).

Derivation of the multiplier update in Eq. 18 (SWyK)

  • We added the derivations of the actor loss and the dual update (Appendix F).

Presentation of the sliding window, the entropy bonus, and the dispersion (fDay)

  • We revised Section 4.3 to clarify how the sliding window is built and used.
  • We added the sliding window size ∣W∣|W| and the multiplier update interval uu to the implementation settings (Table 6, Appendix A.3).
  • We included the entropy bonus in the actor loss (Eq. 17).
  • We defined the dispersion (Eq. 19) before discussing it and stated that a smaller dispersion is desirable (Section 5.3).

Minor issues (fDay)

  • We replaced "physically impossible events" with "unrealizable events".
  • We removed the underbraces in Eq. 10.
  • We defined the value after the slash in the tables as CVaR$_{0.1}$ (Section 5.1).

As the discussion period is still ongoing, we welcome any additional feedback, questions, or responses regarding our submission. We will do our best to respond promptly and thoroughly. Once again, we sincerely thank all reviewers for their time and dedication.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.