- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a model-free off-policy actor–critic framework with stochastic policies for dynamic expectile and CVaR. It introduces a surrogate policy update that avoids transition perturbation, proves optimal value convergence under tabular softmax parameterization and exact policy evaluation, and provides critic learning convergence results. Experiments on different tasks evaluate sample efficiency and risk-averse behavior.
Critical strengths
-
Theoretical justification. The analysis establishes that the surrogate update can achieve optimal value convergence despite not being the true risk-sensitive policy gradient. This provides the foundation for simplifying policy optimization under the chosen dynamic-risk objective.
-
Practical validation. The algorithm design combines risk-sensitive value learning with off-policy actor updates. Experiments demonstrate risk-averse behavior and improved learning efficiency in several tasks, supporting the practical impact of this approach.
Critical weaknesses
-
Policy-convergence claims exceed the theoretical result. The authors establish optimal-value convergence but do not establish the behavior of the policy sequence when optimal stochastic policies are not unique. This distinction limits the support for the paper’s stronger convergence claims.
-
The baseline comparison is limited. Although the proposed algorithms outperform the included baselines in several settings, comparisons with additional standard risk-neutral methods, such as vanilla SAC and PPO, would help assess the robustness of the reported empirical advantages.
Questions for authors
-
When optimal policies are nonunique, what can be established about the behavior of the policy sequence? Does this have implications for the algorithm design?
-
Could the authors provide comparisons with additional risk-neutral RL baselines, such as vanilla SAC and PPO?
-
Could the authors clarify how expected return and CVaR at different evaluation levels change as the training risk level varies? For example, for each policy trained with , what are its expected return and CVaR at evaluation levels 0.2, 0.3, and 0.4?
5 comments
Thank you for the response and additional experiments. Two concerns remain.
First, the paper claims convergence to an optimal policy, whereas the response establishes that every limit point is optimal. When optimal policies are nonunique, this does not establish that the policy sequence itself converges. Please either provide the additional argument or revise the convergence claims accordingly.
Second, the heavily overlapping learning curves make the empirical advantages difficult to assess. To demonstrate the benefits of risk-sensitive learning, I would expect plots of expected return and CVaR against the training risk level, at multiple CVaR evaluation levels. If the intended claim is improved sample efficiency, the scope of that claim also needs clarification: SAC learns faster and achieves higher expected return and static CVaR on HalfCheetah. The current results therefore do not establish a consistent efficiency advantage over the added baseline.
These concerns leave me unconvinced that the current presentation sufficiently supports the paper’s theoretical and empirical claims.
We thank the reviewer for the additional feedback.
Claim. Thanks for this suggestion. We have revised the claim in introduction to "achieves convergence to the optimal value".
Experiment. We would like to clarify the motivation behind our experimental design and how the results should be interpreted. Following previous work (Luo et al., 2024; Mead et al., 2025), we adopt domains with identifiable risk-averse behaviors to examine whether the proposed algorithms indeed learn risk-averse policies. For instance, in Maze and Cliffwalk, risk-averse behavior corresponds to choosing a longer path, whereas in LunarLander and InvertedPendulum, it corresponds to avoiding regions with noisy rewards. While static CVaR can serve as a proxy for evaluating risk-sensitive performance, it does not necessarily characterize the underlying behavior of a policy. Nevertheless, we report it as a complementary evaluation metric.
In all four domains presented in the main text, only our proposed algorithms exhibit clear risk-averse behaviors (see Fig. 2(a), (b), (d), and (g)). Regarding the overlapping learning curves, we acknowledge that the curves for static CVaR at the 0.2 level in LunarLander appear to overlap. However, none of SAC, EPG or PPO learns to avoid the noisy reward region, as evidenced by their visitation rates of approximately 0.5 in Fig. 2(d). This observation highlights the importance of examining risk-averse behaviors in addition to static CVaR when evaluating risk-sensitive learning algorithms.
The purpose of including HalfCheetah in the appendix is to demonstrate the applicability of our algorithms to more complex domains with higher-dimensional continuous action spaces. However, since this environment was originally designed to evaluate robustness, identifiable risk-averse behaviors are difficult to establish, and we therefore report static CVaR as a reference metric. The superior performance of SAC over EPG in this domain also suggests that incorporating risk-sensitive objectives into the SAC framework could be particularly valuable, which we consider an interesting direction for future research.
In response to the reviewer's suggestion, we have additionally included learning curves of static CVaR at evaluation levels of 0.3 and 0.4 when our algorithms are trained at different risk levels in Fig 11 and 12. We would like to emphasize, however, that the risk level used for training under dynamic risk measures should not be directly identified with the risk level used for evaluation under static risk measures, as these are mathematically distinct objectives.
Reply to Reviewer HxHv
We thank the reviewer for the feedback. We address the main concerns below.
(W1,Q1) When optimal policies are non-unique. We confirm that the proof of Theorem 3.3 (convergence to the optimal value function) does not require the optimal policy to be unique. When multiple optimal actions exist, every limit point of the policy sequence πt is supported on the set of optimal actions, since we establish that πt(a∣s)→0 for all a∈Is+∪Is−. Consequently, every limit point is an optimal policy.
(W2, Q2) Risk-neutral baseline SAC and PPO. We have included SAC and PPO as additional baselines in our deep learning experiments. The results are presented in the updated Fig. 2(c)–(h), Fig. 6, and Fig. 10. In LunarLander and Inverted Pendulum, both SAC and PPO exhibit behavior similar to EPG in terms of visiting noisy regions (i.e., more often compared with Exp-AC and CVaR-AC). In HalfCheetah, SAC learns substantially faster than the other algorithms and achieves better tail performance after 1 million steps. This is noticeable in Figure 10.a where SAC quickly and steadily outperforms EPG in terms of the risk neutral objective and throughout the training. This suggests that SAC's superior performance may primarily stem from its more efficient learning mechanism rather than its ability to learn risk-averse behavior. Moreover, since the characteristics of a risk-averse policy in HalfCheetah are not well understood, it is difficult to assess whether SAC exhibits genuinely risk-averse behavior. In conclusion, Figure 10 does confirm that our risk averse adaptation of EPG does identify lower risk policies (in terms of CVaR) than its risk neutral variant. The observations in HalfCheetah further suggest that incorporating risk-sensitive objectives into the SAC framework might be quite valuable for some environments and is an interesting direction for future research.
(Q3) Performance under different risk level. We thank the reviewer for the question. The expected return and static CVaR at evaluation level 0.2 for policies trained under different risk levels are already reported in Fig. 5 of the original manuscript.
We would like to clarify that our optimization objective is dynamic risk, which is difficult to evaluate directly in practice. We therefore report static CVaR at level 0.2 as an empirical proxy, since it can be readily estimated from sampled returns. This evaluation protocol is also adopted by Rigter et al. (2023), who optimize dynamic CVaR but evaluate policy performance using a specific static CVaR. Importantly, dynamic and static CVaR are distinct risk measures, and their values are not directly comparable even at the same risk level α. Similarly, the risk level α in dynamic expectile does not directly correspond to the evaluation level in static CVaR, as they characterize different risk measures. Consequently, the training risk level α should not be interpreted as the evaluation level of static CVaR.