acceptodds
Under review as a conference paper at ICLR 2027

Actor-Critic Algorithm for Dynamic Expectile and CVaR

Abstract

Optimizing dynamic risk with stochastic policies is challenging in both policy updates and value learning. The former typically requires transition perturbation, while the latter may rely on model-based approaches. To address these challenges, we propose a surrogate policy gradient without transition perturbation under softmax policy parameterization. We further develop model-free value learning methods for dynamic expectile and conditional value-at-risk (CVaR) by leveraging elicitability. Finally, inspired by Expected SARSA and Expected Policy Gradient, the first model-free off-policy actor-critic algorithm with stochastic policies is constructed for dynamic expectile and CVaR. Empirical results in domains with verifiable risk-averse behavior show that our algorithm can learn risk-averse policy and consistently outperforms other existing methods.

Primary area:
reinforcement learning
Keywords:
Risk-sensitive reinforcement learning, dynamic risk, model-free off-policy actor-critic

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_HxHvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a model-free off-policy actor–critic framework with stochastic policies for dynamic expectile and CVaR. It introduces a surrogate policy update that avoids transition perturbation, proves optimal value convergence under tabular softmax parameterization and exact policy evaluation, and provides critic learning convergence results. Experiments on different tasks evaluate sample efficiency and risk-averse behavior.

Critical strengths
  1. Theoretical justification. The analysis establishes that the surrogate update can achieve optimal value convergence despite not being the true risk-sensitive policy gradient. This provides the foundation for simplifying policy optimization under the chosen dynamic-risk objective.

  2. Practical validation. The algorithm design combines risk-sensitive value learning with off-policy actor updates. Experiments demonstrate risk-averse behavior and improved learning efficiency in several tasks, supporting the practical impact of this approach.

Critical weaknesses
  1. Policy-convergence claims exceed the theoretical result. The authors establish optimal-value convergence but do not establish the behavior of the policy sequence when optimal stochastic policies are not unique. This distinction limits the support for the paper’s stronger convergence claims.

  2. The baseline comparison is limited. Although the proposed algorithms outperform the included baselines in several settings, comparisons with additional standard risk-neutral methods, such as vanilla SAC and PPO, would help assess the robustness of the reported empirical advantages.

Questions for authors
  1. When optimal policies are nonunique, what can be established about the behavior of the policy sequence? Does this have implications for the algorithm design?

  2. Could the authors provide comparisons with additional risk-neutral RL baselines, such as vanilla SAC and PPO?

  3. Could the authors clarify how expected return and CVaR at different evaluation levels change as the training risk level varies? For example, for each policy trained with α∈0.2,0.3,0.4\alpha\in\\{0.2,0.3,0.4\\}, what are its expected return and CVaR at evaluation levels 0.2, 0.3, and 0.4?

5 comments
AuthorsOpenReview ↗

Reply to Reviewer HxHv

We thank the reviewer for the feedback. We address the main concerns below.

(W1,Q1) When optimal policies are non-unique. We confirm that the proof of Theorem 3.3 (convergence to the optimal value function) does not require the optimal policy to be unique. When multiple optimal actions exist, every limit point of the policy sequence πt\pi_t is supported on the set of optimal actions, since we establish that πt(a∣s)→0\pi_t(a\mid s)\to 0 for all a∈Is+∪Is−a\in I_s^+\cup I_s^-. Consequently, every limit point is an optimal policy.

(W2, Q2) Risk-neutral baseline SAC and PPO. We have included SAC and PPO as additional baselines in our deep learning experiments. The results are presented in the updated Fig. 2(c)–(h), Fig. 6, and Fig. 10. In LunarLander and Inverted Pendulum, both SAC and PPO exhibit behavior similar to EPG in terms of visiting noisy regions (i.e., more often compared with Exp-AC and CVaR-AC). In HalfCheetah, SAC learns substantially faster than the other algorithms and achieves better tail performance after 1 million steps. This is noticeable in Figure 10.a where SAC quickly and steadily outperforms EPG in terms of the risk neutral objective and throughout the training. This suggests that SAC's superior performance may primarily stem from its more efficient learning mechanism rather than its ability to learn risk-averse behavior. Moreover, since the characteristics of a risk-averse policy in HalfCheetah are not well understood, it is difficult to assess whether SAC exhibits genuinely risk-averse behavior. In conclusion, Figure 10 does confirm that our risk averse adaptation of EPG does identify lower risk policies (in terms of CVaR) than its risk neutral variant. The observations in HalfCheetah further suggest that incorporating risk-sensitive objectives into the SAC framework might be quite valuable for some environments and is an interesting direction for future research.

(Q3) Performance under different risk level. We thank the reviewer for the question. The expected return and static CVaR at evaluation level 0.2 for policies trained under different risk levels are already reported in Fig. 5 of the original manuscript.

We would like to clarify that our optimization objective is dynamic risk, which is difficult to evaluate directly in practice. We therefore report static CVaR at level 0.2 as an empirical proxy, since it can be readily estimated from sampled returns. This evaluation protocol is also adopted by Rigter et al. (2023), who optimize dynamic CVaR but evaluate policy performance using a specific static CVaR. Importantly, dynamic and static CVaR are distinct risk measures, and their values are not directly comparable even at the same risk level α\alpha. Similarly, the risk level α\alpha in dynamic expectile does not directly correspond to the evaluation level in static CVaR, as they characterize different risk measures. Consequently, the training risk level α\alpha should not be interpreted as the evaluation level of static CVaR.

Thank you for the response and additional experiments. Two concerns remain.

First, the paper claims convergence to an optimal policy, whereas the response establishes that every limit point is optimal. When optimal policies are nonunique, this does not establish that the policy sequence itself converges. Please either provide the additional argument or revise the convergence claims accordingly.

Second, the heavily overlapping learning curves make the empirical advantages difficult to assess. To demonstrate the benefits of risk-sensitive learning, I would expect plots of expected return and CVaR against the training risk level, at multiple CVaR evaluation levels. If the intended claim is improved sample efficiency, the scope of that claim also needs clarification: SAC learns faster and achieves higher expected return and static CVaR on HalfCheetah. The current results therefore do not establish a consistent efficiency advantage over the added baseline.

These concerns leave me unconvinced that the current presentation sufficiently supports the paper’s theoretical and empirical claims.

We thank the reviewer for the additional feedback.

Claim. Thanks for this suggestion. We have revised the claim in introduction to "achieves convergence to the optimal value".

Experiment. We would like to clarify the motivation behind our experimental design and how the results should be interpreted. Following previous work (Luo et al., 2024; Mead et al., 2025), we adopt domains with identifiable risk-averse behaviors to examine whether the proposed algorithms indeed learn risk-averse policies. For instance, in Maze and Cliffwalk, risk-averse behavior corresponds to choosing a longer path, whereas in LunarLander and InvertedPendulum, it corresponds to avoiding regions with noisy rewards. While static CVaR can serve as a proxy for evaluating risk-sensitive performance, it does not necessarily characterize the underlying behavior of a policy. Nevertheless, we report it as a complementary evaluation metric.

In all four domains presented in the main text, only our proposed algorithms exhibit clear risk-averse behaviors (see Fig. 2(a), (b), (d), and (g)). Regarding the overlapping learning curves, we acknowledge that the curves for static CVaR at the 0.2 level in LunarLander appear to overlap. However, none of SAC, EPG or PPO learns to avoid the noisy reward region, as evidenced by their visitation rates of approximately 0.5 in Fig. 2(d). This observation highlights the importance of examining risk-averse behaviors in addition to static CVaR when evaluating risk-sensitive learning algorithms.

The purpose of including HalfCheetah in the appendix is to demonstrate the applicability of our algorithms to more complex domains with higher-dimensional continuous action spaces. However, since this environment was originally designed to evaluate robustness, identifiable risk-averse behaviors are difficult to establish, and we therefore report static CVaR as a reference metric. The superior performance of SAC over EPG in this domain also suggests that incorporating risk-sensitive objectives into the SAC framework could be particularly valuable, which we consider an interesting direction for future research.

In response to the reviewer's suggestion, we have additionally included learning curves of static CVaR at evaluation levels of 0.3 and 0.4 when our algorithms are trained at different risk levels in Fig 11 and 12. We would like to emphasize, however, that the risk level used for training under dynamic risk measures should not be directly identified with the risk level used for evaluation under static risk measures, as these are mathematically distinct objectives.

Reviewer_icajOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a model-free off-policy actor-critic with stochastic policies to optimize dynamic risk defined by expectile or conditional value-at-risk (CVaR). To avoid reweighting transition probabilities, it introduces a surrogate update that uses a risk-sensitive Q-function in an expression with the same form as the standard policy gradient. Assuming tabular softmax policies and an exact Q-function, this paper shows monotonic improvement and convergence to the optimal value for this update. For value learning, this paper uses expectile updates and updates that learn value-at-risk (VaR) and CVaR at different time scales. It analyzes their convergence under a fixed policy. In numerical experiments, the method learns risk-averse policies faster than Exp-PPO in tabular environments. With function approximation, the method also learns risk-averse policies within training budgets where the compared on-policy methods fail to learn reasonable policies.

Critical strengths
  • S1 (Theoretical guarantees for the surrogate policy update): For tabular softmax policies with an exact Q-function, this paper shows monotonic improvement and convergence to the optimal value for a surrogate update that avoids reweighting transition probabilities. The results also apply to coherent risk measures other than expectile and CVaR.

  • S2 (Effectiveness shown in numerical experiments): The method learns faster than Exp-PPO in tabular environments. With function approximation, the method also learns risk-averse policies within training budgets where the compared on-policy methods fail to learn reasonable policies.

Critical weaknesses
  • W1 (Insufficient evaluation of the practical benefits of the proposed update): This paper does not present comparisons with existing risk-averse off-policy methods in continuous-action environments or ablations that separate the effects of policy updates and experience replay. The numerical experiments support the effectiveness of the full method, but the limited comparisons make it difficult to assess the practical benefits of the proposed update over existing learning methods.

  • W2 (Gap between the theoretical guarantees and the proposed algorithm): The theoretical analysis is limited to the tabular setting and treats policy updates with an exact Q-function and value learning under a fixed policy separately. This paper does not show convergence of the full proposed algorithm that learns Q and the policy together, so the theoretical guarantee for the implemented method is partial.

Questions for authors
  • Q1: Could you provide comparison results with existing risk-averse off-policy methods in continuous-action environments, or results from using the proposed update on-policy? I would like to understand what practical benefits the proposed update offers over existing methods.

  • Q2: For the tabular case where Q and the policy are updated together, could you explain the additional conditions needed to combine the convergence results for policy updates and value learning? I would like to understand how far the theoretical results apply to the proposed learning procedure.

1 comment
AuthorsOpenReview ↗

Reply for Reviewer icaj

(W1,Q1) Ablation study. We thank the reviewer for the question. We would like to clarify that our proposed actor-critic framework is primarily designed and theoretically analyzed for discrete action spaces. Its extension to continuous action is presented as a practical heuristic rather than a theoretically justified component. Therefore, we believe that comparisons in discrete-action environments are more directly relevant to evaluating the proposed policy update.

This being said, we added additional on-policy training results (i.e., train Exp-AC and CVaR-AC without a reply buffer) in Fig. 8 and discussed in Appendix E.3.2 in the updated manuscript. Since we use EPG-style policy update, which does not employ importance sampling, the policy is only updated once per data collection in the on-policy setting. In contrast, Exp-PPO uses importance sampling to enable multiple policy updates using the same collected data. Consequently, on-policy learning exhibit slower learning than Exp-PPO in both domains, as well as their off-policy counterparts.

Nevertheless, a key advantage of the EPG-style policy update is that it does not require the sampled states to follow the current policy's visitation distribution, allowing policy learning to leverage data collected by behavior policies through experience replay. Therefore, off-policy learning is a more natural and sample-efficient choice for our proposed framework, and constitutes one of its key practical advantages.

(W2,Q2) Convergence of the actor-critic. We thank the reviewer for raising this concern. Our analysis follows the policy-iteration template standard for deriving actor-critic methods, e.g., the soft policy iteration in soft actor-critic paper (Haarnoja et al. 2018). We establish (i) convergence of the risk-sensitive critic for a fixed policy (Props. 3.4–3.5) and (ii) monotone improvement and global convergence of the surrogate actor update with an exact critic (Thms. 3.2–3.3).

For the simultaneous updates of QQ and the policy, existing convergence analyses of risk-neutral actor-critic methods typically require additional conditions, such as appropriate step-size schedules, sufficient exploration, and boundedness of the iterates (Konda & Tsitsiklis, 2003; Bhatnagar et al., 2009; Zhang et al., 2020). While classical analyses primarily establish convergence to stationary points, more recent studies have established global optimality for specific actor-critic algorithms under additional assumptions (Zhang et al., 2022). However, these results do not directly extend to our risk-sensitive setting, which involves nonlinear risk-sensitive Bellman operators and, in the case of CVaR, coupled quantile and CVaR updates. Establishing convergence of the fully coupled risk-sensitive actor-critic algorithm therefore requires further analysis. We leave this as an interesting direction for future research and will clarify the scope of our theoretical guarantees in the revised manuscript.

[Konda & Tsitsiklis, 2003]On Actor-Critic Algorithms. SIAM Journal on Control and Optimization, 42(4), 1143–1166, 2003.

[Bhatnagar et al., 2009] Natural Actor–Critic Algorithms. Automatica, 45(11), 2471–2482, 2009.

[Zhang et al., 2020] Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation. ICML, 2020.

[Zhang et al., 2022] Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatch. Journal of Machine Learning Research, 23(343), 1–91, 2022.

Reviewer_dzJkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides a model-free actor-critic framework for dynamic expectile and CVaR with stochastic policies. For tabular case, the paper provides monotinic improvement and asymptotic optimality for the proposed policy gradient approach. The authors next develop algorithm in Section 3.3 and numerical study demonstrate risk-sensitive behavior in small navigation tasks and modified control environment.

Critical strengths
  • The experiments are solid and sound to show the effectiveness of this framework
  • the presentation is clear for the audience to understand the setup.
Critical weaknesses
  1. Although this paper presents much theoretical foundation of the proposed algorithm, I cannot see the provided theory yields global convergence of the proposed actor critic algorithm. It requires exact policy evaluation at each iteration, independent tabular logits, and a small enough constant actor sep. The provided separate theorems cannot lead to the convergence of the proposed algorithm in Section 3.3
  2. The proof of Proposition 3.5 seems to be wrong: The central argument in Appendix C.2 is to introduce Eq. (18), which is a generalization of Eq. (11). However, Eq. (11) cannot be reformulated as desired.
  3. For the algorithm in Section 3.3, the authors approximate V(S') by the mean of ten sampled Q values, then place that random mean inside an asymmetric expectile update. Please analyze the bias of this heuristic.
Questions for authors

How the provided framework distinguishes itself from existing literature, such as:

  • On the theory of policy gradient methods: Optimality, approximation, and distribution shift
  • Soft robust MDPs and risk-sensitive MDPs: Equivalence, policy gradient, and sample complexity
  • Deep reinforcement learning for option pricing and hedging under dynamic expectile risk measures
  • Conditionally elicitable dynamic risk measures for deep reinforcement learning
  • Learning diverse risk preferences in population-based self-play
4 comments
AuthorsOpenReview ↗

Reply to Reviewer dzJk (Part 1)

We thank the reviewer for the feedback. We address the main concerns below.

(W1) Convergence of the actor-critic. We thank the reviewer for raising this concern. Our analysis follows the policy-iteration template standard for deriving actor-critic methods, e.g., the soft policy iteration in soft actor-critic paper (Haarnoja et al. 2018). We establish (i) convergence of the risk-sensitive critic for a fixed policy (Props. 3.4–3.5) and (ii) monotone improvement and global convergence of the surrogate actor update with an exact critic (Thms. 3.2–3.3).

Establishing convergence of the fully coupled actor-critic algorithm with replay buffers, target networks, and neural function approximation remains challenging even in the risk-neutral setting. Existing convergence guarantees typically rely on restrictive assumptions, such as linear function approximation, two-timescale step sizes, and projected actor updates, and generally establish convergence to stationary points (Konda & Tsitsiklis, 2003; Bhatnagar et al., 2009; Zhang et al., 2020). We therefore leave the convergence analysis of the fully coupled actor-critic algorithm as an interesting direction for future research. This open question will be made clear in the final version of this paper.

[Haarnoja et al. 2018] Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. ICML 2018.

[Konda & Tsitsiklis 2003] On Actor-Critic Algorithms. SIAM journal on Control and Optimization 42.4 (2003): 1143-1166.

[Bhatnagar et al. 2009] Natural Actor-Critic Algorithms. Automatica, 45(11), 2471–2482.

[Zhang et al. 2020] Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation. ICML 2020.

(W2) Proposition 3.5. We have carefully re-examined the constructive proof presented in lines 1103–1114 for reformulating Eq. 11 in the form of Eq. 18, and can confirm that there is no issue with our proof. We would appreciate it if the reviewer could clarify which step of the construction they believe is incorrect, so that we can provide a more detailed explanation of our arguments.

(W3) Value learning in continuous action space. First, we would like to emphasize that this paper's primary focus is on MDPs with discrete actions where existing (deterministic) policy optimization methods cannot be used. In this setting, V(s′)V(s') can be calculated exactly by enumeration. This being said, our implementation replaces exact calculation with a sample average approximation of 10 sampled actions in order to be flexible enough to handle both environments with discrete (Maze, CliffWalk) and continuous (LunarLander, InvertedPendulum) actions. We agree with the reviewer that this approximation introduces some bias in the training and we now provide a related discussion in the revised paper (see new section E.4). Specifically, one might first interpret the introduced bias as encouraging the critic to estimate the fixed point to T^πQ(s,a)=ρ(r~(s,a)+γ(1/N)∑i=1NQ(s~′,a~i′))\hat{\mathcal{T}}^\pi Q(s,a)=\rho\big(\tilde{r}(s,a)+\gamma (1/N)\sum_{i=1}^NQ(\tilde{s}',\tilde{a}_i')\big). where the risk measure accounts for the uncertainty in both r~(s,a)\tilde{r}(s,a), s~′\tilde{s}', and the sample actions a~i′\tilde{a}'_i. By properties of law-invariance and concavity of ρ\rho, we show that the biased fixed point satisfies Q^π≤Qπ\hat{Q}^\pi\leq Q^\pi.

Moreover, we discuss ways of removing this bias. 1) As discussed by Ciosek & Whiteson (2020), if the critic is quadratic with respect to the action, the expectation over actions can be computed in closed form, eliminating the need for action sampling. 2) Employing multilevel Monte Carlo methods (Blanchet & Glynn, 2015). This has been done in other risk-aware RL literature that employ nested risk measures (see Wang et al. 2023). Exploring these two strategies remains an interesting direction for future research.

Detailed explanations are presented in E.4 in appendix of revised manuscript.

[Blanchet & Glynn 2015] Unbiased Monte Carlo for optimization and functions of expectations via multi-level randomization. IEEE Winter Simulation Conference, 2015.

[Wang et al. 2023] S. Wang, N. Si, J. Blanchet, and Z. Zhou. A finite sample complexity bound for distributionally robust Q-learning. AISTATS 2023.

AuthorsOpenReview ↗

Reply to Reviewer dzJk (Part 2)

(Q1) How the provided framework distinguishes itself from existing literature. We thank the reviewer for highlighting these relevant works. All the papers listed were already discussed in the original manuscript. We are happy to summarize the connections to and differences from our work below.

[Agarwal et al. (2021), On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift] As stated in line 199, our convergence analysis builds upon the arguments of Agarwal et al. (2021) and extends them to the risk-averse setting. The key technical differences are detailed in Remark B.9.

[Zhang et al. (2024), Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexity] As discussed in line 140, we adopt the same formulation of dynamic risk (Eq. 6) as Zhang et al. (2024). However, while Zhang et al. (2024) highlight the challenges of constructing a model-free stochastic policy gradient estimator under this formulation, our work addresses this challenge by developing a model-free actor-critic algorithm.

[Marzban et al. (2023), Deep Reinforcement Learning for Option Pricing and Hedging under Dynamic Expectile Risk Measures] As acknowledged in line 237, the dynamic expectile value update (Eq. 10) was previously considered by Marzban et al. (2023) within a deterministic policy gradient (DPG) framework, without a convergence analysis of the critic update. In contrast, we establish its convergence in Proposition 3.4.

[Coache et al. (2023), Conditionally Elicitable Dynamic Risk Measures for Deep Reinforcement Learning] In the paragraph beginning at line 239, we discuss the motivation for our two-timescale learning rule and its differences from Coache et al. (2023). Furthermore, in Section 3.4, we explain that their on-policy learning approach does not directly extend to TD-style off-policy learning with replay buffers, which is supported by our method.

[Jiang et al. (2024), Learning Diverse Risk Preferences in Population-Based Self-Play] In Appendix D, we establish that the objective optimized by Jiang et al. (2024) corresponds to dynamic expectile risk, a connection that is not explicitly identified in their original paper. Furthermore, as discussed in Section 3.4, their on-policy learning approach does not directly extend to TD-style off-policy learning with replay buffers, whereas our method supports such updates.

Thank you for your efforts. My concern regarding the proof is that, can you reformulate Eq. (11) as Eq. (18)? If a_n or b_n is zeta_t^q(s,a), zeta_t^Q(s,a), it depends on time, state, which does not fall into the formulation of (18)?

We thank the reviewer for pointing out this issue. We agree that our original reformulation of Eq. (11) was incomplete. In particular, the coordinate-dependent learning rates ζnq(s,a)\zeta_n^q(s,a) and ζnQ(s,a)\zeta_n^Q(s,a) cannot be directly identified with the scalar step sizes ana_n and bnb_n in Eq. (18). We have revised Proposition 3.5 and its proof in Appendix C.2 to address this issue.

Specifically, we now assume that state-action pairs are sampled independently from a fixed distribution μ\mu with full support. We introduce two deterministic scalar step-size sequences {ζˉnq}\{\bar{\zeta}_n^q\} and {ζˉnQ}\{\bar{\zeta}_n^Q\}, satisfying the two-timescale conditions in Borkar (2023). At each iteration, only the sampled state-action pair is updated.

Under this sampling assumption, we can incorporate the randomness of coordinate selection into the mean fields and martingale difference noise. Please see the reformulation between line 1110 to 1132 in the revised manuscript. This formulation now matches Eq. (18), with deterministic scalar step sizes. We have also updated the ODE analysis accordingly.

We acknowledge that the revised proposition establishes convergence under the stated fixed i.i.d. sampling assumption, rather than for arbitrary asynchronous sampling schemes. While this additional assumption restricts the scope of our theoretical guarantee, we believe it is reasonable in the tabular policy evaluation setting, where state-action pairs can be independently sampled from a fixed distribution with full support, provided that fresh transition samples are available for the selected pairs. We have made this scope explicit in the revised statement of Proposition 3.5. We appreciate the reviewer for identifying this gap, which helped us clarify the assumptions and strengthen the rigor of the convergence analysis.

Reviewer_j8aYOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce an Actor-Critic reinforcement algorithm for optimizing dynamic risk with stochastic policies. The authors focus on two risk measures, expectile and conditional value-at-risk (CvaR). The authors provide empirical evaluation of the proposed methods on four examples. The paper is well written and easy to follow.

The paper's main algorithmic idea is a surrogate stochastic-policy update that avoids explicitly computing the risk-envelope transition perturbations used by prior risk-sensitive policy-gradient methods. It combines this update with expectile regression or a two-timescale VaR/CVaR value-learning rule. The authors use expected-policy-gradient and replay-buffer ideas for off-policy learning.

Critical strengths
  • Solves a well-defined problem: Using stochastic policies and model-free, off-policy value learning to dynamic risk objectives.
  • The analysis in the paper provides monotonicity and convergence-to-optimal-value results for a finite tabular setting under stated assumptions.
  • The paper provides stochastic-approximation convergence results for its value-learning rules.
  • The CVaR approach separates quantile and CVaR estimation and states the timescale and boundedness assumptions needed for convergence.
  • The experiments are well designed and include risk-neutral and risk-sensitive comparisons.
  • The proposed methods can learn risk-averse behavior in cases where some baselines cannot do the same.
Critical weaknesses

None

Questions for authors

None

1 comment

We thank the reviewer for the positive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.