Actor-Critic Algorithm for Dynamic Expectile and CVaR

Optimizing dynamic risk with stochastic policies is challenging in both policy updates and value learning. The former typically requires transition perturbation, while the latter may rely on model-based approaches. To address these challenges, we propose a surrogate policy gradient without transition perturbation under softmax policy parameterization. We further develop model-free value learning methods for dynamic expectile and conditional value-at-risk (CVaR) by leveraging elicitability. Finally, inspired by Expected SARSA and Expected Policy Gradient, the first model-free off-policy actor-critic algorithm with stochastic policies is constructed for dynamic expectile and CVaR. Empirical results in domains with verifiable risk-averse behavior show that our algorithm can learn risk-averse policy and consistently outperforms other existing methods.