- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposed an off-policy actor-critic framework to reduce high variance of policy gradient in risk control. Specifically, the framework adopts a flow-based model to learn the value distribution and two separate critics to learn the mean and risk values of the distribution. The value distribution and two scalar critics are trained independently. Two actors are further trained (one produces the risk-neutral action, one produces the risk-averse action) by using value signals from the two scalar critics. I think the main idea of this paper is that explicitly learning a scalar Q-function from the return distribution, rather than repeatedly estimating Q through sampled returns during policy updates, can reduce the variance of actor gradients and improve training stability, as supported by the experimental results.
Critical strengths
The Q-head idea is simple and straightforward to reduce actor gradient variance when update signals coming from the learned return distribution. This mechanism is not restricted to flow-based critics and can be incorporated into other distributional RL methods.
Critical weaknesses
-
Writing and presentation. I would like to start with the writing quality. After reading the manuscript, I feel much of the content should be written by an LLM, and the authors also acknowledge the use of generative AI. However, I feel the paper is not well written overall. The abstract is not concise, the connections between sections are weak. The current presentation feels more like stacking different components of the framework, mainly explaining how each network is trained, rather than a coherent description of the proposed method. As a result, the motivations behind certain algorithmic design are not clear (see point 2). Section 3 (Preliminaries) is too brief to provide useful background on distributional RL and conditional flow matching. in Eq. (4) and in Eq. (8) are not defined before their usage.
-
Motivation for using a flow model. Since the main finding of the paper is that explicitly learning a scalar Q-function from the return distribution can reduce the variance of policy gradients, what is the motivation for using a flow model to represent the return distribution? In principle, any architecture capable of modeling the return distribution should be sufficient, such as QR-DQN or IQN.
-
Definition of the risk measure. The risk value computation in Eq. (8) is unclear to me, and no reference is provided. Is this formulation proposed by the authors or adopted from prior work? This risk definition can violate the monotonicity property ( implies ), which is a fundamental property of risk measures. Here is a simple counterexample. . , . , . According to Eq. (8), , despite .
-
Selection of the target return distribution. In line 142, when training the return distribution, the target distribution is selected based on the smaller sample mean. The motivation for this operation is not explained, and the choice is not intuitive. If the goal of the algorithm is to reduce risk, selecting the distribution with the smaller mean does not necessarily imply that it is preferable from a risk-sensitive perspective.
-
Lack of a rigorous risk-sensitive RL formulation. Overall, the proposed framework appears to be more of a heuristic than a rigorous risk-sensitive RL algorithm. As discussed by Lim and Malik (2022), updating the actor using CVaR values extracted from a distributional critic does not necessarily optimize static or dynamic CVaR. The proposed framework further complicates this issue by training two actors: for risk-neutral action selection and for risk-averse action selection. According to Algorithm 1, the data come from . As a result, is not even optimizing a risk objective associated with its own policy, but a risk value defined under . This makes the overall framework difficult to interpret and understand.
Questions for authors
See point 2, 3, 4 in weaknesses.