acceptodds
Under review as a conference paper at ICLR 2027

Generative Bellman Flow: Mitigating the CVaR Gradient Bottleneck in Risk-Sensitive Reinforcement Learning

Abstract

Distributional reinforcement learning provides the expressivity to model full return distributions, yet this has not consistently translated into reliable tail-risk policy optimization in continuous control. We identify a CVaR actor-gradient bottleneck: implicit-quantile critics can represent the lower tail, but actor gradients derived from re-sampled tail queries can become noisy when catastrophic and non-catastrophic outcome modes induce well-separated action gradients. To address this, we introduce GBF-SAC, a Soft Actor-Critic instantiation of Generative Bellman Flow. Its conditional flow-matching critic models mean-policy continuation returns and generates a shared sample set for each state-action pair, from which GBF-SAC amortizes the mean and lower-tail statistics into differentiable Q-heads. These heads provide the mean and risk-sensitive actors with smoother scalar objectives rather than repeatedly re-sampled tail estimates. On a suite of risk-sensitive MuJoCo continuous-control tasks with a velocity-dependent crash wrapper, GBF-SAC improves the reward-crash trade-off on the main MuJoCo suite while maintaining competitive mean return. An IQN+Q-Head ablation further isolates the mechanism, indicating that the reward-safety gains arise from the interaction between stable tail representations and the Q-head bridge. Because the critic retains a full learned return distribution, the same trained model also supports diagnostic post-hoc risk scoring at deployment without retraining.

Primary area:
reinforcement learning
Keywords:
Distributional Reinforcement Learning, Risk-Sensitive Reinforcement Learning, CVaR Optimization, Flow Matching, Generative Critic, Soft Actor-Critic, Safe Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dciqOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposed an off-policy actor-critic framework to reduce high variance of policy gradient in risk control. Specifically, the framework adopts a flow-based model to learn the value distribution and two separate critics to learn the mean and risk values of the distribution. The value distribution and two scalar critics are trained independently. Two actors are further trained (one produces the risk-neutral action, one produces the risk-averse action) by using value signals from the two scalar critics. I think the main idea of this paper is that explicitly learning a scalar Q-function from the return distribution, rather than repeatedly estimating Q through sampled returns during policy updates, can reduce the variance of actor gradients and improve training stability, as supported by the experimental results.

Critical strengths

The Q-head idea is simple and straightforward to reduce actor gradient variance when update signals coming from the learned return distribution. This mechanism is not restricted to flow-based critics and can be incorporated into other distributional RL methods.

Critical weaknesses
  1. Writing and presentation. I would like to start with the writing quality. After reading the manuscript, I feel much of the content should be written by an LLM, and the authors also acknowledge the use of generative AI. However, I feel the paper is not well written overall. The abstract is not concise, the connections between sections are weak. The current presentation feels more like stacking different components of the framework, mainly explaining how each network is trained, rather than a coherent description of the proposed method. As a result, the motivations behind certain algorithmic design are not clear (see point 2). Section 3 (Preliminaries) is too brief to provide useful background on distributional RL and conditional flow matching. x′x' in Eq. (4) and zz in Eq. (8) are not defined before their usage.

  2. Motivation for using a flow model. Since the main finding of the paper is that explicitly learning a scalar Q-function from the return distribution can reduce the variance of policy gradients, what is the motivation for using a flow model to represent the return distribution? In principle, any architecture capable of modeling the return distribution should be sufficient, such as QR-DQN or IQN.

  3. Definition of the risk measure. The risk value computation in Eq. (8) is unclear to me, and no reference is provided. Is this formulation proposed by the authors or adopted from prior work? This risk definition can violate the monotonicity property (X≥YX \geq Y implies ρ(X)≥ρ(Y)\rho(X) \geq \rho(Y)), which is a fundamental property of risk measures. Here is a simple counterexample. P(ω1)=P(ω2)=0.5P(\omega_1)=P(\omega_2)=0.5. X(ω1)=−0.75X(\omega_1)=-0.75, X(ω2)=1X(\omega_2)=1. Y(ω1)=−0.75Y(\omega_1)=-0.75, Y(ω2)=0.75Y(\omega_2)=0.75. According to Eq. (8), ρ(X)≈−0.699<ρ(Y)≈−0.679\rho(X) \approx -0.699 < \rho(Y) \approx -0.679, despite X≥YX \geq Y.

  4. Selection of the target return distribution. In line 142, when training the return distribution, the target distribution is selected based on the smaller sample mean. The motivation for this operation is not explained, and the choice is not intuitive. If the goal of the algorithm is to reduce risk, selecting the distribution with the smaller mean does not necessarily imply that it is preferable from a risk-sensitive perspective.

  5. Lack of a rigorous risk-sensitive RL formulation. Overall, the proposed framework appears to be more of a heuristic than a rigorous risk-sensitive RL algorithm. As discussed by Lim and Malik (2022), updating the actor using CVaR values extracted from a distributional critic does not necessarily optimize static or dynamic CVaR. The proposed framework further complicates this issue by training two actors: πμ\pi_\mu for risk-neutral action selection and πc\pi_c for risk-averse action selection. According to Algorithm 1, the data come from πμ\pi_\mu. As a result, πc\pi_c is not even optimizing a risk objective associated with its own policy, but a risk value defined under πμ\pi_\mu. This makes the overall framework difficult to interpret and understand.

Questions for authors

See point 2, 3, 4 in weaknesses.

Reviewer_ugfaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GBF-SAC fits a conditional flow model to an entropy-regularized return distribution under mean-policy continuation. Generated returns supervise persistent scalar heads for the mean and a lower-tail statistic, and separate mean and risk actors differentiate through these heads. The aim is to remove fresh tail-query noise from actor updates while retaining the full return distribution for diagnostic scoring. The paper analyzes conditional query variance and head approximation error, and evaluates the approach on risk-sensitive MuJoCo tasks, controlled distributions, and an additional safety task.

Critical strengths
  1. Learning a persistent scalar objective from distribution samples is a useful actor-interface design. The IQN+Q-head control provides direct evidence for its value: on HalfCheetah, risk-policy reward increases from -139 to 1834 while observed crashes remain at 0% (Table 10). This is more informative than comparing complete algorithms alone. The theory also distinguishes fresh-query variance from approximation error and explicitly recognizes that fixed-grid evaluation can remove the same source of randomness.

  2. The deployment evaluation goes beyond critic scores. The fixed-alpha study uses five independent training seeds and 1,000 common episodes per seed and execution mode, with uncertainty reported across training runs. On Ant, the risk actor improves complete-trajectory CVaR relative to its paired mean actor in every seed under both execution modes (Tables 3–4). The full eight-comparison family and Holm adjustment in Table 12, together with the reported mixed results, make the empirical evidence easier to assess.

Critical weaknesses
  1. The evidence supports the scalar-head interface more clearly than the claimed contribution of the flow representation. As Section 6.1 notes, the IQN+head–GBF comparison changes both the return representation and the tail target, so the remaining reward gap does not isolate a flow-specific benefit. Table 13 is particularly important: IQN+head and Flow+head both have essentially zero fresh-query variance, but their gradient MAEs are 0.0118 and 1.13, respectively. This does not invalidate GBF's control results, but it leaves unexplained why a substantially less accurate gradient in the controlled experiment supports the proposed mechanism. The real-task gradient diagnostic is limited to a short, single-seed run, with consecutive-gradient cosines near zero for both methods. A comparison holding the tail statistic and head configuration fixed is needed to distinguish improved representation from a different objective and the benefit of amortization.

  2. Evidence for improved canonical CVaR optimization remains inconclusive. The four-task benchmark uses the exponential score in Eq. (8), whereas IQN uses CVaR at alpha=0.1. The fixed-alpha control is a useful addition, but Eq. (9) is still a smoothed proxy: Table 6 reports target deviations of 0.160 standard deviations on average and up to 0.549 at the chosen width. Target choice also materially changes performance in Table 14. For actual trajectory CVaR, all four GBF-alpha–QR256 confidence intervals in Table 16 include zero, with favorable outcomes in only two or three of five seeds. These results allow for a benefit, but do not establish a consistent CVaR advantage. Together with the unresolved attribution above, this leaves the main contribution less well supported than the reward–crash results initially suggest.

Questions for authors
  1. Can the IQN+head and Flow+head comparison be made with the same tail statistic, head architecture, and training settings, with sample counts and computation reported explicitly? Paired trajectory-CVaR and reward–crash results under that control would materially affect my assessment. The existing fixed-grid comparisons are useful, but do not isolate this difference.

  2. In Table 13, what objective defines the oracle gradient, and do both heads regress the same statistic? Please explain the 0.0118 versus 1.13 gradient-MAE gap. If comparable diagnostics are available on the control tasks, do they show that reduced query variance compensates for head or target bias? This would help connect the controlled result to the observed policy improvement.

  3. Please specify the exact crash-probability function and clipping/termination rules in Appendix C, the WCSAC cost definition and constraint threshold, and the reward–cost scalarization coefficient for SafetyPointGoal1. These details are needed to reproduce the risk settings and assess the compared operating points.

Reviewer_wEunOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Generative Bellman Flow (GBF-SAC), a risk-sensitive RL method based on SAC. The paper points out that sampling lower-tail quantiles from a distributional critic can lead to high variance in actor gradients, especially when different return modes have very different gradients. To address this problem, the method uses a flow-matching critic to model the return distribution, and trains two Q-heads to estimate the mean return and a lower-tail risk measure. The mean actor and risk actor are then trained using these Q-heads, instead of directly taking gradients through sampled returns. The paper also provides some theoretical analysis on the gradient variance and Q-head approximation error. Experiments on four MuJoCo tasks show that the proposed method can achieve better reward and safety trade-off than several baselines.

Critical strengths

This paper studies an interesting problem of reducing actor gradient variance in risk-sensitive RL. The idea of using separate Q-heads for mean and risk-sensitive actor updates is interesting. The IQN+Q-head ablation is also useful, as it shows that adding Q-heads to an existing distributional RL method can significantly improve the risk-sensitive policy performance, without changing the return distribution model. This provides some evidnece that the proposed actor interface can be effective.

Critical weaknesses

I don't have major concerns about this paper, but I have two minor concerns that are discussed in the Questions section below.

Questions for authors
  1. It would be good to have more empirical results beyond the four MuJoCo locomotion tasks. Although the paper includes an additional experiment on SafetyPointGoal1, the main evaluation is still limited to a small set of tasks. It would be useful to see how well the proposed method generalizes to other risk-sensitive RL environments.
  2. The main experiments use the weighted tail-risk target in Eq. (8), instead of standard CVaR. Could the paper explain more about the motivation of this choice? Although Appendix A.2 provides some analysis of the parameter κα\kappa_\alpha, I would also like to see how sensitive the actual RL performance is to the choice of κ\kappa in Eq. (8).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.