acceptodds
Withdrawn from ICLR 2027

Q Learning with Normalizing Flows

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Q-learning algorithms are the dominant approach for reinforcement learning in discrete action spaces, but adapting them for continuous action spaces requires an intractable continuous max to find optimal actions. Maximum entropy reinforcement learning replaces the max with a softmax that allocates action density proportionally to the exponentiated Q-function, but without an efficient to maximize the Q-function this alone does not suffice for continuous Q-learning. This paper introduces Q Normalizing Flows (QNF), a class of methods that directly represent the softmax density via normalizing flows, allowing effective semi-gradient Q-learning in continuous action spaces. This makes a rich body of representations immediately available, including those most commonly used in off-policy actor critic methods, without coupling heavily to specialized architectures for action selection. Q Normalizing Flows perform similarly to widely used actor-critic algorithms for continuous RL while training with the pure semi-gradient action evaluation loss.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Generative Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_sG2rOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces Q Normalizing Flows (QNF), an extension of Q-learning to continuous-action spaces in maximum entropy reinforcement learning (RL). QNF learns a normalizing flow distribution to represent the softmax density over action values, using action samples from the density in the soft Q-learning target. The paper focuses on an empirical characterization of QNF, comparing it to popular off-policy baselines like CrossQ and Soft Actor-Critic across 8 environments from the DeepMind Control Suite.

Critical strengths

None

Critical weaknesses

Main Argument

I have two main concerns with the paper. First, the paper overall lacks polish and omits crucial details required for a thorough characterization of the accuracy of the paper and the method presented. Second, the empirical study lacks rigour, calling into question some of the claims made and the performance reported.

Details and Polish

Critical Details are Missing Regarding the Method Presented The paper omits critical details about the method developed. By the end of Section 4, which outlines the proposed method, I was not able to decipher what the QNF method is.

The method of learning action-value functions QQ is unclear. The beginning of Section 4 notes that QNF parameterizes the action-value QQ function as a normalizing flow density. How does this work, since QQ is a real-valued function? Immediately after this statement, the paper defines Qϕ,θ(s,a)=Vϕ(s)+αlog⁡πθ(a∣s)Q_{\phi, \theta}(s, a) = V_\phi(s) + \alpha \log\pi_\theta(a \mid s) which seems to be a real-valued function, not a distribution. The parameterization of QQ using normalizing flows is explained in the cited work on MEow (Chao et al, 2024), but none of that is explained here nor is it clear from the pseudocode in this paper that this previous approach is being followed.

Later, Section 4.1 outlines the update to the learned QQ function but leaves multiple symbols undefined. First VϕV_\phi is never defined, but I assume it is a state-value function estimate. In Equations 13 and 14, ϕ0\phi_0, ϕ1\phi_1, ϕˉ0\bar \phi_0, and ϕˉ1\bar \phi_1 are all left undefined. What do these refer to? My assumption was that they are parameters of the QQ and VV functions in Equations 13 and 14. But this is confusing as later, Algorithm 1 defines ϕˉ0\bar \phi_0 and ϕˉ1\bar \phi_1 as target networks. Further, the paragraph mentions that only two critics were used, but the equations seem to contain four. Are the VV functions computed according to the expectation in Equation 12? If so, how is the expectation over the continuous-action space evaluated? The paper mentions that an EMA delay rate of τ\tau is used. What is this EMA delay rate?

The method of policy improvement is also unclear. Section 4.1 outlines that

VϕV_\phi and πθ\pi_\theta can be updated using the same TD loss as in soft Q learning

Then later, in section 4.3, another policy improvement operator is introduced, but critical details are omitted. Section 4.3 outlines a policy target, AVϕA_{V_{\phi}}, but doesn't say how the policy is updated toward this target. It also provides no justification for the policy target or explanation on where the target comes from. It seems to be the exponent of a re-centered softmax policy akin to Equation 11, but why this target is used is not clear. Is a flow matching distribution over softmax targets learned? Why is the policy target AVϕA_{V_\phi} reasonable? Are policies learned using the method in Section 4.1 or Section 4.3?

Other Detail and Polish Issues The paper omits numerous other details or otherwise lacks polish. The operator Δ\Delta is not defined. HπH^\pi is only defined implicitly as policy entropy, and sometimes it is evaluated on states and sometimes on the policy itself, without the superscript. Equations 3, 4, and 5 mix evaluating the action-value function QQ on actions-only as Q(a)Q(a) and on state-action pairs as Q(s,a)Q(s, a). Multiple claims require citations but lack any. For example

Q-learning algorithms are often favored in discrete reinforcement learning for their simplicity, interpretability, performance, and theoretical ability to learn from arbitrarily distributed data.

Action value learning actor critic algorithms are the most sample efficient model-free continuous reinforcement learning algorithms today.

Entropy regularization in reinforcement learning can speed up learning by encouraging exploration, and improve gradient conditioning by smoothing action distributions.

Empirical Study

The empirical study also lacks crucial details In Section 5 (Experiments), the paper mentions that the experiments characterize the QMANSF realization of QNF but does not define what this realization is. What is this realization of QNF? How does it relate to the QNF algorithm already defined and shown in Algorithm 1? Presumably, it uses a different kind of flow distribution, but this is never clarified

The empirical study lacks rigour The empirical analysis presented in the paper is questionable. The paper used only 5 experimental repetitions across all experiments, which is likely far too few for an accurate estimation of central tendency and variation [1]. It is well-known that these kinds of deep RL algorithms exhibit extreme variance in return distributions, which are typically non-normal [1,2,3], requiring many experimental repetitions for a rigorous empirical characterization. In fact, it has been established that using up to 10 random seeds can lead to statistically confidently incorrect conclusions for these kinds of algorithms [2]. While I do recognize that the norm of empirical deep RL work is to use relatively few experimental repetitions, this norm does not reflect scientific rigour. Bootstrap confidence intervals are reported throughout, which make no distributional assumptions. But, they are known to be sensitive to the number of experimental repetitions and I am not confident that the reported shaded regions are accurate characterizations of the underlying return data [3].

The paper does make claims with overlapping confidence intervals. For example, the paper claims

In the flow architecture comparison Figure 3 we find that QMANSF is consistently among the best performers

But throughout Figure 3, nearly all shaded regions overlap across experiments. In relation to Figure 2, the paper discusses how CrossQ collapses entropy even without sufficient return data while QNF does not, but as far as I can tell no Figures show entropy data.

I would explicitly point out that I am not concerned that the QMANSF algorithm does not reach state-of-the-art performance across the considered environments, only matching (or sometimes not quite attaining) the performance of the baselines. Negative results can be quite informative. My concern is the lack of empirical rigour for an algorithm that is not well-explained.

Small Things

These did not affect the scoring of the paper.

  • L54: states that a density represents a timestep
  • L64: Equation 1 is missing tt lower limit.
  • L68: references a non-parametric max operator. What is a parametric max operator?
  • L76: convergence is a technical term which is almost definitely not what is meant here
  • L107: The text before Equation 6 defines PfP_f, but the Equation uses PP
  • L107: Px0P_{x_0} is a bit misleading, since x0x_0 plays no role here. Why not P0P_0?
  • L138: wire-fitting(Korkmaz et al., 2025) -> wire-fitting (Korkmaz et al., 2025)
  • L160: What is a V?
  • Often, the paper says in CITATION for example the one outlined in Mnih et al. (2015). It should say by rather than in.
  • L203 defines a TD loss for α\alpha, but it is not a TD loss on α\alpha.

References

[1] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, David Meger. Deep Reinforcement Learning that Matters. AAAI, 2018.

[2] Andrew Patterson, Samuel Neumann, Martha White, Adam White. Empirical Design in Reinforcement Learning. JMLR, 2024.

[3] Cédric Colas, Olivier Sigaud, Pierre-Yves Oudeyer. How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments.

[4] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv, 2017.

[5] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, Sergey Levine. Soft Actor-Critic Algorithms and Applications. arXiv, 2019.

Questions for authors

It would be difficult to address the current issues in the paper, since significant changes would be needed to understand the proposed approach. The key question right now is a clearer explanation of the algorithm itself.

Reviewer_e4GGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Q Normalizing Flows (QNF), which represent the policy with a normalizing flow so that the policy density is the Gibbs distribution of an implicit QQ-function defined by the flow and a scalar value head, thereby removing a separate SAC-style actor-extraction network while retaining off-policy soft TD learning with twin critics and target networks. The critic target follows the usual soft TD form with clipped double-QQ; the flow is updated with semi-gradient TD rather than a SAC actor loss. Entropy temperature α\alpha is tuned by a closed-form least-squares fit on log-policy and advantage pairs, and a greedification term with weight λ\lambda pushes the policy toward colder distributions when TD error is small. The main architecture is QMANSF. Experiments on eight DeepMind Control tasks use 500500k steps and five seeds, reporting IQM with bootstrap confidence intervals against SAC, CrossQ, FMER, FPO, and MEow. QNF is competitive on some tasks but underperforms SAC and CrossQ on exploitation-heavy locomotion and on hopper-stand; the authors acknowledge higher compute and flow instability. There is no convergence or bias analysis for semi-gradient TD with flows, and no wall-clock comparison to Gaussian actor-critic methods.

Critical strengths
  1. Clear exposition of how a flow can represent a Gibbs policy and how soft TD targets attach to the same pipeline used in actor-critic entropy methods.
  2. Ablations for flow type, fixed vs autotuned α\alpha, and greedification show the authors tested components of their training recipe, not only a single configuration.
  3. The limitations section honestly notes simulation-only scope, compute, and instability.
Critical weaknesses
  1. Equations 7–14 are textbook soft optimality and soft TD; the paper's delta is architectural plus OLS α\alpha and greedification, not a new Bellman framework or QQ-learning principle.
  2. Normalizing flows require sampling and log-determinant Jacobians across layers; the paper does not provide wall-clock or FLOPs-matched comparisons, yet stronger baselines win on exploitation tasks where precise control matters.
  3. Stating performance similar to widely used actor-critic methods is inconsistent with reported IQMs where SAC/CrossQ are ahead; the paper reads like a SAC-variant submission without consistent empirical upside.
  4. hopper-stand show QNF fails to match CrossQ's entropy adaptation; greedification can wait for reward rather than explore—undermining “general continuous QQ-learning” messaging.
  5. One hyperparameter sheet across eight environments and MEow derived from Hopper defaults with manual entropy adjustments make it hard to attribute results to the flow idea rather than tuning.
  6. Without bias analysis or comparison to full-gradient updates, it is unclear the update is well-founded beyond engineering trial-and-error.
Questions for authors
  1. Provide wall-clock and samples-to-threshold tables vs SAC/CrossQ on all eight tasks; does QNF ever dominate when compute is matched?
  2. Ablate Gaussian actor + OLS α\alpha + greed (no flow) vs QMANSF without OLS/greed to separate flow vs tricks.
  3. On cheetah-run, walker-run, hopper-stand, report final IQM, CIs, and simple tests vs the best baseline at matched steps and matched wall-time.
  4. Can you justify semi-gradient TD with empirical bias measurements vs Trust-PCL-style full gradients on a diagnostic domain?
  5. Will code reproduce exact SAC/CrossQ/MEow numbers, including α\alpha initialization and entropy schedules?
  6. Why should the community adopt a more expensive policy class when simpler actors win on multiple headline tasks?
Reviewer_Xb8tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Q Normalizing Flows (QNF), a framework for continuous-action soft Q-learning that parameterizes the Q-function as (Q(s,a)=V(s)+αlog⁡π(a∣s))(Q(s,a)=V(s)+\alpha\log\pi(a|s)), where the policy is represented by a normalizing flow. This construction makes the softmax policy and soft value directly tractable, avoiding the separate policy-extraction optimization used in conventional actor-critic methods. The authors additionally introduce a closed-form temperature-tuning procedure and a greedification term for policy improvement.

Experiments on DeepMind Control tasks demonstrate competitive performance on several environments, although QNF does not consistently match stronger actor-critic baselines.

Critical strengths
  1. Interesting and potentially impactful formulation. Representing Q-values through a normalized policy density is an elegant approach to continuous-action Q-learning. It could motivate alternative RL algorithms and more expressive Q-function representations.

  2. Unified value and policy learning. The method directly trains the value network and flow policy through TD learning without requiring a separate actor-improvement optimization. This offers an appealing alternative to conventional actor-critic training.

  3. Compatibility with expressive policy architectures. The framework supports multiple normalizing flow architectures, and the experiments investigate several choices rather than relying on one specialized model.

  4. Useful initial experimental investigation. The authors compare against established methods including SAC, CrossQ, and MEow, and provide ablations for temperature tuning, greedification, and flow architectures.

Critical weaknesses
  1. Limited demonstration of practical advantages. While the method performs competitively on some tasks, it generally does not surpass the strongest baselines and substantially underperforms in some environments. The experiments do not yet establish a clear practical advantage that compensates for this performance gap.

  2. Limited experimental scope. Evaluation is restricted to a relatively small collection of DeepMind Control tasks, primarily with dense rewards. It remains unclear how the method scales to higher-dimensional action spaces, sparse rewards, or more challenging environments.

  3. Insufficient justification for policy improvement. Section 4.3 introduces a centered and RMS-normalized advantage as a greedification direction. However, the mathematical motivation for this particular construction, especially its claimed cross-entropy neutrality and relationship to policy improvement, is not sufficiently clear.

  4. Unclear computational tradeoffs. One benefit of eliminating separate actor training is potentially memory and runtime benefits, but normalizing flows can also be computationally expensive. The experiments do not quantify whether the proposed simplification improves training efficiency in practice.

  5. Presentation clarity. Section 4.3 is difficult to follow, particularly the definition and interpretation of the centered advantage (\tilde A). Figure 1 is also difficult to interpret because its axes, vector directions, and geometric relationships are not explained sufficiently clearly.

Questions for authors
  1. Practical benefits: Beyond removing a separate actor-improvement objective, what practical advantages should QNF offer over SAC or CrossQ? Can these advantages be demonstrated experimentally, for example through training efficiency, robustness, or performance on tasks benefiting from expressive action distributions?

  2. Policy improvement: What is the theoretical motivation for centering and RMS-normalizing the advantage in Equations 18–20? In particular, in what sense does this direction preserve cross-entropy, and under what conditions should it improve the policy?

  3. Experimental generalization: Can the authors evaluate QNF on additional environments, especially tasks with higher-dimensional action spaces or sparse rewards?

  4. Computational cost: How does QNF compare with SAC and CrossQ in wall-clock training time, memory consumption, and computational cost per update?

  5. Relationship to prior work: Could the authors further clarify the methodological differences between QNF and Trust-PCL or MEow, and identify which proposed components are responsible for the observed improvements?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.