- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces Q Normalizing Flows (QNF), an extension of Q-learning to continuous-action spaces in maximum entropy reinforcement learning (RL). QNF learns a normalizing flow distribution to represent the softmax density over action values, using action samples from the density in the soft Q-learning target. The paper focuses on an empirical characterization of QNF, comparing it to popular off-policy baselines like CrossQ and Soft Actor-Critic across 8 environments from the DeepMind Control Suite.
Critical strengths
None
Critical weaknesses
Main Argument
I have two main concerns with the paper. First, the paper overall lacks polish and omits crucial details required for a thorough characterization of the accuracy of the paper and the method presented. Second, the empirical study lacks rigour, calling into question some of the claims made and the performance reported.
Details and Polish
Critical Details are Missing Regarding the Method Presented The paper omits critical details about the method developed. By the end of Section 4, which outlines the proposed method, I was not able to decipher what the QNF method is.
The method of learning action-value functions is unclear. The beginning of Section 4 notes that QNF parameterizes the action-value function as a normalizing flow density. How does this work, since is a real-valued function? Immediately after this statement, the paper defines which seems to be a real-valued function, not a distribution. The parameterization of using normalizing flows is explained in the cited work on MEow (Chao et al, 2024), but none of that is explained here nor is it clear from the pseudocode in this paper that this previous approach is being followed.
Later, Section 4.1 outlines the update to the learned function but leaves multiple symbols undefined. First is never defined, but I assume it is a state-value function estimate. In Equations 13 and 14, , , , and are all left undefined. What do these refer to? My assumption was that they are parameters of the and functions in Equations 13 and 14. But this is confusing as later, Algorithm 1 defines and as target networks. Further, the paragraph mentions that only two critics were used, but the equations seem to contain four. Are the functions computed according to the expectation in Equation 12? If so, how is the expectation over the continuous-action space evaluated? The paper mentions that an EMA delay rate of is used. What is this EMA delay rate?
The method of policy improvement is also unclear. Section 4.1 outlines that
and can be updated using the same TD loss as in soft Q learning
Then later, in section 4.3, another policy improvement operator is introduced, but critical details are omitted. Section 4.3 outlines a policy target, , but doesn't say how the policy is updated toward this target. It also provides no justification for the policy target or explanation on where the target comes from. It seems to be the exponent of a re-centered softmax policy akin to Equation 11, but why this target is used is not clear. Is a flow matching distribution over softmax targets learned? Why is the policy target reasonable? Are policies learned using the method in Section 4.1 or Section 4.3?
Other Detail and Polish Issues The paper omits numerous other details or otherwise lacks polish. The operator is not defined. is only defined implicitly as policy entropy, and sometimes it is evaluated on states and sometimes on the policy itself, without the superscript. Equations 3, 4, and 5 mix evaluating the action-value function on actions-only as and on state-action pairs as . Multiple claims require citations but lack any. For example
Q-learning algorithms are often favored in discrete reinforcement learning for their simplicity, interpretability, performance, and theoretical ability to learn from arbitrarily distributed data.
Action value learning actor critic algorithms are the most sample efficient model-free continuous reinforcement learning algorithms today.
Entropy regularization in reinforcement learning can speed up learning by encouraging exploration, and improve gradient conditioning by smoothing action distributions.
Empirical Study
The empirical study also lacks crucial details In Section 5 (Experiments), the paper mentions that the experiments characterize the QMANSF realization of QNF but does not define what this realization is. What is this realization of QNF? How does it relate to the QNF algorithm already defined and shown in Algorithm 1? Presumably, it uses a different kind of flow distribution, but this is never clarified
The empirical study lacks rigour The empirical analysis presented in the paper is questionable. The paper used only 5 experimental repetitions across all experiments, which is likely far too few for an accurate estimation of central tendency and variation [1]. It is well-known that these kinds of deep RL algorithms exhibit extreme variance in return distributions, which are typically non-normal [1,2,3], requiring many experimental repetitions for a rigorous empirical characterization. In fact, it has been established that using up to 10 random seeds can lead to statistically confidently incorrect conclusions for these kinds of algorithms [2]. While I do recognize that the norm of empirical deep RL work is to use relatively few experimental repetitions, this norm does not reflect scientific rigour. Bootstrap confidence intervals are reported throughout, which make no distributional assumptions. But, they are known to be sensitive to the number of experimental repetitions and I am not confident that the reported shaded regions are accurate characterizations of the underlying return data [3].
The paper does make claims with overlapping confidence intervals. For example, the paper claims
In the flow architecture comparison Figure 3 we find that QMANSF is consistently among the best performers
But throughout Figure 3, nearly all shaded regions overlap across experiments. In relation to Figure 2, the paper discusses how CrossQ collapses entropy even without sufficient return data while QNF does not, but as far as I can tell no Figures show entropy data.
I would explicitly point out that I am not concerned that the QMANSF algorithm does not reach state-of-the-art performance across the considered environments, only matching (or sometimes not quite attaining) the performance of the baselines. Negative results can be quite informative. My concern is the lack of empirical rigour for an algorithm that is not well-explained.
Small Things
These did not affect the scoring of the paper.
- L54: states that a density represents a timestep
- L64: Equation 1 is missing lower limit.
- L68: references a non-parametric max operator. What is a parametric max operator?
- L76: convergence is a technical term which is almost definitely not what is meant here
- L107: The text before Equation 6 defines , but the Equation uses
- L107: is a bit misleading, since plays no role here. Why not ?
- L138: wire-fitting(Korkmaz et al., 2025) -> wire-fitting (Korkmaz et al., 2025)
- L160: What is a V?
- Often, the paper says in CITATION for example the one outlined in Mnih et al. (2015). It should say by rather than in.
- L203 defines a TD loss for , but it is not a TD loss on .
References
[1] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, David Meger. Deep Reinforcement Learning that Matters. AAAI, 2018.
[2] Andrew Patterson, Samuel Neumann, Martha White, Adam White. Empirical Design in Reinforcement Learning. JMLR, 2024.
[3] Cédric Colas, Olivier Sigaud, Pierre-Yves Oudeyer. How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments.
[4] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv, 2017.
[5] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, Sergey Levine. Soft Actor-Critic Algorithms and Applications. arXiv, 2019.
Questions for authors
It would be difficult to address the current issues in the paper, since significant changes would be needed to understand the proposed approach. The key question right now is a clearer explanation of the algorithm itself.