- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Distributional Flow Critic (DFC), which uses flow matching to model the full state-action return distribution rather than directly regressing its mean. The learned distribution is distilled into a quantile-conditioned one-step critic, whose averaged quantile predictions guide policy optimization within Flow Q-Learning without back propagating through the flow ODE. The paper also presents a contraction result for exact quantile-projected Bellman evaluation under a fixed policy. Experiments across 93 D4RL and OGBench tasks report improved value estimation and policy performance, achieving the highest scores among the compared methods in 13 of 17 domain groups, with additional gains in offline-to-online adaptation and applicability to other actor–critic algorithms.
Critical strengths
- Study how distributional critics improve offline RL performance, where these critics are theoretically more expressive than expected-value estimates.
- Iterative flow sampling for the value is performed only during the value learning phase.
- Extensive experiments and ablation studies are conducted.
Critical weaknesses
- The writing does not clearly motivate why we should use distributional critics. Later ablation studies (Section 5.3) provide evidence that distributional critics improve performance, but this is not the main motivation of the research.
- There is something theoretically confusing when calculating the critic’s bootstrapping target. The value estimate calculated with iterative flow sampling should match the noise input dimension, i.e., . However, the bootstrapping target adds a 1-dimensional to this.
Questions for authors
- Q.1. I find the authors’ observation on “Value estimation accuracy” in Section 5.3 very interesting (Figure 4). However, the current policy training in Section 3.3 actually seems to take an average over all quantiles, meaning it is still maximizing the expected value in theory. What happens if we optimize the policy to only maximize the upper quantiles of the value? Compare C51, IQN, and DFC.
- Q.2. Why did you choose the Huber loss? What happens if we just use the quantile loss or the expectile loss? Does the performance degrade?
- Q.3. I am a little skeptical about the computational efficiency of training stated in Figure 3. According to the loss for , the computational burden increases quadratically with respect to the number of quantiles, i.e., . Compared to FQL, how does the training runtime of DFC change when sweeping from 51, 101, 251, and 1001?
- Q.4. Can the authors break down the dimensionality of each variable during value training? What is the dimensionality of , and what is it exactly modeling mathematically?