acceptodds
Withdrawn from ICLR 2027

Unleashing Flow Policies with Distributional Critics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generative flow models have recently emerged as a powerful paradigm for policy learning in offline reinforcement learning, capable of capturing complex, multimodal action distributions inherent in diverse datasets. However, the full potential of these expressive actors is often bottlenecked by their critics, which typically learn a single scalar estimate of the expected return. We argue that directly regressing this mean value is suboptimal, as it often fails to capture the underlying stochasticity of returns, leading to high-variance and inaccurate value estimates. To address this, we introduce the Distributional Flow Critic (DFC), a novel architecture that utilizes flow matching to learn the complete state-action return distribution. Crucially, we demonstrate that explicitly modeling this distribution acts as a robust representation regularizer: by capturing the comprehensive outcome distribution rather than compressing complex dynamics into a single scalar, DFC yields a much higher-fidelity value signal. We extensively validate our approach across diverse tasks from the D4RL and OGBench benchmarks, spanning locomotion and manipulation. Results show that DFC achieves the highest score on out of domain groups, with an average domain-wise score ratio of relative to the strongest baseline in each group. DFC further excels in offline-to-online adaptation, achieving the best or tied-best fine-tuned performance on out of domains.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FzyAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Distributional Flow Critic (DFC), which uses flow matching to model the full state-action return distribution rather than directly regressing its mean. The learned distribution is distilled into a quantile-conditioned one-step critic, whose averaged quantile predictions guide policy optimization within Flow Q-Learning without back propagating through the flow ODE. The paper also presents a contraction result for exact quantile-projected Bellman evaluation under a fixed policy. Experiments across 93 D4RL and OGBench tasks report improved value estimation and policy performance, achieving the highest scores among the compared methods in 13 of 17 domain groups, with additional gains in offline-to-online adaptation and applicability to other actor–critic algorithms.

Critical strengths
  • Study how distributional critics improve offline RL performance, where these critics are theoretically more expressive than expected-value estimates.
  • Iterative flow sampling for the value is performed only during the value learning phase.
  • Extensive experiments and ablation studies are conducted.
Critical weaknesses
  • The writing does not clearly motivate why we should use distributional critics. Later ablation studies (Section 5.3) provide evidence that distributional critics improve performance, but this is not the main motivation of the research.
  • There is something theoretically confusing when calculating the critic’s bootstrapping target. The value estimate calculated with iterative flow sampling should match the noise input dimension, i.e., dd. However, the bootstrapping target adds a 1-dimensional rr to this.
Questions for authors
  • Q.1. I find the authors’ observation on “Value estimation accuracy” in Section 5.3 very interesting (Figure 4). However, the current policy training in Section 3.3 actually seems to take an average over all quantiles, meaning it is still maximizing the expected value in theory. What happens if we optimize the policy to only maximize the upper quantiles of the value? Compare C51, IQN, and DFC.
  • Q.2. Why did you choose the Huber loss? What happens if we just use the quantile loss or the expectile loss? Does the performance degrade?
  • Q.3. I am a little skeptical about the computational efficiency of training stated in Figure 3. According to the loss for ψ\psi, the computational burden increases quadratically with respect to the number of quantiles, i.e., MM. Compared to FQL, how does the training runtime of DFC change when sweeping MM from 51, 101, 251, and 1001?
  • Q.4. Can the authors break down the dimensionality of each variable during value training? What is the dimensionality of z~j\tilde z_j, and what is it exactly modeling mathematically?
Reviewer_F1xpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Distributional Flow Critic (DFC) for offline and offline-to-online reinforcement learning. A flow teacher learns return distributions and distills them into a quantile-conditioned critic, whose mean guides an FQL actor without differentiating through the critic’s ODE solver. Across 93 OGBench and D4RL tasks, DFC achieves the highest mean score in 13 of 17 domain groups and best or tied-best fine-tuned performance on four of five tasks in Table 2.

Critical strengths
  1. The architecture addresses a concrete optimization cost. Sections 3.1–3.3 separate distribution learning from the differentiable value estimate used by the actor. Stop-gradient teacher sampling and quantile distillation remove the need to differentiate through the flow critic during policy optimization. Algorithm 1 retains FQL’s one-step deployed policy, making the absence of additional critic computation at deployment credible.
  2. The evaluation is broad and includes substantial improvements. Tables 1, 5, and 6 cover navigation, manipulation, and visual observations, with eight seeds for state-based experiments and four for visual experiments. Improvements include 73.4 versus 60.2 for the strongest baseline on antsoccer-arena and 54.2 versus 42.0 on visual-scene. The headline counts and 1.09 score ratio agree with Table 1.
  3. The component ablations are informative. Table 7 evaluates both removing distillation and removing the flow teacher across 68 tasks. DFC exceeds the quantile-only critic on 53 offline tasks and ties on the remaining 15, providing evidence that the teacher contributes beyond simply adopting a quantile critic. The direct-flow comparison also supports using a separate student for actor optimization.
  4. The experiments examine applicability beyond the main configuration. Table 8 tests larger scalar ensembles, while Table 9 reports improvements from inserting DFC into ReBRAC and IQL on eight and nine of ten tasks, respectively. These results support further investigation of DFC as a reusable critic component.
Critical weaknesses
  1. The incremental contribution over the closest methods is insufficiently established. Appendix D discusses Value Flows and floq, but neither is evaluated. Value Flows already learns return distributions with flow matching, while floq distills a flow critic for efficient policy optimization. A comparison with these methods—and a scalar student distilled from DFC’s same distributional teacher—is important to establish the specific benefit of quantile distillation.
  2. The central explanation for the gains remains under-supported. The claimed representation-regularization benefit is not isolated from additional capacity, computation, or teacher–student optimization. The MAE improvement in Section 5.3 is promising, but its tasks, state-action sampling, evaluation policy, normalization, and uncertainty are unspecified. Figure 4 also lacks sufficient labels and protocol details. These diagnostics therefore do not yet convincingly establish that DFC supplies more accurate values and useful action gradients.
Questions for authors
  1. Can you compare against Value Flows and floq under matched protocols, and test a scalar student trained from the same DFC teacher? Does quantile distillation retain an advantage after controlling for capacity and computation?
  2. Can you fully specify and reproduce the MAE and distribution-fitting evaluations, separately measuring teacher and student errors at policy-selected actions?
Reviewer_RXz1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper replaces FQL's scalar critic with a flow-matching model of the return distribution, then distills it into a quantile-conditioned one-step critic so the actor never has to backprop through the ODE. The actor maximizes the mean of the predicted quantiles, otherwise following FQL. Experiments cover 93 OGBench and D4RL tasks, offline and offline-to-online, plus ablations on the two-stage design, ensembles, M, and plugging the critic into ReBRAC and IQL.

Critical strengths
  • The evaluation seems broad (93 tasks, including 25 pixel-based ones) and per-task results are reported.
  • The distillation trick is clean and practical, and inference cost is unchanged.
  • The ablations are the right ones to run in principle (FC, DC, ensembles, transfer to other algorithms).
Critical weaknesses

Novelty seems limited, and the closest prior work is not compared. A distributional flow-matching critic is the core of Value Flows (Dong et al., ICLR 2026), and distilling a flow critic into a one-step critic for policy extraction is what floq (Agrawalla et al., ICLR 2026) does. Both are published at ICLR 2026 (as the paper's own references state), and to my knowledge both report OGBench results, so calling them "concurrent" in the appendix doesn't really hold. What is left is mainly that the student is a quantile critic rather than a scalar one. Proposition 1 is the known QR-DQN contraction result and says nothing about the actual algorithm.

The main hypothesis is never really tested. The paper attributes the gains to distributional "representation regularization", but the actor only ever uses the mean, and I see no analysis of representations. There are simpler explanations that need to be ruled out. The flow critic's target is built from M = 51 next actions per transition (Algorithm 1, line 5), while FQL, as far as I know, samples a single next action, so DFC's target variance is lower before any distributional modeling comes in. Notably, the quantile-only ablation (DC) already captures most of the gain over FQL on many tasks (scene task2: FQL 76, DC 94, DFC 96; cube-double task2: 36, 49, 49), and the flow teacher mostly helps on puzzle tasks (puzzle-4x4 task1: 34, 44, 68). I'd want to see (a) a scalar critic with the same 51-action target, and (b) the flow teacher distilled into a scalar critic instead of a quantile critic. It is also unclear whether DFC keeps FQL's two-critic aggregation or pessimism. Finally, OGBench dynamics are close to deterministic, so if return stochasticity matters, an experiment with added dynamics or reward noise would make the case.

The distributional baselines look broken. C51 with the FQL actor is reasonable on D4RL Adroit (48.2) but scores 0 to 7 on every OGBench domain, which suggests a value support that doesn't match OGBench's return range (roughly -100 to 0 at γ = 0.99) rather than a real result. IQN with the FQL actor is far worse than the authors' own quantile-only ablation (DC), e.g. scene task2: 1 vs 94, antmaze-large task2: 0 vs 69. Either DC differs from IQN in some way the paper doesn't explain, or IQN is under-tuned. EWFM is also near zero on humanoidmaze, cube-double, and both puzzle domains.

Hyperparameters and tables don't line up. The paper says DFC uses FQL's BC coefficient (Appendix B.1), but Table 4 lists α = 300 for DFC vs 1000 for FQL on both puzzle domains, which include DFC's largest gain over FQL (puzzle-4x4, 17.2 to 35.0). Table 4 lists M = 101 for several tasks, yet the main-table numbers match the M = 51 column of Table 10. Table 10 also doesn't support "robust to M" (cube-double 49 vs 74, puzzle-3x3 21 vs 52, pen-human 61 vs 84). FQL with two critics in Table 8 doesn't match FQL in Table 6 (door-human 6 vs 0). ReBRAC with the DFC critic gets 98 on puzzle-3x3 and 74 on antmaze-giant, far above FQL with DFC, which is never discussed.

The statistics are thin. Wins are counted task by task without significance tests, and several "best" domains are within one SD (visual-antmaze-medium 88.2±6 vs 87.4±4, visual-antmaze-teleport 14.6±4 vs 13.6±4). The headline "average ratio of 1.09" is pulled around by low-score domains (visual-puzzle-3x3 contributes 1.60, humanoidmaze-large 0.51). rliable-style aggregates with confidence intervals would be more convincing. On offline-to-online, DFC barely moves on several tasks after 1M online steps (puzzle-3x3 task2 36 to 37, scene task4 20 to 20), which reads more like "doesn't collapse" than "good initialization."

Value-accuracy evidence is anecdotal. The MAE comparison doesn't say which tasks, states, or units were used, and the return-distribution figure shows only three state-action pairs.

Questions for authors
  1. How exactly does DC differ from the IQN baseline, and why is the performance gap so large?
  2. Which α and M produced each reported number, and were FC and DC run with the same settings?
  3. How does DFC aggregate Q values, and does it use any pessimism?
  4. Can you report the MAE protocol in full and W1 aggregated over many states?
  5. What explains the ReBRAC + DFC result on puzzle-3x3?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.