acceptodds
Under review as a conference paper at ICLR 2027

Value Forcing: Aligning Return Distributions at All Noise Levels

Abstract

Learning accurate value functions is central to reinforcement learning (RL), yet remains challenging because supervision is limited and indirect: ground-truth values are unavailable, and each transition in the data only provides an immediate reward that can be used to construct bootstrapped targets as supervision. This challenge is particularly pronounced when data is limited or reward signals are sparse, as is common in real-world applications such as robotics. We argue that the key to accurate value learning in this *limited* data regime is extracting richer learning signals from the available data. In this paper, we introduce *Value Forcing*, a new approach that uses flow matching to estimate return distributions that enables accurate value learning from limited data. Our key insight is that we can extract richer learning signals by enforcing Bellman consistency at every noise level. Empirically, Value Forcing improves the accuracy of estimating the return distribution, and prevents the distribution from collapsing to a single value, as occurs with prior methods. We instantiate Value Forcing with two practical value-learning algorithms: VF, an actor-critic method, and IVF, an implicit value-learning method. We evaluate both algorithms across long-horizon tasks in the offline RL setting and find that they outperform prior value-learning methods, with particularly large gains when only limited data is available.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, distributional reinforcement learning, value learning, flow matching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MfxKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Value Forcing, a flow-based distributional value-learning method with a noise-dependent Bellman operator. An independent Gaussian correction preserves the initial Gaussian distribution while recovering the standard distributional Bellman equation at the endpoint. The authors derive velocity-regression objectives and instantiate explicit VF and expectile-weighted implicit IVF algorithms. Experiments evaluate return-distribution accuracy and policy performance across 20 OGBench tasks, with the strongest comparative gains at reduced dataset fractions.

Critical strengths
  1. A principled refinement of existing Bellman flow formulations. Flow-based return modeling and intermediate-time Bellman supervision already appear in Value Flows, while PCBF addresses source-distribution mismatch through coupled paths. Against this background, the paper’s specific contribution is a corrected operator enforcing consistency between intermediate marginal distributions. Its endpoint compatibility and fixed-policy contraction provide a coherent foundation for the resulting objectives (Sec. 4; Appendix A.1).
  2. Complementary distribution-level and control evaluations. Held-out return distributions and Wasserstein distances directly assess the critic, complementing success rates across multiple tasks and data budgets. The improvements on several distribution diagnostics and the reduced-data gains, particularly for IVF, provide useful evidence that the formulation can benefit learning beyond merely increasing model expressivity (Fig. 2; Tables 2–5).
Critical weaknesses
  1. The distribution-level evaluation is too localized to substantiate the broader accuracy claim. The return-distribution diagnostics evaluate only one state–action pair per domain under a fixed data-collection policy (Sec. 7.1; Appendix D.4). This does not establish accuracy across the state–action distribution encountered during control or under the policies induced by the learned critics. Improvements are also uneven: on cube-quadruple, VF’s Wasserstein distance is 179±6179\pm6, compared with 184±9184\pm9 for PCBF and 57±657\pm6 for the oracle. Consequently, the evidence for broadly preventing distribution collapse—and for connecting this improvement to policy performance—remains limited.
  2. The performance comparison lacks sufficient accounting for computational cost. VF evaluates actions using multiple generated returns and iterative flow integration. Its evaluation configuration uses 16 action candidates, 16 return samples per candidate, and 10 integration steps, while competing methods use different sampling and integration settings (Appendix D.3). Although these operations can be batched, equal gradient-step counts do not establish comparable training or inference costs. Without runtime measurements or matched-compute results, it is difficult to assess the practical cost of the reported gains over simpler critics.
Questions for authors
  1. Distribution accuracy: How were the four diagnostic state–action pairs selected, and how representative are they? Do you have results across multiple held-out pairs, particularly under the learned policies, showing that improved distribution accuracy consistently accompanies better action selection or policy performance?
  2. Computational cost: What are the training time and action-selection latency of VF and IVF relative to the strongest baselines on identical hardware? Under matched runtime budgets, or with fewer return samples and integration steps, how much of the reported performance advantage remains?
4 comments

We thank the reviewer for the detailed feedback and suggestions. The reviewer raised questions about (1) state selection in our didactic experiments, (2) the computational cost of training and evaluating VF and IVF, and (3) their performance under reduced runtime budgets. To address (1), we add experiments on two additional randomly sampled states per domain, showing that VF consistently achieves lower mean 11-Wasserstein distances than all five baselines. For (2), our timing measurements on identical hardware show that VF and IVF train faster than the evaluated flow-based baselines, with comparable action-selection latency. For (3), our existing ablations show that VF and IVF maintain similar success rates with fewer return samples and integration steps. Please let us know if you have any other concerns.

How were the four diagnostic state–action pairs selected, and how representative are they?

We randomly select one state from the 100100M offline dataset for each of the four OGBench domains. We then execute its fixed starting action chunk in the dataset and then roll out the data-collection policy to collect training and evaluation trajectories. We select a single state from each domain for better visualization and to show the return histograms. We have revised Sec. 6.1 and Appendix D.4 (blue text) to clarify the state selection.

Do you have results across multiple held-out pairs, particularly under the learned policies, showing that improved distribution accuracy consistently accompanies better action selection or policy performance?

As suggested by the reviewer, we extend the didactic experiments in Fig. 2 to two additional randomly sampled (s,achunk)(s, a_{\text{chunk}}) pairs per domain. We follow the same training and evaluation protocol as in Appendix D.4. The table below reports the 11-Wasserstein distances averaged over the four domains for each set, with the final row averaging all three sets. Visualizations of the additional pairs are included in Appendix Fig. 6.

State setOracleVF (Ours)FlowCriticValue FlowsPCBFC51IQN
Original states (Fig. 2)49±449 \pm 482±582 \pm 5167±5167 \pm 5295±5295 \pm 5157±5157 \pm 5469±5469 \pm 5189±4189 \pm 4
Additional states 1 (Fig. 6a)35±435 \pm 454±354 \pm 3131±4131 \pm 4256±4256 \pm 4112±4112 \pm 4432±4432 \pm 4151±3151 \pm 3
Additional states 2 (Fig. 6b)39±439 \pm 458±358 \pm 3143±4143 \pm 4239±4239 \pm 4138±4138 \pm 4472±4472 \pm 4171±3171 \pm 3
Average over all three sets41±241 \pm 265±265 \pm 2147±2147 \pm 2263±3263 \pm 3136±2136 \pm 2458±3458 \pm 3170±2170 \pm 2

These results suggest that VF consistently achieves lower mean 11-Wasserstein distances than all five baselines at every evaluated pair.

Regarding evaluation under learned policies, different methods learn different policies, which induce different ground-truth return distributions. Comparing estimation errors under these policies therefore reflects differences in both the estimators and the policies being evaluated. We use a common scripted policy to isolate return-estimation accuracy from these policy differences.

Empirically, VF’s lower return estimation errors in Figs. 2 and 6 accompany its performance gains in Fig. 3, suggesting a positive relationship. However, this relationship is not guaranteed: rejection sampling depends on the ranking of candidate actions by estimated QQ-values, so lower distributional error need not improve action selection.

What are the training time and action-selection latency of VF and IVF relative to the strongest baselines on identical hardware?

As suggested by the reviewer, we conduct ablation experiments comparing the training time and action-selection latency of VF and IVF to selected baselines. These experiments help us assess the computational cost of their gains in success rates.

Specifically, we measure (1) the wall-clock time for 100100K gradient updates using the common training settings in Table 6, and (2) the wall-clock time to select a single action chunk. All measurements use the same NVIDIA H200 GPU. We compare VF and IVF against FlowCritic, Value Flows, C51, and QC among explicit methods, and Implicit C51 and Implicit QC among implicit methods. We report the mean of timings over 4 tasks and 4 random seeds in the table below.

MethodTraining time (mins)Time to sample 11 action chunk (ms)
VF (E, Ours)13.613.61.451.45
FlowCritic (E)18.018.01.501.50
Value Flows (E)22.522.51.221.22
C51 (E)15.515.50.920.92
QC (E)15.415.40.910.91
IVF (I, Ours)11.211.21.431.43
Implicit C51 (I)5.65.60.910.91
Implicit QC (I)5.65.60.980.98

“E/I” denotes explicit/implicit methods. Lower times indicate faster training and action selection. VF trains faster than all four explicit baselines, with action-selection latency comparable to FlowCritic and Value Flows. IVF also trains faster than these flow-based critics, but takes twice as long as Implicit C51 and Implicit QC. This reflects the additional return-flow integration used to compute expectile weights during training. Its action-selection latency is likewise comparable to the flow-based critics, but approximately 1.5×1.5 \times that of the implicit baselines, as it integrates the return flow to estimate QQ.

Under matched runtime budgets, or with fewer return samples and integration steps, how much of the reported performance advantage remains?

We have examined the number of return samples and integration steps in Fig. 9 and Appendix B.2, using the same four ablation tasks above. We have added a new paragraph at the end of Sec. 5 to emphasize these ablations.

These ablation experiments show that VF and IVF maintain similar success rates across different runtime budgets during training, including using only one return sample and one integration step. Therefore, reducing runtime budgets has minor effects on the Value Forcing algorithms. We select the default training hyperparameters (Appendix D.3), number of return flow steps =10= 10 for both VF and IVF, and number of return samples M=1M = 1 for VF and M=16M = 16 for IVF, to match the compute budgets for baselines.

Thank you for the clarifications and additional experiments. The random state-selection protocol and expanded diagnostic experiments alleviate my concerns about the original examples. The timing results also clarify the computational trade-offs: VF trains faster than the evaluated explicit baselines, while IVF incurs additional cost relative to the implicit baselines. The additional OCBench results strengthen the empirical evaluation, particularly VF’s improvement from 60% to 67% over the strongest evaluated explicit baseline with 25% of the data.

Some limitations remain. The connection between distribution-estimation accuracy and action-selection quality is still indirect, and comparative performance under matched wall-clock budgets remains insufficiently established, especially for IVF. Nevertheless, the response sufficiently alleviates my main concerns. Considering the methodological contribution and expanded empirical evidence, I am raising my rating from 2 to 3.

Thank you for your thoughtful reply and for confirming that our response has addressed your main concerns. We will incorporate the additional experiments and clarifications into the final version, while mentioning the limitations.

Reviewer_hnoAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Value Forcing, a flow-matching-based approach to distributional value learning that enforces Bellman consistency at all noise levels through a Gaussian-corrected Flow Bellman operator. The authors develop two offline RL algorithms, VF and IVF, and evaluate them on 20 OGBench tasks, demonstrating improved return distribution estimation and policy performance, particularly in limited-data settings.

Critical strengths

S1. The paper introduces a well-motivated Flow Bellman operator, supported by theoretical analysis of its contraction property and unique fixed point.

S2. The paper provides extensive experiments across multiple offline RL tasks and dataset sizes, including comparisons with relevant baselines, return distribution analyses and ablation studies.

Critical weaknesses

W1. Insufficient validation of the core contribution. PCBF already addresses the Gaussian source-boundary inconsistency through shared-noise coupling, while the paper lacks direct ablations to isolate the contribution of Gaussian correction and the intermediate-noise Bellman consistency.

W2. Insufficient justification of IVF's theoretical guarantees. The paper claims that expectile-weighted flow matching recovers expectile values and approaches the optimal value within the dataset support. However, the analysis relies on zero regression loss at the fixed point, which may not hold for general action-dependent return distributions, leaving the applicability of these guarantees unclear.

W3. Unclear fairness of experimental comparisons. Reward scaling substantially affects performance, but equivalent normalization across baselines is not clearly established. Differences in Q-value estimation and evaluation compute may also affect the comparisons.

Questions for authors

Q1. Can the authors separately ablate Gaussian correction and intermediate-noise Bellman constraints to clarify their contributions beyond existing methods?

Q2. Can the authors justify the zero-loss assumption or clarify whether the theoretical guarantees remain valid when the optimal regression loss is nonzero?

Q3. Do all baselines use equivalent reward normalization? How do differences in Q-value estimation and evaluation compute affect the reported performance?

3 comments

We thank the reviewer for the detailed feedback and suggestions. The reviewer raised questions about (1) the contributions of Gaussian correction and Bellman consistency at intermediate noise levels, (2) the fairness and computational cost of the comparisons, and (3) the assumptions underlying the theoretical guarantees. To address (1), we include new ablations showing that both components improve average policy performance. For (2), we compare baselines with and without reward scaling: most methods perform similarly, while Value Flows performs substantially worse with scaling. We also highlight existing ablations showing that VF and IVF maintain similar success rates with fewer return samples and integration steps. For (3), we clarify that our theoretical analyses allow nonzero optimal regression loss and have removed the corresponding zero-loss statements. Please let us know if you have any other concerns.

Can the authors separately ablate Gaussian correction and intermediate-noise Bellman constraints to clarify their contributions beyond existing methods?

As suggested by the reviewer, we compare VF and IVF with two separate ablations: (1) removing the Gaussian correction from both the noisy input and velocity target (Eq. 13), and (2) enforcing Bellman consistency only at the endpoints (τ∈{0,1}\tau \in \{0,1\} in Eqs. 12, 13, and 16). Following Sec. 6.3, we conduct these experiments on task 2 and task 4 of cube-triple\texttt{cube-triple} and puzzle-4x4\texttt{puzzle-4x4}, keeping all other training and evaluation settings unchanged.

Ablation of Gaussian correction:

TaskVFVF w/o Gaussian correctionIVFIVF w/o Gaussian correction
cube-triple-task2\texttt{cube-triple-task2}87±587 \pm 583±2183 \pm 2194±894 \pm 888±1588 \pm 15
cube-triple-task4\texttt{cube-triple-task4}59±2259 \pm 2228±528 \pm 554±1454 \pm 1440±2440 \pm 24
puzzle-4x4-task2\texttt{puzzle-4x4-task2}99±299 \pm 295±1095 \pm 1079±1279 \pm 1253±2553 \pm 25
puzzle-4x4-task4\texttt{puzzle-4x4-task4}100±0100 \pm 0100±0100 \pm 078±378 \pm 360±860 \pm 8
Average86±686 \pm 676±576 \pm 576±576 \pm 560±1660 \pm 16

Removing Gaussian correction reduces the average success by 1010 for VF and 1616 for IVF. These results support the contribution of Gaussian correction to policy performance.

Ablation of intermediate Bellman constraints:

TaskVFVF, endpoints onlyIVFIVF, endpoints only
cube-triple-task2\texttt{cube-triple-task2}87±587 \pm 50±00 \pm 094±894 \pm 80±00 \pm 0
cube-triple-task4\texttt{cube-triple-task4}59±2259 \pm 220±00 \pm 054±1454 \pm 147±57 \pm 5
puzzle-4x4-task2\texttt{puzzle-4x4-task2}99±299 \pm 299±299 \pm 279±1279 \pm 1298±298 \pm 2
puzzle-4x4-task4\texttt{puzzle-4x4-task4}100±0100 \pm 0100±0100 \pm 078±378 \pm 398±298 \pm 2
Average86±686 \pm 650±050 \pm 076±576 \pm 551±251 \pm 2

Restricting Bellman consistency to the endpoints (τ∈{0,1}\tau \in \{0,1\}) reduces average success by 3636 for VF and 2525 for IVF. Although the endpoint-only supervision leaves VF's mean success unchanged and improves IVF's mean success on puzzle-4x4\texttt{puzzle-4x4}, the average performance over all four evaluated tasks decreases.

Together, these ablations show that both Gaussian correction and Bellman constraints at intermediate noise levels are important components of Value Forcing. The flow Bellman equation in Sec. 4.1 is a principled way to learn the return distribution.

Do all baselines use equivalent reward normalization?

The baselines in Fig. 3 use unscaled rewards, whereas VF and IVF multiply rewards by 1−γ1-\gamma. To assess this difference, we compare selected baselines with and without the same scaling: Value Flows, C51, and QC among explicit methods, and Implicit C51 and Implicit QC among implicit methods. We again evaluate task 2 and task 4 of cube-triple\texttt{cube-triple} and puzzle-4x4\texttt{puzzle-4x4}, keeping all other training and evaluation settings unchanged. Below, we report the success rates averaged over the 44 tasks and use “E/I” to denote explicit/implicit methods.

Methodw/o reward scalingw/ reward scaling
VF (E, Ours)11±411 \pm 486±686 \pm 6
Value Flows (E)75±775 \pm 711±511 \pm 5
C51 (E)79±479 \pm 478±978 \pm 9
QC (E)82±782 \pm 782±582 \pm 5
IVF (I, Ours)11±211 \pm 276±576 \pm 5
Implicit C51 (I)82±482 \pm 483±383 \pm 3
Implicit QC (I)64±1064 \pm 1070±170 \pm 1

In this table, the selected baselines perform similarly with and without reward scaling, with one exception: adding reward scaling dramatically decreases the performance of Value Flows. These results suggest that reward normalization does not significantly affect the performance of most baselines.

How do differences in Q-value estimation and evaluation compute affect the reported performance?

We interpret this question as asking whether the compute budget for sampling returns during training affects the performance of VF and IVF. Please let us know if the reviewer intended a different aspect.

We have examined the number of return samples and integration steps during training in Fig. 9 and Appendix B.2, using the same four ablation tasks as in Sec. 6.3. We have added a new paragraph at the end of Sec. 5 to emphasize these ablations.

These ablation experiments show that VF and IVF maintain similar success rates across different runtime budgets during training, including using only one return sample and one integration step. Therefore, reducing runtime budgets has minor effects on the Value Forcing algorithms. We select the default training hyperparameters (Appendix D.3), number of return flow steps =10= 10 for both VF and IVF, number of return samples M=1M = 1 for VF and M=16M = 16 for IVF, to match the compute budgets for baselines.

Can the authors justify the zero-loss assumption or clarify whether the theoretical guarantees remain valid when the optimal regression loss is nonzero?

Our analysis in Appendix A.2 assumes exact minimization of both regression objectives and convergence of the velocities to a self-consistent fixed point. The minimum regression losses need not be zero. We have removed the zero regression loss in Eq. 22 and Eq. 23 to clarify.

Dear Reviewer hnoA,

Thank you for your time and effort in reviewing our paper and for your thoughtful comments and questions.

With the discussion period nearing its end, we wanted to follow up on our responses and revisions. We would be happy to clarify any points or address any remaining concerns you may have.

Best regards,

The Authors

Reviewer_kYXeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This work introduces a new method to model the return distribution in offline RL using flow matching. It proposes a new Bellman operator applied at every noise level and proves that this operator is a contraction at every noise level for a fixed policy. It instantiates this idea with two methods: Value Forcing (VF) and Implicit Value Forcing (IVF). It compares performance in policy evaluation, where VF sometimes obtains a substantial improvement over the other flow baselines. It also benchmarks VF and IVF on offline control tasks, where they obtain a small on average but consistent advantage.

Critical strengths
  • The work is mostly well written and easy to follow.
  • The idea of value forcing at every noise level, if new, is both simple and elegant. The proof of contraction for the new Bellman operator is a very nice addition.
  • The available results show a clear advantage over the flow-based baselines in policy evaluation and a small to moderate advantage on control tasks.
  • The ablations in the main part answer important questions.
Critical weaknesses
  • The motivation, specific to temporal-difference learning, for value forcing at every noise level is a bit weak in my opinion. Additional supervision is provided to train the neural networks, but why this should help specifically in temporal-difference learning deserves more explanation. (Related question below.)
  • It is not obvious to me why a flow is needed in the first place to model a complex 1D distribution. C51 and IQN can be easily scaled up by using more bins, more quantiles, or larger networks.
  • Other strong baselines in distributional RL would strengthen the comparison, especially on the distribution parametrization side. The return distribution can be modelled using sample-based parametrizations, which could easily be scaled up (https://arxiv.org/abs/2001.02652, https://arxiv.org/abs/2007.12354).
  • Figure 2 does not include the standard baselines of distributional RL (IQN and C51), which have very good performance on control. A well-tuned C51, QR-DQN, IQN, or sample-based method could perform very well here.
  • When the main variant of the method using V and Q flows is introduced, it is not evident to me why this is the one you picked. The ablation shows that using both has little impact. I think this should be motivated.

Smaller remarks:

  • The conclusion is very rushed.
  • The paragraph on IVF is very rushed. The results in Appendix A.2 seem important but are only given a few lines in the main text.
  • Figure 3: the colours of the bars are hard to distinguish.
  • Places that lack a reference to support the claims:
    • "This scheme has limited expressivity and may give a sparse supervision signal when used as the target for TD learning."
    • "Learning to distinguish these two actions and their outcomes can enable more effective use of the available data."

If some of these remarks are addressed, I am open to raising my score.

Questions for authors
  • Is there a mathematical argument, specific to temporal difference, for why enforcing consistency at every noise level should lead to more accurate or faster policy evaluation, especially since the return distribution we ultimately want is at the flow endpoint (τ = 1) ?
  • Why would a flow be needed to model a complex 1D distribution in the first place ? Being clear about this early on in the main part would be beneficial.
  • In Figures 2 and 3, are the baselines evaluated fairly in terms of 1) architectures and 2) number of parameters ? Where it makes sense, did you try the same architecture with different training algorithms ?
  • In Appendix D.3, you mention using the twin Q trick from TD3. I do not understand how you apply this in the distributional case or which prior work this follows. Is this related to TQC? If so, how does it work with flows ?
5 comments

We thank the reviewer for the thoughtful feedback and suggestions. The main concerns are (1) the motivations for enforcing Bellman consistency at every noise level and using flows to model the 1D return distribution, (2) the breadth and fairness of the empirical comparisons, and (3) algorithmic design and implementation choices. To address (1), we have added theoretical analysis showing that Bellman consistency at every noise level can reduce TD gradient variance. For (2), we added comparisons with C51 and IQN on return prediction, and with MMDRL on selected offline RL tasks. We also compare parameter counts to assess fairness. For (3), we clarify the role of VV-flow in VF and IVF, and our use of ensemble averaging. Please let us know if you have any other concerns.

Is there a mathematical argument, specific to temporal difference, for why enforcing consistency at every noise level should lead to more accurate or faster policy evaluation, especially since the return distribution we ultimately want is at the flow endpoint (τ = 1)?

Yes. We have added new theoretical results in Appendix A.3 showing that Value Forcing can reduce gradient variance compared with flow matching to TD targets constructed from endpoint return samples (τ=1\tau=1). Our analysis adapts the variance reduction argument in Theorem 7 of Farebrother et al. (2025) [1].

We formally define both objectives and state the assumptions used in the comparison. Under a shared, fixed bootstrap model and the interpolation-path assumption in Eq. 27, we show that

\text\{Var\}(g^{\text\{VF\}}) + \mathbb{E} \left[\\|g^{\mathrm{End}}-g^{\mathrm{VF}}\\|_2^2\right] \geq \operatorname{Var}(g^{\mathrm{VF}}),$$ where $g^{\mathrm{End}}$ and $g^{\mathrm{VF}}$ are the gradient estimates of the endpoint-only and Value Forcing objectives, respectively, and $\operatorname{Var}$ denotes total gradient variance. Please see Appendix A.3 for the full derivation. > Figure 2 does not include the standard baselines of distributional RL (IQN and C51), which have very good performance on control. Thank you for the suggestion. We have added C51 and IQN to Fig. 2 using the same training trajectories and evaluation protocol as the other methods. Following Appendix D.3, we use $51$ categorical atoms for C51 and $8$ sampled quantile fractions for IQN. The table below reports $1$-Wasserstein distances averaged over the four evaluated domains. The complete results can be found in the revised Fig. 2. || Oracle | VF (Ours) | FlowCritic | Value Flows | PCBF | C51 | IQN | | :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | $1$-Wasserstein distances | $49 \pm 4$ | $82 \pm 5$ | $167 \pm 5$ | $295 \pm 5$ | $157 \pm 5$ | $469 \pm 5$ | $189 \pm 4$ | These results suggest that VF still achieves the lowest mean distance among the five baselines after adding C51 and IQN: VF $= 82$, versus C51 $= 469$ and IQN $= 189$. The advantage of using VF to estimate the full return distribution extends to standard categorical and quantile baselines under the evaluated settings. > Why would a flow be needed to model a complex 1D distribution in the first place? Being clear about this early on in the main part would be beneficial. We agree that flow matching is not strictly necessary to model the 1D return distribution. However, prior distributional RL methods such as C51 and QR-DQN require a large number of categorical atoms [2] or quantiles [3] to accurately estimate the return distribution (see empirical studies below), which is computationally expensive (see experiments below). Flow matching provides an expressive continuous representation without specifying categorical bins or a fixed set of quantile locations. This motivation also appears in prior work [4, 5], and we have revised the introduction (blue text) to clarify it. To study the effects of the number of categorical atoms on C51, we conduct ablation experiments using the same setting as in Fig. 2. Specifically, we vary C51’s number of atoms within $\\{51 \text{(default)}, 201, 1001 \\}$. Below, we report the $1$-Wasserstein distance averaged over the four domains and the average training time on an NVIDIA A100 GPU. | **Method** | **Number of atoms** | **$1$-Wasserstein distance** | **Training time (min)** | | :--- | :--- | ---: | ---: | | **VF (Ours)** | — | $82 \pm 5$ | $2.10$ | | C51 | $51$ (default) | $469 \pm 5$ | $2.04$ | | | $201$ | $319 \pm 4$ | $2.12$ | | | $1001$ | $240 \pm 6$ | $2.46$ | Increasing C51 from $51$ to $1001$ atoms reduces its mean $1$-Wasserstein distance by $49%$, while increasing training time from $2.04$ to $2.46$ minutes, approximately $1.21\times$. This narrows the accuracy gap, but C51 still has a higher error than VF ($240$ vs. $82$) and takes longer to train ($2.46$ vs. $2.10$ minutes). These results support flow matching as an effective choice for accurate return estimation in this setting.

Other strong baselines in distributional RL would strengthen the comparison, especially on the distribution parametrization side. The return distribution can be modelled using sample-based parametrizations, which could easily be scaled up (https://arxiv.org/abs/2001.02652, https://arxiv.org/abs/2007.12354).

As suggested by the reviewer, we include a comparison against MMDRL [6], which models distributions using particles trained with maximum mean discrepancy, on selected OGBench domains. MMDRL is the strongest explicit baseline in this comparison, while VF achieves higher average success (VF=0.606\text{VF} = 0.606 vs. MMDRL=0.554\text{MMDRL} = 0.554). We describe the experiment details below.

MMDRL is an explicit method because it learns an explicit policy to sample the TD targets. We also use flow rejection sampling to parameterize its policy, which is consistent with other methods. We conduct experiments on 10 tasks from OGBench domains cube-triple\texttt{cube-triple} and cube-quadruple\texttt{cube-quadruple} with 25%25\% and 100%100\% of the data and report the mean and standard deviation across four seeds:

DomainDatasetValue Flows (E)C51 (E)QC (E)MMDRL (E)VF (E)│C51 (I)Implicit QC (I)IVF (I)
cube-triple\texttt{cube-triple}2525\\%74±674 \pm 671±671 \pm 668±1168 \pm 1179±1079 \pm 1090±3\mathbf{90 \pm 3}│66±666 \pm 656±656 \pm 686±7\mathbf{86 \pm 7}
100100\\%64±864 \pm 886±4\mathbf{86 \pm 4}81±881 \pm 889±6\mathbf{89 \pm 6}88±7\mathbf{88 \pm 7}│90±6\mathbf{90 \pm 6}87±10\mathbf{87 \pm 10}87±5\mathbf{87 \pm 5}
cube-quadruple\texttt{cube-quadruple}2525\\%17±917 \pm 916±916 \pm 98±68 \pm 620±520 \pm 525±9\mathbf{25 \pm 9}│9±69 \pm 67±87 \pm 820±5\mathbf{20 \pm 5}
100100\\%28±828 \pm 817±1517 \pm 1527±827 \pm 835±635 \pm 640±20\mathbf{40 \pm 20}│19±419 \pm 431±831 \pm 836±12\mathbf{36 \pm 12}
Average—46±646 \pm 647±347 \pm 346±446 \pm 455±255 \pm 261±9\mathbf{61 \pm 9}│46±246 \pm 245±345 \pm 357±3\mathbf{57 \pm 3}

Bold indicates success rates at or above 9595% of the best in the row. “E/I” denotes explicit/implicit methods.

In Figures 2 and 3, are the baselines evaluated fairly in terms of 1) architectures and 2) number of parameters? Where it makes sense, did you try the same architecture with different training algorithms?

Yes, whenever compatible, we use the common MLP backbone and hyperparameters in Table 6. Table 7 lists domain-specific hyperparameters.

To make the differences in model size transparent, we compare the number of trainable parameters of different methods averaged over the 44 OGBench domains in the Table below. We use “E/I” to denote explicit/implicit methods. Our methods VF (vQv_Q only) and IVF have the number of trainable parameters comparable to most explicit and implicit baselines, respectively.

MethodVF (E, Ours)VF, vQv_Q only (E, Ours)Floq (E)FlowCritic (E)Value Flows (E)PCBF (E)C51 (E)IQN (E)QC (E)│IVF (I, Ours)DIVL (I)C51 (I)IQN (I)QC (I)
Parameters (M)16.3916.399.919.9114.3814.386.626.629.919.919.919.919.989.989.729.729.879.87│9.919.9115.1415.1413.2513.2513.6713.6713.0913.09

VF uses 16.3916.39M parameters because it learns a separate VV-velocity. Its QQ-only variant reduces this to 9.919.91M, matching Value Flows and PCBF. IVF also uses 9.919.91M parameters, fewer than all listed implicit baselines.

When the main variant of the method using V and Q flows is introduced, it is not evident to me why this is the one you picked. The ablation shows that using both has little impact. I think this should be motivated.

We agree that a separate VV-flow is optional for VF, since the QQ-flow can directly provide bootstrapping targets. Our IVF formulation, however, requires both flows to compute the expectile weight in Eq. 15. Applying expectile regression directly to QQ-flow can instead favor “lucky” transitions with high returns, as discussed in Sec. 4.2 of Kostrikov et al. (2021) [7]. We added a paragraph in Sec. 4.2 (blue text) to clarify this motivation and emphasize that learning only a QQ-flow is a valid implementation of VF.

In Appendix D.3, you mention using the twin Q trick from TD3. I do not understand how you apply this in the distributional case or which prior work this follows. Is this related to TQC? If so, how does it work with flows?

The twin Q trick refers to learning an ensemble of two VV and QQ velocities and averaging the predictions from them: we compute the return prediction or Q-estimate from each ensemble member in parallel using the same initial noise and take the mean of the final prediction. Our aggregation uses the mean, whereas TD3 uses the minimum. Prior flow-based distributional RL methods [4, 8] also adopt this trick. Specifically, we use ensemble averaging for:

  1. At line 4 in Algorithm 1, when performing rejection sampling in VF, we compute QQ-estimate from each ensemble member and take the average.
  2. At line 4 in Algorithm 1, we sample the noisy return zτz^\tau in VF from each ensemble member and take the average.
  3. At line 6 in Algorithm 1, when computing the expectile weight in IVF, we compute both VV-estimate and QQ-estimate from each ensemble member and take the average.
  4. At line 6 in Algorithm 1, we sample the noisy return zτz^\tau in IVF from each ensemble member and take the average.

We have revised the VF and IVF paragraphs in Appendix D.3 to clarify these aggregation rules.

The conclusion is very rushed. The paragraph on IVF is very rushed. The results in Appendix A.2 seem important but are only given a few lines in the main text.

We have expanded the conclusion (blue text) to discuss limitations and future directions. We have also expanded the IVF paragraph in Sec. 5 (blue text) to explain the expectile weighting and highlight the main results from Appendix A.2: the connection to the scalar expectile regression loss in Kostrikov et al. (2021) [7] and the recovery of the optimal value within the dataset support as μ→1\mu \to 1.

Figure 3: the colours of the bars are hard to distinguish.

We have revised Fig. 3 with more distinct colors and added hatching for methods without flow matching to make the bars easier to distinguish.

Places that lack a reference to support the claims: "This scheme has limited expressivity and may give a sparse supervision signal when used as the target for TD learning." "Learning to distinguish these two actions and their outcomes can enable more effective use of the available data."

Thanks for the suggestions. We have added a reference to Bellemare et al. (2017) [2] in the first sentence and explicitly cited Wang et al. (2023) [9] in the second sentence.

[1] Farebrother, J., Pirotta, M., Tirinzoni, A., Munos, R., Lazaric, A., & Touati, A. (2025). Temporal difference flows. arXiv preprint arXiv:2503.09817.

[2] Bellemare, M. G., Dabney, W., & Munos, R. (2017, July). A distributional perspective on reinforcement learning. In International conference on machine learning (pp. 449-458). Pmlr.

[3] Dabney, W., Rowland, M., Bellemare, M., & Munos, R. (2018, April). Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI conference on artificial intelligence (Vol. 32, No. 1).

[4] Dong, P., Zheng, C., Finn, C., Sadigh, D., & Eysenbach, B. (2026, April). Value flows. In International Conference on Learning Representations (Vol. 2026, pp. 128614-128646).

[5] Zhong, S., Ding, S., Diao, H., Wang, X., Teh, K. C., & Peng, B. (2026). Flowcritic: Bridging value estimation with flow matching in reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6] Nguyen-Tang, T., Gupta, S., & Venkatesh, S. (2021, May). Distributional reinforcement learning via moment matching. In Proceedings of the AAAI conference on artificial intelligence (Vol. 35, No. 10, pp. 9144-9152).

[7] Kostrikov, I., Nair, A., & Levine, S. (2021). Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169.

[8] Agrawalla, B., Nauman, M., Agrawal, K., & Kumar, A. (2026, April). floq: Training critics via flow-matching for scaling compute in value-based rl. In International Conference on Learning Representations (Vol. 2026, pp. 54569-54595).

[9] Wang, K., Zhou, K., Wu, R., Kallus, N., & Sun, W. (2023). The benefits of being distributional: Small-loss bounds for reinforcement learning. Advances in neural information processing systems, 36, 2275-2312.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Additional benchmark results

We have expanded our evaluation to six OCBench [1] tasks from domains block-lite-single\texttt{block-lite-single}, block-lite-double\texttt{block-lite-double}, and chamber-lite-easy\texttt{chamber-lite-easy}, using 25%25\% and 100%100\% of the available data. The table below reports final success rates after 500500K gradient steps over 44 random seeds. Below, we report the mean and standard deviation across four seeds, averaged over the six tasks.

DatasetValue Flows (E)C51 (E)QC (E)VF (E, Ours)│C51 (I)QC (I)IVF (I, Ours)
25%25\%60±360 \pm 358±358 \pm 345±545 \pm 567±2\mathbf{67 \pm 2}│51±351 \pm 357±3\mathbf{57 \pm 3}58±2\mathbf{58 \pm 2}
100%100\%85±285 \pm 291±2\mathbf{91 \pm 2}62±562 \pm 592±1\mathbf{92 \pm 1}│86±286 \pm 280±180 \pm 191±1\mathbf{91 \pm 1}

Bold indicates success rates at or above 9595\\% of the best in the row. “E/I” denotes explicit/implicit methods.

VF achieves the highest mean success among the evaluated explicit methods at both dataset sizes, with a larger advantage when data is limited: 6767 versus 6060 for the strongest baseline at 2525\\% data. IVF performs comparably to the strongest implicit baseline at 2525\\% data and achieves higher mean success with the full dataset (9191 versus 8686). These results are included in the revised Fig. 3 and Tables 3 and 5, with experimental details in Appendix D.1.

[1] Park, S., & Levine, S. (2026). Behavioral Cloning Mystery. arXiv preprint arXiv:2610.07056.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.