- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes Value Forcing, a flow-based distributional value-learning method with a noise-dependent Bellman operator. An independent Gaussian correction preserves the initial Gaussian distribution while recovering the standard distributional Bellman equation at the endpoint. The authors derive velocity-regression objectives and instantiate explicit VF and expectile-weighted implicit IVF algorithms. Experiments evaluate return-distribution accuracy and policy performance across 20 OGBench tasks, with the strongest comparative gains at reduced dataset fractions.
Critical strengths
- A principled refinement of existing Bellman flow formulations. Flow-based return modeling and intermediate-time Bellman supervision already appear in Value Flows, while PCBF addresses source-distribution mismatch through coupled paths. Against this background, the paper’s specific contribution is a corrected operator enforcing consistency between intermediate marginal distributions. Its endpoint compatibility and fixed-policy contraction provide a coherent foundation for the resulting objectives (Sec. 4; Appendix A.1).
- Complementary distribution-level and control evaluations. Held-out return distributions and Wasserstein distances directly assess the critic, complementing success rates across multiple tasks and data budgets. The improvements on several distribution diagnostics and the reduced-data gains, particularly for IVF, provide useful evidence that the formulation can benefit learning beyond merely increasing model expressivity (Fig. 2; Tables 2–5).
Critical weaknesses
- The distribution-level evaluation is too localized to substantiate the broader accuracy claim. The return-distribution diagnostics evaluate only one state–action pair per domain under a fixed data-collection policy (Sec. 7.1; Appendix D.4). This does not establish accuracy across the state–action distribution encountered during control or under the policies induced by the learned critics. Improvements are also uneven: on cube-quadruple, VF’s Wasserstein distance is , compared with for PCBF and for the oracle. Consequently, the evidence for broadly preventing distribution collapse—and for connecting this improvement to policy performance—remains limited.
- The performance comparison lacks sufficient accounting for computational cost. VF evaluates actions using multiple generated returns and iterative flow integration. Its evaluation configuration uses 16 action candidates, 16 return samples per candidate, and 10 integration steps, while competing methods use different sampling and integration settings (Appendix D.3). Although these operations can be batched, equal gradient-step counts do not establish comparable training or inference costs. Without runtime measurements or matched-compute results, it is difficult to assess the practical cost of the reported gains over simpler critics.
Questions for authors
- Distribution accuracy: How were the four diagnostic state–action pairs selected, and how representative are they? Do you have results across multiple held-out pairs, particularly under the learned policies, showing that improved distribution accuracy consistently accompanies better action selection or policy performance?
- Computational cost: What are the training time and action-selection latency of VF and IVF relative to the strongest baselines on identical hardware? Under matched runtime budgets, or with fewer return samples and integration steps, how much of the reported performance advantage remains?
4 comments
What are the training time and action-selection latency of VF and IVF relative to the strongest baselines on identical hardware?
As suggested by the reviewer, we conduct ablation experiments comparing the training time and action-selection latency of VF and IVF to selected baselines. These experiments help us assess the computational cost of their gains in success rates.
Specifically, we measure (1) the wall-clock time for K gradient updates using the common training settings in Table 6, and (2) the wall-clock time to select a single action chunk. All measurements use the same NVIDIA H200 GPU. We compare VF and IVF against FlowCritic, Value Flows, C51, and QC among explicit methods, and Implicit C51 and Implicit QC among implicit methods. We report the mean of timings over 4 tasks and 4 random seeds in the table below.
| Method | Training time (mins) | Time to sample action chunk (ms) |
|---|---|---|
| VF (E, Ours) | ||
| FlowCritic (E) | ||
| Value Flows (E) | ||
| C51 (E) | ||
| QC (E) | ||
| IVF (I, Ours) | ||
| Implicit C51 (I) | ||
| Implicit QC (I) |
“E/I” denotes explicit/implicit methods. Lower times indicate faster training and action selection. VF trains faster than all four explicit baselines, with action-selection latency comparable to FlowCritic and Value Flows. IVF also trains faster than these flow-based critics, but takes twice as long as Implicit C51 and Implicit QC. This reflects the additional return-flow integration used to compute expectile weights during training. Its action-selection latency is likewise comparable to the flow-based critics, but approximately that of the implicit baselines, as it integrates the return flow to estimate .
Under matched runtime budgets, or with fewer return samples and integration steps, how much of the reported performance advantage remains?
We have examined the number of return samples and integration steps in Fig. 9 and Appendix B.2, using the same four ablation tasks above. We have added a new paragraph at the end of Sec. 5 to emphasize these ablations.
These ablation experiments show that VF and IVF maintain similar success rates across different runtime budgets during training, including using only one return sample and one integration step. Therefore, reducing runtime budgets has minor effects on the Value Forcing algorithms. We select the default training hyperparameters (Appendix D.3), number of return flow steps for both VF and IVF, and number of return samples for VF and for IVF, to match the compute budgets for baselines.
Thank you for the clarifications and additional experiments. The random state-selection protocol and expanded diagnostic experiments alleviate my concerns about the original examples. The timing results also clarify the computational trade-offs: VF trains faster than the evaluated explicit baselines, while IVF incurs additional cost relative to the implicit baselines. The additional OCBench results strengthen the empirical evaluation, particularly VF’s improvement from 60% to 67% over the strongest evaluated explicit baseline with 25% of the data.
Some limitations remain. The connection between distribution-estimation accuracy and action-selection quality is still indirect, and comparative performance under matched wall-clock budgets remains insufficiently established, especially for IVF. Nevertheless, the response sufficiently alleviates my main concerns. Considering the methodological contribution and expanded empirical evidence, I am raising my rating from 2 to 3.
Thank you for your thoughtful reply and for confirming that our response has addressed your main concerns. We will incorporate the additional experiments and clarifications into the final version, while mentioning the limitations.
We thank the reviewer for the detailed feedback and suggestions. The reviewer raised questions about (1) state selection in our didactic experiments, (2) the computational cost of training and evaluating VF and IVF, and (3) their performance under reduced runtime budgets. To address (1), we add experiments on two additional randomly sampled states per domain, showing that VF consistently achieves lower mean 1-Wasserstein distances than all five baselines. For (2), our timing measurements on identical hardware show that VF and IVF train faster than the evaluated flow-based baselines, with comparable action-selection latency. For (3), our existing ablations show that VF and IVF maintain similar success rates with fewer return samples and integration steps. Please let us know if you have any other concerns.
We randomly select one state from the 100M offline dataset for each of the four OGBench domains. We then execute its fixed starting action chunk in the dataset and then roll out the data-collection policy to collect training and evaluation trajectories. We select a single state from each domain for better visualization and to show the return histograms. We have revised Sec. 6.1 and Appendix D.4 (blue text) to clarify the state selection.
As suggested by the reviewer, we extend the didactic experiments in Fig. 2 to two additional randomly sampled (s,achunk) pairs per domain. We follow the same training and evaluation protocol as in Appendix D.4. The table below reports the 1-Wasserstein distances averaged over the four domains for each set, with the final row averaging all three sets. Visualizations of the additional pairs are included in Appendix Fig. 6.
These results suggest that VF consistently achieves lower mean 1-Wasserstein distances than all five baselines at every evaluated pair.
Regarding evaluation under learned policies, different methods learn different policies, which induce different ground-truth return distributions. Comparing estimation errors under these policies therefore reflects differences in both the estimators and the policies being evaluated. We use a common scripted policy to isolate return-estimation accuracy from these policy differences.
Empirically, VF’s lower return estimation errors in Figs. 2 and 6 accompany its performance gains in Fig. 3, suggesting a positive relationship. However, this relationship is not guaranteed: rejection sampling depends on the ranking of candidate actions by estimated Q-values, so lower distributional error need not improve action selection.