- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
For training flow-based critics in offline RL, this paper identifies structural biases in existing methods that deviates from the ground-truth return distributions and proposes Retimed Bellman Flows(ReBF). ReBF demonstrates that expressing the student flow input as an affine transformation of the successor flow input and the immediate reward requires retiming the teacher to , together with decoupling the successor return generation noise from the student's initial noise. It is proved that these conditions are required to yield conditionally unbiased velocity bootstrapping. Across every synthetic Markov reward processes and OGBench, ReBF consistently achieves lower Wasserstein distances to the ground-truth return distributions than existing baselines. These improvements were translated into better downstream offline RL performance on several OGBench and D4RL benchmarks.
Critical strengths
- The paper developed a theoretical guarantee of unbiasedness beyond empirical design.
- The authors improve the explainability of the proposed design knobs by additionally constructing interpolations parameterized by (control variate) and (noise correlation).
- Figure 2 reveals the alignment of theoretical attributes in ReBF and actual empirical results on learning unbiased return distribution.
Critical weaknesses
- In Table 3, it seems that ReBF doesn't achieve the best performance in all of later offline RL with learned flow critic, compared to its dominance in learning unbiased return distribution; Value Flows admit orders of higher Wasserstein distance than ReBF, but still achieves better performance in several tasks.
Questions for authors
- Could the authors provide any explanations on why ReBF's lower Wasserstein distance does not necessarily lead into better offline RL performance?
4 comments
Dear Reviewer K8dC,
Thank you for your time and for your thoughtful questions. As the discussion period is nearing its end, we would like to briefly follow up on our response.
In the revised manuscript, we clarified that ReBF corrects a structural inconsistency in velocity bootstrapping that persists even under exact regression. To address how these distributional improvements affect action selection, we added (i) Long-horizon control, (ii) Risk-sensitive control on six Stochastic-D4RL datasets, and (iii) Limited-data control experiments to demonstrate the performance of our algorithm.
These results show that improved critic fidelity translates into control gains when the task carries real downside risk. We hope this could address your concern; if so, we would greatly appreciate it if you would consider updating your score and confidence. We would be happy to answer any remaining questions.
The Authors
I thank the authors for their responses to address my concerns, and I will accordingly update my review if necessary.
Many thanks for your recognition and support of our research work. We are happy that our response has addressed your concerns. Again, we greatly appreciate your valuable comments, which have significantly improved the quality of our paper.
Response to Reviewer K8dC
We thank the reviewer for identifying the distinction between distributional fidelity and policy performance, and we explain below when the improved critic translates into better control.
Response. The key distinction is that W₁ evaluates the full return distribution, whereas the risk-neutral extractor used in Table 3 primarily depends on the relative expected returns of a finite set of candidate actions. Improving the variance or tails can substantially reduce W₁ without changing the selected action; conversely, a less accurate distribution, such as that of Value Flows, may still preserve the action ranking needed for a particular task. End-to-end performance also depends on candidate-action sampling, policy extraction, finite-step ODE integration, function approximation, and offline data coverage. Therefore, critic fidelity and policy performance need not be monotonically related.
The tasks in Table 3 provide a standardized comparison, but they largely use deterministic dynamics and risk-neutral action selection, so they do not isolate when distributional fidelity should matter. In response to a related question from Reviewer MDpt, we added targeted evaluations in which only the critic differs; the full setup is in our response to Reviewer MDpt, and we summarize the most relevant results below.
Long-horizon control with mean and lower-tail extractors. On antmaze-teleport and humanoidmaze-medium (Table 10 in the revised manuscript), ReBF improves the average success rate of PCBF with the expected-return extractor. When the same critics select actions by CVaR at level α = 0.1, PCBF drops sharply, whereas ReBF largely retains its success rate. This is the pattern that the explanation above predicts: a mean-based decision is relatively insensitive to errors in the shape of the distribution, whereas a decision that reads the lower tail exposes them.
Risk-sensitive control on Stochastic-D4RL. On six datasets with stochastic hazards (Table 9 in the revised manuscript), ReBF attains a 3.8–7.3× lower W₁ and a higher CVaR (α = 0.1) return than PCBF on every dataset. The largest gains occur on the medium-replay datasets with substantial failure risk, where ReBF also reduces the fall rate from 0.85 to 0.03 on Hopper and from 0.29 to 0.05 on Walker2d; on the medium-expert datasets, where falls are rare, the gains are smaller but consistent.
Value Flows. To test the reviewer's observation, we also ran Value Flows on Stochastic-D4RL under the same evaluation protocol. Its W₁ is higher than that of ReBF on every dataset, by 1.5–14×, and its effect on control depends on risk. On the medium-expert datasets, where falls are rare, Value Flows is close to ReBF (CVaR returns of about 1590, 2150, and 1070, versus 1640, 2220, and 1070 for ReBF), so a less accurate critic can still rank actions well. On the failure-prone medium-replay datasets, however, it falls well behind: its CVaR return is about 600 on Hopper, −25 on Walker2d, and 190 on HalfCheetah, compared with about 1030, 1170, and 380 for ReBF, and its fall rate reaches 0.97 on Hopper and 0.73 on Walker2d, higher than that of PCBF (0.85 and 0.29) and far above that of ReBF (0.03 and 0.05).
This conditional pattern explains the apparent discrepancy in Table 3: improved distributional fidelity may matter little when the policy uses only the mean or when failures are rare, but it becomes consequential when the task carries real downside risk and decisions depend on the lower tail. We therefore do not claim that a lower W₁ guarantees state-of-the-art performance on every task; rather, the additional results show that ReBF's improved critic fidelity translates into control gains when downside risk is relevant to action selection.