acceptodds
Under review as a conference paper at ICLR 2027

Retimed Bellman Flows: Escaping the Impossible Triangle of Velocity Bootstrapping

Abstract

Flow critics learn return distributions by transporting Gaussian noise to Bellman endpoints via continuous velocity fields. While velocity bootstrapping stabilizes training by querying a successor teacher, existing methods face a structural dilemma: on straight paths, no residual-free same-time affine mapping can preserve Gaussian initial noise while maintaining an unbiased target. To overcome this limitation, we introduce Retimed Bellman Flows (ReBF). ReBF queries the teacher critic at a dynamically shifted earlier flow time, aligning intermediate student and teacher trajectories. By combining this retimed clock with fresh, decoupled noise generation, ReBF constructs a provably conditionally unbiased velocity target that preserves the Bellman fixed point and contracts under Wasserstein distances. Empirically, ReBF reduces distance to ground-truth return distributions by up to on synthetic MRPs and outperforms existing flow critics across 38 challenging OGBench and D4RL offline reinforcement learning tasks.

Primary area:
reinforcement learning
Keywords:
flow matching, velocity bootstrapping, Bellman equation, flow based critics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_K8dCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

For training flow-based critics in offline RL, this paper identifies structural biases in existing methods that deviates from the ground-truth return distributions and proposes Retimed Bellman Flows(ReBF). ReBF demonstrates that expressing the student flow input xtx_t as an affine transformation of the successor flow input xτx_{\tau} and the immediate reward requires retiming the teacher to τ=γt/(1−(1−γ)t)\tau=\gamma t/(1-(1-\gamma)t), together with decoupling the successor return generation noise from the student's initial noise. It is proved that these conditions are required to yield conditionally unbiased velocity bootstrapping. Across every synthetic Markov reward processes and OGBench, ReBF consistently achieves lower Wasserstein distances to the ground-truth return distributions than existing baselines. These improvements were translated into better downstream offline RL performance on several OGBench and D4RL benchmarks.

Critical strengths
  • The paper developed a theoretical guarantee of unbiasedness beyond empirical design.
  • The authors improve the explainability of the proposed design knobs by additionally constructing interpolations parameterized by κ\kappa(control variate) and ρ1\rho_1(noise correlation).
  • Figure 2 reveals the alignment of theoretical attributes in ReBF and actual empirical results on learning unbiased return distribution.
Critical weaknesses
  • In Table 3, it seems that ReBF doesn't achieve the best performance in all of later offline RL with learned flow critic, compared to its dominance in learning unbiased return distribution; Value Flows admit orders of higher Wasserstein distance than ReBF, but still achieves better performance in several tasks.
Questions for authors
  • Could the authors provide any explanations on why ReBF's lower Wasserstein distance does not necessarily lead into better offline RL performance?
4 comments
AuthorsOpenReview ↗

Response to Reviewer K8dC

We thank the reviewer for identifying the distinction between distributional fidelity and policy performance, and we explain below when the improved critic translates into better control.

Q1. In Table 3, it seems that ReBF doesn't achieve the best performance in all of later offline RL with learned flow critic, compared to its dominance in learning unbiased return distribution; Value Flows admit orders of higher Wasserstein distance than ReBF, but still achieves better performance in several tasks. Could the authors provide any explanations on why ReBF's lower Wasserstein distance does not necessarily lead into better offline RL performance?

Response. The key distinction is that W₁ evaluates the full return distribution, whereas the risk-neutral extractor used in Table 3 primarily depends on the relative expected returns of a finite set of candidate actions. Improving the variance or tails can substantially reduce W₁ without changing the selected action; conversely, a less accurate distribution, such as that of Value Flows, may still preserve the action ranking needed for a particular task. End-to-end performance also depends on candidate-action sampling, policy extraction, finite-step ODE integration, function approximation, and offline data coverage. Therefore, critic fidelity and policy performance need not be monotonically related.

The tasks in Table 3 provide a standardized comparison, but they largely use deterministic dynamics and risk-neutral action selection, so they do not isolate when distributional fidelity should matter. In response to a related question from Reviewer MDpt, we added targeted evaluations in which only the critic differs; the full setup is in our response to Reviewer MDpt, and we summarize the most relevant results below.

Long-horizon control with mean and lower-tail extractors. On antmaze-teleport and humanoidmaze-medium (Table 10 in the revised manuscript), ReBF improves the average success rate of PCBF with the expected-return extractor. When the same critics select actions by CVaR at level α = 0.1, PCBF drops sharply, whereas ReBF largely retains its success rate. This is the pattern that the explanation above predicts: a mean-based decision is relatively insensitive to errors in the shape of the distribution, whereas a decision that reads the lower tail exposes them.

Average success (%)Mean: PCBFMean: ReBFCVaR α=0.1: PCBFCVaR α=0.1: ReBF
antmaze-teleport30.035.48.531.7
humanoidmaze-medium13.420.810.618.4

Risk-sensitive control on Stochastic-D4RL. On six datasets with stochastic hazards (Table 9 in the revised manuscript), ReBF attains a 3.8–7.3× lower W₁ and a higher CVaR (α = 0.1) return than PCBF on every dataset. The largest gains occur on the medium-replay datasets with substantial failure risk, where ReBF also reduces the fall rate from 0.85 to 0.03 on Hopper and from 0.29 to 0.05 on Walker2d; on the medium-expert datasets, where falls are rare, the gains are smaller but consistent.

DatasetCVaR return: PCBFCVaR return: ReBFW₁: PCBFW₁: ReBF
Hopper-MR−15102912925
Walker2d-MR110117024834
HalfCheetah-MR32638429863
Hopper-ME140516379420
Walker2d-ME199522169024
HalfCheetah-ME1040107120629

Value Flows. To test the reviewer's observation, we also ran Value Flows on Stochastic-D4RL under the same evaluation protocol. Its W₁ is higher than that of ReBF on every dataset, by 1.5–14×, and its effect on control depends on risk. On the medium-expert datasets, where falls are rare, Value Flows is close to ReBF (CVaR returns of about 1590, 2150, and 1070, versus 1640, 2220, and 1070 for ReBF), so a less accurate critic can still rank actions well. On the failure-prone medium-replay datasets, however, it falls well behind: its CVaR return is about 600 on Hopper, −25 on Walker2d, and 190 on HalfCheetah, compared with about 1030, 1170, and 380 for ReBF, and its fall rate reaches 0.97 on Hopper and 0.73 on Walker2d, higher than that of PCBF (0.85 and 0.29) and far above that of ReBF (0.03 and 0.05).

This conditional pattern explains the apparent discrepancy in Table 3: improved distributional fidelity may matter little when the policy uses only the mean or when failures are rare, but it becomes consequential when the task carries real downside risk and decisions depend on the lower tail. We therefore do not claim that a lower W₁ guarantees state-of-the-art performance on every task; rather, the additional results show that ReBF's improved critic fidelity translates into control gains when downside risk is relevant to action selection.

Dear Reviewer K8dC,

Thank you for your time and for your thoughtful questions. As the discussion period is nearing its end, we would like to briefly follow up on our response.

In the revised manuscript, we clarified that ReBF corrects a structural inconsistency in velocity bootstrapping that persists even under exact regression. To address how these distributional improvements affect action selection, we added (i) Long-horizon control, (ii) Risk-sensitive control on six Stochastic-D4RL datasets, and (iii) Limited-data control experiments to demonstrate the performance of our algorithm.

These results show that improved critic fidelity translates into control gains when the task carries real downside risk. We hope this could address your concern; if so, we would greatly appreciate it if you would consider updating your score and confidence. We would be happy to answer any remaining questions.

The Authors

I thank the authors for their responses to address my concerns, and I will accordingly update my review if necessary.

Many thanks for your recognition and support of our research work. We are happy that our response has addressed your concerns. Again, we greatly appreciate your valuable comments, which have significantly improved the quality of our paper.

Reviewer_MDptOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies velocity bootstrapping for flow-based distributional critics. It identifies compatibility issues between the Bellman transformation, the interpolation path, and the noise coupling used to query a teacher critic. The proposed method, Retimed Bellman Flows (ReBF), combines a transformed flow time with independent generation and interpolation noise to obtain a consistent bootstrapping target under the stated assumptions. The paper develops theoretical results on target consistency and policy evaluation, and evaluates the method through controlled distribution-estimation experiments, extensive ablations, and offline RL benchmarks on OGBench and D4RL.

Critical strengths

The paper investigates the proposed mechanism carefully. The theoretical analysis is supported by controlled experiments that examine the effects of retiming, noise coupling, and correction strength. These experiments help explain the behavior of the method beyond reporting aggregate benchmark scores. I appreciate the substantial effort devoted to connecting the derivations with observable failures of existing updates.

The evaluation of return-distribution fidelity is a meaningful strength. Table 2 and the accompanying visualizations examine aspects of the learned distribution that would be missed by evaluating only its mean. The improvements in Wasserstein distance provide evidence that ReBF can produce more accurate distributional critics, even when competing methods estimate the expected return reasonably well.

Critical weaknesses

Despite the thorough analysis, I remain uncertain whether the contribution represents a sufficiently substantial advance over existing flow-based distributional RL methods, particularly PCBF. The central algorithmic changes concern the time and location at which the teacher is queried and the coupling between the relevant noise variables. The paper explains why these changes are appropriate for its formulation, but the broader significance of this refinement is less convincing. Establishing consistency within the specified interpolation and bootstrapping construction is valuable; however, I would like a clearer account of what substantial limitation of prior methods this resolves in practice, beyond improving the accuracy of the same underlying estimation procedure.

The downstream results in Table 3 do not establish a substantial or consistent advantage over the strongest existing methods. There are encouraging gains on cube-double-play and puzzle-4x4-play, but the improvement on scene-play is small, and ReBF does not outperform the strongest baseline on cube-triple-play, D4RL Adroit, or either visual domain. On visual-cube-double-play, it matches PCBF at 3 while Value Flows reaches 13. These results support a competitive method, but provide a less compelling case for its practical significance than the distribution-estimation results in Table 2. I recognize that better distribution estimation need not automatically improve expected-return maximization. Nevertheless, this leaves an important gap between the demonstrated correction of the critic and the broader value of the proposed method.

Questions for authors

(a) The paper provides a mathematical motivation for retiming and independent noise. Could the authors clarify which aspect of this result should be regarded as a substantial conceptual advance over PCBF and related methods, rather than a correction or refinement of their bootstrapping construction? In particular, under what practically relevant conditions should these changes make a consequential difference relative to a well-tuned existing method?

(b) How should the stronger distribution-estimation results in Table 2 be reconciled with the mixed policy-performance results in Table 3? Does the available evidence indicate that the benefits are limited by policy extraction, by the use of an expected-return objective, or by other sources of approximation error? An explanation supported by the existing experiments or additional diagnostics would materially affect my assessment.

I appreciate the thorough experimental and theoretical investigation. My current reservation is that the methodological advance appears incremental, while the downstream gains are not sufficiently strong or consistent to establish its broader significance. I would reconsider this assessment if the authors can make a more convincing case for the conceptual importance or practical consequences of the proposed changes.

5 comments
AuthorsOpenReview ↗

Response to Reviewer MDpt (1/4): Critic Fidelity vs. End-to-End Control

We thank the reviewer for the careful assessment. The reviewer raises two concerns: the significance of the contribution relative to PCBF, and the gap between distributional accuracy in Table 2 and policy performance in Table 3. We address both concerns below through further clarification and additional experiments.

Q1. What substantial conceptual limitation of PCBF and related methods does ReBF resolve beyond refining their bootstrapping construction, and under what practically relevant conditions should this correction produce consequential gains over a well-tuned existing method? Why do ReBF's stronger distribution-estimation results in Table 2 not consistently translate into better policy performance in Table 3? Is this gap primarily due to risk-neutral policy extraction, the expected-return objective, or other approximation errors, and can this explanation be supported by additional diagnostics or experiments?

Response. We thank the reviewer for raising this important distinction. ReBF addresses a structural inconsistency in velocity bootstrapping: an accurate successor return distribution does not necessarily make the reused critic’s intermediate velocity a valid Bellman target. Incompatible teacher queries and noise coupling can introduce systematic distributional error even under exact population regression. Under the stated assumptions, retiming together with independent noise restores consistency with the distributional Bellman operator.

The practical consequences depend on how the learned distribution is used. The policy extractor in Table 3 is risk-neutral: it ranks candidate actions by estimated expected return. Wasserstein distance instead evaluates the full return distribution. Correcting dispersion or lower-tail errors can therefore substantially improve distributional accuracy without changing the ordering of action means. Long horizons can propagate these distributional errors without necessarily introducing a mean error. Their consequences for action selection become more direct under risk-sensitive objectives, which explicitly depend on lower-tail outcomes.

The OGBench and D4RL tasks in Table 3 (domain averages) and its per-task breakdown in Table 13 were included primarily to provide a standardized comparison with existing flow-based methods, including PCBF and Value Flows. Domain averages can obscure improvements on individual tasks with different difficulty levels. For example, ReBF improves over PCBF by 30 percentage points on cube-triple task 1 (5555 vs. 2525), 18 points on cube-double task 5 (8383 vs. 6565), and 8 points on puzzle-4x4 task 4 (4343 vs. 3535). These task-level gains are less visible after averaging across tasks with ceiling or near-zero performance, although the overall results remain mixed.

Some of these benchmarks, such as scene-play and cube-triple, already contain sequential dependencies. However, their final performance jointly reflects manipulation skill composition, combinatorial generalization, offline coverage, and policy extraction, making it difficult to isolate the practical consequence of accumulated critic error. Moreover, their risk-neutral evaluation does not directly test whether improved lower-tail accuracy supports better decisions under risk.

These considerations motivate long-horizon decision-making under uncertainty as a particularly relevant setting for distributional consistency. We therefore add two complementary evaluations, presented in the following comments:

  • (i) Long-horizon control: retaining the same expected-return extractor and candidate-action budget to test whether the correction improves standard control in demanding sequential settings.
  • (ii) Risk-sensitive control: using the same CVaR-based extractor to test whether improved distributional estimates translate into better decisions under stochastic hazards.

We additionally report limited-data control as a supplementary evaluation of robustness to restricted offline coverage.

AuthorsOpenReview ↗

Response to Reviewer MDpt (2/4): Long-Horizon Control

(i) Long-horizon control. We select antmaze-teleport and humanoidmaze-medium because they provide two complementary long-horizon settings. Antmaze-teleport requires sustained navigation over episodes of up to approximately 1,000 steps and contains stochastic teleporters, some of which lead to dead ends; it therefore combines long credit-assignment paths with variable return outcomes. Humanoidmaze-medium requires high-level maze navigation together with low-level control of a 21-DoF humanoid over episodes of up to approximately 2,000 steps. In both environments, success information must propagate through hundreds or thousands of Bellman backups. Both methods select actions among the same 16 behavior-policy candidates, and we compare three selection rules that read the learned distribution differently: its mean, its CVaR at level α = 0.25, and its CVaR at level α = 0.1. We report the final checkpoint with 200 evaluation episodes per task.

Table 10 in the revised manuscript: Long-horizon control. Success rate (%) when the action is chosen among K = 16 candidates by the mean, CVaR (α = 0.25), or CVaR (α = 0.1) of the learned return distribution. The better value of each pair is in bold.

1. antmaze-teleport environment

TaskMean (PCBF)Mean (ReBF)CVaR α=0.25 (PCBF)CVaR α=0.25 (ReBF)CVaR α=0.1 (PCBF)CVaR α=0.1 (ReBF)
T124.536.514.535.011.535.5
T224.039.52.013.00.59.5
T343.539.525.536.021.538.5
T445.544.510.041.55.045.0
T512.517.03.537.04.030.0
Avg30.035.411.132.58.531.7

2. humanoidmaze-medium environment

TaskMean (PCBF)Mean (ReBF)CVaR α=0.25 (PCBF)CVaR α=0.25 (ReBF)CVaR α=0.1 (PCBF)CVaR α=0.1 (ReBF)
T14.00.00.00.00.00.0
T232.547.057.069.549.074.0
T314.516.50.04.00.54.5
T41.51.00.00.00.00.0
T514.539.53.518.03.513.5
Avg13.420.812.118.310.618.4

With the expected-return extractor, ReBF reaches an average success rate of 35.4% on antmaze-teleport and 20.8% on humanoidmaze-medium, compared with 30.0% and 13.4% for PCBF. The difference grows when the decision reads the lower tail of the distribution. Under CVaR with α = 0.25 and α = 0.1, ReBF keeps its success rate on antmaze-teleport (32.5% and 31.7%), whereas PCBF drops to 11.1% and 8.5%; on humanoidmaze-medium, ReBF is better or tied on every task under both lower-tail extractors. Because antmaze-teleport contains stochastic teleporters that can lead to dead ends, a critic with a miscalibrated lower tail misranks candidates once the decision depends on that tail.

With the expected-return extractor, ReBF reaches an average success rate of 35.4% on antmaze-teleport and 20.8% on humanoidmaze-medium, compared with 30.0% and 13.4% for PCBF; the gains concentrate on antmaze T1, T2, and T5 and humanoidmaze T2 and T5, whereas the remaining tasks are close. The difference grows when the decision reads the lower tail of the distribution. Moving from the mean to CVaR with α = 0.1 lowers ReBF's average only from 35.4% to 31.7% on antmaze-teleport, whereas PCBF falls from 30.0% to 8.5%; on humanoidmaze-medium, ReBF is better or tied on every task under both lower-tail extractors and reaches 74.0% on T2. The size of this drop is itself a diagnostic: a critic whose lower tail is consistent with its mean ranks candidates similarly under both rules, whereas a miscalibrated tail reorders them. Antmaze-teleport, whose stochastic teleporters can lead to dead ends, is exactly the setting where such reordering selects risky actions.

AuthorsOpenReview ↗

Response to Reviewer MDpt (3/4): Risk-Sensitive Control on Stochastic-D4RL

(ii) Risk-sensitive control. The benchmarks in Table 3 have largely deterministic dynamics and select actions by expected return, so they do not test whether distributional fidelity matters when the return distribution itself carries risk. We therefore evaluate on six Stochastic-D4RL datasets (Hopper, Walker2d, and HalfCheetah; medium-replay and medium-expert). In these environments, rewards are relabeled with stochastic hazards: entering an unsafe region (excessive torso pitch for Hopper and Walker2d, excessive forward velocity for HalfCheetah) incurs a large penalty with a fixed probability, and on Hopper and Walker2d excessive pitch also terminates the episode early (a fall). The same action can therefore lead to substantially different returns, and the return distribution is broad and heavy-tailed.

Table 9 in the revised manuscript: Risk-sensitive control on Stochastic-D4RL. W₁ is the distance between the critic and the Monte Carlo return distribution of the method's own policy, averaged over 8 start states; Std. pred./MC reports the predicted and Monte Carlo return standard deviations. Fall rate is not defined for HalfCheetah, whose hazard does not terminate episodes.

DatasetMethodCVaR α=0.1 return ↑Fall rate ↓W₁ ↓Std. pred./MC
Hopper-medium-replayPCBF−15±89-15 \pm 890.85±0.100.85 \pm 0.10129±68129 \pm 6834.9 / 62.1
ReBF1029±29\mathbf{1029 \pm 29}0.03±0.03\mathbf{0.03 \pm 0.03}25±7\mathbf{25 \pm 7}1.7 / 2.9
Walker2d-medium-replayPCBF110±30110 \pm 300.29±0.060.29 \pm 0.06248±72248 \pm 7266.3 / 46.4
ReBF1170±278\mathbf{1170 \pm 278}0.05±0.03\mathbf{0.05 \pm 0.03}34±4\mathbf{34 \pm 4}23.7 / 30.8
HalfCheetah-medium-replayPCBF326±66326 \pm 66–298±29298 \pm 2936.1 / 42.6
ReBF384±28\mathbf{384 \pm 28}–63±9\mathbf{63 \pm 9}47.4 / 39.6
Hopper-medium-expertPCBF1405±321405 \pm 320.05±0.080.05 \pm 0.0894±1294 \pm 1211.3 / 3.5
ReBF1637±26\mathbf{1637 \pm 26}0.00±0.00\mathbf{0.00 \pm 0.00}20±1\mathbf{20 \pm 1}3.4 / 2.9
Walker2d-medium-expertPCBF1995±2851995 \pm 2850.00±0.000.00 \pm 0.0090±4290 \pm 4228.4 / 21.6
ReBF2216±22\mathbf{2216 \pm 22}0.00±0.000.00 \pm 0.0024±7\mathbf{24 \pm 7}12.9 / 10.0
HalfCheetah-medium-expertPCBF1040±381040 \pm 38–206±68206 \pm 6867.2 / 41.9
ReBF1071±11\mathbf{1071 \pm 11}–29±8\mathbf{29 \pm 8}18.4 / 40.3

In this table, ReBF attains a lower W₁ than PCBF on all six datasets, by 3.8–7.3×, and a higher mean CVaR (α = 0.1) return on every dataset. The largest control gains occur on the two medium-replay datasets with substantial failure risk. On Hopper, ReBF achieves a CVaR return of 1029 with a fall rate of 0.03, compared with −15 and 0.85 for PCBF. On Walker2d, ReBF achieves 1170 and 0.05, compared with 110 and 0.29. On the medium-expert datasets, where falls are rare, the gains are smaller but consistent.

Fig. 10 in the revised manuscript visualizes where these W₁ errors come from. The PCBF critic places most of its mass well below the Monte Carlo distribution of its own policy and often spreads into a long left tail; on Walker2d-medium-replay, for example, its predicted returns extend from about −400 to 0, whereas the actual returns of its policy lie between about 0 and 250. Its error is therefore a shift in the location of the distribution, not only a tail error. The ReBF critic, in contrast, overlaps the Monte Carlo distribution and tracks its shape on every dataset. Because the CVaR extractor ranks candidate actions by the lower tail, a tail dominated by spurious low returns cannot separate safe actions from risky ones, which is consistent with PCBF's higher fall rates. Thus, improved critic fidelity has the largest control effect when lower-tail risk influences action selection, and a smaller effect when failure risk is low.

Reviewer_2zCUOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper provides a patch up to some challenges to using flow matching methods as distributional RL critics. As I understand it, they use a frozen copy of the critic at the next s,as,a pair as a teacher for the current vtv_t in the flow matching problem, such that the time derivative of the interpolant uses a sample X1′∼η(s′,a′)X_1' \sim \eta(s', a'). Usually this induces a bias if done in the standard sense because the teacher flow and student flow use the same times.

The paper proposes an amendment to this bias so that the proper tower property can be used in the loss by changing the SNR schedules for the teacher and the flow appropriately and using an indpendent base noise between the two such that no specific coupling needs to be used in the flow matching setup.

Doing this, you still get the same Bellman fixed point but clean up some of the biases in training. They test it on a variety of toy problems and a bunch of offline RL control task, and measure signficant improvements over untuned baselines.

Critical strengths

This paper presents a neat and tidy fix to clear up the biases in the standard objective one might arrive at with e.g. PCBF. As I understand it, fixing the interpolant schedule error between the teacher and student allows the tower property to be properly applied in Theorem 5.3.

I like the toy evaluation for actually testing the implications of the pointed out errors and the fix.

Critical weaknesses

I think the framing a bit oversells the novelty here. This is basically just changing the interpolant schedules. It does clean up the bias in the approach, but as explained a bit later, this bias seems small for γ\gamma close to 1 and the whole impossibility triangle seems a bit oversold.

One important thing I'd need the authors to clarify is why they think this re-timing is central at all to the performances presented. As I understand it, in real world problems, the discount γ\gamma is usually around like 0.990.99. This means there's close to no change in the interpolant schedules between the student and the teacher. If I do the math out I get that t−τ(t)=.0025t - \tau(t) = .0025 at that γ\gamma, so I don't understand. Do you just expect this to compound significantly even at this discount?

Doesn't this mean that the majority of the gains is seen from refreshing the gaussian noise i.e. ρ1=0\rho_1 = 0? The title of the paper is only well supported by the toy experiments where γ\gamma can be set much lower. This should be ablated on the real experiments.

Also, are the improvement claims against PCBF fair? the 7x quoted is on the non-tuned gain version. I understand the gain tuning would be great to remove, but you also quote that with it, the gain drops to like 1-2x across most of your tests in Table 2.

Questions for authors

Can you do an ablation where you don't to the time rescheduling but still independent noise on the actual discount in real problems e.g. the 0.99 one? I'd be curious to see if most of the benefit for the larger experiments is coming from that.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 2zCU (1/4): Summary and Conceptual Contribution

We thank Reviewer 2zCU for the careful reading and precise technical questions regarding our framing, the effect of retiming at realistic discounts, the role of independent noise, and the fairness of the PCBF comparison. In response, we clarify that our contribution is identifying the structural inconsistency of same-time velocity bootstrapping and deriving the corresponding Bellman-consistent correction, rather than the complexity of its implementation. We add an exact Gaussian analysis showing that, although the one-backup mismatch vanishes as γ~→1\tilde{\gamma}\to1, its fixed-point effect can remain nonzero. Following the reviewer's suggestion, we also add a controlled OGBench ablation that isolates retiming while holding ρ1=0\rho_1=0 and all other settings fixed, showing a measurable but domain-dependent effect at realistic discounts. Beyond this requested ablation, we add a fixed-policy long-horizon evaluation that removes policy-extraction effects; at an effective discount of γ~≈0.995\tilde{\gamma}\approx0.995, where the per-backup clock offset is only about 0.00120.0012, retiming alone reduces W1W_1 by 1.31.3–1.8×1.8\times on humanoidmaze-medium, humanoidmaze-large, and antmaze-giant. Finally, we qualify the original 7.7×7.7\times claim and report the comparison against tuned PCBF, for which ReBF achieves consistent 1.191.19–1.75×1.75\times improvements in distributional accuracy.

Q1. I think the framing a bit oversells the novelty here. This is basically just changing the interpolant schedules. It does clean up the bias in the approach, but as explained a bit later, this bias seems small for γ close to 1 and the whole impossibility triangle seems a bit oversold.

Response. We agree that the resulting implementation is compact, but we believe implementation complexity should be distinguished from the underlying conceptual contribution. Our work begins by identifying a previously uncharacterized inconsistency in same-time velocity bootstrapping: even with an exact teacher and exact regression, its induced backup need not have the Bellman return distribution as its fixed point. More specifically, we constructed a toy example that there is a finite gap of learned distribution even γ is arbitrary close to 1. The reason is that when γ is close to 1, the effective horizon scales 1/(1-γ), which can greatly amplify the error of each bellman step update, due to the well-known compounding error problem of bellman equation. See Why retiming matters part for the construction of this case.

Therefore, unbiasedness is important since a small amount of bias will be amplified throughout the recursively through the bellman equation through the long horizon. The problem become more severe in the case of sparse reward long horizon task. We show that retimed clock reduce this bias introduced by same-time couple to zero, and can greatly increase the distributional accuracy of the learned critics, which can in turn improve the downstream tasks.

AuthorsOpenReview ↗

Response to Reviewer 2zCU (2/4): Why Retiming Matters

Q2. One important thing I'd need the authors to clarify why they think this re-timing is central at all to the performances presented. In real world problems, the discount γ\gamma is usually around like 0.99. This means there's close to no change in the interpolant schedules between the student and the teacher. If I do the math out I get that t−τ(t)=0.0025t-\tau(t)=0.0025 at that γ\gamma so I don't understand. Do you just expect this to compound significantly even at this discount?

Response. At γ~=0.99\tilde\gamma=0.99, the maximum clock difference is approximately 0.00250.0025. However, repeated bootstrapping can amplify this one-step error. We have constructed a toy example that a finite gap 0.33×0.33 \times exists when γ→1\gamma \rightarrow 1, even though the one step error e1→0e_1 \rightarrow 0, the amplification rate e∞/e1→+∞e_{\infty}/e_1 \rightarrow +\infty.

Consider a single nonterminal state with iid rewards Rk∼N(rˉ,σR2)R_k\sim\mathcal N(\bar r,\sigma_R^2), σR>0\sigma_R>0, and 0<γ~<10<\tilde\gamma<1. The true return law is

m^\star=\frac{\bar r}{1-\tilde\gamma},\qquad (s^\star)^2=\frac{\sigma_R^2}{1-\tilde\gamma^2}.$$ We use a standard Gaussian base, fresh independent noise ($\rho_1=0$), full correction ($\kappa=1$), exact population regression, and exact ODE generation. Initialize both methods at the canonical teacher $v_{\eta^\star}(t,x)=\mathbb E[U-Z\mid(1-t)Z+tU=x]$, with independent $U\sim\eta^\star$ and $Z\sim\mathcal N(0,1)$. **Theorem 5.6 (Gaussian self-loop).** Same-time affine updates converge to a unique affine fixed point with return law $\mathcal N(m^\star,(1+\tilde\gamma)(s^\star)^2)$. Its one-backup and fixed-point $W_1$ errors, divided by $s^\star$, are $$e_1=\sqrt{2/\pi}\bigl(e^{\tilde\gamma(1-\tilde\gamma)/2}-1\bigr),\qquad e_\infty=\sqrt{2/\pi}\bigl(\sqrt{1+\tilde\gamma}-1\bigr).$$ These follow from Gaussian conditioning and $W_1=\sqrt{2/\pi}|s_1-s_2|$ for equal-mean Gaussian laws. **ReBF recovers $\eta^\star$ exactly.** Let $Y=R+\tilde\gamma U$, with $R$ independent of $(U,Z)$. Set $\alpha=1-(1-\tilde\gamma)t$, $\tau=\tilde\gamma t/\alpha$, and $W_\tau=(1-\tau)Z+\tau U$. Retiming gives $$X_t=(1-t)Z+tY=tR+\alpha W_\tau,\qquad Y-Z=R-(1-\tilde\gamma)W_\tau+\frac{\tilde\gamma}{\alpha}(U-Z).$$ Conditioning on $(R,W_\tau)$ replaces $U-Z$ by the canonical teacher $v_{\eta^\star}(\tau,W_\tau)$; the tower property then gives the regressed field $\mathbb E[Y-Z\mid X_t=x]$. Its exact ODE generates $$\mathcal L(Y)=\mathcal N\left(\bar r+\tilde\gamma m^\star, \sigma_R^2+\tilde\gamma^2(s^\star)^2\right) =\mathcal N(m^\star,(s^\star)^2)=\eta^\star.$$ The teacher remains canonical, so every subsequent update also preserves $\eta^\star$: both errors are exactly zero under these assumptions. For same-time querying, if $\gamma \rightarrow 1$, the one-step error $e_1\to0$, **but the error after infinite horizon goes to a constant**, $e_\infty\to\sqrt{2/\pi}(\sqrt2-1)\approx0.3305$, and the amplification rate can grow to infinit $$\frac{e_\infty}{e_1}\sim\frac{2(\sqrt2-1)}{1-\tilde\gamma} \rightarrow \infty \quad\text{as }\tilde\gamma\uparrow1.$$ At $\tilde\gamma=0.99$, the normalized error grows from $0.00396$ after one backup to $0.32767$ at the fixed point, an amplification of $82.8$. At $0.999$, these become $0.000399$, $0.33021$, and $828.3$. The mean is correct; the persistent error is in distributional scale. The proposition covers the independent-noise case. To separate the contributions of the query clock and the noise coupling, Fig. 8 (in revised manuscript) in Appendix D.1.1 compares all four combinations, including the two shared-noise variants ($\rho_1=1$) over $\tilde\gamma\in[0.5,0.999]$. Panel (a) shows that the one-backup errors of all three biased variants shrink with the clock offset as $\tilde\gamma\to1$. Panel (b) shows that their fixed-point errors instead level off at nonzero values ($0.33$, $0.19$, and $0.06$). Panel (c) shows that the ratio between the two grows with the effective horizon $1/(1-\tilde\gamma)$. Table 7 reports the exact values at $\tilde\gamma=0.99$ and $0.999$, with errors normalized by $s^\star$. The independent-noise rows match the closed forms above, whereas the shared-noise rows are exact population computations that the proposition does not cover. ReBF's zeros hold up to ODE-grid residuals below $2\times10^{-5}$ at $\tilde\gamma=0.99$, which decrease under refinement. **Table 7.** | Method | One-backup $W_1/s^\star$ | Fixed-point $W_1/s^\star$ | Fixed-point $\mathrm{sd}/s^\star$ | Fixed-point $W_1/s^\star$ at $0.999$ | |---|---:|---:|---:|---:| | Same-time, $\rho_1=1$ | $1.7\times10^{-3}$ | $0.062$ | $0.92$ | $0.062$ | | Same-time, $\rho_1=0$ | $4.0\times10^{-3}$ | $0.33$ | $1.41$ | $0.33$ | | Retimed, $\rho_1=1$ | $3.4\times10^{-3}$ | $0.19$ | $0.77$ | $0.19$ | | **ReBF (retimed, $\rho_1=0$)** | *0** | **0** | **1.00** | **0** | This example proves that a small clock mismatch can produce persistent fixed-point bias
AuthorsOpenReview ↗

Response to Reviewer 2zCU (3/4): Long-Horizon Retiming Ablations

Q3. Doesn't this mean that the majority of the gains is seen from refreshing the gaussian noise i.e. ρ1=0\rho_1 = 0? The title of the paper is only well supported by the toy experiments where γ\gamma can be set much lower. This should be ablated on the real experiments.

Response. The exact Gaussian analysis in our response to Q2 already separates the effects of retiming and noise coupling. At γ~=0.99\tilde{\gamma}=0.99, using independent noise without retiming (same-time, ρ1=0\rho_1=0) leaves a normalized fixed-point error of 0.330.33, while retiming with shared noise (retimed, ρ1=1\rho_1=1) leaves an error of 0.190.19. Only ReBF, which combines the retimed lift with independent noise, recovers the exact fixed point. Thus, independent noise alone is not sufficient in the controlled analysis.

The following trained critics using the solatire example, which uses γ≤0.95\gamma\le0.95, so we also trained ReBF and the same-time variant (both ρ1=0\rho_1=0; pre-registered, 10 paired seeds) on toys with known return laws at γ≥0.99\gamma\ge0.99, with effective horizon H=1/(1−γ(1−p))H=1/(1-\gamma(1-p)) set by Solitaire's termination probability pp.

Table. W1/sd⋆W_1/\mathrm{sd}^\star to the exact return law; bold: paired 95% CI excludes 0.

Settingγ\gammaHHSame-timeReBF
Solitaire, p=1/20p=1/200.99170.0450.046
Solitaire, p=1/100p=1/1000.99500.1310.050
Solitaire, p=1/300p=1/3000.99750.1770.050
Solitaire, p=1/300p=1/3000.9951200.1400.040

At each discount the gap grows with HH, although the clock offset is at most about 0.00250.0025 per backup. At H=50H=50, the same-time variant with independent noise reaches W1/sd⋆=0.131W_1/\mathrm{sd}^\star=0.131, whereas ReBF reaches 0.0500.050.

To directly assess whether retiming contributes beyond independent noise at realistic discounts, we followed the reviewer's suggestion and added a controlled ablation to the OGBench experiments. We compare ReBF with an otherwise identical variant without retiming, denoted ReBF w/o retiming. The latter replaces the retimed lift (τ,Γ/α)(\tau,\Gamma/\alpha) with its same-time counterpart (t,Γ)(t,\Gamma). Both variants share the same network architecture, training procedure, and policy-extraction method.

Table 8 in Appendix D.1.2: Retiming ablation on OGBench. Both variants use ρ1=0\rho_1=0 and differ only in the query clock. The higher success rate in each row is shown in bold.

TaskReBF w/o retimingReBF
cube-double task197±2\mathbf{97 \pm 2}97±3\mathbf{97 \pm 3}
cube-double task270±1170 \pm 1182±9\mathbf{82 \pm 9}
cube-double task365±365 \pm 393±2\mathbf{93 \pm 2}
cube-double task427±10\mathbf{27 \pm 10}25±625 \pm 6
cube-double task556±1356 \pm 1383±8\mathbf{83 \pm 8}
cube-double average (γ~=0.995\tilde{\gamma}=0.995)63±763 \pm 776±6\mathbf{76 \pm 6}
puzzle-4x4 task139±439 \pm 446±13\mathbf{46 \pm 13}
puzzle-4x4 task231±5\mathbf{31 \pm 5}29±329 \pm 3
puzzle-4x4 task344±444 \pm 445±10\mathbf{45 \pm 10}
puzzle-4x4 task437±337 \pm 343±8\mathbf{43 \pm 8}
puzzle-4x4 task510±610 \pm 615±4\mathbf{15 \pm 4}
puzzle-4x4 average (γ~=0.99\tilde{\gamma}=0.99)32±232 \pm 236±8\mathbf{36 \pm 8}

The table shows that retiming provides an additional benefit beyond independent noise, although its effect is domain-dependent. On cube-double, removing retiming reduces the average success rate from 76%76\% to 63%63\%, with the largest drops on task 3 (93%→65%93\%\to65\%) and task 5 (83%→56%83\%\to56\%). On puzzle-4x4, the average drops from 36%36\% to 32%32\%, but this gap lies within seed variance. Therefore, independent noise alone does not recover ReBF's performance on cube-double, and the observed gains cannot be attributed solely to setting ρ1=0\rho_1=0.

The OGBench ablation shows that retiming affects downstream control; however, policy performance also depends on action selection and optimization. We therefore add a fixed-policy long-horizon evaluation that isolates the effect of retiming on the learned critic. At the effective backup discount γ~=0.9995≈0.995\tilde{\gamma}=0.999^5\approx0.995 (five-step action chunks), the maximum clock offset is about 0.00120.0012, which is even smaller than in the reviewer's example. ReBF and ReBF w/o retiming share the same policy, data, noise coupling (ρ1=0\rho_1=0), and training procedure, and differ only in the query clock. Fig. 9 in Appendix D.1.3 shows that retiming reduces W1W_1 by 1.31.3–1.8×1.8\times on humanoidmaze-medium, humanoidmaze-large, and antmaze-giant, and it brings the predicted return spread closer to the Monte Carlo reference in all three environments. Thus, even at γ~≈0.995\tilde{\gamma}\approx0.995, the small per-backup clock mismatch produces a measurable accumulated error in the learned critic.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear AC,

We thank the Area Chair and all three reviewers for their careful assessment and for recognizing the value of our work that identified a structural inconsistency in same-time velocity bootstrapping and to derive a Bellman-consistent correction based on a retimed query together with compatible noise coupling.

The reviewers' concerns were

  • (1) whether the gains primarily arise from independent noise instead of retiming (Reviewer 2zCU),
  • (2) whether ReBF constitutes a substantial advance over PCBF (Reviewer MDpt),
  • (3) why improved distributional fidelity does not uniformly yield the best control performance. (Reviewer MDpt, K8dC)

Regarding (1), we have shown that retiming indeed is an essential design. We demonstrate this in three aspects. First, we added an exact analysis showing that a small one-backup mismatch can accumulate into a non-vanishing fixed-point error. Second, following the reviewer's request, we added a controlled OGBench ablation that isolates retiming while holding all other settings fixed. Third, we additionally added a fixed-policy long-horizon evaluation that removes policy-extraction effects and shows that retiming reduces W1W_1 by 1.31.3–1.8×1.8\times at realistic discounts.

Regarding (2), We clarify that ReBF at least presents three substantial benefits over PCBF: (i) long-horizon control, (ii) risk-sensitive control, and (iii) limited-data scenario.

On long-horizon control with the same expected-return extractor, ReBF surpasses the success rate of PCBF by 5.4% on antmaze-teleport and 7.4% on humanoidmaze-medium, and the gap widens (by 23.2% and 7.8%) with CVaR0.1\text{CVaR}_{0.1} extractor.

On six risk-sensitive Stochastic-D4RL datasets with a CVaR extractor, ReBF attains a 3.83.8–7.3×7.3\times lower W1W_1 and a higher CVaR return than PCBF on every dataset, with the largest gains in CVaR return and safety on the failure-prone datasets.

On limited-data scenario, with only 25% samples, ReBF achieves a higher success rate on nine of the ten tasks, demonstrating ReBF can improve end-to-end control when offline coverage is limited. Among 10 cube-double tasks, ReBF (14.4%) doubles the average success rate of PCBF (7.2%), and on 10 puzzle-4x4 tasks, ReBF improves the average success rate from 16.8% to 28.4%.

Regarding (3), the control performance is affected by multiple factors, e.g., flow matching ODE steps, policy extraction, realistic environment, etc. Therefore, distributional fidelity alone is not sufficient to determine the final result. We have shown that, on long-horizon tasks, e.g., antmaze-teleport, switching policy extractor from mean to CVaR makes PCBF performance drastically worse while ReBF remains stable. Besides, on risk-sensitive control, ReBF achieves a 3.8–7.3× lower W1 and a higher CVaR return than PCBF on every dataset, and it reduces the fall rate from 0.85 to 0.03 on Hopper-medium-replay, that is, ReBF's critic captures the risk of falling, whereas PCBF's does not. These results show that distributional fidelity matters most when the task carries real downside risk and the decision depends on the lower tail.

We hope this response clarifies our contribution and addresses the reviewers' main concerns. We sincerely thank the Area Chair and reviewers for the time, careful assessment, and constructive feedback.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.