acceptodds
Under review as a conference paper at ICLR 2027

Flow-time Bellman Matching for Offline Distributional Reinforcement Learning

Abstract

Distributional reinforcement learning models the full return distribution rather than only its expectation, providing a richer representation of future returns for value learning. Flow matching provides a flexible framework for modeling return distributions, but existing approaches typically impose Bellman consistency only on the terminal return distribution, leaving the temporal structure of the transport process unused. We introduce Flow-time Bellman Matching (FBM), which extends Bellman regression from the terminal return distribution to the entire return flow by constructing Bellman targets from the successor flow at each corresponding flow time. We show that the proposed flow-time Bellman operator is consistent with the standard distributional Bellman operator at the endpoint. To enable efficient evaluation, we further distill the return flow into a one-step quantile critic using an exact correspondence between Gaussian source noise and return quantiles. Experiments on OGBench and D4RL show that achieves the best or near-best performance on 25 of 37 state-based tasks, while controlled analyses demonstrate the contribution of flow-time Bellman supervision and quantile distillation.

Primary area:
reinforcement learning
Keywords:
Offline Reinforcement Learning, Flow Matching, Distributional Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_91JaOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Flow-Time Bellman Matching (FBM) for offline distributional reinforcement learning. Its Bellman update combines a successor return at the same flow time with a time-scaled reward and Gaussian correction, preserving the Gaussian source and recovering the usual Bellman update at the endpoint. A scalar flow supplies same-quantile targets for a one-step critic, which also receives a quantile Bellman loss and ranks actions from a learned behavior proposal. Validation comprises OGBench/Adroit control results, six analytic return-distribution experiments, and an auxiliary-loss ablation.

Critical strengths
  1. The source-preserving path construction is well motivated. Equations (8)-(11) give a concrete way to reconcile intermediate-time Bellman updates with the prescribed Gaussian source. Propositions 1-2 provide a coherent population-level, fixed-policy foundation. The order-preserving scalar ODE argument in Section 4.2/Appendix B.4 also gives a principled quantile coordinate for distillation.

  2. The analytic distribution-learning study provides useful controlled evidence. Appendix F.1 matches initialization, transition batches, flow times, noise, networks, and optimization across FBM and endpoint CFM. Tables 2/6 report improvements across several distinct return structures with five seeds. The appendix also explicitly reports rare-mode deterioration and unresolved tail/mode errors, which helps establish the scope of the observed benefit.

Critical weaknesses
  1. The central contrast with the closest cited method needs revision. The abstract and Introduction motivate FBM by describing existing flow critics as relying on terminal Bellman supervision. However, Value Flows (Dong et al.; Sections 4.1-4.2, Equations (2), (3), and (5)) already formulates Bellman updates at intermediate flow times and a corresponding DCFM objective; it additionally uses an endpoint BCFM term for stability. FBM's time-scaled reward and Gaussian-source-preserving correction are concrete differences, but the paper does not adequately explain their incremental theoretical and algorithmic contribution relative to that construction. This distinction is central to assessing significance. Reference: https://arxiv.org/html/2510.07650v4

  2. The control and efficiency evidence does not sufficiently isolate the proposed contributions. The matched FBM-versus-endpoint-CFM experiment is conducted on analytic fixed-policy processes, while the control comparison evaluates complete algorithms with different critic objectives/readouts. Figure 4 varies the auxiliary quantile Bellman weight but does not isolate the flow-time update itself or establish the necessity of the flow teacher. In addition, Section 5 asks about computational savings from distillation, but no direct latency or speed-accuracy comparison is reported. Value Flows already estimates expected Q through its initial vector field (Section 4.3, Equation (6)), so comparison against that readout is especially relevant. These gaps leave the source of the control gains and the practical benefit of the additional teacher/student system uncertain.

My current recommendation is weak rejection because the advance over the closest baseline and its contribution to practical control are not sufficiently established.

Questions for authors
  1. Please specify the mathematical and implementation differences between FBM and the intermediate-time DCFM plus endpoint BCFM objectives of Value Flows. Which losses, uncertainty weighting, and policy readout were used in the reported Value Flows baseline?

  2. On representative control tasks, what happens when only the FBM teacher objective is replaced by endpoint CFM while keeping the student, distillation, auxiliary quantile loss, proposal, and sampling budgets fixed? Is the reported IQN baseline exactly the corresponding student-only ablation with the teacher/distillation removed, or does it change other components?

  3. What are the measured action-selection latency and control/value-approximation tradeoffs for the distilled critic, direct flow evaluation, and the baseline's initial-vector-field readout under matched candidate budgets? Please include the additional training cost of the teacher/student system.

These questions concern the existing contribution and evaluation; their answers could materially change my recommendation.

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer 91ja

Q1. Difference from Value Flows' DCFM + BCFM.

FBM differs from Value Flows by ensuring Gaussian-source preservation and exact path–velocity consistency within a single intermediate-time Bellman construction, whereas Value Flows relies on separate DCFM and BCFM objectives. We identify a specific inconsistency in Value Flows' Proposition 1 (Appendix B.1): the transition from step (c) to (d) overlooks the coordinate scaling ∂z=γ−1∂x\partial_z=\gamma^{-1}\partial_x, where x=(z−r)/γx=(z-r)/\gamma. Consequently, its Eq. (3) omits the factor γ\gamma required by the continuity equation. Indeed, for Zt=r+γZt′Z_t=r+\gamma Z'_t, the correct velocity is Z˙t=γvk(Zt′,t)\dot Z_t=\gamma v_k(Z'_t,t), whereas DCFM uses vk(Zt′,t)v_k(Z'_t,t). Thus, the claimed correspondence between its Bellman-updated density and velocity field does not generally hold. Consistent with this issue, Value Flows reports that DCFM-only training produces a divergent vector field and performs poorly without BCFM regularization (Appendix C.1 and Figure 4). FBM instead derives its velocity directly from a source-preserving Bellman path, ensuring both pathwise consistency and the correct boundary conditions.

Our Value Flows baseline follows the original implementation, combining uncertainty-weighted DCFM and BCFM objectives, retaining the original uncertainty weighting and behavior-flow-based candidate generation, and estimating action values through the initial-vector-field readout Q^(s,a)=Eϵ∼N(0,1)[vθ(ϵ,0,s,a)]\widehat Q(s,a)=\mathbb E_{\epsilon\sim\mathcal N(0,1)}[v_\theta(\epsilon,0,s,a)].

Q2. Endpoint CFM teacher replacement.

We provide additional OGBench experiments in which only the FBM teacher objective is replaced with standard endpoint CFM, while keeping the student, distillation, auxiliary quantile Bellman loss, action proposal, and sampling budgets unchanged. We clarify that IQN is an independent baseline, not a student-only ablation of FBM. Even after removing the teacher and distillation, our remaining quantile critic is not equivalent to IQN: IQN uses randomly paired quantile samples for Bellman regression, whereas our method employs a different quantile-training formulation without such random pairing. Therefore, the reported IQN results should not be interpreted as an exact teacher-removal ablation.

OGBench TaskFBM (Ours)Endpoint CFM + Student
cube-double-task194 ± 396 ± 2
cube-double-task282 ± 479 ± 5
puzzle-3x3-task1100 ± 092 ± 3
puzzle-3x3-task2100 ± 027 ± 7
puzzle-4x4-task174 ± 228 ± 0
puzzle-4x4-task216 ± 17 ± 1
Average77.754.9

Q3. Training and inference efficiency.

We report training wall-clock time and action-selection latency for FBM and the baselines under matched hardware and candidate budgets. We additionally compare FBM's distilled student with direct flow evaluation and Value Flows' initial-vector-field readout to quantify the inference-efficiency gains and additional training cost.

MethodTraining Time (h)Inference (ms)
BC0.61151.9610
IQL1.16103.1700
ReBRAC1.13203.1485
FBRAC1.27553.0615
FQL1.18602.9560
IFQL1.15703.0930
C511.30403.1535
IQN4.47501.9615
CODAC3.05552.9695
Value Flows4.07303.1685
FBM (Ours)7.30102.9470
Reviewer_VvsaOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary
  1. The paper proposes Flow-Time Bellman Matching (FBM), a flow-based distributional critic for offline reinforcement learning. Rather than constructing a conditional-flow-matching target only from a sampled terminal Bellman return, FBM defines a time-indexed Bellman operator, where the additional Gaussian innovation preserves the prescribed Gaussian source distribution at t=0t=0, while B1πB_1^\pi recovers the standard distributional Bellman operator. The paper proves that BtπB_t^\pi is a γ\gamma-contraction in maximal Wasserstein distance at every flow time and derives a flow-matching regression target using the successor vector field.

For efficient policy extraction, the paper exploits monotonicity of one-dimensional ODE flows: the Gaussian source rank τ=Φ(ξ)\tau=\Phi(\xi) is preserved by the flow. This implicit flow critic is distilled into a feed-forward quantile critic. The student is trained using both same-rank flow distillation and a conventional all-pairs quantile Bellman objective. Candidate actions sampled from a learned behavior flow are ranked using the estimated mean of this student critic.

  1. The empirical evaluation includes analytically tractable Markov reward processes and offline-control benchmarks from OGBench and D4RL. The controlled experiments report improved Wasserstein accuracy relative to endpoint conditional flow matching, while the complete method obtains strong aggregate offline-control results.
Critical strengths

The major strength is the construction

BtπZt=tr+γZt′+(1−t)1−γ2ηB_t^\pi Z_t = t r+\gamma Z_t' +(1-t)\sqrt{1-\gamma^2}\eta

simultaneously preserves the standard Gaussian source at t=0t=0, recovers the ordinary distributional Bellman operator at t=1t=1, and yields a closed γ\gamma-contractive update at every intermediate flow time. This is mathematically clean and addresses a genuine source-boundary difficulty in naive pointwise Bellman transformations of flow paths.

Critical weaknesses

1. The novelty relative to published PCBF is insufficiently established.

The paper has addressed the major challenge, design a mechanism to satisfy both gaussian noise at t=0 and bellman consistent target at t=1. However, PCBF identifies the exact challenge and already proposed source-consistent Bellman-interpolated paths, successor-velocity supervision at intermediate flow times, a λ\lambda-parameterized control variate, a distribution-free bias decomposition, a linear-Gaussian bias–variance analysis, and Bellman contraction results.

More over, in the later section, PCBF has already considered a correlated source X0=ρX0′+1−ρ2ηX_0 = \rho X_0' + \sqrt{1-\rho^2}\eta and the straight bridges Zt′=(1−t)X0′+tX1′Z_t'=(1-t)X_0'+tX_1' and xt=(1−t)X0+t(r+γX1′)x_t=(1-t)X_0+t(r+\gamma X_1'). Substituting the source and using γZt′=γ(1−t)X0′+γtX1′\gamma Z_t'=\gamma(1-t)X_0'+\gamma t X_1' gives

xt=tr+γZt′+(1−t)(ρ−γ)X0′+(1−t)1−ρ2ηx_t = tr + \gamma Z_t' + (1-t)(\rho-\gamma)X_0' + (1-t)\sqrt{1-\rho^2}\eta.

The residual (1−t)(ρ−γ)X0′(1-t)(\rho-\gamma)X_0' is the explicit cross-conditioning term. It is proportional to ρ−γ\rho-\gamma and vanishes for every tt exactly when ρ=γ\rho=\gamma, which leaves the FBM state xt=tr+γZt′+(1−t)1−γ2ηx_t = tr + \gamma Z_t' + (1-t)\sqrt{1-\gamma^2}\eta.

The same identity produces the velocity. Let Δ′=X1′−X0′\Delta'=X_1'-X_0'. Then X1′=Zt′+(1−t)Δ′X_1'=Z_t'+(1-t)\Delta' and X0′=Zt′−tΔ′X_0'=Z_t'-t\Delta', so the sample Bellman velocity expands as

r+γX1′−X0=r+(γ−ρ)Zt′+[γ(1−t)+ρt]Δ′−1−ρ2ηr+\gamma X_1'-X_0 = r+(\gamma-\rho)Z_t' + \bigl[\gamma(1-t)+\rho t\bigr]\Delta' - \sqrt{1-\rho^2}\eta.

PCBF's control variate replaces the successor displacement by the successor velocity,

utλ=(r+γX1′−X0)+λ(v(Zt′,t)−Δ′)u_t^\lambda = (r+\gamma X_1'-X_0) + \lambda\bigl(v(Z_t',t)-\Delta'\bigr).

The coefficient of Δ′\Delta' is the variance-minimizing choice λ⋆(t)=γ(1−t)+ρt\lambda^\star(t)=\gamma(1-t)+\rho t. Substituting λ=λ⋆(t)\lambda=\lambda^\star(t) cancels Δ′\Delta' and leaves

utλ⋆=r+(γ−ρ)Zt′+λ⋆(t)v(Zt′,t)−1−ρ2ηu_t^{\lambda^\star} = r + (\gamma-\rho)Z_t' + \lambda^\star(t) v(Z_t',t) - \sqrt{1-\rho^2}\eta.

Thus ρ=γ\rho=\gamma removes the cross-conditioning term and forces λ⋆(t)=γ\lambda^\star(t)=\gamma for every tt, so ut=r+γv(Zt′,t)−1−γ2ηu_t = r + \gamma v(Z_t',t) - \sqrt{1-\gamma^2}\eta. Together with the state identity above, this is exactly FBM's state and velocity target. The algebra uses the straight successor bridge; it does not use FBM's ODE characteristic.

The potentially new distinction is that FBM evaluates the successor's actual ODE characteristic ϕθˉ,t(ξ′)\phi_{\bar\theta,t}(\xi') rather than PCBF's straight source-to-generated-endpoint interpolation, and then promotes this to a closed marginal operator. The manuscript does not isolate this distinction theoretically or experimentally. In the absence of a PCBF-(ρ,λ)=(γ,γ)(\rho,\lambda)=(\gamma,\gamma) baseline, it is unclear whether FBM's gains arise from the source correlation, the particular (γ,γ)(\gamma,\gamma) selection, the fixed velocity coefficient, the actual ODE trajectory, or the auxiliary quantile critic.

I am also not yet convinced that replacing the conditional straight bridge by the model's ODE characteristic is, by itself, a sufficiently consequential contribution. In standard conditional flow matching, the straight conditional path is a tractable training construction: at the population optimum, regressing its conditional velocity yields a marginal vector field whose ODE reproduces the prescribed marginal probability path. Therefore, at an exact canonical solution, moving from a straight conditional-path sample to the corresponding ODE characteristic does not create a new target marginal path; it changes the samplewise coupling and, potentially, the off-manifold training dynamics. Such a difference can matter, but the paper should demonstrate that it matters rather than treating the use of the ODE trajectory as intrinsically superior. There is also a computational trade-off: once a successor endpoint has been generated, the straight interpolant is available in closed form, whereas the ODE characteristic requires numerical integration to the sampled time. This comparison is not one-sided because PCBF itself must integrate the successor flow to obtain its endpoint, while FBM's quantile-distillation stage also requires terminal integration; accordingly, wall-clock and vector-field-evaluation comparisons are needed.

A second issue is that ρ=γ\rho=\gamma should be interpreted as a structural choice for same-time marginal closure, not obviously as the statistically optimal source coupling. For a general correlated source X0=ρX0′+1−ρ2ηX_0=\rho X_0'+\sqrt{1-\rho^2}\eta, the straight Bellman bridge contains an additional residual term (1−t)(ρ−γ)X0′(1-t)(\rho-\gamma)X_0'. Choosing ρ=γ\rho=\gamma uniquely removes this explicit source residual and makes the update close on the same-time successor marginal alone. However, PCBF's analysis already shows that source coupling participates in a bias--variance trade-off: in its linear-Gaussian model, the cross-conditioning bias is proportional to ρ−γ\rho-\gamma, while the variance-minimizing correction is λ⋆(t)=γ(1−t)+ρt\lambda^\star(t)=\gamma(1-t)+\rho t. Hence ρ=γ\rho=\gamma removes that intrinsic bias and yields λ⋆=γ\lambda^\star=\gamma, but it also introduces an independent innovation term of scale 1−γ2\sqrt{1-\gamma^2} in FBM's raw velocity target. A coupling closer to shared noise can reduce this innovation at the price of cross-conditioning bias. Therefore, the relevant question is not simply whether ρ=γ\rho=\gamma is unbiased, but whether it minimizes the finite-sample regression risk or downstream distributional error. The paper currently provides no such optimality result.

This is consequential because the paper currently presents “flow-time Bellman supervision” and successor-velocity variance reduction as central novelties, although those broader ideas are already contained in PCBF. The strongest novelty claim would need to show that the ODE-characteristic/same-time-closure choice offers a demonstrable advantage over the broader PCBF coupling family, not merely that it is algebraically exact under one particular coupling.

2. The same-rank quantile distillation is useful but is not clearly novel, and it is not specific to FBM.

The paper presents the one-dimensional noise--quantile correspondence and same-rank distillation into qψ(s,a,τ)q_\psi(s,a,\tau) as a further contribution. However, the broader idea already appears in FlowIQN (Groom et al., 2026, Quantile-Coupled Flow Matching for Distributional Reinforcement Learning, arXiv:2605.08515), which explicitly uses quantile-coupled flows and discusses same-quantile teacher--student distillation. More fundamentally, FBM's identity qflow(s,a,τ)=ϕ1s,a(Φ−1(τ))q^{\mathrm{flow}}(s,a,\tau) = \phi_1^{s,a}(\Phi^{-1}(\tau)) is a generic property of any regular one-dimensional invertible ODE flow: order preservation converts the Gaussian source rank into the terminal quantile rank. It therefore does not depend on the FBM Bellman operator. A scalar PCBF or value flow critic has the same monotone terminal map and can define exactly the same post-hoc quantile function and be distilled into the same one-step student.

This weakens the paper's novelty decomposition. Quantile distillation may still be a useful engineering component, but it should be presented as an orthogonal readout/compression mechanism rather than as evidence of novelty of the FBM operator. A direct comparison of the same student distilled from FBM and PCBF teachers would help determine whether the student adds anything method-specific. I also do not see why the distillation term needs an asymmetric quantile-Huber loss once the teacher has already supplied the quantile label τ\tau; symmetric Huber or squared same-rank regression would be natural alternatives, and FlowIQN gives a direct one-dimensional Wasserstein interpretation for same-rank squared-error distillation.

3. The downstream evaluation does not isolate the contribution of the FBM operator.

The deployed policy does not directly use the FBM flow critic. It uses a separate quantile critic trained with Ldistill+λQRLQRL_{\mathrm{distill}} + \lambda_{\mathrm{QR}} L_{\mathrm{QR}}, where LQRL_{\mathrm{QR}} is itself a conventional distributional Bellman objective. The student selects the successor actions used in later FBM updates. Thus the practical algorithm is a coupled two-critic control system, not simply an implementation of the fixed-policy operator analyzed in Proposition 1.

The auxiliary quantile Bellman objective could correct or mask errors in the flow teacher. Consequently, the offline-control results do not establish that the FBM operator is better than PCBF, endpoint CFM, or a direct quantile critic under an identical policy-extraction mechanism. The paper needs a cleaner attribution experiment, especially because the controlled distribution-learning experiments compare primarily against endpoint CFM rather than PCBF.

Questions for authors

1. Clarification of the central novelty relative to PCBF.

Under PCBF's generalized correlated-source analysis, let X0=ρX0′+1−ρ2ηX_0=\rho X_0'+\sqrt{1-\rho^2}\eta. For the straight successor bridge Zt′=(1−t)X0′+tX1′Z_t'=(1-t)X_0'+tX_1', setting ρ=γ\rho=\gamma gives (1−t)X0+t(r+γX1′)=tr+γZt′+(1−t)1−γ2η(1-t)X_0+t(r+\gamma X_1') = tr+\gamma Z_t'+(1-t)\sqrt{1-\gamma^2}\eta. Furthermore, choosing PCBF's correction coefficient λ=γ\lambda=\gamma appears to give the FBM-form velocity target.

Is this derivation correct? If so, what precisely is the methodological contribution of FBM beyond the (ρ,λ)=(γ,γ)(\rho,\lambda)=(\gamma,\gamma) specialization of PCBF's published framework?

My current understanding is that the remaining distinction is Zt′,PCBF=(1−t)X0′+tX1′Z_t^{\prime,\mathrm{PCBF}}=(1-t)X_0'+tX_1' versus Zt′,FBM=ϕθˉ,t(X0′)Z_t^{\prime,\mathrm{FBM}}=\phi_{\bar\theta,t}(X_0').

Please explain whether replacing the straight bridge with the learned ODE characteristic gives a formally identifiable advantage, e.g., eliminating PCBF's cross-conditioning bias beyond the Gaussian analysis, defining a genuinely new off-manifold operator, or yielding another property unavailable to PCBF.

A related point is that ρ=γ\rho=\gamma appears to be chosen because it gives exact same-time closure. Is this choice also statistically optimal in any sense, or is it primarily a structural choice within a broader bias--variance trade-off?

2. Applicability of Proposition 3 to the implemented target.

Proposition 3 uses the population conditional velocity of the straight CFM bridge, vsuc⋆(z,t)=E[Z1′−ξ′∣Zt′=z]v_{\mathrm{suc}}^\star(z,t)=\mathbb{E}[Z_1'-\xi'\mid Z_t'=z], while Algorithm 1 evaluates the learned field along its own ODE characteristic, zt′=ϕθˉ,t(ξ′)z_t'=\phi_{\bar\theta,t}(\xi').

Under what assumptions does the variance-reduction statement in Proposition 3 apply to the actual implemented target during iterative training, rather than only to an exact canonical population solution?

In particular, can the authors clarify whether the relevant conditional regression noise E ⁣[Var⁡(Ut∣Xt,H)]\mathbb{E}\!\left[\operatorname{Var}(U_t\mid X_t,H)\right] is reduced by FBM relative to endpoint CFM or PCBF? A theoretical clarification or an existing diagnostic would be sufficient for rebuttal.

3. What component is responsible for the empirical gain?

The final policy-facing critic is trained with both same-rank flow distillation and an independent quantile Bellman loss, Ldistill+λQRLQRL_{\mathrm{distill}}+\lambda_{\mathrm{QR}}L_{\mathrm{QR}}.

Moreover, same-rank flow-to-quantile distillation is not specific to FBM: FlowIQN already uses same-quantile teacher--student supervision, and a scalar PCBF flow can likewise define q(s,a,τ)=ϕ1s,a ⁣(Φ−1(τ))q(s,a,\tau)=\phi_1^{s,a}\!\left(\Phi^{-1}(\tau)\right).

Can the authors clarify how much of the downstream performance comes from the FBM flow operator versus the policy-facing quantile student?

If available, the most informative comparison would be either:

  1. the same quantile student distilled from a PCBF teacher; or
  2. the student trained only with LQRL_{\mathrm{QR}}.

I do not view a full new benchmark sweep as necessary for rebuttal; existing ablations or a clear explanation of what was already controlled would be sufficient to help resolve this attribution issue.

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer Vvsa (Part I/II)

We thank the reviewer for the thoughtful and detailed feedback. For clarity, we organize our responses by specific technical concerns raised across the reviewer's weaknesses and questions.

R1. Methodological distinction from PCBF.

We agree that combining PCBF's correlated-source formulation with ρ=λ=γ\rho=\lambda=\gamma produces an algebraically similar expression. However, FBM is not a special case of PCBF: setting ρ=λ=γ\rho=\lambda=\gamma only yields an algebraically similar velocity target, while PCBF still relies on a control-variate correction to a straight Bellman bridge that may alter its marginal flow, whereas FBM derives the exact velocity of its ODE-characteristic Bellman path.

First, PCBF requires an additional control variate to obtain the FBM-like target. Under ρ=γ\rho=\gamma, the exact derivative of the reviewer's straight Bellman bridge is X˙t=r+γΔ′−1−γ2η\dot X_t=r+\gamma\Delta'-\sqrt{1-\gamma^2}\eta

where Δ′=Z1′−X0′\Delta'=Z'_1-X'_0.

PCBF's FBM-like expression is obtained only after introducing the correction

utγ=X˙t+γ[vθ(Zˉt′,t)−Δ′]u_t^\gamma=\dot X_t+\gamma[v_\theta(\bar Z'_t,t)-\Delta']

with Zˉt′=(1−t)X0′+tZ1′\bar Z'_t=(1-t)X'_0+tZ'_1. This correction is not generally the pathwise derivative of PCBF's constructed bridge. Moreover, as analyzed in PCBF's Propositions 5.6–5.7, a nonzero conditional expectation of the correction changes the marginal velocity field, so the corrected target is not guaranteed to generate the intended marginal flow for an arbitrary learned successor critic.

Second, FBM derives an exact velocity without introducing any control-variate correction. FBM directly constructs XtFBM=tr+γϕθˉ,t(ξ′)+(1−t)1−γ2ηX_t^{\mathrm{FBM}}=tr+\gamma\phi_{\bar\theta,t}(\xi')+(1-t)\sqrt{1-\gamma^2}\eta from the learned successor ODE characteristic. Differentiating this path yields X˙tFBM=r+γvθˉ(ϕθˉ,t(ξ′),t)−1−γ2η\dot X_t^{\mathrm{FBM}}=r+\gamma v_{\bar\theta}(\phi_{\bar\theta,t}(\xi'),t)-\sqrt{1-\gamma^2}\eta exactly. Thus, the velocity target is consistent with its own Bellman path for any learned successor ODE, without requiring the control-variate correction to be conditionally unbiased or the successor critic to have reached the optimal CFM solution.

Third, PCBF's variance-optimality guarantee does not extend to its implemented coupling. The reviewer's correspondence requires the specific choice ρ=λ=γ\rho=\lambda=\gamma. Although PCBF's Appendix C.9 derives λ∗(t)=γ(1−t)+ρt\lambda^*(t)=\gamma(1-t)+\rho t, this variance-minimizing coefficient relies on the assumption that the terminal successor return Z1′Z'_1 is independent of the source noises. This assumption is incompatible with PCBF's actual ODE coupling, where Z1′=ϕ1(X0′)Z'_1=\phi_1(X'_0) is a deterministic function of the source. Hence, the analysis does not establish that λ=γ\lambda=\gamma minimizes variance in the implemented setting. More fundamentally, PCBF obtains its FBM-like target by introducing a control-variate correction to a straight-bridge velocity, whereas FBM derives the coefficient γ\gamma and its velocity target directly by differentiating the Bellman ODE characteristic, ensuring exact pathwise consistency without any variance-optimality assumption.

Fourth, the two methods construct fundamentally different Bellman paths. PCBF uses the straight successor bridge Zˉt′=(1−t)X0′+tϕ1(X0′)\bar Z'_t=(1-t)X'_0+t\phi_1(X'_0) and subsequently modifies its velocity through a control variate. FBM instead constructs its Bellman path directly from the actual successor ODE characteristic Zt′=ϕt(X0′)Z'_t=\phi_t(X'_0) and obtains the velocity by exact differentiation. Therefore, although the resulting expressions can appear algebraically similar under ρ=λ=γ\rho=\lambda=\gamma, PCBF's corrected velocity is not generally the derivative of its constructed path.

R2. Advantage of ODE characteristics.

PCBF obtains the FBM-like velocity expression by modifying the derivative of a different, straight-interpolated path; this modification may change the corresponding marginal flow unless additional consistency conditions hold. FBM instead derives its velocity directly from its own source-preserving ODE-based Bellman path. Therefore, the reviewer's derivation establishes an algebraic resemblance under a particular parameter specialization, not equivalence of the flow constructions, training objectives, or their consistency guarantees.

R3. Structural role of the discount coefficient.

We clarify that the coefficient γ\gamma is structurally determined by FBM's exact same-time Bellman consistency. It uniquely eliminates the source-dependent residual (1−t)(ρ−γ)ξ′(1-t)(\rho-\gamma)\xi' within the considered coupling family, yielding a closed same-time Bellman operator that preserves the Gaussian source, recovers the standard distributional Bellman operator at t=1t=1, and satisfies the γ\gamma-contraction in Proposition 1. These guarantees hold for general return distributions without PCBF's restrictive linear-Gaussian assumptions. While alternative couplings may offer different bias–variance trade-offs, our priority is exact same-time Bellman consistency rather than minimizing estimator variance.

AuthorsOpenReview ↗

Author Response to Reviewer Vvsa (Part II/II)

R4. Performance attribution.

We supplement the OGBench experiments with two ablations: PCBF with ρ=λ=γ\rho=\lambda=\gamma and FBM (Flow-only), which removes the quantile student and auxiliary Bellman loss. The results are reported below. These results show that FBM's performance gains arise from its ODE-characteristic Bellman construction rather than merely from source correlation, coefficient selection, or the auxiliary quantile critic.

OGBench TaskFBM (Full)PCBF (ρ=λ=γ\rho=\lambda=\gamma)FBM (Flow-only)
cube-double-task194 ± 380 ± 292 ± 11
cube-double-task282 ± 457 ± 767 ± 7
puzzle-3x3-task1100 ± 050 ± 4100 ± 0
puzzle-3x3-task2100 ± 023 ± 197 ± 2
puzzle-4x4-task174 ± 228 ± 653 ± 1
puzzle-4x4-task216 ± 12 ± 325 ± 2
Average77.740.072.3

R5. Same-rank quantile distillation.

We respectfully clarify that the reviewer's interpretation of FlowIQN is inaccurate. FlowIQN focuses on optimal source–target coupling through sorting randomly sampled noise and Bellman-target samples; it does not establish FBM's intrinsic Gaussian-noise-to-return-quantile correspondence for an unconditional one-dimensional ODE flow. In contrast, FBM directly obtains the exact quantile label qθ(s,a,τ)=ϕθ,1(Φ−1(τ))q_\theta(s,a,\tau)=\phi_{\theta,1}(\Phi^{-1}(\tau)) through ODE order preservation, without random sample pairing, sorting, or additional quantile-conditioned flow training. This enables direct same-rank teacher–student supervision. Regarding the asymmetric quantile-Huber loss, as explained in Section 4.2 (Eq. 14) and Appendix A.2 (Eq. 27) of our submitted paper, it is a standard quantile-regression technique adopted for numerical stability and robustness, rather than a requirement for establishing the noise–quantile correspondence.

R6. Variance reduction under learned ODEs.

We clarify that Proposition 3 does not require the successor return distribution to have converged to the true Bellman fixed point: its variance inequality holds for any successor return distribution with finite second moment, provided the canonical conditional velocity is used. These assumptions and the variance-reduction proof are already provided in Proposition 3 (Section 4.1) and Appendix B.3 of our submitted paper. At the population CFM optimum, the straight bridge and its corresponding ODE induce identical time-tt marginals, so the variance result applies to ODE-characteristic evaluation without requiring samplewise trajectory equality.

For an imperfect learned velocity, an unconditional variance-reduction guarantee is not claimed; nevertheless, Algorithm 1 always uses the exact derivative of its learned ODE characteristic, preserving pathwise consistency throughout training. We have revised the submitted manuscript to explicitly clarify that Proposition 3 does not guarantee variance reduction for arbitrary learned velocity fields during iterative training.

R7. Contribution of the flow critic.

We conduct additional OGBench experiments removing the quantile student, distillation, and auxiliary quantile Bellman loss, using the FBM flow critic directly for action selection. As shown below, this flow-only variant still achieves competitive performance, demonstrating that FBM's control capability does not depend on the student. Quantile distillation primarily serves as an efficient inference-time readout.

OGBench TaskFBM (Full)FBM (Flow-only)
cube-double-task194 ± 392 ± 11
cube-double-task282 ± 467 ± 7
puzzle-3x3-task1100 ± 0100 ± 0
puzzle-3x3-task2100 ± 097 ± 2
puzzle-4x4-task174 ± 253 ± 1
puzzle-4x4-task216 ± 125 ± 2
Average77.772.3
Reviewer_sfxsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Flow-Time Bellman Matching (FBM), a method to train flow-matching return critics for offline distributional RL. FBM differs from existing flow critics because existing methods first generate a full sample of the next state’s return and compute the target, then train the current flow to move noise toward that target. FBM does not generate the full next-state return and runs the next-state flow only part of the way and then use the next-state flow’s velocity to construct the current flow’s training target. It has results on OGBench and D4RL.

Critical strengths

The critical strength is that it is a simple and well motivated core idea, and the noise splitting trick is a good innovation. The experiment results on OGBench backs up the performance comparing to other flow matching critics. This matters because flow matching critics have shown good performance, and making the performance even better makes it more practical.

Critical weaknesses

The critic that actually selects actions is the distilled quantile critic that is trained with a full quantile Bellman loss, instead of having the flow teacher be the only critic that selects the actions. The matters because it does not isolate the main contribution of the paper which is the new critic formulation .

Questions for authors

What is the wall clock training time compared to other methods such as Value Flows and IQN?

How were hyperparameters selected to ensure fairness?

What is the result of the offline RL pipeline with a standard CFM teacher in place of the FBM teacher and the distillation loss removed?

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer sfxs

W1. Contribution of the flow critic.

To directly isolate the contribution of the FBM critic, we conduct additional OGBench control experiments without the quantile student, distillation, or auxiliary quantile Bellman loss, using the FBM flow critic itself for action selection. This flow-only variant still achieves competitive performance, demonstrating that FBM's control capability does not rely on the student or its independent Bellman objective. Quantile distillation is introduced primarily to provide efficient action-value evaluation at inference time, avoiding repeated ODE integration.

OGBench TaskFBM (Full)FBM (Flow-only)
cube-double-task194 ± 392 ± 11
cube-double-task282 ± 467 ± 7
puzzle-3x3-task1100 ± 0100 ± 0
puzzle-3x3-task2100 ± 097 ± 2
puzzle-4x4-task174 ± 253 ± 1
puzzle-4x4-task216 ± 125 ± 2
Average77.772.3

Q1. Training wall-clock time.

We report the measured wall-clock training times of FBM and the baselines below. All measurements use the same NVIDIA RTX 4090 GPU and a budget of one million gradient steps on the same OGBench task. FBM incurs a modest additional training cost.

MethodTraining Time (h)Inference (ms)
BC0.61151.9610
IQL1.16103.1700
ReBRAC1.13203.1485
FBRAC1.27553.0615
FQL1.18602.9560
IFQL1.15703.0930
C511.30403.1535
IQN4.47501.9615
CODAC3.05552.9695
Value Flows4.07303.1685
FBM (Ours)7.30102.9470

Q2. Hyperparameter-selection fairness.

For all baselines, we follow the hyperparameter settings reported in their original papers as closely as possible to ensure faithful comparisons. FBM introduces relatively few additional hyperparameters, and we prioritize shared configurations rather than extensive task-specific tuning. Our hyperparameter selection protocol and detailed configurations are documented in Appendix D (Tables 3–4) of the submitted paper.

Q3. Endpoint CFM and distillation ablations.

We conduct additional OGBench experiments with distillation and the quantile student removed, using the flow critic directly for action selection. As shown below, standard endpoint CFM performs poorly, whereas FBM achieves competitive performance under the same flow-only pipeline. We additionally include a CFM variant with the quantile student and distillation to compare FBM and CFM both with and without student distillation. These comparisons isolate the contribution of FBM's flow-time Bellman supervision from the effect of quantile distillation.

OGBench TaskFBM (Full)FBM (Flow-only)CFM (Flow-only)CFM (+ Student)
cube-double-task194 ± 392 ± 1196 ± 396 ± 2
cube-double-task282 ± 467 ± 756 ± 1979 ± 5
puzzle-3x3-task1100 ± 0100 ± 081 ± 1092 ± 3
puzzle-3x3-task2100 ± 097 ± 271 ± 727 ± 7
puzzle-4x4-task174 ± 253 ± 135 ± 628 ± 0
puzzle-4x4-task216 ± 125 ± 23 ± 27 ± 1
Average77.772.357.054.9

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.