acceptodds
Under review as a conference paper at ICLR 2027

Flow-Time Bellman Propagation for Generative Policy Improvement

Abstract

Flow policies represent complex action distributions by transforming noise into actions through a sequence of generation steps. To learn useful behavior, these policies must incorporate value information into the generation process. Value-weighted flow matching favors high-value training actions, but leaves the target direction for each noise–action pair unchanged. We propose Flow-Time Bellman Propagation (FTBP), which learns value feedback for intermediate generation states and uses it to train the flow. A continuation potential predicts the critic value of the action obtained by completing the current flow. We train this potential with endpoint critic supervision and local bootstrap targets, avoiding a separate completed rollout for each intermediate target. The learned potential then guides training through successor evaluation (FTBP-S), target shaping (FTBP-T), or their combination (FTBP-C). During online fine-tuning, recorded generation traces provide additional supervision under the same policy objectives. Our analysis relates the two updates and characterizes value propagation for a matched finite-grid sampler. Experiments demonstrate that FTBP delivers strong and consistent improvements in both offline learning and online fine-tuning.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, offline-to-online reinforcement learning, flow matching, bellman potential

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kakMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FTBP to improve flow policies using value feedback during denoising. It learns a value function B that predicts the final action’s Q-value from an intermediate point, using endpoint supervision and local TD targets. B guides training by scoring the next step, adjusting the flow-matching target, or combining both. Online generation traces provide additional supervision. Experiments on OGBench and D4RL AntMaze report improvements in offline learning and online fine-tuning.

Critical strengths

The writing, framework and experiments are very complete. The paper evaluates both offline learning and online fine-tuning, with baseline comparisons and ablations of the main components. These experiments provide useful evidence that the method can improve performance in the tested settings.

Critical weaknesses
  1. My main concern is the reliability of B. The velocity network may exploit overestimated values, but I did not see experiments directly checking B’s prediction accuracy. The theoretical analysis also does not establish that this guidance remains reliable during joint training.

  2. I do not see a clear need for C. S and T both use B’s gradients to guide the velocity, so combining them may simply increase the guidance strength. The current comparisons do not show whether C still helps when this strength is matched, leaving its additional contribution unclear.

Questions for authors

How reliable are B’s predictions during training? What happens if B overestimates values?

5 comments
AuthorsOpenReview ↗

Response to Reviewer kakM (1/3)

We sincerely thank the reviewer for the careful assessment of our work and the thoughtful, constructive comments. We appreciate the recognition of the thoroughness of our framework and experimental evaluation, and of the evidence for performance improvements in the tested settings. The questions help us assess the reliability of the learned potential during joint training, examine the consequences of overestimation, and clarify the additional contribution of combining successor evaluation and target shaping. Our responses to each point are provided below.

R3-1. Prediction accuracy during training

FTBP learns reusable intermediate value feedback through endpoint supervision and local Bellman backups, and uses this feedback to improve flow-policy training. B predicts the critic value of completing the remaining flow from an intermediate state. To evaluate its accuracy, we freeze the current flow, target critic, and target potential at each checkpoint and compute

F(t,s,x)=Qtarget(s,clip⁡(Φt→1θ(s,x))).F(t,s,x)=Q_{\mathrm{target}}(s,\operatorname{clip}(\Phi^\theta_{t\to1}(s,x))).

Here B denotes the target potential used for policy feedback. For this diagnostic reference, the suffix starts at the query time and follows the current velocity field. At time uu, each Euler step uses Δ(u)=min⁡(h,1−u)\Delta(u)=\min(h,1-u) for both time and spatial advancement, reaching t=1t=1 exactly. Clipping is applied only to the final suffix action. The reference ∇xF\nabla_xF differentiates through the complete continuation and endpoint clipping. This specifies the terminal-connected evaluation reference discussed in Appendix B.3; the training backups retain the sampler described in the paper.

Table 1a evaluates generation-grid states. Table 1b evaluates held-out continuous-time queries relevant to C's policy objective. FM interpolation uses xt=(1−t)z+tax_t=(1-t)z+ta, with z∼N(0,I)z\sim\mathcal N(0,I), a replay action aa paired with state ss, and t∼U(0,1)t\sim U(0,1). Flow successor uses xS+=Proj⁡A(xt+hSvθ(t,s,xt))x_S^+=\operatorname{Proj}_{\mathcal A}(x_t+h_Sv_\theta(t,s,x_t)) at tS+=t+hS<1t_S^+=t+h_S < 1. We retain only interior successor queries to avoid terminal-boundary effects.

Table 1a. FTBP-C continuation-field accuracy on generation-grid queries across training.

EnvironmentStageB NRMSE ↓Gradient cosine ↑Positive F gain ↑Pointwise coverage ↑
antmaze-large-navigateOffline 200K0.05530.537190.82%66.85%
Offline 1M0.04840.856593.54%88.39%
Online 100K0.04660.850193.86%90.77%
Online 500K0.04130.874694.76%91.35%
humanoidmaze-medium-navigateOffline 200K0.07170.765699.93%95.90%
Offline 1M0.04960.861299.82%96.41%
Online 100K0.03640.889899.93%96.18%
Online 500K0.03550.889299.95%96.70%
antsoccer-arena-navigateOffline 200K0.05250.677897.71%86.52%
Offline 1M0.04790.892093.09%89.65%
Online 100K0.04780.895194.21%92.37%
Online 500K0.03870.915395.47%93.89%

Table 1b. FTBP-C continuation-field accuracy at continuous-time policy-training queries.

EnvironmentStageQuery distributionB NRMSE ↓Gradient cosine ↑Positive F gain ↑Pointwise coverage ↑
antmaze-large-navigateOffline 1MFM interpolation0.07460.853191.97%85.63%
Online 500KFM interpolation0.07120.862992.76%89.08%
Offline 1MFlow successor0.06930.867092.64%87.25%
Online 500KFlow successor0.06250.874293.81%90.13%
humanoidmaze-medium-navigateOffline 1MFM interpolation0.08040.859198.36%94.72%
Online 500KFM interpolation0.07070.881098.55%95.62%
Offline 1MFlow successor0.07690.872499.47%95.96%
Online 500KFlow successor0.06380.890599.92%96.63%
antsoccer-arena-navigateOffline 1MFM interpolation0.08220.887991.42%86.19%
Online 500KFM interpolation0.06860.908293.58%88.93%
Offline 1MFlow successor0.07340.892692.75%89.07%
Online 500KFlow successor0.06190.918595.14%92.97%

Statistical protocol. All entries in Tables 1a and 1b are means across more than 10 independent training seeds. At each evaluated checkpoint, we use 512 held-out queries for each query distribution. Query-level metrics are first aggregated within each seed and then averaged across seeds, rather than treating individual queries as independent training runs. Environment names denote environment families, each aggregating five single-task goals. Online steps are counted from the start of fine-tuning.

AuthorsOpenReview ↗

Response to Reviewer kakM (2/3)

R3-1. Prediction accuracy during training (continued)

NRMSE is RMSE⁡(B−F)/Std⁡(F)\operatorname{RMSE}(B-F)/\operatorname{Std}(F), using each query distribution's own Std⁡(F)\operatorname{Std}(F); cosine compares raw spatial gradients. Across all distributions, coverage and finite-step gain use the common norm-clipped spatial gradient g=C(∇xB)g=\mathcal C(\nabla_xB). Coverage measures ∥g−∇xF∥<∥g∥\lVert g-\nabla_xF\rVert < \lVert g\rVert with g≠0g\ne0; positive F gain measures F(t,s,x+0.01g/∥g∥)>F(t,s,x)F(t,s,x+0.01g/\lVert g\rVert) > F(t,s,x). These metrics assess the spatial field at each query. S's projection Jacobian enters the velocity gradient and is included in the policy updates tested below.

Table 1a shows lower normalized value error and stronger gradient alignment at the final online checkpoint than early in offline training, with variation between stages. Table 1b extends these measurements to continuous-time queries relevant to policy training. The higher NRMSE on those distributions, together with positive gradient alignment and frequent positive finite-step gains, shows why value and directional accuracy should be assessed separately.

R3-2. Overestimation and joint policy updates

The concern is whether the policy can exploit an optimistic potential estimate and move toward a worse completed action. We examine this in two ways, by maximizing B at fixed model parameters and by comparing policy updates with and without B-dependent feedback.

Potential-ascent stress test. With B, the flow, and the critic frozen, we take 20 normalized B-ascent steps of size 0.02 within an L2 ball of radius 0.1 around each initial query. At fixed (t,s)(t,s), let x0x_0 and x20x_{20} denote the initial and final query points. We recompute

\Delta F=F(t,s,x_{20})-F(t,s,x_0).$$ Misleading ascent means $\Delta B > 10^{-6}$ and $\Delta F < -10^{-6}$. Training combines potential feedback with data-based FM regression, a fixed target potential within each policy update, and gradient-norm clipping in T. The paired experiment measures the effect of this feedback during joint optimization. **Paired policy updates.** We fork C and a feedback-off control from the same checkpoint. The control removes B's policy-feedback terms while retaining all other losses and target updates. B, the critic, and trace supervision remain active in both branches. The branches start from identical model, target-network, and optimizer states and share batches, RNG, and trace schedules. After 100 joint updates, we evaluate actions produced by the one-step actor in each branch using a common critic $Q_0$, frozen at the fork, on 2,048 held-out observations with four shared latent draws each. $$G_{100}=\mathbb E_{s,z}\big[Q_0(s,a^{\mathrm{actor}}_{C,100}(s,z))-Q_0(s,a^{\mathrm{actor}}_{\mathrm{off},100}(s,z))\big].$$ Here $a^{\mathrm{actor}}_{m,100}(s,z)=\operatorname{Proj}_{\mathcal A}f_{\omega_{m,100}}(s,z)$ is the one-step actor action after 100 updates in branch $m$. **Table 2. Stress and policy-update diagnostics at online 500K. Mean ± SD across more than 10 training seeds; pp denotes percentage points.** | Environment | Stress ΔB | Stress ΔF | B↑F↓ | Single-update C−FM (×10⁻⁵) | Joint C−FM (×10⁻⁴) | |---|---:|---:|---:|---:|---:| | antmaze-large-navigate | 0.0741 ± 0.0034 | 0.0341 ± 0.0011 | 4.63% ± 1.21 pp | 4.58 ± 0.92 | 3.57 ± 1.12 | | humanoidmaze-medium-navigate | 0.0491 ± 0.0011 | 0.0300 ± 0.0010 | 1.83% ± 0.54 pp | 3.28 ± 0.64 | 2.33 ± 0.87 | | antsoccer-arena-navigate | 0.0717 ± 0.0021 | 0.0361 ± 0.0023 | 8.26% ± 1.78 pp | 8.39 ± 1.04 | 5.36 ± 1.28 | FM denotes the feedback-off control. Single-update evaluates full-flow actions after one policy-objective update; Joint evaluates one-step actor actions after 100 full updates, with potential, critic, and trace learning retained in both branches. Both columns report C-minus-control critic values under a common frozen evaluator. They measure distinct update protocols and action generators. The stress test produces positive mean ΔF in all three environments. Mean ΔB exceeds mean ΔF, quantifying optimism in the predicted improvement, while B↑F↓ identifies cases where that optimism leads to a worse continuation value. The joint updates give positive mean C−control gains for the one-step actor. Using the same frozen critic attributes the comparison to the resulting actor actions rather than evaluation-critic drift. Together with Tables 1a and 1b, these results assess prediction accuracy, potentially exploitable errors, and how intermediate feedback supplied to flow training affects the executed actor.
AuthorsOpenReview ↗

Response to Reviewer kakM (3/3)

R3-3. C at matched guidance strength

T specifies a target direction; S evaluates the proposed move. T shifts the data-based FM target using the clipped potential gradient at the current interpolation point. This target is detached and fixed during the policy update. S instead differentiates the potential at the successor proposed by the current velocity. Its feedback therefore depends on where that velocity leads and includes the derivative of action projection.

Equation (10) identifies their shared first-order direction in the smooth small-step regime with inactive projection and gradient clipping. At finite steps, the potential gradient can change between the current point and the successor, so the two signals need not be parallel. Projection can suppress S feedback in saturated coordinates, while the FM sample weights affect T's contribution. These differences can change guidance direction and its relative allocation across samples.

C combines these signals within one FM objective. It retains a single regression term and adds both B-dependent contributions. Rescaling either interface alone generally cannot reproduce this combined field. Matching the RMS of the combined signal to that of each individual interface tests whether this structure provides a benefit at the same overall guidance amplitude.

To control amplitude, define the B-dependent per-sample velocity-loss gradient for m∈S,T,Cm\in\\{S,T,C\\} as

\operatorname{RMS}_{\mathcal B}(d_m)=\sqrt{\mathbb E_{\mathcal B}\left[\frac{\lVert d_m\rVert_2^2}{d_a}\right]}.$$ Here, $\ell_{\mathrm{FM}}$ is the common weighted FM objective and $d_a$ is the action dimension. On each minibatch, we match this RMS across S, T, and C, including the loss weights, normalization factors, and projection derivatives. For C, we sum its S and T components within the same model before rescaling the combined signal to the common RMS. All methods use the same seed set, training budgets, and evaluation procedure. We use four tasks to examine whether C's gains in the original experiments persist after matching guidance strength. The offline tasks, antmaze-large-navigate-task4 and cube-double-play-task1, cover navigation and manipulation. The online tasks, antmaze-giant-navigate-task3 and antsoccer-arena-stitch-task5, extend the comparison to larger-scale navigation and the stitching setting. The original results include both a near-tie between C and T and larger separations, allowing us to assess how the contribution of combining S and T varies across settings. **Table 3. Success rates (%) with matched guidance strength. Mean ± SD across more than 10 training seeds.** | Task | Stage | FTBP-S | FTBP-T | FTBP-C | |---|---|---:|---:|---:| | antmaze-large-navigate-task4 | Offline 1M | 87 ± 4 | 92 ± 6 | **93 ± 4** | | cube-double-play-task1 | Offline 1M | 72 ± 7 | 73 ± 6 | **77 ± 6** | | antmaze-giant-navigate-task3 | Online 500K | 57 ± 9 | 57 ± 10 | **78 ± 7** | | antsoccer-arena-stitch-task5 | Online 500K | 51 ± 8 | 52 ± 6 | **57 ± 9** | We evaluate the executed one-step actor at the final checkpoint over 50 episodes per seed. Offline training uses 1M updates; online evaluation follows 500K interactions after offline training. C retains the highest mean at matched velocity-output RMS. The largest separation occurs on antmaze-giant-navigate-task3, with further mean advantages on cube-double-play-task1 and antsoccer-arena-stitch-task5. C and T perform similarly on antmaze-large-navigate-task4. This supports an additional, task-dependent benefit from combining successor evaluation and target shaping when the guidance amplitude is matched. ### **Closing remarks** Together, the experiments support the paper's central claim that a locally trained continuation potential can provide useful intermediate feedback for flow-policy improvement. The accuracy measurements track B as training progresses; the stress test and paired updates assess the consequences of optimizing its predictions; and the matched-strength comparison evaluates C's additional contribution while controlling the velocity-output guidance amplitude. The evidence supports useful feedback with measurable residual errors and a task-dependent benefit from combining S and T. We sincerely appreciate the reviewer's careful evaluation and constructive suggestions. The questions have helped us examine both the reliability of the shared potential and the contribution of the combined feedback mechanism more directly. We would be happy to clarify any aspect of the experiments or address any further questions.
Reviewer_XW1hOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose FTBP, which learns an intermediate flow-time potential intended to approximate the environmental value of completing the remaining flow trajectory. Terminal critic targets and local consistency backups train the potential, with recorded-trace supervision added online. The potential improves the policy through successor evaluation, velocity-target shaping, or their combination. The paper develops fixed-continuation results and reports improvements on several offline and offline-to-online control tasks using an existing one-step actor backbone.

Critical strengths
  1. This work proposed a practical mechanism for supplying intermediate feedback to a generative policy.
  2. The paper included offline and online evaluations, component ablations, and detailed per-task results across OGBench and AntMaze.
Critical weaknesses

please refer to questions.

Questions for authors
  1. Are there existing measurements of the potential's spatial-gradient agreement with the gradient obtained by completing the current flow and evaluating its terminal critic? In particular, what supports the gradient-error condition used in the improvement analysis beyond low value/consistency loss?
  2. It appears to me that the potential-learning objective does not sufficiently establish the guidance field used for policy improvement. As the authors explicitly demonstrate, a potential can satisfy both the local and terminal constraints while retaining incorrect intermediate values and spatial gradients. The empirical return gains are encouraging, is it possible the authors to make more discussion on whether the learned field reliably represents the intended continuation gradient. I would raise my score if this is resolved.
4 comments
AuthorsOpenReview ↗

Response to Reviewer XW1h (1/3)

We sincerely thank the reviewer for the careful reading of our paper and the thoughtful, constructive comments. We appreciate the recognition of FTBP's practical mechanism for intermediate feedback and of the breadth of our offline and online evaluations. The questions help us examine how accurately the learned potential represents the continuation gradient and make the connection between the learning objective, the improvement analysis, and policy feedback more explicit. Our responses to each point are provided below.

R2-1. Direct continuation-gradient measurements

We directly compare the learned spatial gradient with the gradient of the completed-flow continuation value. At each checkpoint, we freeze the current flow, target critic, and target potential and evaluate

F(t,s,x)=Qtarget(s,clip⁡(Φt→1θ(s,x))).F(t,s,x)=Q_{\mathrm{target}}(s,\operatorname{clip}(\Phi^\theta_{t\to1}(s,x))).

Here B denotes the target potential used for policy feedback. For this diagnostic reference, the suffix starts at the query time and follows the current velocity field. At time uu, each Euler step uses Δ(u)=min⁡(h,1−u)\Delta(u)=\min(h,1-u) for both time and spatial advancement, reaching t=1t=1 exactly. Clipping is applied only to the final suffix action. The reference ∇xF\nabla_xF differentiates through the complete continuation and endpoint clipping. This specifies the terminal-connected evaluation reference discussed in Appendix B.3; the training backups retain the sampler described in the paper.

Table 1a evaluates generation-grid states. Table 1b evaluates held-out continuous-time queries relevant to policy optimization, covering two distributions:

  • FM interpolation: We sample a replay pair (s,a)(s,a), z∼N(0,I)z\sim\mathcal N(0,I), and t∼U(0,1)t\sim U(0,1), then construct xt=(1−t)z+tax_t=(1-t)z+ta.
  • Flow successor: Starting from an FM interpolation query xtx_t, we compute xS+=Proj⁡A(xt+hSvθ(t,s,xt))x_S^+=\operatorname{Proj}_{\mathcal A}(x_t+h_Sv_\theta(t,s,x_t)) and evaluate B at tS+=t+hS<1t_S^+=t+h_S < 1. We retain only interior successor queries to avoid terminal-boundary effects.

Table 1a. FTBP-C continuation-field accuracy on generation-grid queries across training.

EnvironmentStageB NRMSE ↓Gradient cosine ↑Positive F gain ↑Pointwise coverage ↑
antmaze-large-navigateOffline 200K0.05530.537190.82%66.85%
Offline 1M0.04840.856593.54%88.39%
Online 100K0.04660.850193.86%90.77%
Online 500K0.04130.874694.76%91.35%
humanoidmaze-medium-navigateOffline 200K0.07170.765699.93%95.90%
Offline 1M0.04960.861299.82%96.41%
Online 100K0.03640.889899.93%96.18%
Online 500K0.03550.889299.95%96.70%
antsoccer-arena-navigateOffline 200K0.05250.677897.71%86.52%
Offline 1M0.04790.892093.09%89.65%
Online 100K0.04780.895194.21%92.37%
Online 500K0.03870.915395.47%93.89%

Table 1b. FTBP-C continuation-field accuracy at continuous-time policy-training queries.

EnvironmentStageQuery distributionB NRMSE ↓Gradient cosine ↑Positive F gain ↑Pointwise coverage ↑
antmaze-large-navigateOffline 1MFM interpolation0.07460.853191.97%85.63%
Online 500KFM interpolation0.07120.862992.76%89.08%
Offline 1MFlow successor0.06930.867092.64%87.25%
Online 500KFlow successor0.06250.874293.81%90.13%
humanoidmaze-medium-navigateOffline 1MFM interpolation0.08040.859198.36%94.72%
Online 500KFM interpolation0.07070.881098.55%95.62%
Offline 1MFlow successor0.07690.872499.47%95.96%
Online 500KFlow successor0.06380.890599.92%96.63%
antsoccer-arena-navigateOffline 1MFM interpolation0.08220.887991.42%86.19%
Online 500KFM interpolation0.06860.908293.58%88.93%
Offline 1MFlow successor0.07340.892692.75%89.07%
Online 500KFlow successor0.06190.918595.14%92.97%

Statistical protocol. All entries in Tables 1a and 1b are means across more than 10 independent training seeds. At each evaluated checkpoint, we use 512 held-out queries for each query distribution. Query-level metrics are first aggregated within each seed and then averaged across seeds, rather than treating individual queries as independent training runs. Environment names denote environment families, each aggregating five single-task goals. Online steps are counted from the start of fine-tuning.

AuthorsOpenReview ↗

Response to Reviewer XW1h (2/3)

R2-1. Direct continuation-gradient measurements (continued)

NRMSE is RMSE⁡(B−F)/Std⁡(F)\operatorname{RMSE}(B-F)/\operatorname{Std}(F), using each query distribution's own Std⁡(F)\operatorname{Std}(F); cosine compares raw spatial gradients. Across all distributions, coverage and finite-step gain use the common norm-clipped spatial gradient g=C(∇xB)g=\mathcal C(\nabla_xB). Coverage measures ∥g−∇xF∥<∥g∥\lVert g-\nabla_xF\rVert < \lVert g\rVert with g≠0g\ne0; positive F gain measures F(t,s,x+0.01g/∥g∥)>F(t,s,x)F(t,s,x+0.01g/\lVert g\rVert) > F(t,s,x). These metrics assess the spatial field at each query. S's projection Jacobian enters the velocity gradient and is included in the policy updates tested below.

Connection to the improvement condition. With ϵ(x)=∥g−∇xF∥\epsilon(x)=\lVert g-\nabla_xF\rVert, Proposition 5 gives, when F has an L-Lipschitz gradient along the tested segment,

F(x+αg)−F(x)≥α∥g∥(∥g∥−ϵ(x))−Lα22∥g∥2.F(x+\alpha g)-F(x)\geq\alpha\lVert g\rVert(\lVert g\rVert-\epsilon(x))-\frac{L\alpha^2}{2}\lVert g\rVert^2.

Coverage tests the gradient-error premise ϵ(x)<∥g∥\epsilon(x) < \lVert g\rVert. Under the smoothness assumption, the bound is positive for 0<α<2(1−ϵ(x)/∥g∥)/L0 < \alpha < 2(1-\epsilon(x)/\lVert g\rVert)/L. The positive-gain column complements this test by directly evaluating a finite perturbation through the full continuation. Since the error condition is sufficient, a positive gain can also occur outside its coverage.

Table 1a shows lower normalized value error and stronger gradient alignment at the final online checkpoint than early in offline training, with variation between stages. Table 1b extends these measurements to continuous-time queries relevant to policy training. The higher NRMSE on those distributions, together with positive gradient alignment and frequent positive finite-step gains, shows why value and directional accuracy should be assessed separately.

R2-2. Learning constraints and useful policy feedback

FTBP learns reusable intermediate value feedback through endpoint supervision and local Bellman backups, and uses this feedback to improve flow-policy training. Recorded traces provide additional supervision during online fine-tuning. The potential makes this feedback available without computing a completed-suffix target for every training query.

The analysis separates value identification from derivative accuracy. Proposition 1 identifies values along terminal-connected matched transitions; Appendix A.4 bounds gradient error through derivative residuals and transition Jacobians. The sin⁡(2πKt)f(x)\sin(2\pi Kt)f(x) example in Appendix B.3, constructed for zero flow and terminal value in an unprojected region, shows why local consistency and terminal supervision alone can leave incorrect between-grid values and gradients. Thus, the accuracy of the learned guidance field is an empirical question in addition to the conditional analysis. Table 1b addresses this question at continuous-time policy-training queries, complementing the grid measurements in Table 1a. We next test the consequences of actively optimizing B and using its feedback in joint policy updates.

AuthorsOpenReview ↗

Response to Reviewer XW1h (3/3)

R2-2. Learning constraints and useful policy feedback (continued)

Potential-ascent stress test. With B, the flow, and the critic frozen, we take 20 normalized B-ascent steps of size 0.02 within an L2 ball of radius 0.1 around each initial query. At fixed (t,s)(t,s), let x0x_0 and x20x_{20} denote the initial and final query points. We recompute

\Delta F=F(t,s,x_{20})-F(t,s,x_0).$$ Misleading ascent means $\Delta B > 10^{-6}$ and $\Delta F < -10^{-6}$. **Paired policy updates.** We fork C and a feedback-off control from the same checkpoint. The control removes B's policy-feedback terms while retaining all other losses and target updates. B, the critic, and trace supervision remain active in both branches. The branches start from identical model, target-network, and optimizer states and share batches, RNG, and trace schedules. After 100 joint updates, we evaluate actions produced by the one-step actor in each branch using a common critic $Q_0$, frozen at the fork, on 2,048 held-out observations with four shared latent draws each. $$G_{100}=\mathbb E_{s,z}\big[Q_0(s,a^{\mathrm{actor}}_{C,100}(s,z))-Q_0(s,a^{\mathrm{actor}}_{\mathrm{off},100}(s,z))\big].$$ Here $a^{\mathrm{actor}}_{m,100}(s,z)=\operatorname{Proj}_{\mathcal A}f_{\omega_{m,100}}(s,z)$ is the one-step actor action after 100 updates in branch $m$. **Table 2. Stress and policy-update diagnostics at online 500K. Mean ± SD across more than 10 training seeds; pp denotes percentage points.** | Environment | Stress ΔB | Stress ΔF | B↑F↓ | Single-update C−FM (×10⁻⁵) | Joint C−FM (×10⁻⁴) | |---|---:|---:|---:|---:|---:| | antmaze-large-navigate | 0.0741 ± 0.0034 | 0.0341 ± 0.0011 | 4.63% ± 1.21 pp | 4.58 ± 0.92 | 3.57 ± 1.12 | | humanoidmaze-medium-navigate | 0.0491 ± 0.0011 | 0.0300 ± 0.0010 | 1.83% ± 0.54 pp | 3.28 ± 0.64 | 2.33 ± 0.87 | | antsoccer-arena-navigate | 0.0717 ± 0.0021 | 0.0361 ± 0.0023 | 8.26% ± 1.78 pp | 8.39 ± 1.04 | 5.36 ± 1.28 | FM denotes the feedback-off control. Single-update evaluates full-flow actions after one policy-objective update; Joint evaluates one-step actor actions after 100 full updates, with potential, critic, and trace learning retained in both branches. Both columns report C-minus-control critic values under a common frozen evaluator. They measure distinct update protocols and action generators. Table 2 shows positive mean continuation gains under potential ascent, while the difference between ΔB and ΔF quantifies optimistic estimates of the gain. The B↑F↓ column identifies the cases where increasing B reduces the recomputed continuation value. The joint updates yield positive mean C−control gains for the one-step actor in all three environments. Because the evaluator is frozen, these differences reflect the resulting actor actions rather than drift in the evaluation critic. They test whether the intermediate feedback supplied to flow training translates into useful changes in the executed actor over the joint-optimization window. ### **Closing remarks** The measurements support the paper's central claim that a locally trained continuation potential can provide useful intermediate feedback for flow-policy improvement. Direct gradient comparisons assess fidelity on the evaluated grid and continuous-time query distributions, the stress test quantifies misleading ascent, and the paired updates measure the contribution of this feedback during joint optimization. These results provide empirical support for the learned field alongside the conditional improvement analysis. We sincerely thank the reviewer again for the time and care devoted to assessing our work. The questions have helped us make the connection between the learning objective, the gradient-error condition, and the observed policy-update benefits more explicit. We would be happy to provide any further clarification or address any additional questions.

Dear Reviewer XW1h,

Thank you again for taking the time to review our paper and share your thoughtful comments. We hope our responses have adequately addressed your concerns.

As the ICLR discussion period is approaching its conclusion, we would be grateful if you could share any further thoughts or feedback on our responses when you have a chance. We would be happy to clarify anything that remains unclear or answer any additional questions.

Thank you again for your time and consideration.

Best regards,

The Authors

Reviewer_qmoPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. This paper proposes Flow-Time Bellman Propagation (FTBP) to improve flow-based generative policies by introducing value feedback at intermediate generation states.

  2. The method learns a continuation potential using terminal critic values and local TD-style updates. Two variants, FTBP-S and FTBP-T, are proposed to incorporate this information into flow matching.

  3. The paper also introduces additional supervision from recorded generation trajectories and provides theoretical analysis. Experiments on OGBench and D4RL show improvements in offline learning and online fine-tuning over several baselines.

Critical strengths
  1. The paper addresses an interesting problem of incorporating value feedback into intermediate flow-generation steps. The proposed approach is reasonable and avoids repeatedly completing the flow to estimate intermediate values.

  2. The overall framework is well motivated and easy to follow. The two proposed variants, FTBP-S and FTBP-T, provide different ways to incorporate value information into flow matching.

  3. The experiments cover several offline and online RL benchmarks and show improvements over relevant baselines. The ablation studies further help explain the contributions of local TD supervision and the combined FTBP-S/T approach.

Critical weaknesses
  1. The proposed framework is reasonable, but some aspects of its novelty and theoretical support need further clarification.

    1.1 The continuation potential in Eq. (5) is similar to the intermediate value formulation in Q-Flow, while the local TD update in Eq. (6) is closely related to BDPO. Although FTBP combines these ideas with velocity-level guidance, the authors should better explain what makes the proposed method fundamentally different from existing approaches.

    1.2 Proposition 1 and Eq. (8) rely on matched transitions on a fixed grid, while the actual implementation uses a different sampling procedure. It remains unclear how this difference affects the theoretical consistency result.

    1.3 The experiments do not directly evaluate whether the learned potential Bψ(t,s,x)B_\psi(t,s,x) and its gradient ∇xBψ(t,s,x)\nabla_x B_\psi(t,s,x) are accurate. Since both are central to the proposed method, further validation is needed to support the reliability of the learned value feedback.

  2. The experiments mainly report policy performance without comparing overall training time or computational cost against Q-Flow and other baselines. Such comparisons would help establish whether the proposed method offers a practical efficiency advantage.

Questions for authors
  1. In Section 4.2, the successor time is defined as tS+=min⁡(t+hS,1)t_S^+=\min(t+h_S,1), while the spatial update still uses the full step size hSh_S. When t+hS>1t+h_S>1, the spatial step is therefore larger than the remaining generation time. Why is this choice preferred over using 1−t1-t as the step size? Does it affect the stability of the value guidance near the endpoint?

  2. Eq. (10) shows that the first-order effects of FTBP-S and FTBP-T match when 2wβ/da=λcBhS2w\beta/d_a=\lambda c_Bh_S. However, w(s,a)w(s,a) varies across training samples, while β\beta and λ\lambda are fixed coefficients. How should this relationship be interpreted when the guidance strengths cannot match across all samples? Does the analysis provide any guidance for selecting these coefficients in practice?

  3. In Eq. (11), all intermediate states from a recorded trajectory share the same environmental TD target yy. However, these trajectories may come from older policies, while the current continuation has changed during fine-tuning. How are these older trajectories sampled and reused during training? Is there any restriction on their age, and how sensitive is the method to the amount of recorded trace supervision?

  4. Section 5 uses the best published results for several baselines, while Appendix D reports reproduced results under a common protocol. The paper also mentions differences in checkpoint evaluation between FTBP and some published baselines. Could the authors clarify how these differences affect the reported comparisons? In particular, are the main conclusions consistent when all methods are evaluated using the same checkpoint-selection and evaluation procedures?

10 comments
AuthorsOpenReview ↗

Response to Reviewer qmoP (1/6)

We sincerely thank the reviewer for the careful assessment of our work and the thoughtful, constructive comments. We appreciate the recognition of the motivation for intermediate value feedback and the complementary roles of successor evaluation and target shaping. The questions help us clarify the method's contribution, connect the analysis to its implementation, and assess the reliability and practical cost of the learned feedback. Our responses to each point are provided below.

R1-1. Relationship to Q-Flow and BDPO

We thank the reviewer for the thoughtful question about the method's contribution. FTBP makes intermediate feedback practical for deterministic flow policies by learning terminal-value predictions through local backups and using the resulting potential in complementary flow-training objectives.

Local backups remove suffix rollouts from interior-target construction. Q-Flow avoids backpropagation through the ODE solver, but its intermediate-value target still requires a forward rollout from the queried state to the terminal action. FTBP replaces this suffix-based target with one local transition and a target-potential evaluation, propagating endpoint critic information through repeated backups. The practical benefit is cheaper construction of each interior target. In the reproduced 35-task comparison in R1-8, FTBP-C improves the aggregate mean success rate over Q-Flow by 6.6 percentage points offline and 8.6 online. Figure 6 further shows that FTBP-C has a lower training-update cost than Q-Flow. These measurements support both the performance benefit and the practical efficiency of the approach in the tested settings.

Terminal-value learning separates feedback from diffusion-path regularization. BDPO learns intermediate values that include accumulated transition-KL penalties relative to a learned behavior diffusion model. This formulation provides behavior regularization for stochastic diffusion policies. Its Gaussian transition-KL construction cannot be transferred unchanged to deterministic flow transitions. FTBP propagates terminal critic values with no running path cost and retains regression to dataset actions through weighted FM. Intermediate supervision therefore requires neither a separate reference diffusion model nor the evaluation of its transition penalties. This separation lets a single terminal-value predictor support both successor evaluation and target shaping.

The locally learned potential supports complementary policy updates. S evaluates the velocity-proposed successor, while T shapes the target at the current interpolation point. Their finite-step feedback differs in evaluation location, weighting, and projection. In our matched-strength experiment (R3-3), we control the RMS of the B-dependent velocity gradients, including C's combined signal. C retains the highest mean success on all four evaluated tasks, supporting a task-dependent benefit beyond increasing guidance amplitude. The direct gradient and paired-update diagnostics in R1-3 further test whether the locally learned potential supplies useful policy feedback. The contribution thus concerns both how intermediate values are learned and how they improve flow-policy training.

R1-2. Fixed-grid analysis and continuous-time training

The reviewer raises an important distinction between the transitions assumed in Proposition 1 and those sampled during training. Proposition 1 considers a fixed flow and critic on a terminal-connected grid, with matching backup and continuation transitions. It bounds continuation-value error by the endpoint error plus the accumulated local residuals. Zero errors identify the continuation value along this grid.

The implementation samples local-backup times continuously from [0,1−h][0,1-h], with h=1/Kh=1/K. A chain starting at a generic continuous time need not reach the supervised endpoint exactly. Appendix B.3 illustrates this in an unprojected region with zero flow and terminal value. The example B~(t,s,x)=sin⁡(2πKt)f(x)\widetilde B(t,s,x)=\sin(2\pi Kt)f(x) satisfies the sampled fixed-step constraints while retaining incorrect intermediate values and gradients. Appendix B.4 identifies a second distinction, since local backups project each successor while generation clips only the final action.

Proposition 1 therefore characterizes value identification for matched, terminal-connected transitions. Gradient accuracy additionally depends on derivative residuals and transition Jacobians, as analyzed in Appendix A.4. For the implemented learner, we assess value and gradient accuracy directly on both generation-grid states and continuous-time policy-training queries. The measurements below provide the empirical connection between these conditional results and the learned feedback.

AuthorsOpenReview ↗

Response to Reviewer qmoP (2/6)

R1-3. Accuracy of the potential and its spatial gradient

We directly evaluate the quantities highlighted by the reviewer. At each checkpoint, we freeze the current flow, target critic, and target potential and define

F(t,s,x)=Qtarget(s,clip⁡(Φt→1θ(s,x))).F(t,s,x)=Q_{\mathrm{target}}(s,\operatorname{clip}(\Phi^\theta_{t\to1}(s,x))).

Here B denotes the target potential used for policy feedback. For this diagnostic reference, the suffix follows the current velocity field from the query time, using Δ(u)=min⁡(h,1−u)\Delta(u)=\min(h,1-u) for both time and spatial advancement and clipping only its final action. The reference ∇xF\nabla_xF differentiates through the complete suffix and endpoint clipping. This terminal-connected evaluation reference leaves the training sampler unchanged.

We evaluate generation-grid queries, FM interpolation queries, and interior S-successor queries. For FM interpolation, we sample a replay pair (s,a)(s,a), noise z∼N(0,I)z\sim\mathcal{N}(0,I), and time t∼U(0,1)t\sim U(0,1), and form

xt=(1−t)z+ta.x_t=(1-t)z+ta.

For interior S-successor queries, we use

xS+=Proj⁡A(xt+hSvθ(t,s,xt)),x_{S}^{+}=\operatorname{Proj}_{\mathcal{A}}\left(x_t+h_Sv_\theta(t,s,x_t)\right),

retaining only queries with t+hS<1t+h_S < 1.

Table 1a. FTBP-C continuation-field accuracy on generation-grid queries at online 500K.

EnvironmentB NRMSE ↓Gradient cosine ↑Positive F gain ↑Pointwise coverage ↑
antmaze-large-navigate0.04130.874694.76%91.35%
humanoidmaze-medium-navigate0.03550.889299.95%96.70%
antsoccer-arena-navigate0.03870.915395.47%93.89%

Table 1b. FTBP-C continuation-field accuracy at continuous-time policy-training queries at online 500K.

EnvironmentQuery distributionB NRMSE ↓Gradient cosine ↑Positive F gain ↑Pointwise coverage ↑
antmaze-large-navigateFM interpolation0.07120.862992.76%89.08%
Flow successor0.06250.874293.81%90.13%
humanoidmaze-medium-navigateFM interpolation0.07070.881098.55%95.62%
Flow successor0.06380.890599.92%96.63%
antsoccer-arena-navigateFM interpolation0.06860.908293.58%88.93%
Flow successor0.06190.918595.14%92.97%

Statistical protocol. Entries in Tables 1a and 1b are means across more than 10 independent training seeds, using 512 held-out queries per distribution at each checkpoint. Query-level metrics are aggregated within each seed before averaging across seeds. Environment names denote families aggregating five single-task goals. Online steps are counted from the start of fine-tuning.

NRMSE is RMSE⁡(B−F)/Std⁡(F)\operatorname{RMSE}(B-F)/\operatorname{Std}(F), using each distribution's own Std⁡(F)\operatorname{Std}(F); cosine compares raw spatial gradients. For the common norm-clipped spatial gradient g=C(∇xB)g=\mathcal C(\nabla_xB), coverage measures ∥g−∇xF∥<∥g∥\lVert g-\nabla_xF\rVert<\lVert g\rVert with g≠0g\ne0, and positive F gain measures F(t,s,x+0.01g/∥g∥)>F(t,s,x)F(t,s,x+0.01g/\lVert g\rVert)>F(t,s,x).

Coverage directly tests the gradient-error premise in Proposition 5. Together with local smoothness, this premise gives a positive improvement bound for sufficiently small steps. The finite-step measurement independently tests whether a perturbation in the learned direction improves the completed-flow critic value. Tables 1a and 1b support useful directional information on both the grid and the sampled continuous-time distributions, including the interior successors evaluated by S.

AuthorsOpenReview ↗

Response to Reviewer qmoP (3/6)

R1-3. Accuracy of the potential and its spatial gradient (continued)

We further test whether potential feedback remains useful when actively optimized. The stress test freezes B, the flow, and the critic, then takes 20 normalized B-ascent steps of size 0.02 within an L2 ball of radius 0.1. We recompute changes in B and F from the initial to the final query. Misleading ascent means ΔB>10−6\Delta B>10^{-6} and ΔF<−10−6\Delta F<-10^{-6}.

For the joint comparison, we fork C and a feedback-off control from the same model, target-network, and optimizer states. The branches share batches, RNG, and trace schedules. The control removes only B's policy-feedback terms, while B, critic, and trace supervision remain active. After 100 full updates, we evaluate their one-step actor actions with a common critic Q0Q_0, frozen at the fork, on 2,048 held-out observations with four shared latent draws each.

Table 2. Stress and policy-update diagnostics at online 500K. Mean ± SD across more than 10 training seeds; pp denotes percentage points.

EnvironmentStress ΔBStress ΔFB↑F↓Single-update C−FM (×10⁻⁵)Joint C−FM (×10⁻⁴)
antmaze-large-navigate0.0741 ± 0.00340.0341 ± 0.00114.63% ± 1.21 pp4.58 ± 0.923.57 ± 1.12
humanoidmaze-medium-navigate0.0491 ± 0.00110.0300 ± 0.00101.83% ± 0.54 pp3.28 ± 0.642.33 ± 0.87
antsoccer-arena-navigate0.0717 ± 0.00210.0361 ± 0.00238.26% ± 1.78 pp8.39 ± 1.045.36 ± 1.28

FM denotes the feedback-off control. Single-update evaluates full-flow actions after one policy-objective update; Joint evaluates one-step actor actions after 100 full updates, with potential, critic, and trace learning retained in both branches. Both columns report C-minus-control critic values under a common frozen evaluator. They measure distinct update protocols and action generators.

The stress test shows positive mean continuation gains while quantifying residual misleading ascent. The paired updates show positive mean C-minus-control gains, including for the executed one-step actor during joint optimization. These measurements connect continuation-field accuracy to the practical contribution of intermediate policy feedback.

R1-4. Training cost and policy execution

We appreciate the request to make the computational comparison clearer. Figure 6 reports matched training-step times in milliseconds for FQL, GFP, Q-Flow, FINO, and FTBP-S/T/C. The FTBP variants remain in a comparable per-update computational range to the flow baselines. Local backups avoid completed-suffix targets for individual interior queries, while the additional potential network and its losses contribute training overhead.

The step-time measurement characterizes update cost; end-to-end wall-clock time also includes interaction, evaluation, and data handling. At execution time, the potential is not queried. The one-step actor uses one forward pass, and full-flow generation uses K velocity evaluations. Thus, the potential supplies additional training feedback without adding network evaluations to either action-generation procedure.

R1-5. Successor steps near the endpoint

The reviewer correctly identifies that the temporal clamp and spatial step have different roles. FTBP-S evaluates

tS+=min⁡(t+hS,1),xS+=Proj⁡A(xt+hSvθ(t,s,xt)).t_S^+=\min(t+h_S,1),\qquad x_S^+=\operatorname{Proj}_{\mathcal A}(x_t+h_Sv_\theta(t,s,x_t)).

The clamp keeps the potential query within its time domain. The fixed spatial step sets the distance over which S evaluates the velocity-proposed move. Near the endpoint, this is a spatial look-ahead scored at the terminal time. Using min⁡(hS,1−t)\min(h_S,1-t) for the spatial step instead makes the displacement vanish as t→1t\to1 and correspondingly reduces the step-size factor in the S velocity gradient.

We compare these choices by changing only S's spatial step, leaving its temporal clamp and the other settings fixed.

Table 3. Effect of S's near-endpoint spatial step on cube-double-play. Mean ± SD across more than 10 training seeds.

MethodSpatial stepOffline 1M SR (%) ↑Online 500K SR (%) ↑
FTBP-ShSh_S43 ± 379 ± 3
FTBP-Smin⁡(hS,1−t)\min(h_S,1-t)39 ± 575 ± 2
FTBP-ChSh_S51 ± 478 ± 2
FTBP-Cmin⁡(hS,1−t)\min(h_S,1-t)48 ± 575 ± 4

SR is success rate. Results aggregate the five cube-double-play goals using the final offline and online checkpoints.

The fixed spatial step has higher mean performance for both S and C in both phases. These results support retaining a finite look-ahead for successor evaluation on this family. Action projection bounds the queried successor.

Comments on the paper

1 comment

Dear Reviewers

The author-reviewer discussion period will close soon. Please read and respond to authors if you have not done yet.

Thank you.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.