- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes FTBP to improve flow policies using value feedback during denoising. It learns a value function B that predicts the final action’s Q-value from an intermediate point, using endpoint supervision and local TD targets. B guides training by scoring the next step, adjusting the flow-matching target, or combining both. Online generation traces provide additional supervision. Experiments on OGBench and D4RL AntMaze report improvements in offline learning and online fine-tuning.
Critical strengths
The writing, framework and experiments are very complete. The paper evaluates both offline learning and online fine-tuning, with baseline comparisons and ablations of the main components. These experiments provide useful evidence that the method can improve performance in the tested settings.
Critical weaknesses
-
My main concern is the reliability of B. The velocity network may exploit overestimated values, but I did not see experiments directly checking B’s prediction accuracy. The theoretical analysis also does not establish that this guidance remains reliable during joint training.
-
I do not see a clear need for C. S and T both use B’s gradients to guide the velocity, so combining them may simply increase the guidance strength. The current comparisons do not show whether C still helps when this strength is matched, leaving its additional contribution unclear.
Questions for authors
How reliable are B’s predictions during training? What happens if B overestimates values?
5 comments
Response to Reviewer kakM (2/3)
R3-1. Prediction accuracy during training (continued)
NRMSE is , using each query distribution's own ; cosine compares raw spatial gradients. Across all distributions, coverage and finite-step gain use the common norm-clipped spatial gradient . Coverage measures with ; positive F gain measures . These metrics assess the spatial field at each query. S's projection Jacobian enters the velocity gradient and is included in the policy updates tested below.
Table 1a shows lower normalized value error and stronger gradient alignment at the final online checkpoint than early in offline training, with variation between stages. Table 1b extends these measurements to continuous-time queries relevant to policy training. The higher NRMSE on those distributions, together with positive gradient alignment and frequent positive finite-step gains, shows why value and directional accuracy should be assessed separately.
R3-2. Overestimation and joint policy updates
The concern is whether the policy can exploit an optimistic potential estimate and move toward a worse completed action. We examine this in two ways, by maximizing B at fixed model parameters and by comparing policy updates with and without B-dependent feedback.
Potential-ascent stress test. With B, the flow, and the critic frozen, we take 20 normalized B-ascent steps of size 0.02 within an L2 ball of radius 0.1 around each initial query. At fixed , let and denote the initial and final query points. We recompute
\Delta F=F(t,s,x_{20})-F(t,s,x_0).$$ Misleading ascent means $\Delta B > 10^{-6}$ and $\Delta F < -10^{-6}$. Training combines potential feedback with data-based FM regression, a fixed target potential within each policy update, and gradient-norm clipping in T. The paired experiment measures the effect of this feedback during joint optimization. **Paired policy updates.** We fork C and a feedback-off control from the same checkpoint. The control removes B's policy-feedback terms while retaining all other losses and target updates. B, the critic, and trace supervision remain active in both branches. The branches start from identical model, target-network, and optimizer states and share batches, RNG, and trace schedules. After 100 joint updates, we evaluate actions produced by the one-step actor in each branch using a common critic $Q_0$, frozen at the fork, on 2,048 held-out observations with four shared latent draws each. $$G_{100}=\mathbb E_{s,z}\big[Q_0(s,a^{\mathrm{actor}}_{C,100}(s,z))-Q_0(s,a^{\mathrm{actor}}_{\mathrm{off},100}(s,z))\big].$$ Here $a^{\mathrm{actor}}_{m,100}(s,z)=\operatorname{Proj}_{\mathcal A}f_{\omega_{m,100}}(s,z)$ is the one-step actor action after 100 updates in branch $m$. **Table 2. Stress and policy-update diagnostics at online 500K. Mean ± SD across more than 10 training seeds; pp denotes percentage points.** | Environment | Stress ΔB | Stress ΔF | B↑F↓ | Single-update C−FM (×10⁻⁵) | Joint C−FM (×10⁻⁴) | |---|---:|---:|---:|---:|---:| | antmaze-large-navigate | 0.0741 ± 0.0034 | 0.0341 ± 0.0011 | 4.63% ± 1.21 pp | 4.58 ± 0.92 | 3.57 ± 1.12 | | humanoidmaze-medium-navigate | 0.0491 ± 0.0011 | 0.0300 ± 0.0010 | 1.83% ± 0.54 pp | 3.28 ± 0.64 | 2.33 ± 0.87 | | antsoccer-arena-navigate | 0.0717 ± 0.0021 | 0.0361 ± 0.0023 | 8.26% ± 1.78 pp | 8.39 ± 1.04 | 5.36 ± 1.28 | FM denotes the feedback-off control. Single-update evaluates full-flow actions after one policy-objective update; Joint evaluates one-step actor actions after 100 full updates, with potential, critic, and trace learning retained in both branches. Both columns report C-minus-control critic values under a common frozen evaluator. They measure distinct update protocols and action generators. The stress test produces positive mean ΔF in all three environments. Mean ΔB exceeds mean ΔF, quantifying optimism in the predicted improvement, while B↑F↓ identifies cases where that optimism leads to a worse continuation value. The joint updates give positive mean C−control gains for the one-step actor. Using the same frozen critic attributes the comparison to the resulting actor actions rather than evaluation-critic drift. Together with Tables 1a and 1b, these results assess prediction accuracy, potentially exploitable errors, and how intermediate feedback supplied to flow training affects the executed actor.Response to Reviewer kakM (3/3)
R3-3. C at matched guidance strength
T specifies a target direction; S evaluates the proposed move. T shifts the data-based FM target using the clipped potential gradient at the current interpolation point. This target is detached and fixed during the policy update. S instead differentiates the potential at the successor proposed by the current velocity. Its feedback therefore depends on where that velocity leads and includes the derivative of action projection.
Equation (10) identifies their shared first-order direction in the smooth small-step regime with inactive projection and gradient clipping. At finite steps, the potential gradient can change between the current point and the successor, so the two signals need not be parallel. Projection can suppress S feedback in saturated coordinates, while the FM sample weights affect T's contribution. These differences can change guidance direction and its relative allocation across samples.
C combines these signals within one FM objective. It retains a single regression term and adds both B-dependent contributions. Rescaling either interface alone generally cannot reproduce this combined field. Matching the RMS of the combined signal to that of each individual interface tests whether this structure provides a benefit at the same overall guidance amplitude.
To control amplitude, define the B-dependent per-sample velocity-loss gradient for as
\operatorname{RMS}_{\mathcal B}(d_m)=\sqrt{\mathbb E_{\mathcal B}\left[\frac{\lVert d_m\rVert_2^2}{d_a}\right]}.$$ Here, $\ell_{\mathrm{FM}}$ is the common weighted FM objective and $d_a$ is the action dimension. On each minibatch, we match this RMS across S, T, and C, including the loss weights, normalization factors, and projection derivatives. For C, we sum its S and T components within the same model before rescaling the combined signal to the common RMS. All methods use the same seed set, training budgets, and evaluation procedure. We use four tasks to examine whether C's gains in the original experiments persist after matching guidance strength. The offline tasks, antmaze-large-navigate-task4 and cube-double-play-task1, cover navigation and manipulation. The online tasks, antmaze-giant-navigate-task3 and antsoccer-arena-stitch-task5, extend the comparison to larger-scale navigation and the stitching setting. The original results include both a near-tie between C and T and larger separations, allowing us to assess how the contribution of combining S and T varies across settings. **Table 3. Success rates (%) with matched guidance strength. Mean ± SD across more than 10 training seeds.** | Task | Stage | FTBP-S | FTBP-T | FTBP-C | |---|---|---:|---:|---:| | antmaze-large-navigate-task4 | Offline 1M | 87 ± 4 | 92 ± 6 | **93 ± 4** | | cube-double-play-task1 | Offline 1M | 72 ± 7 | 73 ± 6 | **77 ± 6** | | antmaze-giant-navigate-task3 | Online 500K | 57 ± 9 | 57 ± 10 | **78 ± 7** | | antsoccer-arena-stitch-task5 | Online 500K | 51 ± 8 | 52 ± 6 | **57 ± 9** | We evaluate the executed one-step actor at the final checkpoint over 50 episodes per seed. Offline training uses 1M updates; online evaluation follows 500K interactions after offline training. C retains the highest mean at matched velocity-output RMS. The largest separation occurs on antmaze-giant-navigate-task3, with further mean advantages on cube-double-play-task1 and antsoccer-arena-stitch-task5. C and T perform similarly on antmaze-large-navigate-task4. This supports an additional, task-dependent benefit from combining successor evaluation and target shaping when the guidance amplitude is matched. ### **Closing remarks** Together, the experiments support the paper's central claim that a locally trained continuation potential can provide useful intermediate feedback for flow-policy improvement. The accuracy measurements track B as training progresses; the stress test and paired updates assess the consequences of optimizing its predictions; and the matched-strength comparison evaluates C's additional contribution while controlling the velocity-output guidance amplitude. The evidence supports useful feedback with measurable residual errors and a task-dependent benefit from combining S and T. We sincerely appreciate the reviewer's careful evaluation and constructive suggestions. The questions have helped us examine both the reliability of the shared potential and the contribution of the combined feedback mechanism more directly. We would be happy to clarify any aspect of the experiments or address any further questions.
Response to Reviewer kakM (1/3)
We sincerely thank the reviewer for the careful assessment of our work and the thoughtful, constructive comments. We appreciate the recognition of the thoroughness of our framework and experimental evaluation, and of the evidence for performance improvements in the tested settings. The questions help us assess the reliability of the learned potential during joint training, examine the consequences of overestimation, and clarify the additional contribution of combining successor evaluation and target shaping. Our responses to each point are provided below.
R3-1. Prediction accuracy during training
FTBP learns reusable intermediate value feedback through endpoint supervision and local Bellman backups, and uses this feedback to improve flow-policy training. B predicts the critic value of completing the remaining flow from an intermediate state. To evaluate its accuracy, we freeze the current flow, target critic, and target potential at each checkpoint and compute
F(t,s,x)=Qtarget(s,clip(Φt→1θ(s,x))).
Here B denotes the target potential used for policy feedback. For this diagnostic reference, the suffix starts at the query time and follows the current velocity field. At time u, each Euler step uses Δ(u)=min(h,1−u) for both time and spatial advancement, reaching t=1 exactly. Clipping is applied only to the final suffix action. The reference ∇xF differentiates through the complete continuation and endpoint clipping. This specifies the terminal-connected evaluation reference discussed in Appendix B.3; the training backups retain the sampler described in the paper.
Table 1a evaluates generation-grid states. Table 1b evaluates held-out continuous-time queries relevant to C's policy objective. FM interpolation uses xt=(1−t)z+ta, with z∼N(0,I), a replay action a paired with state s, and t∼U(0,1). Flow successor uses xS+=ProjA(xt+hSvθ(t,s,xt)) at tS+=t+hS<1. We retain only interior successor queries to avoid terminal-boundary effects.
Table 1a. FTBP-C continuation-field accuracy on generation-grid queries across training.
Table 1b. FTBP-C continuation-field accuracy at continuous-time policy-training queries.
Statistical protocol. All entries in Tables 1a and 1b are means across more than 10 independent training seeds. At each evaluated checkpoint, we use 512 held-out queries for each query distribution. Query-level metrics are first aggregated within each seed and then averaged across seeds, rather than treating individual queries as independent training runs. Environment names denote environment families, each aggregating five single-task goals. Online steps are counted from the start of fine-tuning.