- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on the preference alignment problem for rectified flow models and points out that image-level or state-level preference supervision suffers from flow-blindness. Such preference supervision cannot guarantee the production of rectified trajectories. To address this, the authors propose velocity preference supervision at shared states. The framework consists of two core modules: BT-Flow lifts existing image preferences into velocity preferences at shared states and trains a flow-aware reward model via the Bradley-Terry model. Flow-VPA uses the trained BT-Flow reward model, treats the predicted velocity as a deterministic action, and directly optimizes the velocity field at states visited by on-policy ODE rollouts. The paper further proves that the canonical flow reward of BT-Flow preserves the original endpoint Bradley-Terry preference probability.
Critical strengths
- The problem is clearly defined. The paper explicitly identifies the flow-blindness issue in existing preference alignment and uses BT-Flow to transform image preferences into state-velocity preferences without additional human annotations, making the resulting supervision more consistent with the actual flow sampling process.
- The algorithm design is efficient. The method avoids the SDE conversion and multiple stochastic trajectory sampling that methods such as Flow-GRPO must introduce for likelihood computation, and directly differentiates the deterministic velocity policy, substantially reducing the cost of online sampling and back-propagation.
Critical weaknesses
- The proofs of some propositions rely on theoretical conditions and lack analysis and discussion in real flow-model settings (see Questions 1 and 2).
- The experiments lack key baselines (see Question 3). The evaluation relies on automatic model judgments. The main win rates are judged by GPT-5.6 Luna/Terra, and Image-RM is also from the same source as the training data. There may therefore be automatic judgment bias or reward over-optimization. Large-scale, diverse human preference evaluation is lacking.
Questions for authors
- Proposition 4.1 imposes zero-sum perturbations on the realized velocity sequence to argue for the flow-blindness of endpoint rewards. This formulation degrades the flow model into static sequence and ignores that Euler sampling is a dynamic feedback system driven by the state-dependent velocity field (Eq. 6). That is, a velocity perturbation at step changes the state at , which in turn changes the velocity at the next step. Can the authors discuss the flow-blindness of endpoint rewards based on the dynamic system of a real flow model?
- The proof of Proposition 5.1 is based on the premise of single-step linear extrapolation , establishing the equivalence between the flow reward and the endpoint reward . In practice, however, multiple Euler sampling steps with interval are performed, the pretrained flow field may not be perfectly straight, and factors such as CFG guidance exist. The endpoints obtained by single-step extrapolation and multi-step sampling may therefore deviate. Thus, under multi-step denoising, does Proposition 5.1 still hold? The authors could supplement an analysis of the policy gradient bias caused by using BT-Flow trained on single-step extrapolation to guide multi-step policy updates.
- The authors discuss positive/negative-sample-guided or advantage-guided velocity field alignment methods such as DiffusionNFT, Neighbor GRPO, and AdvantageFlow, but the experiments do not include comparisons with these methods and only compare basic Flow-DPO/Flow-GRPO/Flow-LPO. Readers cannot verify whether "on-policy state velocity optimization" is truly more effective than positive/negative-sample-guided methods such as DiffusionNFT or Neighbor GRPO. In addition, as an ODE reinforcement learning method, the paper also lacks comparison and analysis of its performance and efficiency advantages.
- In Section 4.2, the authors analyze the rectified deviation and preference win rate of the endpoint reward. What are the corresponding metric values for the state–velocity reward?
- During Flow-VPA training, and are required to be based on the same . In practice, however, is sampled from and (line 312). This makes it difficult to guarantee whether the optimization gain originates from or . We suggest that the authors analyze the estimation error caused by this practical operation. For example, they could always choose as and report the corresponding metrics.
5 comments
Q4: Geometry and performance with the state–velocity reward
We now report trajectory geometry and preference performance for Flow-VPA and State-GRPO (Flow-GRPO with State-RM) across five inference budgets. We use from Section 4.2 and Appendix E.5 and report GPT-5.6 Luna win rates against the matched-NFE SD3.5-M reference. The base model's deviation shows how alignment changes geometry.
| NFE | Base deviation | State-GRPO deviation | Flow-VPA deviation | State-GRPO win rate (%) | Flow-VPA win rate (%) |
|---|---|---|---|---|---|
| 10 | 0.04055 | 0.04816 | 0.04144 | 67.6 | 73.0 |
| 20 | 0.04833 | 0.05810 | 0.04788 | 68.8 | 71.6 |
| 40 | 0.05247 | 0.06476 | 0.05104 | 69.4 | 72.8 |
| 60 | 0.05378 | 0.06720 | 0.05218 | 68.6 | 70.0 |
| 80 | 0.05467 | 0.06851 | 0.05292 | 68.8 | 70.2 |
Flow-VPA achieves higher preference win rates and lower trajectory deviation than State-GRPO at every evaluated NFE. Relative to the base model, State-GRPO increases deviation at all five budgets. Flow-VPA reduces deviation at 20–80 NFE and remains close to the base at 10 NFE, where deviation increases slightly.
These results directly show that Flow-VPA's stronger preference alignment is accompanied by lower trajectory deviation than the evaluated state-reward baseline under finite-step sampling.
Q5: The shared-state condition in dual-anchor training
The shared-state condition is satisfied by construction. We sample one anchor and use it for both candidates in that preference comparison. Writing the sampled anchor identity as ,
$ x_t=x_t^A, \qquad v_t^{A,+}=\frac{x_1^+-x_t^A}{1-t}, \qquad v_t^{A,-}=\frac{x_1^--x_t^A}{1-t}. $
The Bradley–Terry loss compares the two rewards at identical . Equations (10)–(12) and Figure 4 specify these within-anchor comparisons. Sampling either anchor does not change the context within a pair. Thus, no shared-state mismatch is introduced by dual-anchor sampling.
We thank the reviewer again for the detailed and insightful review. They help us clarify the role of the theoretical results and strengthen the evidence for the practical algorithm.
Thanks for your detailed response.
Authors provide more details about the experimental results, which address some of my concerns. However, the responses to Q1 and Q2 are still based on the assumption that flow matching is a static system, and indeed, one step denoising can easily get Proposition 5.1.
In Q1, it's straightforward that the rewards of two trajectories are the same, if they share the same endpoint (clear image x_1) without considering their intermediate states. My point is, the zero-sum perturbations in intermediate steps can usually leads to different endpoint, since the states must be re-calculated after the perturbation. I think we cannot assume that the endpoint is unchanged even after intermediate perturbation, is that right?
In Q2, I have have a similar concern, that the flow matching usually contains multiple "denoising" steps. The prove of Proposition 5.1 suppose that the flow matching is one-step "denoising", which I admit it's right under this assumption, while it simplifies the real situation.
We sincerely thank the reviewer for recognizing the motivation and efficiency of direct velocity optimization. We have carefully considered the theoretical and empirical questions and would like to address each of them below.
W1, Q1, Q2: Theoretical Clarification.
1. Endpoint non-identifiability under state-dependent dynamics
State-dependent feedback does not remove endpoint non-identifiability. Proposition 4.1 concerns different trajectories that reach the same endpoint; it does not assert that arbitrary perturbations of a fixed network preserve its endpoint.
Consider two Euler steps of size 1/2. For a fixed source x0, endpoint y, and intermediate state m, define the state-dependent field
$v^{(m)}(x,0,c)=2(m-x), \qquad v^{(m)}(x,\tfrac12,c)=2(y-x). $
The resulting rollout is x0→m→y, with the second velocity evaluated at the state actually reached. Choosing m=(x0+y)/2 gives direct constant-velocity transport; choosing m off that line gives a bent path. Both receive the same endpoint reward.
This construction explicitly accounts for state-dependent feedback. Different velocity fields can reach the same preferred image through different paths, which endpoint reward alone cannot distinguish.
2. Preference preservation under multi-step sampling
Proposition 5.1 is an exact statement about direct completions at a shared state. The endpoint xt+(1−t)vt is the completion obtained by holding the candidate velocity fixed; it is not assumed to equal the endpoint reached by continuing the current policy.
For the constructed candidates vt±=(x1±−xt)/(1−t),
rflow⋆(xt,vt±,t,c)=r⋆(xt+(1−t)vt±,c)=r⋆(x1±,c).
Consequently, the corresponding Bradley–Terry preference probability is preserved at every queried state. This identity does not require a straight learned field or a one-step sampler. The factor 1−t defines the completion used to evaluate the candidate velocity; Δt controls how far the sampler advances before predicting another velocity. Curvature and CFG can change the visited states, predicted velocities, and realized endpoint without invalidating the identity for the constructed candidates.
Regarding the policy-gradient question, Flow-VPA optimizes a local surrogate at fixed on-policy states.
For a differentiable canonical reward,
$ \nabla_v r_{\mathrm{flow}}^\star(x_t,v,t,c) = (1-t)\nabla_x r^\star(x_t+(1-t)v,c). $
This need not equal the gradient of the final image reward through subsequent denoising steps. Proposition 5.1 establishes local preference preservation; it does not claim gradient equivalence or an unbiased estimator of the terminal-reward gradient. Stop-gradient visitation deliberately omits differentiation through the rollout.
W2 and Q3: Additional empirical evidence
1. Automatic evaluation and human agreement
We agree that automatic evaluation remains a limitation. To reduce dependence on a single evaluator, we report GPT-5.6 Luna/Terra judgments and Q-Judger scores from the evaluator released with Qwen-Image-Bench.
Appendix E.4 and Table 4 assess agreement with original human-labeled pairs from the validation-unique and test-unique splits of Pick-a-Pic-v2:
This provides empirical calibration for using these evaluators as preference proxies. We will retain direct human evaluation as a limitation.
2. Forward-process alignment baselines and efficiency
We agree that a direct comparison with a representative forward-process method is important. In our related work, we will also discuss Neighbor GRPO and AdvantageFlow.
To directly resolve the reviewer's concern, We now add a direct comparison with DiffusionNFT trained using Image-RM and cfg=4.5. We also tested CFG=1 in our setup, which consistently produced lower reference win rates, and we theorefore report the stronger tested CFG=4.5 configuration.
The table reports reference win rates after one epoch of online alignment, evaluated at 40 NFE using GPT-5.6 Luna, alongside total online alignment cost. For DiffusionNFT, we follow the training setup reported in its original paper, except for CFG.
Flow-VPA achieves a 9.8-percentage-point higher reference win rate than DiffusionNFT while using 65.6% fewer H100 GPU-hours. These results support Flow-VPA’s favorable preference–compute trade-off over DiffusionNFT.