acceptodds
Under review as a conference paper at ICLR 2027

Rectified Preference Alignment via Deterministic Policy Optimization

Abstract

Preference alignment for text-to-image rectified-flow models commonly relies on image-level rewards. However, such supervision is flow-blind: rectified and non-rectified paths to the same preferred image receive identical rewards. Consequently, alignment can improve endpoint reward without preserving the pretrained flow geometry that supports effective finite-step sampling. This may compromise generation quality and weaken alignment gains as the number of sampling steps decreases. We address this gap with Flow-VPA, a Flow-aware Velocity Preference Alignment framework that combines rectified-transport supervision with deterministic policy optimization. To provide this supervision, we introduce BT-Flow, a Bradley–Terry reward model over state–velocity pairs that lifts existing chosen–rejected image comparisons into local velocity preferences. Conditioned on a shared intermediate state, time, and prompt, BT-Flow learns to favor the velocity defining a straight-line completion to the chosen image over the corresponding velocity toward the rejected image. Building on this reward, Flow-VPA treats predicted velocities as deterministic actions and updates the policy using reward gradients at on-policy states. This enables direct alignment of the deterministic flow without introducing an auxiliary stochastic differential equation, evaluating trajectory likelihoods, or differentiating through the sampler. Experiments on SD3.5-M show that Flow-VPA improves preference alignment and generation quality over representative baselines under practical finite-step sampling, with a better trade-off between alignment performance and policy-training compute.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Rectified Flow Models, Preference Alignment, Deterministic Policy Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LyMsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the preference alignment problem for rectified flow models and points out that image-level or state-level preference supervision suffers from flow-blindness. Such preference supervision cannot guarantee the production of rectified trajectories. To address this, the authors propose velocity preference supervision at shared states. The framework consists of two core modules: BT-Flow lifts existing image preferences into velocity preferences at shared states and trains a flow-aware reward model via the Bradley-Terry model. Flow-VPA uses the trained BT-Flow reward model, treats the predicted velocity as a deterministic action, and directly optimizes the velocity field at states visited by on-policy ODE rollouts. The paper further proves that the canonical flow reward of BT-Flow preserves the original endpoint Bradley-Terry preference probability.

Critical strengths
  1. The problem is clearly defined. The paper explicitly identifies the flow-blindness issue in existing preference alignment and uses BT-Flow to transform image preferences into state-velocity preferences without additional human annotations, making the resulting supervision more consistent with the actual flow sampling process.
  2. The algorithm design is efficient. The method avoids the SDE conversion and multiple stochastic trajectory sampling that methods such as Flow-GRPO must introduce for likelihood computation, and directly differentiates the deterministic velocity policy, substantially reducing the cost of online sampling and back-propagation.
Critical weaknesses
  1. The proofs of some propositions rely on theoretical conditions and lack analysis and discussion in real flow-model settings (see Questions 1 and 2).
  2. The experiments lack key baselines (see Question 3). The evaluation relies on automatic model judgments. The main win rates are judged by GPT-5.6 Luna/Terra, and Image-RM is also from the same source as the training data. There may therefore be automatic judgment bias or reward over-optimization. Large-scale, diverse human preference evaluation is lacking.
Questions for authors
  1. Proposition 4.1 imposes zero-sum perturbations on the realized velocity sequence to argue for the flow-blindness of endpoint rewards. This formulation degrades the flow model into static sequence and ignores that Euler sampling is a dynamic feedback system driven by the state-dependent velocity field vθ(xN(k),k,c)v_\theta (x_N^{(k)}, k, c) ​(Eq. 6). That is, a velocity perturbation at step tt changes the state at t+Δtt + \Delta t, which in turn changes the velocity at the next step. Can the authors discuss the flow-blindness of endpoint rewards based on the dynamic system of a real flow model?
  2. The proof of Proposition 5.1 is based on the premise of single-step linear extrapolation x+(1−t)vx+(1-t)v, establishing the equivalence between the flow reward rflow∗r^{*}_{flow} and the endpoint reward r∗r^{*}. In practice, however, multiple Euler sampling steps with interval Δt\Delta t are performed, the pretrained flow field may not be perfectly straight, and factors such as CFG guidance exist. The endpoints obtained by single-step extrapolation and multi-step sampling may therefore deviate. Thus, under multi-step denoising, does Proposition 5.1 still hold? The authors could supplement an analysis of the policy gradient bias caused by using BT-Flow trained on (1−t)(1-t) single-step extrapolation to guide multi-step Δt\Delta t policy updates.
  3. The authors discuss positive/negative-sample-guided or advantage-guided velocity field alignment methods such as DiffusionNFT, Neighbor GRPO, and AdvantageFlow, but the experiments do not include comparisons with these methods and only compare basic Flow-DPO/Flow-GRPO/Flow-LPO. Readers cannot verify whether "on-policy state velocity optimization" is truly more effective than positive/negative-sample-guided methods such as DiffusionNFT or Neighbor GRPO. In addition, as an ODE reinforcement learning method, the paper also lacks comparison and analysis of its performance and efficiency advantages.
  4. In Section 4.2, the authors analyze the rectified deviation and preference win rate of the endpoint reward. What are the corresponding metric values for the state–velocity reward?
  5. During Flow-VPA training, vt+v_t^+ ​and vt−v_t^− ​are required to be based on the same xtx_t​. In practice, however, xtx_t is sampled from xt+x_t^+ ​and xt−x_t^− ​(line 312). This makes it difficult to guarantee whether the optimization gain originates from vtv_t ​or xtx_t​. We suggest that the authors analyze the estimation error caused by this practical operation. For example, they could always choose xt+x_t^+ ​as xtx_t ​and report the corresponding metrics.
5 comments

We sincerely thank the reviewer for recognizing the motivation and efficiency of direct velocity optimization. We have carefully considered the theoretical and empirical questions and would like to address each of them below.

W1, Q1, Q2: Theoretical Clarification.

1. Endpoint non-identifiability under state-dependent dynamics

State-dependent feedback does not remove endpoint non-identifiability. Proposition 4.1 concerns different trajectories that reach the same endpoint; it does not assert that arbitrary perturbations of a fixed network preserve its endpoint.

Consider two Euler steps of size 1/21/2. For a fixed source x0x_0, endpoint yy, and intermediate state mm, define the state-dependent field

$v^{(m)}(x,0,c)=2(m-x), \qquad v^{(m)}(x,\tfrac12,c)=2(y-x). $

The resulting rollout is x0→m→yx_0\rightarrow m\rightarrow y, with the second velocity evaluated at the state actually reached. Choosing m=(x0+y)/2m=(x_0+y)/2 gives direct constant-velocity transport; choosing mm off that line gives a bent path. Both receive the same endpoint reward.

This construction explicitly accounts for state-dependent feedback. Different velocity fields can reach the same preferred image through different paths, which endpoint reward alone cannot distinguish.

2. Preference preservation under multi-step sampling

Proposition 5.1 is an exact statement about direct completions at a shared state. The endpoint xt+(1−t)vtx_t+(1-t)v_t is the completion obtained by holding the candidate velocity fixed; it is not assumed to equal the endpoint reached by continuing the current policy.

For the constructed candidates vt±=(x1±−xt)/(1−t)v_t^\pm=(x_1^\pm-x_t)/(1-t),

rflow⋆(xt,vt±,t,c)=r⋆(xt+(1−t)vt±,c)=r⋆(x1±,c).r_{\mathrm{flow}}^\star(x_t,v_t^\pm,t,c) = r^\star(x_t+(1-t)v_t^\pm,c) = r^\star(x_1^\pm,c).

Consequently, the corresponding Bradley–Terry preference probability is preserved at every queried state. This identity does not require a straight learned field or a one-step sampler. The factor 1−t1-t defines the completion used to evaluate the candidate velocity; Δt\Delta t controls how far the sampler advances before predicting another velocity. Curvature and CFG can change the visited states, predicted velocities, and realized endpoint without invalidating the identity for the constructed candidates.

Regarding the policy-gradient question, Flow-VPA optimizes a local surrogate at fixed on-policy states.

For a differentiable canonical reward,

$ \nabla_v r_{\mathrm{flow}}^\star(x_t,v,t,c) = (1-t)\nabla_x r^\star(x_t+(1-t)v,c). $

This need not equal the gradient of the final image reward through subsequent denoising steps. Proposition 5.1 establishes local preference preservation; it does not claim gradient equivalence or an unbiased estimator of the terminal-reward gradient. Stop-gradient visitation deliberately omits differentiation through the rollout.

W2 and Q3: Additional empirical evidence

1. Automatic evaluation and human agreement

We agree that automatic evaluation remains a limitation. To reduce dependence on a single evaluator, we report GPT-5.6 Luna/Terra judgments and Q-Judger scores from the evaluator released with Qwen-Image-Bench.

Appendix E.4 and Table 4 assess agreement with original human-labeled pairs from the validation-unique and test-unique splits of Pick-a-Pic-v2:

EvaluatorValidation-unique agreement (%)Test-unique agreement (%)
GPT-5.6 Luna63.5266.27
GPT-5.6 Terra64.3467.33
Q-Judger QAA-F59.1459.67
Q-Judger All555.8160.85

This provides empirical calibration for using these evaluators as preference proxies. We will retain direct human evaluation as a limitation.

2. Forward-process alignment baselines and efficiency

We agree that a direct comparison with a representative forward-process method is important. In our related work, we will also discuss Neighbor GRPO and AdvantageFlow.

To directly resolve the reviewer's concern, We now add a direct comparison with DiffusionNFT trained using Image-RM and cfg=4.5. We also tested CFG=1 in our setup, which consistently produced lower reference win rates, and we theorefore report the stronger tested CFG=4.5 configuration.

The table reports reference win rates after one epoch of online alignment, evaluated at 40 NFE using GPT-5.6 Luna, alongside total online alignment cost. For DiffusionNFT, we follow the training setup reported in its original paper, except for CFG.

MethodReference win rate (%)H100 GPU-hours
DiffusionNFT63.0496.66
Flow-VPA72.8170.94

Flow-VPA achieves a 9.8-percentage-point higher reference win rate than DiffusionNFT while using 65.6% fewer H100 GPU-hours. These results support Flow-VPA’s favorable preference–compute trade-off over DiffusionNFT.

Q4: Geometry and performance with the state–velocity reward

We now report trajectory geometry and preference performance for Flow-VPA and State-GRPO (Flow-GRPO with State-RM) across five inference budgets. We use DrectD_{\mathrm{rect}} from Section 4.2 and Appendix E.5 and report GPT-5.6 Luna win rates against the matched-NFE SD3.5-M reference. The base model's deviation shows how alignment changes geometry.

NFEBase deviationState-GRPO deviationFlow-VPA deviationState-GRPO win rate (%)Flow-VPA win rate (%)
100.040550.048160.0414467.673.0
200.048330.058100.0478868.871.6
400.052470.064760.0510469.472.8
600.053780.067200.0521868.670.0
800.054670.068510.0529268.870.2

Flow-VPA achieves higher preference win rates and lower trajectory deviation than State-GRPO at every evaluated NFE. Relative to the base model, State-GRPO increases deviation at all five budgets. Flow-VPA reduces deviation at 20–80 NFE and remains close to the base at 10 NFE, where deviation increases slightly.

These results directly show that Flow-VPA's stronger preference alignment is accompanied by lower trajectory deviation than the evaluated state-reward baseline under finite-step sampling.

Q5: The shared-state condition in dual-anchor training

The shared-state condition is satisfied by construction. We sample one anchor and use it for both candidates in that preference comparison. Writing the sampled anchor identity as A∈+,−A\in\\{+,-\\},

$ x_t=x_t^A, \qquad v_t^{A,+}=\frac{x_1^+-x_t^A}{1-t}, \qquad v_t^{A,-}=\frac{x_1^--x_t^A}{1-t}. $

The Bradley–Terry loss compares the two rewards at identical xtA,t,cx_t^A,t,c. Equations (10)–(12) and Figure 4 specify these within-anchor comparisons. Sampling either anchor does not change the context within a pair. Thus, no shared-state mismatch is introduced by dual-anchor sampling.


We thank the reviewer again for the detailed and insightful review. They help us clarify the role of the theoretical results and strengthen the evidence for the practical algorithm.

Reviewer_LyMsOpenReview ↗

Thanks for your detailed response.

Authors provide more details about the experimental results, which address some of my concerns. However, the responses to Q1 and Q2 are still based on the assumption that flow matching is a static system, and indeed, one step denoising can easily get Proposition 5.1.

In Q1, it's straightforward that the rewards of two trajectories are the same, if they share the same endpoint (clear image x_1) without considering their intermediate states. My point is, the zero-sum perturbations in intermediate steps can usually leads to different endpoint, since the states must be re-calculated after the perturbation. I think we cannot assume that the endpoint is unchanged even after intermediate perturbation, is that right?

In Q2, I have have a similar concern, that the flow matching usually contains multiple "denoising" steps. The prove of Proposition 5.1 suppose that the flow matching is one-step "denoising", which I admit it's right under this assumption, while it simplifies the real situation.

Reviewer_MVPQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies the "flow-blindness" limitation in preference alignment for rectified-flow models, where endpoint- or state-only rewards fail to preserve straight-line transport geometry and lead to degraded alignment gains under low-NFE sampling. To address this, the authors propose BT-Flow, a Bradley–Terry reward model over state–velocity pairs that lifts offline image comparisons into local velocity preferences favoring straight-line completions toward preferred endpoints. Building on this reward, they introduce Flow-VPA, a likelihood-free deterministic policy optimization method that directly updates predicted velocities using reward gradients at on-policy ODE states without requiring an auxiliary SDE or solver backpropagation. Experiments on SD3.5-M validate that Flow-VPA outperforms representative image- and state-level RL baselines in generation quality and preference win rates, while retaining its advantages down to 10 NFEs at a lower training compute cost.

Critical strengths
  1. The paper identifies the "flow-blindness" issue in rectified-flow alignment and clearly shows why neither endpoint rewards nor dense state rewards preserve straight-line trajectories. Lifting offline image pairs into shared-state velocity preferences via BT-Flow provides a novel and principled way to couple human preference with rectified transport without extra annotations, which directly preserves alignment gains in the practical low-NFE sampling regime.

  2. Treating predicted velocities as deterministic actions allows Flow-VPA to directly optimize the ODE policy via reward gradients at stop-gradient on-policy states, completely bypassing the auxiliary SDE conversion, trajectory likelihood estimation, and solver backpropagation required by prior RL methods. Thus, it achieves both stronger low-NFE preference win rates and substantially lower online training compute than SDE-based baselines like Flow-GRPO.

Critical weaknesses
  1. Redirecting velocity at an intermediate state xtx_t toward a preferred endpoint x1+x_1^+ explicitly constructs a piecewise-bent trajectory x0→xt→x1+x_0 \to x_t \to x_1^+ rather than a straight line. Specifically, in BT-Flow's rejected-anchor construction (Eq. 8) and Flow-VPA's step-wise stop-gradient updates, the intermediate state xtx_t is reached by evolving from x0x_0 along an unaligned or rejected trajectory, yet the reward explicitly trains the velocity at xtx_t to point toward x1+x_1^+ via vt+=(x1+−xt)/(1−t)v_t^+ = (x_1^+ - x_t)/(1-t). Unless all trajectories collapse to a single degenerate optimum x1⋆x_1^\star from t=0t=0 (as unrealistically assumed in Theorem 5.1), steering an intermediate state xtx_t toward a new target x1+x_1^+ mid-way inevitably introduces a sharp kink at xtx_t, suffering from the exact same trajectory-bending issue for which the paper criticizes state-level rewards.

  2. The proposed state-velocity reward is mathematically isomorphic to one-step endpoint lookahead rather than a genuine constraint on multi-step transport geometry. Since BT-Flow synthesizes velocity preferences solely from offline image pairs (x1+,x1−)(x_1^+, x_1^-) via linear subtraction without any trajectory-level supervision, there is a strict bijection between a candidate velocity vtv_t at a given state xtx_t and its one-step extrapolated clean image x^1=xt+(1−t)vt\hat{x}_1 = x_t + (1-t)v_t. Consequently, scoring (xt,vt)(x_t, v_t) provides no additional geometric signal beyond scoring the one-step predicted endpoint (as in DenseGRPO), merely re-parameterizing existing lookahead rewards into decomposed network inputs.

Questions for authors
  1. Given that redirecting an intermediate state xtx_t (such as the rejected anchor xt−x_t^- in Eq. 8) toward x1+x_1^+ inherently creates a bent path x0→xt→x1+x_0 \to x_t \to x_1^+ unless all trajectories collapse to a single endpoint x1⋆x_1^\star from t=0t=0 (as assumed in Theorem 5.1), how does Flow-VPA preserve straight trajectories under multi-modal preferences? Please report the trajectory deviation metric D\mathcal{D} (from Section 4.2) for Flow-VPA, along with quantitative diversity metrics and visual samples across different random seeds, to clarify whether straightness comes at the cost of mode collapse.

  2. Since a velocity vtv_t at a given xtx_t has a strict bijection with the one-step extrapolated endpoint x^1=xt+(1−t)vt\hat{x}_1 = x_t + (1-t)v_t and BT-Flow is trained solely on endpoint pairs (x1+,x1−)(x_1^+, x_1^-), what geometric signal does scoring (xt,vt)(x_t, v_t) capture beyond simply scoring x^1\hat{x}_1? Please clarify how Flow-VPA compares against an image-level reward model fine-tuned on the same extrapolated states x^1\hat{x}_1 and optimized via direct one-step gradient backpropagation.

4 comments

We really appreciate the reviewer's thoughtful feedback and positive assessment of the identified problem as well as our deterministic optimization approach. After carefully considering the concerns you put forward, we would like to address them below.

W1: Does the rejected-anchor construction necessarily introduce a kink?

The rejected anchor defines a local preference comparison; it does not fix the incoming prefix of the learned rollout. Holding a sampled state fixed during an update avoids differentiation through that rollout. Subsequent rollouts are generated anew by the updated velocity field.

The canonical flow objective strictly disfavors the specific kink in the reviewer's example. Consider a two-step path that first follows the rejected interpolation and then completes directly to the chosen image. Let r±=r⋆(x1±,c)r^\pm=r^\star(x_1^\pm,c), with r+>r−r^+>r^-. Summing the two canonical flow rewards gives

Jbent=r−+r+<r++r+=Jdirect.J_{\mathrm{bent}}=r^-+r^+<r^++r^+=J_{\mathrm{direct}}.

Both paths receive the same terminal reward, but the canonical flow objective favors the path whose early direction already completes to the preferred endpoint. This comparison requires a strictly preferred pair, not a unique global maximizer.

W2: Direct completion versus continued denoising

At a fixed state and time, each velocity corresponds to exactly one direct-completion endpoint, x^1=xt+(1−t)vt\hat{x}_1=x_t+(1-t)v_t, and vice versa. We agree with this observation, which is explicit in Proposition 5.1.

The relevant distinction is how the endpoint is obtained. DenseGRPO uses multi-step ODE denoising from intermediate states, scores the resulting images, and uses changes in those scores as feedback. BT-Flow evaluates the completion obtained by holding the candidate velocity unchanged for the remaining time.

In the deterministic example above, continuing each path's denoising process from an intermediate state reaches the chosen image. These continuation scores therefore do not distinguish the detour. During the rejected prefix, however, holding the current velocity fixed leads to the rejected image, so the canonical flow objective assigns the direct path a higher score. It detects an inferior early completion even when later steps correct the endpoint.

An exact image reward evaluated on the same reconstructed endpoints shares this property. DenseGRPO's single-step ablation also uses a related scoring approach. We therefore attribute this geometric distinction to evaluating direct completions along the path. The controls in Q2 separately test the practical benefit of the learned state–velocity representation.

Q1: Multimodal preferences and output diversity

We agree that human preferences can admit multiple equally desirable outputs, beyond the unique-maximizer setting of Theorem 5.1. For equally preferred endpoints, the reward gap in W1 vanishes, so switching between them can still introduce bends. Maintaining target consistency in this setting is a broader challenge: neither terminal reward nor our flow reward alone guarantees it. We will clarify this limitation.

Our contribution remains that Flow-VPA penalizes lower-reward intermediate direct completions even when the final image is unchanged. The practical question is whether this distinction improves trajectory geometry while retaining output diversity comparing to the baseline methods.

We therefore report DrectD_{\mathrm{rect}} and preference performance at 40 NFE, together with within-prompt diversity using LPIPS (100 prompts, with eight seeds per prompt, averaged over all 28 image pairs).

MethodRewardGPT-5.6 Luna win rate (%) ↑\uparrowDrectD_{\mathrm{rect}} ↓\downarrowLPIPS ↑\uparrow
Flow-GRPOImage-RM61.000.05380.6518
Flow-GRPOState-RM69.400.06480.5660
Flow-VPAFlow-RM72.800.05100.6486

Thus, Flow-VPA's higher preference win rate and lower trajectory deviation coexist with higher observed diversity than the State-RM baselines and similar observed diversity to the Image-RM baseline.

Q2: Why not simply use a reconstructed-endpoint reward?

Figure 3(c) already tests Image-RM+REC within Flow-VPA, including direct gradients through the reconstructed endpoint. For the constructed training pairs,

xt+(1−t)vt±=x1±.x_t+(1-t)v_t^\pm=x_1^\pm.

Thus, training an image reward on these reconstructed endpoints uses the original labeled image pairs. To address the remaining architectural confound, we trained an additional conditional endpoint RM matching Flow-RM's backbone. Only the candidate representation changes: the reconstructed endpoint x^1\hat{x}_1 is fed directly into the candidate stream in place of the rescaled-velocity input.

We evaluated the rewards within Flow-VPA under matched alignment settings as presented below. All entries are GPT-5.6 Luna reference win rates on the same 500 held-out prompts at 40 NFE, with K=1K=1.

Exploration scale σmax⁡\sigma_{\max}Flow-RM (%)Conditional endpoint RM (%)Image-RM+REC (%)
067.268.066.8
0.168.068.266.2
0.368.465.662.4
0.570.664.261.0
0.769.463.858.4

The conditional endpoint RM is slightly ahead at zero and low exploration. Flow-RM performs better at every tested scale from σmax⁡=0.3\sigma_{\max}=0.3 onward. The advantage under moderate exploration therefore persists with the conditional architecture matched and gradients retained through reconstruction. This supports a practical benefit of the velocity-residual representation.


We thank the reviewer again for these insightful feedback. They have helped us strengthen the evidence on trajectory geometry and diversity, and clarify the relationship to endpoint-based rewards.

Thank you for the detailed response and the additional clarifications. The authors have successfully addressed most of my concerns. I will keep my score 3: Weak acceptance.

Thank you very much for revisiting our submission. We sincerely appreciate your careful consideration and acknowledgement that your main concerns are resolved.

We will for sure incorporate these clarifications and additional results into the revised manuscript.

Thank you again for your time and valuable engagement.

Best regards,

The Authors

Reviewer_yZJCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Flow-VPA to address the flow-blindness issue in preference alignment for rectified-flow models. The authors introduce BT-Flow, which lifts image-level chosen-rejected preferences into state-velocity preferences, and develop Flow-VPA to directly optimize the velocity field through deterministic policy optimization without requiring auxiliary SDE transitions or trajectory likelihood computation. Experiments on SD3.5-M demonstrate improvements in preference alignment, generation quality, and robustness under low-NFE sampling compared with several existing baselines.

Critical strengths

S1. The paper addresses a meaningful problem in rectified-flow alignment. Incorporating transport geometry into preference optimization is relevant to maintaining generation quality under finite-step sampling.

S2. BT-Flow provides an interesting formulation by explicitly incorporating velocity into preference modeling and comparing different transport directions under a shared state.

S3. Flow-VPA avoids auxiliary SDE transitions and trajectory likelihood computation, offering a potentially more efficient alternative to existing RL-based alignment methods. The experimental results are also encouraging.

Critical weaknesses

W1. Unclear independent contribution of BT-Flow. Since candidate velocities can be directly mapped back to their corresponding endpoints, BT-Flow is theoretically closely related to endpoint reward modeling. It remains unclear whether the improvements come from genuinely additional velocity-level supervision or simply from the reward parameterization and architecture. The current Image-RM+REC ablation does not fully isolate this contribution.

W2. Gap between theoretical analysis and practical optimization. Theorem 5.1 relies on idealized assumptions, including an exact reward model and no reference regularization, whereas the practical algorithm uses an approximate reward model, LoRA, reference regularization, and stop-gradient state visitation. It is unclear whether the theoretical rectified-transport guarantee remains valid under these practical settings.

W3. Limited experimental generalization. Experiments are conducted only on SD3.5-M and rely primarily on automatic preference evaluators such as GPT-5.6. Evaluation on additional model architectures and human preference studies would provide stronger evidence of the method's general applicability.

Questions for authors
  1. What additional information does BT-Flow provide compared with directly evaluating reconstructed endpoints using Image-RM? Can the authors provide stronger ablations to isolate its contribution?

  2. How does BT-Flow generalize when the on-policy state distribution deviates from the synthetic interpolation distribution used during reward training?

  3. Theorem 5.1 does not consider reference regularization, whereas the practical algorithm uses a nonzero regularization coefficient. Does the rectified-transport property still hold under this setting?

3 comments

We deeeply thank the reviewer for the detailed and constructive feedback, and for recognizing the importance of the problem and the formulation of our method. After carefully considering your feedback, we would like to address each of your concerns below.

W1 and Q1: The contribution of BT-Flow and stronger endpoint-lookahead ablations

BT-Flow addresses the transport ambiguity of endpoint-only preference alignment by comparing direct-completion velocities at a shared state, time, and prompt. At fixed state and time, a candidate velocity and its reconstructed endpoint are in bijection, as used explicitly in Proposition 5.1. BT-Flow therefore reuses the existing preference information. Its contribution is how that information is represented and applied to local velocity optimization.

We agree that the comparison in Figure 3(c) does not fully separate the candidate representation from the conditional reward architecture. To address this, we train a conditional endpoint reward with the same backbone as Flow-RM. Its candidate stream receive the reconstructed endpoint x^1=xt+(1−t)vt\hat{x}_1=x_t+(1-t)v_t with the following Bradley–Terry loss:

−Elog⁡σ(rϕ(xt,x^1+,t,c)−rϕ(xt,x^1−,t,c)).-\mathbb E \log\sigma \left( r_\phi(x_t,\hat x_1^+,t,c) - r_\phi(x_t,\hat x_1^-,t,c) \right).

We evaluated all three rewards within Flow-VPA under matched alignment settings. The table reports GPT-5.6 Luna reference win rates against SD3.5-M on the same 500 held-out prompts at 40 NFE. This ablation uses K=1K=1.

Exploration scale σmax⁡\sigma_{\max}Flow-RM (%)Conditional endpoint RM (%)Image-RM+REC (%)
067.268.066.8
0.168.068.266.2
0.368.465.662.4
0.570.664.261.0
0.769.463.858.4

The matched conditional-endpoint control is slightly ahead at zero and low exploration. Flow-RM performs better at every tested scale from σmax⁡=0.3\sigma_{\max}=0.3 onward.

Thus, the advantage under moderate exploration persists when the conditional architecture is matched. This supports a practical benefit of the velocity representation for optimizing explored actions.

W2 and Q3: Connecting the theoretical analysis to practical optimization

We agree that the structural guarantee and the behavior of the implemented algorithm should be distinguished more clearly.

First, the structural result extends to a common reference-deviation constraint. We will revise Theorem 5.1 and its proof as follows.

Theorem 5.1. Fix a source noise x0x_0, a prompt cc, and N≥2N\geq2 Euler steps on a uniform time grid. Suppose the latent reward r⋆(⋅,c)r^\star(\cdot,c) has a unique maximizer x1⋆x_1^\star. Consider the ideal deterministic image and flow reward terms in Eqs. (9) and (13), using r⋆r^\star and its canonical lift rflow⋆r_{\mathrm{flow}}^\star, subject to a common reference-deviation budget Rref(θ)≤ρR_{\mathrm{ref}}(\theta)\leq\rho. Denote their constrained maximizer sets by ΘImage⋆\Theta_{\mathrm{Image}}^\star and ΘFlow⋆\Theta_{\mathrm{Flow}}^\star. If the feasible policy class admits both direct and non-direct rollouts from x0x_0 to x1⋆x_1^\star, then

ΘFlow⋆=ΘImage⋆∩θ:θ induces a direct rectified rollout⊊ΘImage⋆.\Theta_{\mathrm{Flow}}^\star= \Theta_{\mathrm{Image}}^\star \cap \\{ \theta:\theta\text{ induces a direct rectified rollout} \\} \subsetneq \Theta_{\mathrm{Image}}^\star.

So for Q3, the key is that exact rectification remains compatible with reference proximity when optimal direct transport is feasible within the budget.

Second, Section 4.2 demonstrates the practical relevance of this ambiguity. Endpoint-reward alignment changes trajectory geometry, and its preference advantage can weaken as NFE decreases. These observations motivate assessing preference and geometry jointly under finite-step sampling.

Third, we directly evaluate the implemented method with nonzero reference regularization. Figure 3(a) shows robust preference performance as NFE decreases. We additionally report Flow-VPA’s trajectory deviation togther with alignment performance as below.

NFEBase deviationFlow-VPA deviationFlow-VPA win rate (%)
100.040550.0414473.0
200.048330.0478871.6
400.052470.0510472.8
600.053780.0521870.0
800.054670.0529270.2

These measurements support the practical effectiveness of Flow-VPA under its actual training configuration. The theorem explains the structural preference for direct transport; the experiments establish the observed preference and geometry improvements.

W3. Broader experimental validation

We agree that additional backbones and direct human comparisons would strengthen the evidence for general applicability. These evaluations are beyond what we can complete within the rebuttal period, and we will make their absence explicit as a limitation.

The current evaluation combines GPT-5.6 Luna/Terra judgments with Q-Judger scores. Appendix E.4 and Table 4 also assess agreement with original human-labeled pairs in the validation-unique and test-unique splits of Pick-a-Pic-v2:

EvaluatorValidation-unique agreement (%)Test-unique agreement (%)
GPT-5.6 Luna63.5266.27
GPT-5.6 Terra64.3467.33
Q-Judger QAA-F59.1459.67
Q-Judger All555.8160.85

This held-out calibration provides supporting evidence that the evaluators track human preferences.

Q2: Generalization beyond synthetic interpolation states

We evaluated the frozen Flow-RM on the same held-out human-labeled endpoint pairs under three anchor distributions. For each comparison, both candidate velocities were constructed from the same state, time, and prompt. Keeping the labeled endpoints fixed tests preference-ranking transfer as the anchor distribution changes.

Anchor distributionPairwise accuracy (%)
Synthetic interpolation states64.22
States visited by the base policy65.03
States visited by the aligned policy64.34

The observed accuracies remain similar across all three distributions. These results support transfer of the learned preference rankings to the tested base-policy and aligned-policy states.


We thank the reviewer again for the constructive reviewes and questions. They help us clarify the reward representation, the scope of the theory, and the empirical evidence supporting the practical method.

Dear Reviewer yZJC,

As we are now in the final day of the discussion phase, we would greatly appreciate your feedback on whether we have appropriately addressed your concerns via our rebuttal.

If you have any remaining concerns or would like further clarification, please kindly let us know, and we will do our best to resolve them.

Thank you for your time and consideration.

Sincerely,
Authors

Comments on the paper

4 comments
Area_Chair_qCFjOpenReview ↗

Please check authors' rebuttal and update your review and score accordingly

Dear Reviewers,

The authors have submitted their rebuttal and responses to your comments.

Please take a moment to carefully read through their feedback and update your official reviews and ratings accordingly. Thank you for your time and contribution to the review process!

Your AC

Dear AC and Reviewers,

We hope you all had a wonderful weekend!

We much appreciate the following up from the AC, and once again sincerely thank all reviewers for their detailed and constructive feedback. Below is a brief summary of how our rebuttal addresses your concerns:

  1. Reviewer yZJC: We revised the theory to account for reference constraints and added matched endpoint-reward ablations and reward generalization tests.
  2. Reviewer MVPQ: We clarified trajectory kinks and endpoint equivalence, and added matched endpoint-reward ablations alongside joint results on geometry and diversity.
  3. Reviewer LyMs: We clarified state-dependent endpoint non-identifiability, local preference preservation, and shared-anchor training. We also added performance and efficiency comparisons with DiffusionNFT and geometry results for State-GRPO.

We remain fully available to answer any remaining questions and would greatly appreciate your feedback.

Best regards,
The Authors

Dear Reviewers and AC,

We sincerely thank you for the time and effort devoted to evaluating our work and for your constructive feedback.

We have uploaded a revised manuscript, with the changes highlighted. To facilitate the meta-review process, we briefly summarize our contributions and the revisions made in response to your comments.

1. Summary of Contributions

This paper identifies a structural limitation of preference alignment for rectified-flow models: image-level rewards cannot distinguish trajectories reaching the same preferred image, while denser state-level rewards do not explicitly supervise the local transport needed for effective finite-step sampling. We introduce BT-Flow, which lifts existing image comparisons into preferences over velocities at a shared state, and Flow-VPA, which directly aligns these velocities at on-policy states through likelihood-free deterministic policy optimization. Our theoretical analysis show that, under mild assumptions, flow alignment selects direct rectified solutions among image-alignment optima. Experiments on SD3.5-M demonstrate stronger preference alignment, sustained gains down to 10 NFE, and lower online training cost than the evaluated baselines.

2. Revision Summary

  • Revised Theorem 5.1 to incorporate a common reference constraint (Reviewers yZJC).
  • Added matched reconstructed-endpoint reward ablations, supporting the benefit of the velocity representation under moderate exploration (Reviewers yZJC and MVPQ).
  • Added reward-generalization tests on policy-visited states throughout alignment (Reviewer yZJC).
  • Added trajectory-geometry and LPIPS diversity results, showing strong alignment with low rectified deviation and competitive diversity (Reviewers MVPQ and LyMs).
  • Added DiffusionNFT comparisons demonstrating stronger alignment at lower online training cost, and expanded the related-work discussion (Reviewer LyMs).

We thank you again for your constructive feedback, which has helped strengthen both the analysis and empirical evidence. We remain available to address any further questions.

Best regards,
The Authors

Dear Reviewers and AC,

Thank you for your careful consideration of our work and the constructive exchanges throughout the discussion. Before the discussion closes today (AoE), we would welcome your feedback on whether our revisions and clarifications have addressed your remaining concerns regarding:

  1. Reference-constrained analysis in Theorem 5.1.
  2. Reward representation and generalization, supported by matched endpoint-reward ablations and tests on on-policy states.
  3. Trajectory geometry, output diversity, and efficiency, supported by additional measurements and DiffusionNFT comparisons.
  4. Endpoint-reward non-identifiability: distinct SDE-sampled rollouts reaching the same endpoint receive the same image reward (Proposition 4.1).
  5. Lifting endpoint preferences to local velocity preferences, with preference preservation and compatibility with multi-step denoising clarified in Proposition 5.1.

Your insightful reviews have helped us make the theoretical statements more precise and strengthen the experimental support for our method. We would be happy to address any outstanding questions before the deadline.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.