acceptodds
Withdrawn from ICLR 2027

From Preference to Flow: Turning Pairwise Preferences into Local Flow Supervision

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Preference optimization has emerged as an effective post-training paradigm for aligning visual generative models with human preferences. Prior methods usually use pairwise preferences as ranking signals over final outputs or as objectives over the generation trajectory, but they rarely turn such preferences into explicit local supervision for intermediate flow dynamics. Motivated by this gap, we propose **P2Flow (Preference-to-Flow)**, a preference post-training framework that converts pairwise preferences into local state-velocity supervision for intermediate flow dynamics. Given a pretrained flow-based generator, P2Flow uses the latent displacement from the preferred output to the rejected one as a pair-specific preference-violating direction to construct perturbed intermediate states around the preferred flow states, and pairs them with corrective velocity targets whose implied clean endpoint remains anchored to the preferred latent. A timestep-dependent modulation further creates bridge-shaped state perturbations that vanish at both the clean and Gaussian endpoints. This approach retains the standard squared velocity-regression objective of flow matching and requires no additional network modules, reward models, or online rollouts during optimization. In extensive experiments using our 40.8K high-confidence human-VLM consensus preference pairs, P2Flow consistently outperforms supervised fine-tuning and strong preference-optimization baselines on Wan2.1-T2V-14B and LingBot-Video-Dense across three benchmarks, showing clear improvements in both automatic metrics and human evaluations.

Primary area:
generative models
Keywords:
Preference Optimization, Post-Training, Generative Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YBKyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes P2Flow, a preference post-training method for flow-based video generation models. Instead of using DPO-style ranking losses, P2Flow perturbs intermediate states along the winner-to-loser latent direction and trains the model with a standard flow-matching MSE loss to predict corrective velocities pointing back to the preferred sample. The authors also introduce a 40.8K video preference dataset annotated via human–VLM consensus and show that P2Flow outperforms SFT and prior DPO baselines on two large video backbones.

Critical strengths
  1. The method is lightweight and easy to integrate into existing training pipelines, as it reduces preference learning to a standard flow-matching MSE loss requiring only one forward pass per step and no reference or reward models in memory.

  2. The paper contributes a well-curated 40.8K video preference dataset built with semantic balancing across 20K prompt clusters, candidate videos from 8 modern generators, and strict consensus filtering between Gemini-3 and three human annotators.

Critical weaknesses
  1. Formulating preference optimization purely as an MSE velocity regression anchored to the preferred sample z_w acts more like data-augmented behavior cloning than true preference alignment. First, it explicitly breaks global rectified-flow trajectory consistency (as shown in Equation 9, where the tangent of the constructed state path does not match the target velocity), which may distort the learned vector field. Second, because the objective only regresses toward fixed offline winners, it lacks exploration to discover better generations beyond the training data. Third, the method is largely heuristic and lacks rigorous theoretical justification connecting this local MSE regression to formal preference optimization or reward maximization.
  2. Since candidate videos for a given prompt are generated independently with different initial noise, the preferred sample z_w and rejected sample z_l often have completely different spatial layouts, camera motions, and object structures. Treating their linear subtraction in VAE latent space as a local preference-violating direction is geometrically questionable; perturbing states toward a structurally misaligned video and forcing the model to regress back to the preferred sample via L2 loss can push intermediate states off-manifold and induce target-averaging artifacts.
Questions for authors
  1. Since the constructed velocity target breaks global trajectory consistency (Equation 9), does P2Flow increase trajectory curvature and degrade generation quality under few-step ODE sampling (e.g., 5 to 15 steps) compared to the pretrained model and DPO baselines?
  2. Because preferred and rejected videos come from different models with misaligned spatial layouts, could the authors directly visualize the decoded images of the intermediate weighted features (z_t^pref) across different timesteps? This would clarify whether combining structurally different latents produces meaningful features or severe ghosting artifacts.
  3. Is the proposed method specifically tailored to video generation? Can it be directly applied to general flow-matching models such as image generation?
Reviewer_fdSHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes P2Flow, a post-training method for flow-based text-to-video generators. Given a preferred and a rejected video under the same prompt, P2Flow uses their latent displacement to perturb an intermediate state on the preferred sample's flow path and constructs a velocity target whose implied clean endpoint remains the preferred latent. The method retains the standard squared flow-matching loss and does not require a reward model or online rollouts. The paper also introduces 40.8K preference pairs obtained by strict agreement among Gemini-3 and three human annotators, and evaluates P2Flow on Wan2.1-T2V-14B and LingBot-Video-Dense using VBench, VideoJAM, a held-out DiverseT2V-Bench, automatic metrics, and a small human study. P2Flow is reported to outperform SFT, Vanilla DPO, DenseDPO, and ViPO.

Critical strengths

The paper gives a simple and technically coherent construction for converting pairwise data into state–velocity examples while retaining the standard flow-matching regression interface. The explicit distinction between local supervision and a global flow trajectory is a useful clarification.

Critical weaknesses
  1. The central latent-direction assumption is unverified. For videos generated by different models, z_l - z_w can mix quality with content, identity, camera, motion, style, and source-model artifacts. Strict consensus labels do not make this displacement a disentangled quality direction. The current direction ablations show only that alternative perturbations have different effects; they do not establish when the proposed direction is semantically valid or why it should generalize. This weakens the main interpretation of P2Flow.

  2. The evidence does not establish improvement of the global generation trajectory. The identity z_t^pref - t v_pref = z_w follows from the way the training pair is constructed. Because the constructed states are not required to form a globally consistent flow path and v_pref is not their tangent, low endpoint-recovery error does not imply successful multi-step sampling toward the preferred video. The sample-correction experiment is not specified in enough detail to resolve this issue.

  3. The experimental comparisons do not isolate the method's contribution. SFT uses a much smaller preferred-only subset, whereas P2Flow and the other preference methods use all retained pairs. The paper does not equalize examples, optimization steps or tokens, compute, or tuning budgets. Baseline adaptation details and hyperparameters are also missing, and the comparison does not include several directly related flow-preference methods. The reported improvements therefore cannot be cleanly attributed to P2Flow.

  4. The quantitative claims lack uncertainty estimates. Table 2 reports single values without seeds, variance, confidence intervals, or significance tests. The held-out benchmark and human study are relatively small, and the aggregation used for VideoAlign Overall is not justified. Stronger prompt-level statistical analysis and independent human validation are needed.

Questions for authors
  1. What evidence supports the interpretation that z_l - z_w represents a quality-degrading direction rather than changes in content, identity, camera, motion, style, or source-model artifacts? Can you provide controlled attribute-level pairs, the exact normalization used for the displacement, and correlations with independent human attribute judgments?

  2. Can you provide an equal-budget comparison in which SFT, Vanilla DPO, DenseDPO, ViPO, and P2Flow use the same number of examples, optimization steps or tokens, LoRA parameter count, and hyperparameter-search budget? Please also report the number of unique prompts and winner videos used by each method.

  3. How are DenseDPO and the other baselines adapted to cross-model, full-video pairs? Please provide the implementation details, tuned hyperparameters, training duration, random seeds, and results across multiple runs.

  4. Can you evaluate trajectory-level behavior using multiple solvers, step sizes, and rollout lengths, and compare P2Flow with a standard winner-reconstruction objective or an equivalent state-perturbation formulation? Please fully specify the sample-correction protocol, including sample counts, rejected-sample selection, noise levels, solver, and number of steps.

  5. Please report prompt-level confidence intervals or significance tests, inter-annotator agreement, the exact VideoJAM sample count, and the relationship between VideoAlign and independent human judgments. How often does P2Flow improve text alignment at the expense of motion or visual quality?

  6. Please provide the encoder, preprocessing, latent-scaling, timestep and perturbation-strength sampling, loss-mixing, LoRA-injection, training-step, and sampling configurations. Are the candidate videos and prompt data legally redistributable under the terms of the source models, services, and datasets?

Reviewer_e1goOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes P2Flow (Preference-to-Flow), a preference post-training framework that converts pairwise preferences into local state–velocity supervision for intermediate flow dynamics.

Critical strengths
  1. P2Flow turns winner–loser pairs into winner-anchored local state–velocity targets, aligning preference learning with the flow-matching objective without extra modules, reward models, or online rollouts.

  2. The paper builds a large-scale, high-confidence video preference dataset with SOTA generators and strict human–VLM consensus, improving scale, coverage, diversity, and label quality over prior resources.

  3. Extensive experiments on Wan2.1-T2V-14B and LingBot-Video-Dense verify the superiority of P2Flow over SFT and strong preference-optimization baselines across multiple benchmarks and human evaluations.

Critical weaknesses

I am somewhat concerned that prior DPO-style methods such as VideoDPO [1] construct positive–negative pairs from outputs of the same model, whereas you generate videos with many different models, including much stronger ones like Seedance 2.0, and then form preference pairs across them. Might this introduce a bias in how the pairs are matched? Could you run an experiment on prior datasets (e.g., constructing positive–negative pairs from videos generated by the same model) to verify that P2Flow remains effective under that setting?

[1] Liu R, Wu H, Zheng Z, et al. Videodpo: Omni-preference alignment for video diffusion generation[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 8009-8019.

Questions for authors

My major concern lies in weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.