acceptodds
Under review as a conference paper at ICLR 2027

FlowCPO: A Unified Divergence View of Preference Alignment for Flow Models

Abstract

Preference alignment for flow and diffusion models now spans online reinforcement learning and offline preference optimization, but the relation between these methods remains unclear. In particular, existing forward-process alignment methods require fresh samples from the current model, while offline methods based on fixed preference pairs rely primarily on positive-only fine-tuning or DPO-style likelihood-ratio surrogates. We organize these approaches through a divergence-based framework and introduce FlowCPO, an offline forward-KL objective that uses both preferred and dispreferred samples without online rollouts. For linear interpolation, we show under explicit regularity conditions that the forward-KL objective is bounded by a contrastive flow matching loss, yielding a tractable surrogate on fixed data. We further show that this loss is nonnegative, whereas the signed regression loss of simplified FlowDPO can be unbounded below. In the in-domain setting, FlowCPO achieves higher mean GenEval and OCR scores than the evaluated baselines, reaching 0.84 and 0.87 versus 0.81 and 0.74 for FlowDPO at CFG 3.0. In the out-of-domain setting, the results are mixed, with the best GenEval result but lower reward scores than RFT on several metrics.

Primary area:
reinforcement learning
Keywords:
Flow Matching, Diffusion Model, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wRkKOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose FlowCPO, an offline preference alignment method for flow matching models. The key idea is to organize existing alignment methods along two axes, the sampling regime and the direction of the idealized KL objective, and then to fill the one combination that no prior method occupies: a forward-KL objective optimized offline using both preferred and dispreferred samples. The authors turn that objective into two regression terms, one fitting the preferred samples and one pushing the model away from the dispreferred samples, so that training reduces to flow-matching regression on fixed preference pairs. The experiments fine-tune Stable Diffusion 3.5 Medium with LoRA on in-domain and out-of-domain offline preference data, comparing against two offline baselines, RFT and FlowDPO, and evaluating on GenEval, an OCR metric, and several preference metrics.

Critical strengths
  1. The paper is clearly written, the results are easy to audit, and the training and data-construction protocol is documented in enough detail to be reimplemented.
  2. Hyperparameter coverage is thorough, and every fine-tuned entry is a mean plus or minus a standard deviation over five seeds (Tabs. 5-8), which very few papers in this area report.
Critical weaknesses
  1. The method is essentially an offline version of DiffusionNFT, and DiffusionNFT's own paper already ran similar experiments. At λ=1\lambda = 1 the objective of Eq. 11 is DiffusionNFT's loss with the soft optimality weight replaced by a hard 0/10/1 label on a two-sample group, and the two share the same mixed velocity fields, the same regression form, the same EMA reference, and even the same symbols β\beta and η\eta. DiffusionNFT is off-policy by design, so freezing its sampling policy already reaches the offline end of the same spectrum. The original paper studies that limit and reports that it is stable but converges impractically slowly. What "offline" adds here is only that the data are produced ahead of time, which shifts the cost rather than removing it. Constructing the in-domain set still requires 16 samples for roughly 95K prompts from the frozen reference, so the total compute is not obviously lower, and the data pipeline becomes a component that has to be built and maintained in its own right.
  2. Key baselines are missing. Only RFT and FlowDPO are compared. DiffusionNFT, AWM and RAM are online methods, but moving them into this setting is trivial. Once the fixed preference pairs are assigned 0/1 rewards, all three can be trained directly on the buffer that has already been generated, without any new rollout. In fact the λ=1\lambda = 1 row of Tab. 4 already is DiffusionNFT under hard labels, which the paper does not point out. Without that comparison on the same data, there is no evidence that the changes made here improve on the method the objective is taken from.
Questions for authors
  1. What exactly differs from an offline DiffusionNFT? At λ=1\lambda = 1 the objective in Eq. 11 is DiffusionNFT's loss with the optimality weight reduced to a hard 0/10/1 label on a two-sample group, while the mixed velocity fields, the EMA reference and the hyperparameters β\beta and η\eta are all the same. DiffusionNFT's Figure 8 already sweeps the EMA coefficient and reports that holding the sampling policy nearly fixed is stable but converges impractically slowly. Please state precisely which component of Eq. 11 is not present in DiffusionNFT.
  2. What is the advantage of offline over online? DiffusionNFT reports a GenEval score of 0.98 on SD3.5-Medium within 1k steps and without CFG, against 0.84 here at 2k steps with CFG 3.0, and it is ahead on the other metrics as well. If moving the same objective offline costs that much accuracy while still requiring 16 samples for roughly 95K prompts up front, what is the case for it?
  3. Baselines on the same offline data. Please add DiffusionNFT, AWM and RAM trained on the buffer you have already generated, in both the in-domain and the OOD setting. Once the fixed pairs carry 0/10/1 rewards, all three run there with no new rollout, and the DiffusionNFT row should use graded weights over all 16 candidates, whose scores the pipeline has already computed. Without that row there is no evidence that the changes made here improve on the objective they are taken from.
  4. What does the dispreferred branch contribute? Out of domain at CFG 4.5, RFT beats FlowCPO on all eight metrics and FlowCPO falls below the untuned base model on three of them, yet RFT is simply Eq. 11 with λ=0\lambda = 0. Outside the metric used to filter the training data, what evidence remains that the branch helps? App. D.1 states that the OOD images come from FLUX.1-Dev and SD3.5-Large, and Tab. 3 scores both above the SD3.5-M reference, so one possibility is that pushing away from those losers also pushes the model away from content better than its own.
5 comments

We sincerely thank the reviewer for the careful review and rigorous questions.

W2 & Q3: Regarding the missing baselines which are offline version of AWM and RAM with 0/1 rewards.

Thank you for raising this question.

Our claim is that the offline version of AWM and RAM is actually RFT, which is already included in current experiments.

In the following, we will provide the theoretical proof under our proposed divergence-based framework.

  • AWM: In App. C.3, we have proved that

    LAWM=Eπθ[w(t)r(x0,c)∥vθ−ut∥22]L_{AWM} = E_{\pi_\theta}[w(t)r(x_0, c)\| v_\theta - u_t\|^2_2]

    When we hardcode the reward and set reward as 1 for preferred samples, 0 for dispreferred samples, then we have:

    Loffline−AWM=Eπθ+[w(t)⋅1⋅∥vθ−ut∥22]+Eπθ−[w(t)⋅0⋅∥vθ−ut∥22]=Eπθ+[w(t)∥vθ−ut∥22]L_{offline-AWM} = E_{\pi_\theta^+}[w(t)\cdot 1 \cdot \| v_\theta - u_t\|^2_2] + E_{\pi_\theta^-}[w(t)\cdot 0 \cdot \| v_\theta - u_t\|^2_2]=E_{\pi_\theta^+}[w(t)\| v_\theta - u_t\|^2_2] which is actually RFT.

  • RAM: In App. C.4, we have proved the equivalent loss of RAM is

    L2=Eπθ[r(x0,c)∥vθ−ut∥22]+Eπθ[∥vθ−vref∥22]L_2 = E_{\pi_\theta}[r(x_0, c)\| v_\theta - u_t\|^2_2] + E_{\pi_\theta}[\| v_\theta - v_{ref}\|^2_2]

    When we hardcode the reward and set reward as 1 for preferred samples, 0 for dispreferred samples, then we have:

    Loffline−2=Eπθ+[∥vθ−ut∥22]+Eπθ[∥vθ−vref∥22]L_{offline-2} = E_{\pi_\theta^+}[\| v_\theta - u_t\|^2_2] + E_{\pi_\theta}[\| v_\theta - v_{ref}\|^2_2]

    which is actually RFT plus a regularization term.

Finally, we emphasize that we have cover all the baselines which can be divided into three types using eq. 12:

  • zero negative regularization term: RFT
  • minus negative regularization term: FlowDPO
  • plus negative regularization term: FlowCPO

W1 & Q1: Regarding the difference between FlowCPO and offline DiffusionNFT

Thank you for pointing out this issue.

We admit that FlowCPO is an offline version of DiffusionNFT and have claimed it with Tab.1 since the introduction. DiffusionNFT has provided a few experimental results to discuss the offline DiffusionNFT with soft reward label rather than hard 0/1 label.

The differences:

  • we theoretically propose the divergence-based framework and establish the relationship between FlowGRPO, DiffusionNFT and their offline version.
  • theoretically, we provide an analysis on the difference between FlowCPO and FlowDPO in App. B.5.
  • empirically, we deeply explore the effectiveness of difference types of negative regularization in RQ1 and RQ2 of experiments.

We believe that both the theoretical and empirical results can help community better understand the differences among three types of negative regularization (RFT, FlowCPO and FlowDPO), which are not included in DiffusionNFT paper.

Q2:Regarding the advantage of offline methods over online methods

Thank you for pointing this out.

Studying offline methods is valuable from both practical and methodological perspectives:

  1. Practical relevance: As industrial products accumulate increasingly large offline datasets, offline methods provide a way to reuse these data efficiently.
  2. Methodological value: Offline learning is an important part of reinforcement learning. Developing its theoretical foundations contributes to a more complete understanding of the field.

Offline methods also offer two practical advantages:

  1. Training efficiency: Offline methods avoid online sampling during optimization, which can substantially reduce training time, especially when generating new samples is computationally expensive.
  2. Control over data quality: Offline datasets can be curated and filtered before training, allowing reward hacking to be addressed through both data selection and algorithm design. In online methods, newly generated samples continually enter the training process, making it more challenging to maintain consistent control over data quality.

Q4:Why in-domain dispreferred samples help while out-of-domain dispreferred samples don’t?

Thank you for raising this important question.

We will first theoretically explain why in-domain dispreferred samples help while OOD dispreferred samples don’t help, and then provide empirical evidences from prior studies.

  • Theoretical explanation: the conclusion here is that the in-domain dispreferred samples can strengthen the update signal, while OOD dispreferred samples may diminish this signal. (We provide detailed analysis in App. B.6.)
    • Flow Matching:

      LFM(θ)=Eπref[∥vθ−ut∥22]=12Eπ+[∥vθ−ut∥22]+12Eπ−[∥vθ−ut∥22]L_{FM}(\theta) = E_{\pi_{ref}}[\| v_\theta -u_t\|^2_2] = \frac{1}{2}E_{\pi^+}[\| v_\theta -u_t\|^2_2] + \frac{1}{2}E_{\pi^-}[\| v_\theta -u_t\|^2_2]

      Here we need to emphasize that the π+\pi^+ and π−\pi^- are from πref\pi_{ref}, i.e. the samples are in-domain.

      Let g+(θ)=Eπ+[2(vθ−ut)⋅∇vθ], g−(θ)=Eπ−[2(vθ−ut)⋅∇vθ]g_+(\theta) = E_{\pi^+}[2 (v_\theta -u_t) \cdot \nabla v_\theta],\ g_-(\theta) = E_{\pi^-}[2 (v_\theta -u_t) \cdot \nabla v_\theta], then for optimal solution θ0\theta_0, we have ∇LFM(θ0)=0⇒g+(θ0)=−g−(θ0)\nabla L_{FM}(\theta_0) = 0 \Rightarrow g_+(\theta_0) = -g_-(\theta_0).

    • RFT:

      LRFT(θ)=Eπ+[∥vθ−ut∥22]L_{RFT}(\theta) = E_{\pi^+}[\| v_\theta -u_t\|^2_2]

      Then for the first update from θ=θ0\theta = \theta_0 , we have ∇LRFT(θ0)=g+(θ0)\nabla L_{RFT}(\theta_0) = g_+(\theta_0).

      This makes the model to align with the preferred samples.

    • Simplied FlowDPO:

      LDPO(θ)=Eπ+[∥vθ−ut∥22]−Eπ−[∥vθ−ut∥22]L_{DPO}(\theta) = E_{\pi^+}[\| v_\theta -u_t\|^2_2] - E_{\pi^-}[\| v_\theta -u_t\|^2_2].

      Then for the first update from θ=θ0\theta = \theta_0 and in-domain samples, we have ∇LDPO(θ0)=g+(θ0)−g−(θ0)\nabla L_{DPO}(\theta_0) = g_+(\theta_0) - g_-(\theta_0).

      • If the π+\pi^+ and π−\pi^- are from πref\pi_{ref}, i.e. the samples are in-domain, then g+(θ0)=−g−(θ0)g_+(\theta_0) = -g_-(\theta_0) and thus ∇LDPO(θ0)=2g+(θ0)\nabla L_{DPO}(\theta_0) = 2g_+(\theta_0), which means in-domain dispreferred samples can strengthen the update signal.
      • If the π+\pi^+ and π−\pi^- are not from πref\pi_{ref}, i.e. the samples are OOD, then g−(θ0)g_-(\theta_0) may diminish the signal of g+(θ0)g_+(\theta_0).
    • FlowCPO:

      LCPO(θ)=Eπ+[∥μθ−ut∥22]+Eπ−[∥νθ−ut∥22]L_{CPO}(\theta) = E_{\pi^+}[\| \mu_\theta -u_t\|^2_2] + E_{\pi^-}[\| \nu_\theta -u_t\|^2_2],

      where μθ=(1−β)vθ0+βvθ\mu_\theta = (1 - \beta)v_{\theta_0} + \beta v_\theta and νθ=(1+β)vθ0−βvθ\nu_\theta = (1 + \beta)v_{\theta_0} - \beta v_\theta.

      Then for the first update from θ=θ0\theta = \theta_0 and in-domain samples, we have ∇LCPO(θ0)=βg+(θ0)−βg−(θ0)\nabla L_{CPO}(\theta_0) = \beta g_+(\theta_0) - \beta g_-(\theta_0).

      • If the π+\pi^+ and π−\pi^- are from πref\pi_{ref}, i.e. the samples are in-domain, then g+(θ0)=−g−(θ0)g_+(\theta_0) = -g_-(\theta_0) and thus ∇LCPO(θ0)=2βg+(θ0)\nabla L_{CPO}(\theta_0) = 2\beta g_+(\theta_0), which means in-domain dispreferred samples can strengthen the update signal.
      • If the π+\pi^+ and π−\pi^- are not from πref\pi_{ref}, i.e. the samples are OOD, then g−(θ0)g_-(\theta_0) may diminish the signal of g+(θ0)g_+(\theta_0).
  • Empirical Evidence from prior studies:
    • In Figure 5 of [2], pure SFT and SFT-then-RL with stronger SFT initialization achieve higher Pass@1 than the 5% SFT + 95% RL configuration. This suggests that adequate supervised adaptation can be important for effective RL on OOD tasks.
    • In Section 6.2.1 and Figure 4 of [3], stronger SFT initialization also improves the final cross-task (OOD) evaluation performance after RL. This further supports the importance of the starting model.
    • In the last paragraph (“DPO outline”) of Section 4 in [1], the authors recommend fitting the reference model to preferred completions when the original model is unavailable, to mitigate distribution shift before applying DPO.
[1] Direct preference optimization: Your language model is secretly a reward model. NIPS 2023
[2] Reinforcement Learning for Out-of-Distribution Reasoning in LLMs: An Empirical Study on Diagnosis-Related Group Coding, NIPS 2025 Spotlight
[3] Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning, ACL 2026
AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer,

Thank you again for reviewing our paper. We would like to follow up on our response to your review and would appreciate your feedback on whether any of your concerns remain unresolved.

As the discussion period is nearing its end, we would be happy to provide any further clarification that would be helpful for your assessment.

Thank you for your time and consideration.

Best regards,
The Authors

Reviewer_yA28OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FlowCPO introduces an offline alignment objective using coupled positive and mirrored negative flow branches. It derives a forward-KL regression bound for flow-matching and reports improved in-domain results on SD3.5-M, with mixed out-of-domain performance.

Critical strengths
  1. Rigorous mathematical derivations. The paper clearly states its assumptions and derives divergence correlations and offline regression bounds, providing a solid mathematical foundation for the two-branch objective.
  2. Reliable in-domain performance improvements. FlowCPO outperforms FlowDPO on the GenEval and OCR tasks, and its ablation experiments on different regularization mechanisms are rigorously designed and thoroughly executed.
  3. The paper is well-organized and written in clear, accessible language.
Critical weaknesses
  1. Empirical results are insufficient to fully demonstrate the advantages of this method. Although there is a significant improvement in in-domain OCR performance, the improvement in GenEval performance is modest. More importantly, FlowCPO underperforms the simpler RFT baseline model—which includes only positive examples—on multiple out-of-domain (OOD) preference metrics, making the practical benefits of the additional dual-branch design unclear.
  2. This innovation appears to be incremental. As discussed in Appendix C.2, the positive/mirrored negative regression structure is very similar to that of DiffusionNFT. The main new contributions lie in its adaptability to fixed preference data and its divergence-based theoretical justification. While these contributions are certainly useful, the mixed OOD results mean that the practical value of this extension has not yet been fully demonstrated.
Questions for authors
  1. Why do models that incorporate negative samples perform worse than RFT on multiple OOD performance metrics?
  2. Under what data conditions does a two-branch design consistently outperform fine-tuning using only positive samples?
7 comments

We sincerely appreciate the reviewer for the careful review and recognition on our theoretical derivation and in-domain empirical improvements.

W1.1: Regarding the advantage of FlowCPO over FlowDPO

Thank you for your concern on the advantage of FlowCPO.

We provide both theoretical and empirical results to support the advantage of FlowCPO.

Theoretically, we provide an analysis between the FlowCPO and FlowDPO in App B.5 and suggest the advantage that FlowCPO may be more stable than FlowDPO because FlowCPO has lower bound.

Empirically, we provide addition experimental results on FLUX.1-13B. This experiment is reimplemented in FlowFactory[4] codebase. The batch size per GPU is 8 and the experiment runs on 8 GPU.

  • Table Conclusion: we consider FlowCPO is more stable than FlowDPO.
  • For the degradation of quality of FlowDPO, we consider it is because
    • Unstable algorithm: FlowDPO is less stable than FlowCPO under current same experimental setup.
    • Large model size: the model size of FLUX.1-13B is very large.
CFGTrainStepGenEvalOCRPickscoreClipscoreUniRwdHPS v3
FLUX.1-13B1-0.38510.230921.48170.25010.51132.9467
3.5-0.63180.611822.79290.27760.656011.0171
FlowDPO12000.44400.2821.65870.24830.51914.2611
112200.53510.36920.84660.25860.44532.7595
3.52000.61160.604822.78250.27560.660211.3142
3.512200.61640.541421.29680.26640.49937.3950
FlowCPO111000.60600.223421.43910.25050.49903.2108
3.511000.76710.606322.75530.27840.658711.2727

W1.2 & Q1 & W2: In the OOD setting, why the performances of FlowCPO and FlowDPO are worse than RFT.

Thank you for raising this important question.

We will first theoretically explain why in-domain dispreferred samples help while OOD dispreferred samples don’t help, and then provide empirical evidences from prior studies.

  • Theoretical explanation: the conclusion here is that the in-domain dispreferred samples can strengthen the update signal, while OOD dispreferred samples may diminish this signal. (We provide detailed analysis in App. B.6.)
    • Flow Matching:

      LFM(θ)=Eπref[∥vθ−ut∥22]=12Eπ+[∥vθ−ut∥22]+12Eπ−[∥vθ−ut∥22]L_{FM}(\theta) = E_{\pi_{ref}}[\| v_\theta -u_t\|^2_2] = \frac{1}{2}E_{\pi^+}[\| v_\theta -u_t\|^2_2] + \frac{1}{2}E_{\pi^-}[\| v_\theta -u_t\|^2_2]

      Here we need to emphasize that the π+\pi^+ and π−\pi^- are from πref\pi_{ref}, i.e. the samples are in-domain.

      Let g+(θ)=Eπ+[2(vθ−ut)⋅∇vθ], g−(θ)=Eπ−[2(vθ−ut)⋅∇vθ]g_+(\theta) = E_{\pi^+}[2 (v_\theta -u_t) \cdot \nabla v_\theta],\ g_-(\theta) = E_{\pi^-}[2 (v_\theta -u_t) \cdot \nabla v_\theta], then for optimal solution θ0\theta_0, we have ∇LFM(θ0)=0⇒g+(θ0)=−g−(θ0)\nabla L_{FM}(\theta_0) = 0 \Rightarrow g_+(\theta_0) = -g_-(\theta_0).

    • RFT:

      LRFT(θ)=Eπ+[∥vθ−ut∥22]L_{RFT}(\theta) = E_{\pi^+}[\| v_\theta -u_t\|^2_2]

      Then for the first update from θ=θ0\theta = \theta_0 , we have ∇LRFT(θ0)=g+(θ0)\nabla L_{RFT}(\theta_0) = g_+(\theta_0).

      This makes the model to align with the preferred samples.

    • Simplied FlowDPO:

      LDPO(θ)=Eπ+[∥vθ−ut∥22]−Eπ−[∥vθ−ut∥22]L_{DPO}(\theta) = E_{\pi^+}[\| v_\theta -u_t\|^2_2] - E_{\pi^-}[\| v_\theta -u_t\|^2_2].

      Then for the first update from θ=θ0\theta = \theta_0 and in-domain samples, we have ∇LDPO(θ0)=g+(θ0)−g−(θ0)\nabla L_{DPO}(\theta_0) = g_+(\theta_0) - g_-(\theta_0).

      • If the π+\pi^+ and π−\pi^- are from πref\pi_{ref}, i.e. the samples are in-domain, then g+(θ0)=−g−(θ0)g_+(\theta_0) = -g_-(\theta_0) and thus ∇LDPO(θ0)=2g+(θ0)\nabla L_{DPO}(\theta_0) = 2g_+(\theta_0), which means in-domain dispreferred samples can strengthen the update signal.
      • If the π+\pi^+ and π−\pi^- are not from πref\pi_{ref}, i.e. the samples are OOD, then g−(θ0)g_-(\theta_0) may diminish the signal of g+(θ0)g_+(\theta_0).
    • FlowCPO:

      LCPO(θ)=Eπ+[∥μθ−ut∥22]+Eπ−[∥νθ−ut∥22]L_{CPO}(\theta) = E_{\pi^+}[\| \mu_\theta -u_t\|^2_2] + E_{\pi^-}[\| \nu_\theta -u_t\|^2_2],

      where μθ=(1−β)vθ0+βvθ\mu_\theta = (1 - \beta)v_{\theta_0} + \beta v_\theta and νθ=(1+β)vθ0−βvθ\nu_\theta = (1 + \beta)v_{\theta_0} - \beta v_\theta.

      Then for the first update from θ=θ0\theta = \theta_0 and in-domain samples, we have ∇LCPO(θ0)=βg+(θ0)−βg−(θ0)\nabla L_{CPO}(\theta_0) = \beta g_+(\theta_0) - \beta g_-(\theta_0).

      • If the π+\pi^+ and π−\pi^- are from πref\pi_{ref}, i.e. the samples are in-domain, then g+(θ0)=−g−(θ0)g_+(\theta_0) = -g_-(\theta_0) and thus ∇LCPO(θ0)=2βg+(θ0)\nabla L_{CPO}(\theta_0) = 2\beta g_+(\theta_0), which means in-domain dispreferred samples can strengthen the update signal.
      • If the π+\pi^+ and π−\pi^- are not from πref\pi_{ref}, i.e. the samples are OOD, then g−(θ0)g_-(\theta_0) may diminish the signal of g+(θ0)g_+(\theta_0).
  • Empirical Evidence from prior studies:
    • In Figure 5 of [2], pure SFT and SFT-then-RL with stronger SFT initialization achieve higher Pass@1 than the 5% SFT + 95% RL configuration. This suggests that adequate supervised adaptation can be important for effective RL on OOD tasks.
    • In Section 6.2.1 and Figure 4 of [3], stronger SFT initialization also improves the final cross-task (OOD) evaluation performance after RL. This further supports the importance of the starting model.
    • In the last paragraph (“DPO outline”) of Section 4 in [1], the authors recommend fitting the reference model to preferred completions when the original model is unavailable, to mitigate distribution shift before applying DPO.

Q2:Under what data conditions does the two-branch design consistently outperform positive-only fine-tuning?

In the theoretical analysis and empirical evidence of answer to W1.2 & Q1 & W2, we can know that:

  • when the data are in-domain, the two-branch design consistently outperform positive-only fine-tuning.
  • when the data are OOD, we can first make it be in-domain by applying SFT (i.e. Flow Matching loss) and then use RLHF methods.
[1] Direct preference optimization: Your language model is secretly a reward model. NIPS 2023
[2] Reinforcement Learning for Out-of-Distribution Reasoning in LLMs: An Empirical Study on Diagnosis-Related Group Coding, NIPS 2025 Spotlight
[3] Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning, ACL 2026
[4] Flow-Factory: A Unified Framework for Reinforcement Learning in Flow-Matching Models, 2026
AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer,

Thank you again for reviewing our paper. We would like to follow up on our response to your review and would appreciate your feedback on whether any of your concerns remain unresolved.

As the discussion period is nearing its end, we would be happy to provide any further clarification that would be helpful for your assessment.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_ukMMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper organizes preference alignment for flow and diffusion models along two axes: whether training samples come from the current model (online) or from a fixed dataset (offline), and whether the idealized objective is a forward or a reverse KL divergence. It proposes FlowCPO to fill the missing offline forward-KL case. Starting from a control-as-inference view of reward, the paper rewrites expected reward as a contrastive reverse-KL objective, reverses the KL direction to obtain an offline objective, and models the preferred and dispreferred branches as two velocity fields placed symmetrically around an EMA copy of the model. Under linear interpolation and uniform regularity assumptions, this forward-KL objective is upper-bounded by a contrastive flow-matching loss that can be computed from fixed preference pairs, with no online sampling. Unlike the signed loss of simplified FlowDPO, this loss is nonnegative. On SD3.5-M, FlowCPO improves in-domain GenEval and OCR scores over RFT and FlowDPO; out-of-domain results are mixed.

Critical strengths
  • The divergence view (Tables 1 and 2) organizes the field clearly. It treats FlowGRPO, DiffusionNFT, FlowDPO, RFT and SFT as special cases of one general objective, and identifies a real gap: an offline forward-KL method that uses both preferred and dispreferred data.
  • The derivation is explicit and self-consistent, covering the reward as a contrastive log-density ratio, the EMA-centered two-branch parameterization, and the flow-matching upper bound with clearly stated assumptions.
  • The method is simple and needs no online sampling. The nonnegativity argument against simplified FlowDPO is a reasonable motivation.
Critical weaknesses
  • The central claim, that using dispreferred samples helps, does not hold consistently. The advantage appears in-domain, but out-of-domain, FlowCPO is worse than the positive-only RFT on several reward metrics (Table 3). Since the whole point of the method is to use both sides of each preference pair, this is a substantive limitation. The paper acknowledges it but does not explain why it happens or when the negative branch can be trusted.
  • All fine-tuning results use a single backbone. Every fine-tuning experiment is on SD3.5-M. Since the paper presents a general framework, at least one other flow backbone is needed to show the results are not specific to SD3.5-M.
Questions for authors
  • Can you explain the out-of-domain results? Do they correlate with the mismatch between the model that generated the preference data and the reference model? In that setting, does the negative branch actively hurt performance, or does it simply stop helping?

  • Does the in-domain advantage over FlowDPO and RFT hold on a second backbone?

2 comments

We sincerely thank the reviewer for their careful review and recognizing the value of this divergence view.

W1 & Q1: Why in-domain dispreferred samples help while out-of-domain dispreferred samples don’t?

Thank you for raising this important question.

We will first theoretically explain why in-domain dispreferred samples help while OOD dispreferred samples don’t help, and then provide empirical evidences from prior studies.

  • Theoretical explanation: the conclusion here is that the in-domain dispreferred samples can strengthen the update signal, while OOD dispreferred samples may diminish this signal. (We provide detailed analysis in App. B.6.)
    • Flow Matching:

      LFM(θ)=Eπref[∥vθ−ut∥22]=12Eπ+[∥vθ−ut∥22]+12Eπ−[∥vθ−ut∥22]L_{FM}(\theta) = E_{\pi_{ref}}[\| v_\theta -u_t\|^2_2] = \frac{1}{2}E_{\pi^+}[\| v_\theta -u_t\|^2_2] + \frac{1}{2}E_{\pi^-}[\| v_\theta -u_t\|^2_2]

      Here we need to emphasize that the π+\pi^+ and π−\pi^- are from πref\pi_{ref}, i.e. the samples are in-domain.

      Let g+(θ)=Eπ+[2(vθ−ut)⋅∇vθ], g−(θ)=Eπ−[2(vθ−ut)⋅∇vθ]g_+(\theta) = E_{\pi^+}[2 (v_\theta -u_t) \cdot \nabla v_\theta],\ g_-(\theta) = E_{\pi^-}[2 (v_\theta -u_t) \cdot \nabla v_\theta], then for optimal solution θ0\theta_0, we have ∇LFM(θ0)=0⇒g+(θ0)=−g−(θ0)\nabla L_{FM}(\theta_0) = 0 \Rightarrow g_+(\theta_0) = -g_-(\theta_0).

    • RFT:

      LRFT(θ)=Eπ+[∥vθ−ut∥22]L_{RFT}(\theta) = E_{\pi^+}[\| v_\theta -u_t\|^2_2]

      Then for the first update from θ=θ0\theta = \theta_0 , we have ∇LRFT(θ0)=g+(θ0)\nabla L_{RFT}(\theta_0) = g_+(\theta_0).

      This makes the model to align with the preferred samples.

    • Simplied FlowDPO:

      LDPO(θ)=Eπ+[∥vθ−ut∥22]−Eπ−[∥vθ−ut∥22]L_{DPO}(\theta) = E_{\pi^+}[\| v_\theta -u_t\|^2_2] - E_{\pi^-}[\| v_\theta -u_t\|^2_2].

      Then for the first update from θ=θ0\theta = \theta_0 and in-domain samples, we have ∇LDPO(θ0)=g+(θ0)−g−(θ0)\nabla L_{DPO}(\theta_0) = g_+(\theta_0) - g_-(\theta_0).

      • If the π+\pi^+ and π−\pi^- are from πref\pi_{ref}, i.e. the samples are in-domain, then g+(θ0)=−g−(θ0)g_+(\theta_0) = -g_-(\theta_0) and thus ∇LDPO(θ0)=2g+(θ0)\nabla L_{DPO}(\theta_0) = 2g_+(\theta_0), which means in-domain dispreferred samples can strengthen the update signal.
      • If the π+\pi^+ and π−\pi^- are not from πref\pi_{ref}, i.e. the samples are OOD, then g−(θ0)g_-(\theta_0) may diminish the signal of g+(θ0)g_+(\theta_0).
    • FlowCPO:

      LCPO(θ)=Eπ+[∥μθ−ut∥22]+Eπ−[∥νθ−ut∥22]L_{CPO}(\theta) = E_{\pi^+}[\| \mu_\theta -u_t\|^2_2] + E_{\pi^-}[\| \nu_\theta -u_t\|^2_2],

      where μθ=(1−β)vθ0+βvθ\mu_\theta = (1 - \beta)v_{\theta_0} + \beta v_\theta and νθ=(1+β)vθ0−βvθ\nu_\theta = (1 + \beta)v_{\theta_0} - \beta v_\theta.

      Then for the first update from θ=θ0\theta = \theta_0 and in-domain samples, we have ∇LCPO(θ0)=βg+(θ0)−βg−(θ0)\nabla L_{CPO}(\theta_0) = \beta g_+(\theta_0) - \beta g_-(\theta_0).

      • If the π+\pi^+ and π−\pi^- are from πref\pi_{ref}, i.e. the samples are in-domain, then g+(θ0)=−g−(θ0)g_+(\theta_0) = -g_-(\theta_0) and thus ∇LCPO(θ0)=2βg+(θ0)\nabla L_{CPO}(\theta_0) = 2\beta g_+(\theta_0), which means in-domain dispreferred samples can strengthen the update signal.
      • If the π+\pi^+ and π−\pi^- are not from πref\pi_{ref}, i.e. the samples are OOD, then g−(θ0)g_-(\theta_0) may diminish the signal of g+(θ0)g_+(\theta_0).
  • Empirical Evidence from prior studies:
    • In Figure 5 of [2], pure SFT and SFT-then-RL with stronger SFT initialization achieve higher Pass@1 than the 5% SFT + 95% RL configuration. This suggests that adequate supervised adaptation can be important for effective RL on OOD tasks.
    • In Section 6.2.1 and Figure 4 of [3], stronger SFT initialization also improves the final cross-task (OOD) evaluation performance after RL. This further supports the importance of the starting model.
    • In the last paragraph (“DPO outline”) of Section 4 in [1], the authors recommend fitting the reference model to preferred completions when the original model is unavailable, to mitigate distribution shift before applying DPO.

W2 & Q2: regarding about the generalization of FlowCPO beyond SD3.5-M in in-domain setting.

Thank you for your concern on the generalization of FlowCPO.

We provide both theoretical and empirical results to support the generalization of FlowCPO.

Theoretically, we provide theoretical results in App B.5 to

  • support the effectiveness of FlowCPO by comparing it with FlowDPO .
  • suggest the advantage that FlowCPO may be more stable than FlowDPO because FlowCPO has lower bound.

Empirically, we provide addition experimental results on FLUX.1-13B. This experiment is reimplemented in FlowFactory[4] codebase. The batch size per GPU is 8 and the experiment runs on 8 GPU.

  • Table Conclusion: we consider FlowCPO is more stable than FlowDPO.
  • For the degradation of quality of FlowDPO, we consider it is because
    • Unstable algorithm: FlowDPO is less stable than FlowCPO under current same experimental setup.
    • Large model size: the model size of FLUX.1-13B is very large.
CFGTrainStepGenEvalOCRPickscoreClipscoreUniRwdHPS v3
FLUX.1-13B1-0.38510.230921.48170.25010.51132.9467
3.5-0.63180.611822.79290.27760.656011.0171
FlowDPO12000.44400.2821.65870.24830.51914.2611
112200.53510.36920.84660.25860.44532.7595
3.52000.61160.604822.78250.27560.660211.3142
3.512200.61640.541421.29680.26640.49937.3950
FlowCPO111000.60600.223421.43910.25050.49903.2108
3.511000.76710.606322.75530.27840.658711.2727
[1] Direct preference optimization: Your language model is secretly a reward model. NIPS 2023
[2] Reinforcement Learning for Out-of-Distribution Reasoning in LLMs: An Empirical Study on Diagnosis-Related Group Coding, NIPS 2025 Spotlight
[3] Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning, ACL 2026
[4] Flow-Factory: A Unified Framework for Reinforcement Learning in Flow-Matching Models, 2026

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank the reviewers and Area Chair for their constructive discussion and careful overseeing.

1. The ratings currently shown on OpenReview are as follows.

ReviewerInitial RatingPost-Rebuttal RatingReviewer Comments
ukMM3 (Weak Accept)Maintained 3 (Weak Accept)Maintained the initial weak-accept rating.
yA282 (Weak Reject)Maintained 2 (Weak Reject)Recognized that work’s mathematical rigor and its excellent guidance for off-policy flow-based RL, , but retained concerns about SOTA performance of FlowCPO, particularly in the OOD and FLUX.1-13B experiments, for which we have provided theoretical explanation.
wRkK2 (Weak Reject)Raised to 3 (Weak Accept)Appreciated the unified divergence-based framework, the explanation of the dispreferred branch, and the AWM–RFT connection, and stated that these clarifications addressed their main concerns.

2. We summarize our contributions and responses to the main concerns below.

1. Main contributions

  • Theoretically, we propose a unified distributional view complementary to stochastic optimal control. It organizes flow-based alignment methods by sampling regimes, divergence directions and optimized processes.
  • Methodologically, we formulate FlowCPO as an offline forward-KL counterpart to DiffusionNFT, further analyze its relationship to simplified FlowDPO.
  • Empirically, we evaluate the effects of different types of negative regularization and their performance over in-domain and out-of-domain data setting.

2. Responses to reviewer concerns

(a) Novelty and relationship to DiffusionNFT (yA28, wRkK).

  • We acknowledge that FlowCPO is an offline formulation of DiffusionNFT and have claimed it with Tab. 1 since the introduction.
  • However, our main contributions are:
    • the unified divergence-based framework,
    • the theoretical analysis between FlowCPO and FlowDPO (App. B.5),
    • the empirical analysis of different types of negative regularization (Eq. 12).

(b) When dispreferred samples help and why OOD performance is mixed (ukMM, yA28, wRkK).

  • Theoretically, we add an analysis in App B.6 to show that in-domain dispreferred samples can reinforce preferred updates, while OOD dispreferred samples can weaken this effect.
  • Empirically, we provide supporting evidence from prior researches: stronger supervised initialization can improve subsequent RL performance on OOD tasks, while the original DPO work recommends adapting the reference model to preferred data to mitigate distribution shift. These researches provide complementary support for our empirical results.

(c) Empirical benefits and generalization beyond SD3.5-M (yA28, ukMM).

  • Theoretically, App B.5 establishes that FlowCPO’s loss is nonnegative, removing the unbounded-below behavior possible in simplified FlowDPO’s signed objective. This motivates improved optimization behavior.
  • Empirically,
    • on SD3.5-M (2.5B) at CFG 3.0, FlowCPO achieves mean GenEval and OCR scores of 0.84 and 0.87, compared with 0.81 and 0.74 for FlowDPO under the same offline protocol.
    • On FLUX.1 (13B) at CFG 3.5, FlowCPO achieves a GenEval score of 0.7671, compared with 0.6318 for the pretrained model, supporting its feasibility on a larger, second backbone.

(d) Coverage of missing offline baselines (wRkK).

  • Theoretically, under the connection in App. C.3 and App. C.4, both AWM and RAM can reduce to RFT.
  • Empirically, we have covered all the baselines which can be divided into three types using Eq. 12:
    • zero negative regularization term: RFT,
    • minus negative regularization term: FlowDPO,
    • plus negative regularization term: FlowCPO.

(e) Advantages of offline methods over online methods (wRkK).

  • Training efficiency: Offline methods avoid online sampling during optimization, which can substantially reduce training time, when generating new samples is computationally expensive.
  • Control over data quality: Offline methods allow preference data to be inspected and filtered before optimization, providing direct control over the training dataset and its reward signals.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.