- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose FlowCPO, an offline preference alignment method for flow matching models. The key idea is to organize existing alignment methods along two axes, the sampling regime and the direction of the idealized KL objective, and then to fill the one combination that no prior method occupies: a forward-KL objective optimized offline using both preferred and dispreferred samples. The authors turn that objective into two regression terms, one fitting the preferred samples and one pushing the model away from the dispreferred samples, so that training reduces to flow-matching regression on fixed preference pairs. The experiments fine-tune Stable Diffusion 3.5 Medium with LoRA on in-domain and out-of-domain offline preference data, comparing against two offline baselines, RFT and FlowDPO, and evaluating on GenEval, an OCR metric, and several preference metrics.
Critical strengths
- The paper is clearly written, the results are easy to audit, and the training and data-construction protocol is documented in enough detail to be reimplemented.
- Hyperparameter coverage is thorough, and every fine-tuned entry is a mean plus or minus a standard deviation over five seeds (Tabs. 5-8), which very few papers in this area report.
Critical weaknesses
- The method is essentially an offline version of DiffusionNFT, and DiffusionNFT's own paper already ran similar experiments. At the objective of Eq. 11 is DiffusionNFT's loss with the soft optimality weight replaced by a hard label on a two-sample group, and the two share the same mixed velocity fields, the same regression form, the same EMA reference, and even the same symbols and . DiffusionNFT is off-policy by design, so freezing its sampling policy already reaches the offline end of the same spectrum. The original paper studies that limit and reports that it is stable but converges impractically slowly. What "offline" adds here is only that the data are produced ahead of time, which shifts the cost rather than removing it. Constructing the in-domain set still requires 16 samples for roughly 95K prompts from the frozen reference, so the total compute is not obviously lower, and the data pipeline becomes a component that has to be built and maintained in its own right.
- Key baselines are missing. Only RFT and FlowDPO are compared. DiffusionNFT, AWM and RAM are online methods, but moving them into this setting is trivial. Once the fixed preference pairs are assigned 0/1 rewards, all three can be trained directly on the buffer that has already been generated, without any new rollout. In fact the row of Tab. 4 already is DiffusionNFT under hard labels, which the paper does not point out. Without that comparison on the same data, there is no evidence that the changes made here improve on the method the objective is taken from.
Questions for authors
- What exactly differs from an offline DiffusionNFT? At the objective in Eq. 11 is DiffusionNFT's loss with the optimality weight reduced to a hard label on a two-sample group, while the mixed velocity fields, the EMA reference and the hyperparameters and are all the same. DiffusionNFT's Figure 8 already sweeps the EMA coefficient and reports that holding the sampling policy nearly fixed is stable but converges impractically slowly. Please state precisely which component of Eq. 11 is not present in DiffusionNFT.
- What is the advantage of offline over online? DiffusionNFT reports a GenEval score of 0.98 on SD3.5-Medium within 1k steps and without CFG, against 0.84 here at 2k steps with CFG 3.0, and it is ahead on the other metrics as well. If moving the same objective offline costs that much accuracy while still requiring 16 samples for roughly 95K prompts up front, what is the case for it?
- Baselines on the same offline data. Please add DiffusionNFT, AWM and RAM trained on the buffer you have already generated, in both the in-domain and the OOD setting. Once the fixed pairs carry rewards, all three run there with no new rollout, and the DiffusionNFT row should use graded weights over all 16 candidates, whose scores the pipeline has already computed. Without that row there is no evidence that the changes made here improve on the objective they are taken from.
- What does the dispreferred branch contribute? Out of domain at CFG 4.5, RFT beats FlowCPO on all eight metrics and FlowCPO falls below the untuned base model on three of them, yet RFT is simply Eq. 11 with . Outside the metric used to filter the training data, what evidence remains that the branch helps? App. D.1 states that the OOD images come from FLUX.1-Dev and SD3.5-Large, and Tab. 3 scores both above the SD3.5-M reference, so one possibility is that pushing away from those losers also pushes the model away from content better than its own.
5 comments
Q4:Why in-domain dispreferred samples help while out-of-domain dispreferred samples don’t?
Thank you for raising this important question.
We will first theoretically explain why in-domain dispreferred samples help while OOD dispreferred samples don’t help, and then provide empirical evidences from prior studies.
- Theoretical explanation: the conclusion here is that the in-domain dispreferred samples can strengthen the update signal, while OOD dispreferred samples may diminish this signal. (We provide detailed analysis in App. B.6.)
-
Flow Matching:
Here we need to emphasize that the and are from , i.e. the samples are in-domain.
Let , then for optimal solution , we have .
-
RFT:
Then for the first update from , we have .
This makes the model to align with the preferred samples.
-
Simplied FlowDPO:
.
Then for the first update from and in-domain samples, we have .
- If the and are from , i.e. the samples are in-domain, then and thus , which means in-domain dispreferred samples can strengthen the update signal.
- If the and are not from , i.e. the samples are OOD, then may diminish the signal of .
-
FlowCPO:
,
where and .
Then for the first update from and in-domain samples, we have .
- If the and are from , i.e. the samples are in-domain, then and thus , which means in-domain dispreferred samples can strengthen the update signal.
- If the and are not from , i.e. the samples are OOD, then may diminish the signal of .
-
- Empirical Evidence from prior studies:
- In Figure 5 of [2], pure SFT and SFT-then-RL with stronger SFT initialization achieve higher Pass@1 than the 5% SFT + 95% RL configuration. This suggests that adequate supervised adaptation can be important for effective RL on OOD tasks.
- In Section 6.2.1 and Figure 4 of [3], stronger SFT initialization also improves the final cross-task (OOD) evaluation performance after RL. This further supports the importance of the starting model.
- In the last paragraph (“DPO outline”) of Section 4 in [1], the authors recommend fitting the reference model to preferred completions when the original model is unavailable, to mitigate distribution shift before applying DPO.
[1] Direct preference optimization: Your language model is secretly a reward model. NIPS 2023
[2] Reinforcement Learning for Out-of-Distribution Reasoning in LLMs: An Empirical Study on Diagnosis-Related Group Coding, NIPS 2025 Spotlight
[3] Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning, ACL 2026
Follow-up on our response
Dear Reviewer,
Thank you again for reviewing our paper. We would like to follow up on our response to your review and would appreciate your feedback on whether any of your concerns remain unresolved.
As the discussion period is nearing its end, we would be happy to provide any further clarification that would be helpful for your assessment.
Thank you for your time and consideration.
Best regards,
The Authors
We sincerely thank the reviewer for the careful review and rigorous questions.
Thank you for raising this question.
Our claim is that the offline version of AWM and RAM is actually RFT, which is already included in current experiments.
In the following, we will provide the theoretical proof under our proposed divergence-based framework.
AWM: In App. C.3, we have proved that
LAWM=Eπθ[w(t)r(x0,c)∥vθ−ut∥22]
When we hardcode the reward and set reward as 1 for preferred samples, 0 for dispreferred samples, then we have:
Loffline−AWM=Eπθ+[w(t)⋅1⋅∥vθ−ut∥22]+Eπθ−[w(t)⋅0⋅∥vθ−ut∥22]=Eπθ+[w(t)∥vθ−ut∥22] which is actually RFT.
RAM: In App. C.4, we have proved the equivalent loss of RAM is
L2=Eπθ[r(x0,c)∥vθ−ut∥22]+Eπθ[∥vθ−vref∥22]
When we hardcode the reward and set reward as 1 for preferred samples, 0 for dispreferred samples, then we have:
Loffline−2=Eπθ+[∥vθ−ut∥22]+Eπθ[∥vθ−vref∥22]
which is actually RFT plus a regularization term.
Finally, we emphasize that we have cover all the baselines which can be divided into three types using eq. 12:
Thank you for pointing out this issue.
We admit that FlowCPO is an offline version of DiffusionNFT and have claimed it with Tab.1 since the introduction. DiffusionNFT has provided a few experimental results to discuss the offline DiffusionNFT with soft reward label rather than hard 0/1 label.
The differences:
We believe that both the theoretical and empirical results can help community better understand the differences among three types of negative regularization (RFT, FlowCPO and FlowDPO), which are not included in DiffusionNFT paper.
Thank you for pointing this out.
Studying offline methods is valuable from both practical and methodological perspectives:
Offline methods also offer two practical advantages: