acceptodds
Under review as a conference paper at ICLR 2027

Towards General Preference Alignment: Nash Preference Optimization for Diffusion Models

Abstract

Reinforcement learning from human feedback (RLHF) has been popular for aligning text-to-image (T2I) diffusion models with human preferences. Direct Preference Optimization (DPO) offers a computationally efficient approach that avoids explicit reward modeling and has been widely adopted in diffusion alignment. However, existing preference-based methods for diffusion alignment still rely on reward-induced preference signals and typically assume that human preferences can be adequately modeled by the Bradley–Terry–Luce (BTL) model, which may fail to capture the full complexity of human preferences. In this paper, we formulate diffusion alignment from a game-theoretic perspective. We propose Diffusion Nash Preference Optimization (Diff.-NPO), an intuitive general preference framework for diffusion alignment. Diff.-NPO encourages the current policy to play against itself to achieve self-improvement and lead to a better alignment. Empirically, we demonstrate the effectiveness of Diff.-NPO on the text-to-image generation task via various metrics. Diff.-NPO consistently outperforms existing preference-based diffusion alignment methods

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
diffusion model,image generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gPGxOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Diff.-NPO, a diffusion model preference alignment method based on Nash Learning. In response to the problem that the traditional DPO method relies on the BTL model and it is difficult to express non-transmitive preferences, the author extends INPO to the diffusion model, optimize the generation strategy through online self-play. The method adopts the strategy update based on Mirror Descent, introduces the previous round of strategy and fixed reference strategy for regularization, and designs Logistic Loss based on denoising error. Experiments show that Diff.-NPO has achieved good preference alignment results on SD1.5 and SDXL.

Critical strengths
  1. This paper focuses on the problem that the BTL model is difficult to deal with intransitive preferences, and tries to achieve more general diffusion model preference alignment through Nash Learning.

  2. The author extends the Mirror Descent update of INPO to the diffusion model and derives the tractable training objective based on the denoising error.

  3. The paper conducted multiple groups of comparative experiments on SD1.5 and SDXL, and provided ablation analysis of regularization coefficients and loss functions.

Critical weaknesses
  1. The training data is constructed through the average ranking of the automatic scoring model, which is still transisive and may not be able to verify the advantage of the method in handling non-transmitative preferences.

  2. Although the author provides a dissipation study on the regularization ratio and loss function, it is still unclear whether the performance improvement comes from Nash's learning itself or the dual strategy regularization and logical goals proposed by Nash.

  3. The introduction is relatively long and can be shortened to better highlight the main contribution. In Section 3, the connection between the theoretical Nash update and the actual logistics goal can be explained more clearly.

Questions for authors
  1. What are the differences between Diff-NPO and other methods in training time, calculation cost and preferred data usage? Can you supplement the comparison under the same calculation budget?

Actually, I am not an expert in the research field of this paper. I initialized my rating based on my understanding of this paper, and I will adjust my rating according to the author's response and the opinions of other reviewers.

5 comments

We thank the reviewer for the thoughtful feedback and for recognizing our motivation, the extension of Nash learning to diffusion models, and the experimental evaluation. Below, we clarify the preference construction and the roles of the objective’s components, and provide additional comparisons of computational cost and data usage. We also appreciate the suggestions on presentation and will revise the introduction and Section 3 to make the method easier to follow.

[W1] Non-transitive preferences

We would like to clarify how our evaluator ensemble defines a pairwise preference function. For a prompt xx and an image yy, let Rm(x,y)R_m(x,y) denote the scalar score assigned by evaluator mm, with M=5M=5 evaluators in our experiments. The pairwise preference function associated with this ensemble is Pens(y≻y′∣x):=1M∑m=1M[1{Rm(x,y)>Rm(x,y′)}+121{Rm(x,y)=Rm(x,y′)}].\mathbb P_{\mathrm{ens}}(y\succ y'\mid x):=\frac{1}{M}\sum_{m=1}^{M}\left[\mathbf 1\{R_m(x,y)>R_m(x,y')\}+\frac12\mathbf 1\{R_m(x,y)=R_m(x,y')\}\right]. This is the fraction of evaluators preferring yy to y′y', with ties receiving weight 1/21/2. Although each evaluator assigns a scalar score, averaging their pairwise comparisons need not yield a BTL representation Pens(y≻y′∣x)=σ(r(x,y)−r(x,y′))\mathbb P_{\mathrm{ens}}(y\succ y'\mid x)=\sigma(r(x,y)-r(x,y')) for a single scalar reward function rr, where σ(t)=(1+e−t)−1\sigma(t)=(1+e^{-t})^{-1}. For example, consider three images A,B,CA,B,C for a fixed prompt xx and five evaluators, with two ranking A≻B≻CA\succ B\succ C, two ranking B≻C≻AB\succ C\succ A, and one ranking C≻A≻BC\succ A\succ B. Their ensemble preferences satisfy Pens(A≻B∣x)=3/5\mathbb P_{\mathrm{ens}}(A\succ B\mid x)=3/5, Pens(B≻C∣x)=4/5\mathbb P_{\mathrm{ens}}(B\succ C\mid x)=4/5, and Pens(C≻A∣x)=3/5\mathbb P_{\mathrm{ens}}(C\succ A\mid x)=3/5. This illustrates that the ensemble preference function can be non-transitive and establishes that our approach accommodates cyclic preferences, although demonstrating their presence and their effect on performance in our experiments requires direct empirical evidence.

Nevertheless, following your suggestion, we have conducted an additional set of experiments to address this concern more directly. We have now added a random-judge experiment, together with explicit verification of non-transitivity.

Preference construction. At each training step, we uniformly select one of five evaluators—PickScore, HPSv2, CLIP, ImageReward, and Aesthetic Score—to provide preference labels. For a fixed image pair, the underlying probability that A is preferred to B is therefore the fraction of these five evaluators preferring A. Although each evaluator provides a scalar ranking, their aggregated pairwise preferences can be non-transitive. Both Online-DPO and Diff.-NPO use this random-judge procedure. HPSv3 is used for evaluation only and is not one of the training judges.

Verification of non-transitivity. For each prompt, we examine its five generated images and check whether there exist A, B, and C such that P(A ≻ B), P(B ≻ C), and P(C ≻ A) are all greater than 1/2, following the definition in our introduction.

Image sourcePairs with judge disagreement (%)Prompts with non-transitive preferences (%)
Original Sana83.022.2
Online-DPO, random judge84.122.6
Diff.-NPO, random judge83.720.8

Non-transitive preferences are observed in 20.8–22.6% of the evaluated prompts, using five candidates per prompt. This verifies that the random-judge oracle exhibits non-transitivity on generated images.

Alignment performance under this oracle. The following table reports mean raw scores and win rates against the original Sana-0.6B. Both trained methods use random-judge feedback.

MethodMetric typePickScoreHPSv2CLIPImageRewardAestheticsHPSv3
Original SanaRaw score22.1828.4735.521.09746.02937.993
Online-DPORaw score22.2128.5335.621.11626.04198.042
Online-DPOWin rate (%)60.362.855.159.257.555.2
Diff.-NPORaw score22.2228.5735.631.12406.04628.068
Diff.-NPOWin rate (%)63.467.653.662.360.556.8

Diff.-NPO achieves higher mean raw scores on all six evaluators and higher win rates against the original backbone on five of six. Paired tests on the same prompts and seeds give p < 0.05 on PickScore, HPSv2, ImageReward, and HPSv3; the differences on CLIP and Aesthetics are not statistically significant. We appreciate the suggestion and will include the setup, diagnostics, and results in the revised manuscript.

[W2] Relationship between Nash learning, regularization, and the logistic objective.

Thank you for raising this point. The original Nash-derived squared-loss formulation already includes regularization toward both the reference and previous policies. We retain these terms unchanged when replacing the squared loss with our logistic objective. The loss-function ablation therefore compares the two objectives under the same dual-policy regularization and shows that the logistic objective yields better alignment performance. We will clarify this relationship in the revision.

[W3] Introduction length and the connection between the theoretical update and practical objective.

Thank you for this helpful suggestion. We agree and will shorten the introduction to highlight the main contributions more directly. We will also revise Section 3 to make the progression from the theoretical mirror-descent update, through the choice of logistic surrogate, to the tractable denoising objective more explicit, clearly identifying the practical design choices and approximation steps.

[Q1] Training time, computational cost, and preference-data usage.

Thank you for requesting this comparison. We provide runtime and data-usage measurements, together with performance comparisons at multiple training stages under comparable computational costs.

Training cost and preference-data usage. All methods use Sana-0.6B with 2,048 preference pairs per optimizer step. Runtimes are normalized to 4 × A100 GPUs.

MethodTime per stepRuntime relative to Offline DPOPreference-data usage per step
Offline DPO66 s1×2,048 human-labelled pairs from Pick-a-Pic v1
Online-DPO732 s11.1×2,048 prompts × 8 generated candidates, ranked by five reward models
Diff.-NPO738 s11.2×Same as Online-DPO

Diff.-NPO adds only 0.8% runtime overhead over Online-DPO, corresponding to an additional forward pass through the previous policy when computing the loss. Both online methods require approximately 11× more runtime per step than Offline DPO because of candidate generation and scoring. In return, they refresh preference data from the evolving policy and require no human-labelled preference pairs during alignment.

Performance under comparable computational budgets. Given their nearly identical per-step runtime and identical preference-data construction, comparing Diff.-NPO and Online-DPO at the same optimizer steps provides a comparison under closely matched computational budgets. The following table reports win rates (%) against the original Sana backbone, with Offline DPO included for reference.

StepMethodPickScoreHPSv2CLIPAestheticsImageRewardHPSv3
270Offline DPO59.056.650.851.053.054.8
90Online-DPO61.565.953.359.961.856.0
90Diff.-NPO62.468.853.862.663.355.9
140Online-DPO63.266.053.660.660.657.3
140Diff.-NPO66.069.755.562.964.357.8
170Online-DPO63.467.454.560.763.759.6
170Diff.-NPO65.771.056.062.465.360.2
200Online-DPO62.867.254.859.661.955.2
200Diff.-NPO66.271.353.962.464.859.8

Direct paired comparisons. We report the raw scores of both methods at the same training steps. Parentheses in the Diff.-NPO rows show p-values from paired sign-flip tests against Online-DPO at the corresponding step.

StepMethodPickScoreHPSv2CLIPAestheticsImageRewardHPSv3
90Online-DPO22.2028.5335.616.04231.11818.0409
90Diff.-NPO22.22 (<0.001)28.57 (<0.001)35.63 (0.148)6.0499 (<0.001)1.1277 (<0.001)8.0598 (0.027)
140Online-DPO22.2228.5535.636.04521.12258.0662
140Diff.-NPO22.24 (<0.001)28.60 (<0.001)35.67 (0.018)6.0573 (<0.001)1.1386 (<0.001)8.0995 (<0.001)
170Online-DPO22.2228.5735.616.04651.12958.0891
170Diff.-NPO22.24 (<0.001)28.62 (<0.001)35.70 (<0.001)6.0566 (<0.001)1.1408 (<0.001)8.1179 (0.006)
200Online-DPO22.2328.5735.646.04881.12508.0614
200Diff.-NPO22.25 (<0.001)28.62 (<0.001)35.67 (0.234)6.0607 (<0.001)1.1397 (<0.001)8.1241 (<0.001)

Diff.-NPO achieves higher mean raw scores on all six evaluators at every evaluated step. The paired differences reach nominal p < 0.05 in 22 of 24 comparisons.

Reviewer_QWepOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an extension of Nash-based preference optimization for RLHF of text-to-image models. The authors extend the work demonstrated in NLP (INPO) and formulate the objective such that it is tractable for diffusion model post-training. The method opposes traditional Bradley-Terry type optimization by optimizing the win-rate of the current policy compared to the past policy (with KL penalty wrt to reference model). Thus, it requires a reward model/oracle. The results seems promising, but the evaluations are very outdated (for instance HPSv3 existed as of last August but authors use v2 despite mostly training their own baselines). Overall, the method seems interesting but there is limited novelty and the outdated metrics used in evaluation raise concern.

Critical strengths
  • Well written paper with clear derivation
  • Interesting extension of Nash-based RLHF to T2I domain, which to my knowledge is new (although the paper extends INPO, an existing work for LLMs).
Critical weaknesses
  • Outdated evaluations. Authors benchmark with SDXL and SD-1.5 which is over 3 years old (and there are other T2I models available). Additionally, metrics such as HPSv2 seem questionable as v3 has existed for over a year. Also, I am concerned that the only metric reported is win-rate and not the raw reward score. The method is essentially optimizing for win-rate, and only showing win-rate can be misleading as an improved reward by a trivial margin (say 1e-20) is considered a win though most humans users would consider that a 'tie'.
  • Limited comparison with offline methods.The proposed method requires a reward model and in the image domain it seems that this is less necessary given image models are quite strong after pre-training and DPO-like methods allow us to directly optimize for RLHF on the data. The Appendix has a small comparison with Diff-DPO, but what about methods like Diff-KTO (that work around some of the transitivity issues in BT modelling assumptions)? Thus, a quantitative comparison with offline methods and the pros/cons (e.g. FLOPs versus performance) would be useful.
Questions for authors
  • What are the raw reward scores for Diff-NPO and the baselines?
  • Do you have up-to-date numbers using say HPSv3? Newer T2I models?
  • Could you provide a quantitative-backed analysis of using Diff-NPO versus an offline method? Is the additional training FLOPs worth it, if this pairwise data is available (and somewhat abundant if you look at the likes of HF datasets)?
4 comments

We thank the reviewer for the constructive feedback and for recognizing the clarity of our derivation and the value of extending Nash-based alignment to text-to-image generation. In response to the concerns, we provide additional Sana and HPSv3 evaluations, a Diffusion-KTO comparison, and a quantitative analysis with offline methods. We also clarify the raw scores and offline comparisons already reported in the submission.

[W1/Q1/Q2] Raw metric scores, HPSv3 and New T2I Flow-matching Model

Tables 1 and 2 in the submitted paper already report raw scores alongside win rates for PickScore, HPSv2, CLIP, ImageReward, and Aesthetics on SD1.5 and SDXL. We apologize that this was not sufficiently clear in the presentation.

We agree that the win rate alone does not convey the magnitude of an improvement. The additional evaluations below therefore also include raw scores, including HPSv3.

We have added experiments on Sana-0.6B, a flow-matching text-to-image model. Both online methods use average-rank preference construction.

The table reports mean raw scores and win rates against the original Sana backbone.

MethodMetric typePickScoreHPSv2CLIPAestheticsImageRewardHPSv3
Original SanaRaw score22.1828.4735.526.02931.09747.9932
Offline DPORaw score22.2028.4935.536.03211.10048.0567
Offline DPOWin rate (%)59.056.650.851.053.054.8
Online-DPORaw score22.2328.5735.646.04881.12508.0614
Online-DPOWin rate (%)62.867.254.859.661.955.2
Diff.-NPORaw score22.2528.6235.676.06071.13978.1241
Diff.-NPOWin rate (%)66.271.353.962.464.859.8

Diff.-NPO achieves higher mean raw scores than both DPO baselines across all six evaluators. These results extend the evidence to a flow-matching backbone and HPSv3.

HPSv3 evaluation on SD1.5. We additionally report HPSv3 win rates against the original SD1.5 backbone and mean raw scores for the methods below.

MethodHPSv3 win rate vs. SD1.5 (%)HPSv3 raw score
Original SD1.5–5.706
Online-SFT50.45.721
Online-DPO74.47.347
SPIN79.17.843
SePPO82.87.863
Diff.-NPO75.37.481

[W2/Q3] Diffusion-KTO comparison, and a quantitative analysis with offline methods

Thank you for this helpful suggestion. We would like to clarify two benefits that motivate our use of online alignment.

Further improvement after pretraining. Our experiments show that online alignment can provide larger gains than offline training. We observe a similar trend on Sana, suggesting that strong pretrained models can still benefit from online preference feedback.

Less dependence on a fixed preference dataset. Given prompts and an available preference oracle, online alignment can begin without first collecting human-labelled image pairs. It also refreshes comparisons as the model improves, allowing training to follow the evolving policy and potentially accommodate new prompts or preference criteria.

Additional Diffusion-KTO baselines. We have expanded the comparison to include the Diffusion-KTO. For a fair comparison, we re-evaluated both Diffusion-KTO and Diff.-NPO using the same evaluation protocol as Table 1 of the paper on Pick-a-Pic. The following tables report win rates (%) against the original SD1.5 backbone and raw scores.

MethodPickScoreHPSv2CLIPAestheticsImageRewardHPSv3
Diffusion-KTO73.181.357.068.876.675.5
Diff.-NPO82.585.259.481.085.475.3

Diff.-NPO outperforms Diffusion-KTO on five of six evaluators

MethodPickScoreHPSv2CLIPAestheticsImageRewardHPSv3
Original SD1.520.6826.8033.405.49460.04705.706
Diffusion-KTO21.1927.8234.275.70600.61457.603
Diff.-NPO21.4728.0534.615.85710.88227.481

Diff.-NPO achieves higher mean raw scores on five of six evaluators We also appreciate the reviewer’s point about computational cost. and we provide the quantitative cost–performance comparison below to make this trade-off explicit.

Offline comparison and the computation–performance trade-off.

Computational cost. All Sana methods use 2,048 preference pairs per optimizer step. The following table reports measured runtime, normalized to 4 × A100 GPUs, and denoiser FLOPs.

MethodTime per stepEstimated denoiser FLOPs per pairDenoiser FLOPs relative to Offline DPO
Offline DPO66 s8F1×
Online-DPO732 s168F21×
Diff.-NPO738 s170F21.25×

Here, F denotes one denoiser forward evaluation, and a backward evaluation is approximated as 2F. Both online methods require 160 forward evaluations per pair for candidate generation. Diff.-NPO additionally evaluates the previous policy on the winning and losing images. Relative to Online-DPO, Diff.-NPO adds only 1.2% estimated denoiser FLOPs and 0.8% measured runtime.

Performance gains. Appendix E.3, Table 5 already provides a quantitative comparison with Offline SFT and Offline DPO on SD1.5. The following table reports win rates (%) against the original Sana backbone and raw scores.

Win rates against the original Sana backbone:

MethodPickScoreHPSv2CLIPAestheticsImageRewardHPSv3
Offline DPO59.056.650.851.053.054.8
Online-DPO62.867.254.859.661.955.2
Diff.-NPO66.271.353.962.464.859.8

Raw metric scores.

MethodPickScoreHPSv2CLIPAestheticsImageRewardHPSv3
Original Sana22.1828.4735.526.02931.09747.9932
Offline DPO22.2028.4935.536.03211.10048.0567
Online-DPO22.2328.5735.646.04881.12508.0614
Diff.-NPO22.2528.6235.676.06071.13978.1241

Diff.-NPO achieves higher win rates than Offline DPO on all six evaluators and Online-DPO on five of six. The mean raw scores reported above are also higher than both baselines across all six evaluators.

Together with the computational measurements, these results illustrate the trade-off: Offline DPO is substantially cheaper when labelled pairs are available, while Diff.-NPO obtains additional alignment gains at higher computational cost. Within the online setting, Diff.-NPO improves over Online-DPO with minimal additional computation.

AuthorsOpenReview ↗

Additional experiments on SD3.5 Medium

Additional experiments on SD3.5 Medium.

Thank you again for your helpful feedback. To further address your concerns about the backbone and reward metrics, we have added experiments on SD3.5 Medium. We compare Diff.-NPO with Online-DPO and report both raw scores and win rates, including HPSv3.

Mean raw scores. Parentheses in the Diff.-NPO row report p-values for raw-score comparisons with Online-DPO.

MethodPickScoreHPSv2CLIPAestheticsImageRewardHPSv3
Original SD3.5 Medium21.9428.0435.505.8930.8398.139
Online-DPO22.2028.6035.826.0081.1038.579
Diff.-NPO22.29 (7.3e-14)28.69 (3.1e-08)35.88 (0.285)6.033 (3.4e-04)1.121 (0.057)8.868 (2.1e-19)

Win rates against the original SD3.5 Medium backbone (%).

MethodPickScoreHPSv2CLIPAestheticsImageRewardHPSv3
Online-DPO66.376.152.764.273.458.8
Diff.-NPO72.879.853.868.874.666.0

We will include these additional results in the revised manuscript.

Dear Reviewer, Thank you again for carefully reading our paper and providing feedback. As the deadline is approaching, it would be very helpful if you could kindly let us know whether you have any other concerns. We will try to address them as soon as possible.

Reviewer_Qhj6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Diffusion Nash Preference Optimization (Diff.-NPO), a preference alignment framework for text-to-image diffusion models. Motivated by the limitations of the Bradley–Terry–Luce model in representing non-transitive preferences, the authors extend Nash learning to diffusion models through an online mirror-descent formulation. Experiments on SD1.5 and SDXL across multiple benchmarks demonstrate improvements over existing preference alignment methods, including Diffusion-DPO.

Critical strengths
  1. This paper effectively extends Nash preference optimization to diffusion models. This paper explores the problem of general preference alignment—which goes beyond the traditional Bradley-Terry-Luce (BTL) assumption—representing a significant direction in the field of alignment for generative models.

  2. The proposed objective combines regularization toward a fixed reference policy and the previous policy, providing a mechanism for balancing policy stability and iterative improvement.

  3. The proposed method achieves higer average win rates than the evaluated baselines on both SD1.5 and SDXL across Pick-a-Pic, Parti-Prompts, and HPSv2.

Critical weaknesses
  1. There is a lack of sufficient verification and analysis regarding the core motivation of "general preferences" and "non-transitive preferences". One of the primary motivations of this paper is to address the limitations of preference optimization based on the BTL model when it comes to modeling non-transitive preferences. However, the preference pairs used for training are constructed based on the average rankings derived from five scalar scoring models—a process that inherently assumes transitive ordering. Consequently, the experiments fail to directly demonstrate the advantages of the proposed method in scenarios involving truly non-transitive preferences. This significantly weakens the connection between the paper’s core motivation and its empirical contributions.

  2. The paper provides limited baseline comparisons, and the experimental analysis is insufficient. Although the paper compares its approach against several preference optimization methods, the SDXL experiments include only Online-SFT and Online-DPO as fine-tuning baselines. Furthermore, experimental results regarding flow-matching methods are missing. The paper provides only win-rate results, failing to report preference metrics (such as PS, HPS, AES, IM, IR, and CLIP) for these methods.

Questions for authors
  1. Given that training preferences are constructed based on average scalar rankings, how exactly does the proposed method demonstrate an advantage when handling non-transitive preferences? Could the authors provide some experiments that involve cyclic preference structures?

  2. Further theoretical analysis of Best-Worst sample selection.

8 comments

We thank the reviewer for the constructive feedback and for recognizing the importance of general preference alignment and our consistent empirical improvements. Below, we address the concerns about non-transitive preferences and Best–Worst sample selection, provide additional flow-matching and baseline evaluations, and clarify the raw metric scores reported in the submission.

[W1/Q1] Non-transitive preferences

We would like to clarify how our evaluator ensemble defines a pairwise preference function. For a prompt xx and an image yy, let Rm(x,y)R_m(x,y) denote the scalar score assigned by evaluator mm, with M=5M=5 evaluators in our experiments. The pairwise preference function associated with this ensemble is Pens(y≻y′∣x):=1M∑m=1M[1{Rm(x,y)>Rm(x,y′)}+121{Rm(x,y)=Rm(x,y′)}].\mathbb P_{\mathrm{ens}}(y\succ y'\mid x):=\frac{1}{M}\sum_{m=1}^{M}\left[\mathbf 1\{R_m(x,y)>R_m(x,y')\}+\frac12\mathbf 1\{R_m(x,y)=R_m(x,y')\}\right]. This is the fraction of evaluators preferring yy to y′y', with ties receiving weight 1/21/2. Although each evaluator assigns a scalar score, averaging their pairwise comparisons need not yield a BTL representation Pens(y≻y′∣x)=σ(r(x,y)−r(x,y′))\mathbb P_{\mathrm{ens}}(y\succ y'\mid x)=\sigma(r(x,y)-r(x,y')) for a single scalar reward function rr, where σ(t)=(1+e−t)−1\sigma(t)=(1+e^{-t})^{-1}. For example, consider three images A,B,CA,B,C for a fixed prompt xx and five evaluators, with two ranking A≻B≻CA\succ B\succ C, two ranking B≻C≻AB\succ C\succ A, and one ranking C≻A≻BC\succ A\succ B. Their ensemble preferences satisfy Pens(A≻B∣x)=3/5\mathbb P_{\mathrm{ens}}(A\succ B\mid x)=3/5, Pens(B≻C∣x)=4/5\mathbb P_{\mathrm{ens}}(B\succ C\mid x)=4/5, and Pens(C≻A∣x)=3/5\mathbb P_{\mathrm{ens}}(C\succ A\mid x)=3/5. This illustrates that the ensemble preference function can be non-transitive and establishes that our approach accommodates cyclic preferences, although demonstrating their presence and their effect on performance in our experiments requires direct empirical evidence.

Nevertheless, following your suggestion, we have conducted an additional set of experiments to address this concern more directly. We have now added a random-judge experiment, together with explicit verification of non-transitivity.

Preference construction. At each training step, we uniformly select one of five evaluators—PickScore, HPSv2, CLIP, ImageReward, and Aesthetic Score—to provide preference labels. For a fixed image pair, the underlying probability that A is preferred to B is therefore the fraction of these five evaluators preferring A. Although each evaluator provides a scalar ranking, their aggregated pairwise preferences can be non-transitive. Both Online-DPO and Diff.-NPO use this random-judge procedure. HPSv3 is used for evaluation only and is not one of the training judges.

Verification of non-transitivity. For each prompt, we examine its five generated images and check whether there exist A, B, and C such that P(A ≻ B), P(B ≻ C), and P(C ≻ A) are all greater than 1/2, following the definition in our introduction.

Image sourcePairs with judge disagreement (%)Prompts with non-transitive preferences (%)
Original Sana83.022.2
Online-DPO, random judge84.122.6
Diff.-NPO, random judge83.720.8

Non-transitive preferences are observed in 20.8–22.6% of the evaluated prompts, using five candidates per prompt. This verifies that the random-judge oracle exhibits non-transitivity on generated images.

Alignment performance under this oracle. The following table reports mean raw scores and win rates against the original Sana-0.6B. Both trained methods use random-judge feedback.

MethodMetric typePickScoreHPSv2CLIPImageRewardAestheticsHPSv3
Original SanaRaw score22.1828.4735.521.09746.02937.993
Online-DPORaw score22.2128.5335.621.11626.04198.042
Online-DPOWin rate (%)60.362.855.159.257.555.2
Diff.-NPORaw score22.2228.5735.631.12406.04628.068
Diff.-NPOWin rate (%)63.467.653.662.360.556.8

Diff.-NPO achieves higher mean raw scores on all six evaluators and higher win rates against the original backbone on five of six. Paired tests on the same prompts and seeds give p < 0.05 on PickScore, HPSv2, ImageReward, and HPSv3; the differences on CLIP and Aesthetics are not statistically significant. We appreciate the suggestion and will include the setup, diagnostics, and results in the revised manuscript.

AuthorsOpenReview ↗

Baseline coverage, flow-matching evaluation, and raw metric scores

[W2] Baseline coverage, flow-matching evaluation, and raw metric scores

Thank you for suggesting broader experimental comparisons. Our SD1.5 evaluation includes official checkpoints released by the authors of SPIN, SePPO, and Diffusion-KTO. Corresponding SDXL checkpoints were not available to us, and reproducing these baselines on SDXL would require additional training that is difficult to complete within the rebuttal period. For SDXL, we include Online-DPO, which was competitive with the other baselines across our SD1.5 benchmarks, alongside Online-SFT. We have also broadened the evaluation with additional experiments on Sana flow-matching model.

Additional SD1.5 baselines. We have expanded the comparison to include the Diffusion-KTO. For a fair comparison, we re-evaluated both Diffusion-KTO and Diff.-NPO using the same evaluation protocol as Table 1 of the paper on Pick-a-Pic. The following tables report win rates (%) against the original SD1.5 backbone and raw scores.

MethodPickScoreHPSv2CLIPAestheticsImageReward
Diffusion-KTO73.181.357.068.876.6
Diff.-NPO82.585.259.481.085.4

Diff.-NPO outperforms Diffusion-KTO on all five evaluators

MethodPickScoreHPSv2CLIPAestheticsImageReward
Original SD1.520.6826.8033.405.49460.0470
Diffusion-KTO21.1927.8234.275.70600.6145
Diff.-NPO21.4728.0534.615.85710.8822

Diff.-NPO achieves higher mean raw scores across all five evaluators.

Flow-matching evaluation. We have also conducted experiments on Sana-0.6B (a flow-matching backbone), comparing Diff.-NPO with Online-DPO and Offline DPO. Both online methods use average-rank preference construction.

Win rates against the original Sana backbone:

MethodPickScoreHPSv2CLIPAestheticsImageRewardHPSv3
Offline DPO59.056.650.851.053.054.8
Online-DPO62.867.254.859.661.955.2
Diff.-NPO66.271.353.962.464.859.8

Raw metric scores. We would also like to clarify that Tables 1 and 2 in the submitted paper report raw scores alongside win rates for SD1.5 and SDXL across PickScore, HPSv2, CLIP, ImageReward, and Aesthetics. We apologize that this was not sufficiently clear in the presentation. For completeness, we report the raw scores for Sana below.

MethodPickScoreHPSv2CLIPAestheticsImageRewardHPSv3
Original Sana22.1828.4735.526.02931.09747.9932
Offline DPO22.2028.4935.536.03211.10048.0567
Online-DPO22.2328.5735.646.04881.12508.0614
Diff.-NPO22.2528.6235.676.06071.13978.1241

On Sana, Diff.-NPO improves win rates over Online-DPO on five of six evaluators, and achieves higher mean raw scores on all six.

AuthorsOpenReview ↗

Further theoretical analysis of Best-Worst sample selection.

[Q2] Further theoretical analysis of Best-Worst sample selection

Response. Thank you very much for your comment. Following your suggestion, we have provided theoretical insights into why the Best-Worst selection strategy adopted here makes sense from an optimization perspective. Related large-margin pair-selection strategies have also been used in the literature; see, for example, Rosset et al. (2024, Section 4). To elaborate briefly, recall the definition of hsh_s at iteration ss in Equation (8). Equation (9) establishes that the ideal population update at iteration s+1s+1 satisfies

hs(ps+1pop;x,y0:T,y0:T′)=Us(x,y0)−Us(x,y0′)η.h_s(p_{s+1}^{\mathrm{pop}};x,\mathbf y_{0:T},\mathbf y'_{0:T})=\frac{U_s(x,y_0)-U_s(x,y_0')}{\eta}.

Now fix ss and hold psp_s fixed. Given a prompt xx and two images y0,y0′y_0,y_0' sampled independently from ps(⋅∣x)p_s(\cdot\mid x), introduce an auxiliary binary comparison label Z∈0,1Z\in\\{0,1\\}, where Z=1Z=1 designates y0y_0 as the preferred image and Z=0Z=0 designates y0′y_0' as the preferred image. Specifically, generate ZZ according to

P(Z=1∣x,y0,y0′)=σ!(βη[Us(x,y0)−Us(x,y0′)])=σ!(βhs(ps+1pop;x,y0:T,y0:T′)).\mathbb P(Z=1\mid x,y_0,y_0')=\sigma\\!\left(\frac{\beta}{\eta}\big[U_s(x,y_0)-U_s(x,y_0')\big]\right)=\sigma\\!\left(\beta h_s(p_{s+1}^{\mathrm{pop}};x,\mathbf y_{0:T},\mathbf y'_{0:T})\right).

Here, UsU_s retains its original definition in terms of the preference oracle; the auxiliary label distribution is introduced only for this analysis. The trajectories y0:T,y0:T′\mathbf y_{0:T},\mathbf y'_{0:T} are obtained by augmenting the clean images through the forward diffusion process, conditionally independently of ZZ given the clean images. We model the conditional label probability, given these trajectories and the prompt, by σ(βhs(p;x,y0:T,y0:T′))\sigma(\beta h_s(p;x,\mathbf y_{0:T},\mathbf y'_{0:T})). Under this model, ps+1popp_{s+1}^{\mathrm{pop}} is a population likelihood maximizer. To verify this, let y0:T+,y0:T−\mathbf y_{0:T}^{+},\mathbf y_{0:T}^{-} denote the trajectories ordered according to the realized auxiliary label ZZ, with the designated winner placed first. Using the antisymmetry of hsh_s, the expected negative log-likelihood has the same logistic form as Equation (10). In the calculation below, we write hs(p)h_s(p) as shorthand for hs(p;x,y0:T,y0:T′)h_s(p;x,\mathbf y_{0:T},\mathbf y'_{0:T}), retaining the original, unordered pair. Then

\begin{aligned}\mathcal L_s(p)&=-\mathbb E\\!\left[\log\sigma\\!\left(\beta h_s(p;x,\mathbf y_{0:T}^{+},\mathbf y_{0:T}^{-})\right)\right]\\\\&=-\mathbb E\\!\left[Z\log\sigma(\beta h_s(p))+(1-Z)\log\sigma(-\beta h_s(p))\right]\\\\&=-\mathbb E\\!\left[\sigma(\beta h_s(p_{s+1}^{\mathrm{pop}}))\log\sigma(\beta h_s(p))+\big(1-\sigma(\beta h_s(p_{s+1}^{\mathrm{pop}}))\big)\log\sigma(-\beta h_s(p))\right]\\\\&=\mathcal L_s(p_{s+1}^{\mathrm{pop}})+\mathbb E\\!\left[\mathrm{KL}\\!\left(\mathrm{Ber}\\!\left(\sigma(\beta h_s(p_{s+1}^{\mathrm{pop}}))\right)\\,\middle\\|\\,\mathrm{Ber}\\!\left(\sigma(\beta h_s(p))\right)\right)\right]\\\\&\geq\mathcal L_s(p_{s+1}^{\mathrm{pop}}).\end{aligned}

Here, Ber(a)\mathrm{Ber}(a) denotes the Bernoulli distribution with success probability aa, and the expectations are taken over the fixed prompt and trajectory-pair distribution and, where applicable, the auxiliary label ZZ. The third equality follows by conditioning on the generated pair, and the fourth uses the decomposition of cross-entropy into conditional entropy and KL divergence. Consequently, ps+1popp_{s+1}^{\mathrm{pop}} minimizes this population negative log-likelihood over any admissible policy class containing it. This establishes population optimality under the auxiliary labeling mechanism; it does not assert that a finite-sample maximum likelihood estimator equals ps+1popp_{s+1}^{\mathrm{pop}}.

Comments on the paper

1 comment

We thank the reviewers for their thoughtful comments and constructive suggestions. Their feedback has helped us strengthen the evidence for Diff.-NPO. Our main additions are:

  1. Preference construction and evidence under non-transitive preferences. We clarified that the ensemble pairwise preference function underlying our average-rank selection can support cyclic preferences and is not restricted to the BTL model. We additionally introduced random-judge experiments, explicitly verified non-transitivity, and demonstrated improvements over Online-DPO under the same preference feedback.

  2. Generalization to flow-matching backbones. We extended the evaluation to Sana and SD3.5 Medium, reporting raw scores, win rates, and statistical comparisons, including HPSv3. The SD3.5 experiments also directly address the follow-up request for an additional backbone.

  3. Broader comparisons and computational analysis. We added comparisons with Diffusion-KTO, quantified training cost and preference-data usage, and compared the online methods under closely matched computational budgets. These additions clarify the performance gains and computational trade-offs.

  4. Additional theoretical support. We provided an analysis of Best-Worst sample selection and clarified the relationship between the Nash-derived regularization and our logistic objective.

We appreciate the reviewers’ constructive feedback and will incorporate these additional experiments, analyses, and clarifications into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.