- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Diff.-NPO, a diffusion model preference alignment method based on Nash Learning. In response to the problem that the traditional DPO method relies on the BTL model and it is difficult to express non-transmitive preferences, the author extends INPO to the diffusion model, optimize the generation strategy through online self-play. The method adopts the strategy update based on Mirror Descent, introduces the previous round of strategy and fixed reference strategy for regularization, and designs Logistic Loss based on denoising error. Experiments show that Diff.-NPO has achieved good preference alignment results on SD1.5 and SDXL.
Critical strengths
-
This paper focuses on the problem that the BTL model is difficult to deal with intransitive preferences, and tries to achieve more general diffusion model preference alignment through Nash Learning.
-
The author extends the Mirror Descent update of INPO to the diffusion model and derives the tractable training objective based on the denoising error.
-
The paper conducted multiple groups of comparative experiments on SD1.5 and SDXL, and provided ablation analysis of regularization coefficients and loss functions.
Critical weaknesses
-
The training data is constructed through the average ranking of the automatic scoring model, which is still transisive and may not be able to verify the advantage of the method in handling non-transmitative preferences.
-
Although the author provides a dissipation study on the regularization ratio and loss function, it is still unclear whether the performance improvement comes from Nash's learning itself or the dual strategy regularization and logical goals proposed by Nash.
-
The introduction is relatively long and can be shortened to better highlight the main contribution. In Section 3, the connection between the theoretical Nash update and the actual logistics goal can be explained more clearly.
Questions for authors
- What are the differences between Diff-NPO and other methods in training time, calculation cost and preferred data usage? Can you supplement the comparison under the same calculation budget?
Actually, I am not an expert in the research field of this paper. I initialized my rating based on my understanding of this paper, and I will adjust my rating according to the author's response and the opinions of other reviewers.
5 comments
[W2] Relationship between Nash learning, regularization, and the logistic objective.
Thank you for raising this point. The original Nash-derived squared-loss formulation already includes regularization toward both the reference and previous policies. We retain these terms unchanged when replacing the squared loss with our logistic objective. The loss-function ablation therefore compares the two objectives under the same dual-policy regularization and shows that the logistic objective yields better alignment performance. We will clarify this relationship in the revision.
[W3] Introduction length and the connection between the theoretical update and practical objective.
Thank you for this helpful suggestion. We agree and will shorten the introduction to highlight the main contributions more directly. We will also revise Section 3 to make the progression from the theoretical mirror-descent update, through the choice of logistic surrogate, to the tractable denoising objective more explicit, clearly identifying the practical design choices and approximation steps.
[Q1] Training time, computational cost, and preference-data usage.
Thank you for requesting this comparison. We provide runtime and data-usage measurements, together with performance comparisons at multiple training stages under comparable computational costs.
Training cost and preference-data usage. All methods use Sana-0.6B with 2,048 preference pairs per optimizer step. Runtimes are normalized to 4 × A100 GPUs.
| Method | Time per step | Runtime relative to Offline DPO | Preference-data usage per step |
|---|---|---|---|
| Offline DPO | 66 s | 1× | 2,048 human-labelled pairs from Pick-a-Pic v1 |
| Online-DPO | 732 s | 11.1× | 2,048 prompts × 8 generated candidates, ranked by five reward models |
| Diff.-NPO | 738 s | 11.2× | Same as Online-DPO |
Diff.-NPO adds only 0.8% runtime overhead over Online-DPO, corresponding to an additional forward pass through the previous policy when computing the loss. Both online methods require approximately 11× more runtime per step than Offline DPO because of candidate generation and scoring. In return, they refresh preference data from the evolving policy and require no human-labelled preference pairs during alignment.
Performance under comparable computational budgets. Given their nearly identical per-step runtime and identical preference-data construction, comparing Diff.-NPO and Online-DPO at the same optimizer steps provides a comparison under closely matched computational budgets. The following table reports win rates (%) against the original Sana backbone, with Offline DPO included for reference.
| Step | Method | PickScore | HPSv2 | CLIP | Aesthetics | ImageReward | HPSv3 |
|---|---|---|---|---|---|---|---|
| 270 | Offline DPO | 59.0 | 56.6 | 50.8 | 51.0 | 53.0 | 54.8 |
| 90 | Online-DPO | 61.5 | 65.9 | 53.3 | 59.9 | 61.8 | 56.0 |
| 90 | Diff.-NPO | 62.4 | 68.8 | 53.8 | 62.6 | 63.3 | 55.9 |
| 140 | Online-DPO | 63.2 | 66.0 | 53.6 | 60.6 | 60.6 | 57.3 |
| 140 | Diff.-NPO | 66.0 | 69.7 | 55.5 | 62.9 | 64.3 | 57.8 |
| 170 | Online-DPO | 63.4 | 67.4 | 54.5 | 60.7 | 63.7 | 59.6 |
| 170 | Diff.-NPO | 65.7 | 71.0 | 56.0 | 62.4 | 65.3 | 60.2 |
| 200 | Online-DPO | 62.8 | 67.2 | 54.8 | 59.6 | 61.9 | 55.2 |
| 200 | Diff.-NPO | 66.2 | 71.3 | 53.9 | 62.4 | 64.8 | 59.8 |
Direct paired comparisons. We report the raw scores of both methods at the same training steps. Parentheses in the Diff.-NPO rows show p-values from paired sign-flip tests against Online-DPO at the corresponding step.
| Step | Method | PickScore | HPSv2 | CLIP | Aesthetics | ImageReward | HPSv3 |
|---|---|---|---|---|---|---|---|
| 90 | Online-DPO | 22.20 | 28.53 | 35.61 | 6.0423 | 1.1181 | 8.0409 |
| 90 | Diff.-NPO | 22.22 (<0.001) | 28.57 (<0.001) | 35.63 (0.148) | 6.0499 (<0.001) | 1.1277 (<0.001) | 8.0598 (0.027) |
| 140 | Online-DPO | 22.22 | 28.55 | 35.63 | 6.0452 | 1.1225 | 8.0662 |
| 140 | Diff.-NPO | 22.24 (<0.001) | 28.60 (<0.001) | 35.67 (0.018) | 6.0573 (<0.001) | 1.1386 (<0.001) | 8.0995 (<0.001) |
| 170 | Online-DPO | 22.22 | 28.57 | 35.61 | 6.0465 | 1.1295 | 8.0891 |
| 170 | Diff.-NPO | 22.24 (<0.001) | 28.62 (<0.001) | 35.70 (<0.001) | 6.0566 (<0.001) | 1.1408 (<0.001) | 8.1179 (0.006) |
| 200 | Online-DPO | 22.23 | 28.57 | 35.64 | 6.0488 | 1.1250 | 8.0614 |
| 200 | Diff.-NPO | 22.25 (<0.001) | 28.62 (<0.001) | 35.67 (0.234) | 6.0607 (<0.001) | 1.1397 (<0.001) | 8.1241 (<0.001) |
Diff.-NPO achieves higher mean raw scores on all six evaluators at every evaluated step. The paired differences reach nominal p < 0.05 in 22 of 24 comparisons.
Response
We thank the reviewer for the thoughtful feedback and for recognizing our motivation, the extension of Nash learning to diffusion models, and the experimental evaluation. Below, we clarify the preference construction and the roles of the objective’s components, and provide additional comparisons of computational cost and data usage. We also appreciate the suggestions on presentation and will revise the introduction and Section 3 to make the method easier to follow.
[W1] Non-transitive preferences
We would like to clarify how our evaluator ensemble defines a pairwise preference function. For a prompt x and an image y, let Rm(x,y) denote the scalar score assigned by evaluator m, with M=5 evaluators in our experiments. The pairwise preference function associated with this ensemble is Pens(y≻y′∣x):=M1∑m=1M[1{Rm(x,y)>Rm(x,y′)}+211{Rm(x,y)=Rm(x,y′)}]. This is the fraction of evaluators preferring y to y′, with ties receiving weight 1/2. Although each evaluator assigns a scalar score, averaging their pairwise comparisons need not yield a BTL representation Pens(y≻y′∣x)=σ(r(x,y)−r(x,y′)) for a single scalar reward function r, where σ(t)=(1+e−t)−1. For example, consider three images A,B,C for a fixed prompt x and five evaluators, with two ranking A≻B≻C, two ranking B≻C≻A, and one ranking C≻A≻B. Their ensemble preferences satisfy Pens(A≻B∣x)=3/5, Pens(B≻C∣x)=4/5, and Pens(C≻A∣x)=3/5. This illustrates that the ensemble preference function can be non-transitive and establishes that our approach accommodates cyclic preferences, although demonstrating their presence and their effect on performance in our experiments requires direct empirical evidence.
Nevertheless, following your suggestion, we have conducted an additional set of experiments to address this concern more directly. We have now added a random-judge experiment, together with explicit verification of non-transitivity.
Preference construction. At each training step, we uniformly select one of five evaluators—PickScore, HPSv2, CLIP, ImageReward, and Aesthetic Score—to provide preference labels. For a fixed image pair, the underlying probability that A is preferred to B is therefore the fraction of these five evaluators preferring A. Although each evaluator provides a scalar ranking, their aggregated pairwise preferences can be non-transitive. Both Online-DPO and Diff.-NPO use this random-judge procedure. HPSv3 is used for evaluation only and is not one of the training judges.
Verification of non-transitivity. For each prompt, we examine its five generated images and check whether there exist A, B, and C such that P(A ≻ B), P(B ≻ C), and P(C ≻ A) are all greater than 1/2, following the definition in our introduction.
Non-transitive preferences are observed in 20.8–22.6% of the evaluated prompts, using five candidates per prompt. This verifies that the random-judge oracle exhibits non-transitivity on generated images.
Alignment performance under this oracle. The following table reports mean raw scores and win rates against the original Sana-0.6B. Both trained methods use random-judge feedback.
Diff.-NPO achieves higher mean raw scores on all six evaluators and higher win rates against the original backbone on five of six. Paired tests on the same prompts and seeds give p < 0.05 on PickScore, HPSv2, ImageReward, and HPSv3; the differences on CLIP and Aesthetics are not statistically significant. We appreciate the suggestion and will include the setup, diagnostics, and results in the revised manuscript.