- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses two limitations of existing preference optimization methods: binary preference modeling cannot adequately capture ties, and human preference annotations may contain noise. The authors propose Noisy Ternary Preference Optimization (NTPO), which replaces the Bradley–Terry (BT) model underlying DPO with the Rao–Kupper (RK) model and explicitly accounts for annotation noise through a transition matrix modeling preference-label flips.
Critical strengths
- The problem is well motivated, and the proposed approach of combining ternary preference modeling with annotation-noise modeling is reasonable.
- The paper provides detailed theoretical derivations and proofs.
- The experimental results support the effectiveness of the proposed method.
Critical weaknesses
- The evaluated backbones are outdated relative to current developments in text-to-image generation. Evaluation on Qwen-Image-2512 and FLUX.2-dev would strengthen the paper. If feasible, additional experiments on Qwen-Image-2.1 and FLUX 3 Image Klein Base would be even more convincing.
- The baseline comparisons are limited and do not sufficiently cover recent relevant methods. For example, the authors should consider comparing against PNAPO from Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairs (ICML 2026).
- In the first example of Figure 1, I find that the preference-optimized outputs exhibit noticeable degradation in image quality compared with the SD1.5 base model, particularly in background sharpness. Across both examples, the differences among the preference optimization methods are not sufficiently compelling. The authors should provide stronger qualitative evidence, such as examples where other methods exhibit clear failure modes that NTPO avoids. The current qualitative results in both the main paper and the appendix do not clearly demonstrate NTPO’s advantages.
- Figure 2(b) contains unusual visual artifacts near the upper-left corners of the bars. These artifacts detract from the presentation and raise questions about how the plots were produced and whether they were manually edited. The authors should clarify their origin and provide clean versions of the plots.
- The training setup requires further justification. The flow-matching models are fine-tuned with LoRA for only 100 steps on eight A800 GPUs. This seems like a small number of parameter updates: have the models actually converged? The authors should show training loss as a function of optimization steps. Given the million-scale size of Pick-a-Pic v2, they should also explain the short training budget and report the actual training-set size and effective batch size. The apparent quality degradation in the first example of Figure 1 raises concerns about whether the optimization objective consistently improves visual quality. I would like to see results for this example from a checkpoint whose training loss has converged.
Questions for authors
- The purpose of the first sentence in lines 184–185 is unclear, and the stated equality appears inconsistent with Eq. (6). Could the authors clarify or correct this statement?
- PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization addresses related issues through clean/flip/tie latent-state modeling. The authors should cite this work and explain how NTPO differs from it, particularly in its treatment of ties and annotation noise.
9 comments
(Weakness 2) Comparison with Recent Methods:PNAPO
(Response 2): We thank the reviewer for pointing out this relevant work. We have cited PNAPO and expanded the discussion in Section 2 (Related Work) of the revised manuscript. We also provide an empirical comparison between our reproduced PNAPO baseline and NTPO in Table 2. Given our limited financial budget, we conduct this comparison using SD3.5-M and FLUX.1-schnell as representative backbones.
Table 2. Performance comparison on the HPSv2 and PartiPrompt benchmarks using SD3.5-M and FLUX.1-schnell.
| HPSv2 | PartiPrompt | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | Method | PS↑ | HPS↑ | AES↑ | IR↑ | CLIP↑ | PS↑ | HPS↑ | AES↑ | IR↑ | CLIP↑ |
| SD3.5-M | Base | 22.95 | 28.88 | 6.091 | 1.165 | 37.04 | 22.58 | 29.08 | 6.053 | 0.887 | 34.01 |
| SD3.5-M | DPO | 23.01 | 29.12 | 6.126 | 1.183 | 37.10 | 22.64 | 29.21 | 6.137 | 0.998 | 34.09 |
| SD3.5-M | PNAPO-rep | 23.23 | 29.38 | 5.962 | 1.208 | 37.13 | 22.71 | 29.33 | 6.144 | 1.137 | 34.15 |
| SD3.5-M | TPO | 23.59 | 30.05 | 6.154 | 1.257 | 37.35 | 23.27 | 30.23 | 6.201 | 1.179 | 34.64 |
| SD3.5-M | NTPO | 24.17 | 30.87 | 6.215 | 1.323 | 37.66 | 23.89 | 31.10 | 6.285 | 1.247 | 35.22 |
| FLUX.1-schnell | Base | 23.01 | 29.59 | 6.103 | 1.218 | 37.78 | 23.15 | 29.42 | 6.135 | 1.051 | 34.68 |
| FLUX.1-schnell | DPO | 23.12 | 30.76 | 6.217 | 1.238 | 38.05 | 23.25 | 30.02 | 6.186 | 1.105 | 34.89 |
| FLUX.1-schnell | PNAPO-rep | 23.26 | 31.68 | 6.358 | 1.249 | 38.24 | 23.48 | 30.27 | 6.218 | 1.216 | 35.06 |
| FLUX.1-schnell | TPO | 23.85 | 32.13 | 6.401 | 1.305 | 38.47 | 23.74 | 30.85 | 6.277 | 1.277 | 35.42 |
| FLUX.1-schnell | NTPO | 24.39 | 32.78 | 6.487 | 1.359 | 38.85 | 24.13 | 31.89 | 6.358 | 1.337 | 35.97 |
Following the suggestion, we have independently reproduced PNAPO and added an empirical comparison with NTPO. We address this coment from two perspectives: (i) the additional empirical comparison, and (ii) the methodological relationship between PNAPO and NTPO.
(1) Independent reproduction and additional comparison. As we could not locate a publicly available official implementation of PNAPO, we independently reproduced the method following the methodological details and experimental configurations reported in its paper. We have added a comparison with NTPO under the same evaluation protocol, as shown in Table 2. The results show that NTPO achieves better overall performance than our reproduced PNAPO baseline, further supporting its effectiveness against this recent method. We label the baseline as “PNAPO-rep” to distinguish our implementation from official results.
(2) Methodological relationship between PNAPO and NTPO. PNAPO and NTPO address distinct aspects of preference alignment. PNAPO leverages tracked generation-time prior noises to improve trajectory estimation and introduces dynamic regularization for rectified-flow optimization. In contrast, NTPO explicitly models ternary preference outcomes—win, loss, and tie—and corrects annotation-label noise. Thus, the “noise” considered by PNAPO refers to the initial noise used for image generation, whereas NTPO addresses corruption in preference labels. We therefore view the two approaches as potentially complementary.
(Weakness 3) Stronger Qualitative Comparisons
(Response 3): We thank the reviewer for this careful observation and constructive suggestion. The softer background in the first example is also observed in DPO, DMPO, -DPO, and TPO, indicating that this behavior is not unique to NTPO and should not be attributed solely to its design. We agree that the current examples do not sufficiently highlight the differences among methods. In the revised manuscript, we will supplement the qualitative evaluation with clearer examples that show more pronounced visual differences among methods and better illustrate NTPO's advantages.
(Weakness 4) Visual Artifacts in Figure 2(b)
(Response 4): We thank the reviewer for identifying these artifacts and raising the concern about the figure-production process. We have checked the reported numerical values against the original experimental records and confirmed that they are accurate and have not been manually altered. We acknowledge that the original figure-preparation and post-processing workflow did not ensure sufficiently reliable graphical presentation. We have therefore redrawn Figure 2(b) using the original experimental results and verified that all bar heights and numerical labels are accurate.
We sincerely thank the reviewer for the thorough reading, constructive feedback, and for recognizing that our problem is well-motivated (Strength 1), our theoretical derivations and proofs are detailed (Strength 2), and our experimental results support the effectiveness of NTPO (Strength 3). We also appreciate the constructive suggestions and address the concerns point by point below.
(Weakness 1) Evaluation on Recent Backbones: Qwen-Image-2512, FLUX.2-dev, Qwen-Image-2.1 and FLUX 3-Image-Klein-Base
(Response 1): We thank the reviewer for this helpful suggestion and agree that evaluating more recent backbones would strengthen the empirical evidence for NTPO. We provide additional experimental results on all four suggested backbones in Table 1.
To address this concern, we rented 8 NVIDIA H200 GPUs, incurring approximately RMB 30,000 in GPU rental costs, and completed experiments on Qwen-Image-2512, FLUX.2-dev, Qwen-Image-2.1, and FLUX-3-Image-Klein-Base. For each backbone, we compare the base model, DPO, TPO, and NTPO on HPSv2 and PartiPrompt using the same five metrics as in the paper. As shown in Table 1, NTPO improves preference alignment across these four backbones, providing further evidence of its effectiveness and applicability to recent text-to-image models.
Table 1: Performance comparison on HPSv2 and PartiPrompt using four recent backbones. Higher scores indicate better performance. The best and second-best results for each backbone and metric are marked in bold and italic, respectively.