acceptodds
Under review as a conference paper at ICLR 2027

Ternary Preference Optimization for Text-to-Image Generation with Noisy Human Feedback

Abstract

Recent text-to-image alignment methods typically rely on two assumptions about human preference data: (1) annotators always strictly prefer one image over another, and (2) preference labels are noise-free and faithfully reflect underlying human preferences. In practice, both assumptions can be violated. For example, approximately 11% of annotations in Pick-a-Pic are ties, indicating that annotators may consider two images equally preferable. Such ties provide meaningful preference information but are typically discarded by binary preference modeling. Meanwhile, human preference labels are inherently noisy due to individual subjectivity and annotator disagreement. To address these limitations, we propose *Noisy Ternary Preference Optimization* (NTPO), a unified framework that explicitly models three preference outcomes—preferred, rejected, and tied—while accounting for noisy and inconsistent annotations. By preserving tie information and explicitly modeling annotation noise, NTPO provides a more faithful formulation of human preference alignment for text-to-image generation. Experiments across multiple benchmarks demonstrate that NTPO improves both alignment quality and robustness compared with existing binary preference optimization methods.

Primary area:
generative models
Keywords:
Preference Alignment, Noisy Human Feedback

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4TT5OpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses two limitations of existing preference optimization methods: binary preference modeling cannot adequately capture ties, and human preference annotations may contain noise. The authors propose Noisy Ternary Preference Optimization (NTPO), which replaces the Bradley–Terry (BT) model underlying DPO with the Rao–Kupper (RK) model and explicitly accounts for annotation noise through a transition matrix modeling preference-label flips.

Critical strengths
  1. The problem is well motivated, and the proposed approach of combining ternary preference modeling with annotation-noise modeling is reasonable.
  2. The paper provides detailed theoretical derivations and proofs.
  3. The experimental results support the effectiveness of the proposed method.
Critical weaknesses
  1. The evaluated backbones are outdated relative to current developments in text-to-image generation. Evaluation on Qwen-Image-2512 and FLUX.2-dev would strengthen the paper. If feasible, additional experiments on Qwen-Image-2.1 and FLUX 3 Image Klein Base would be even more convincing.
  2. The baseline comparisons are limited and do not sufficiently cover recent relevant methods. For example, the authors should consider comparing against PNAPO from Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairs (ICML 2026).
  3. In the first example of Figure 1, I find that the preference-optimized outputs exhibit noticeable degradation in image quality compared with the SD1.5 base model, particularly in background sharpness. Across both examples, the differences among the preference optimization methods are not sufficiently compelling. The authors should provide stronger qualitative evidence, such as examples where other methods exhibit clear failure modes that NTPO avoids. The current qualitative results in both the main paper and the appendix do not clearly demonstrate NTPO’s advantages.
  4. Figure 2(b) contains unusual visual artifacts near the upper-left corners of the bars. These artifacts detract from the presentation and raise questions about how the plots were produced and whether they were manually edited. The authors should clarify their origin and provide clean versions of the plots.
  5. The training setup requires further justification. The flow-matching models are fine-tuned with LoRA for only 100 steps on eight A800 GPUs. This seems like a small number of parameter updates: have the models actually converged? The authors should show training loss as a function of optimization steps. Given the million-scale size of Pick-a-Pic v2, they should also explain the short training budget and report the actual training-set size and effective batch size. The apparent quality degradation in the first example of Figure 1 raises concerns about whether the optimization objective consistently improves visual quality. I would like to see results for this example from a checkpoint whose training loss has converged.
Questions for authors
  1. The purpose of the first sentence in lines 184–185 is unclear, and the stated equality appears inconsistent with Eq. (6). Could the authors clarify or correct this statement?
  2. PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization addresses related issues through clean/flip/tie latent-state modeling. The authors should cite this work and explain how NTPO differs from it, particularly in its treatment of ties and annotation noise.
9 comments

We sincerely thank the reviewer for the thorough reading, constructive feedback, and for recognizing that our problem is well-motivated (Strength 1), our theoretical derivations and proofs are detailed (Strength 2), and our experimental results support the effectiveness of NTPO (Strength 3). We also appreciate the constructive suggestions and address the concerns point by point below.

(Weakness 1) Evaluation on Recent Backbones: Qwen-Image-2512, FLUX.2-dev, Qwen-Image-2.1 and FLUX 3-Image-Klein-Base

(Response 1): We thank the reviewer for this helpful suggestion and agree that evaluating more recent backbones would strengthen the empirical evidence for NTPO. We provide additional experimental results on all four suggested backbones in Table 1.

To address this concern, we rented 8 NVIDIA H200 GPUs, incurring approximately RMB 30,000 in GPU rental costs, and completed experiments on Qwen-Image-2512, FLUX.2-dev, Qwen-Image-2.1, and FLUX-3-Image-Klein-Base. For each backbone, we compare the base model, DPO, TPO, and NTPO on HPSv2 and PartiPrompt using the same five metrics as in the paper. As shown in Table 1, NTPO improves preference alignment across these four backbones, providing further evidence of its effectiveness and applicability to recent text-to-image models.

Table 1: Performance comparison on HPSv2 and PartiPrompt using four recent backbones. Higher scores indicate better performance. The best and second-best results for each backbone and metric are marked in bold and italic, respectively.

HPSv2PartiPrompt
ModelMethodPS↑HPS↑AES↑IR↑CLIP↑PS↑HPS↑AES↑IR↑CLIP↑
Qwen-Image-2512Base23.4332.456.1331.38538.2823.3533.576.1861.30938.17
Qwen-Image-2512DPO23.7533.476.2391.41138.5123.6733.786.3031.35538.32
Qwen-Image-2512TPO23.8934.016.3341.48938.7323.8333.956.3841.39738.45
Qwen-Image-2512NTPO (Ours)24.6334.556.4491.58338.6524.2134.386.4381.48038.63
FLUX.2-devBase23.6030.316.2531.25538.0323.4729.256.2581.21636.29
FLUX.2-devDPO23.8132.556.3711.32438.4123.6531.736.3841.33737.38
FLUX.2-devTPO24.3233.176.4351.41838.5323.9332.476.4171.38938.04
FLUX.2-devNTPO (Ours)24.5833.656.4851.51338.6424.4632.876.4891.44738.53
Qwen-Image-2.1Base23.5532.246.1711.41038.2023.5132.186.1591.43138.13
Qwen-Image-2.1DPO23.8332.526.2791.43538.3123.7633.156.3171.48538.25
Qwen-Image-2.1TPO24.1533.156.4581.48138.5924.0933.676.4191.53638.41
Qwen-Image-2.1NTPO (Ours)24.7133.896.5631.62138.7724.5734.536.5211.56938.68
FLUX-3-Image Klein-BaseBase23.6231.256.2771.37338.2523.4829.696.3051.25436.77
FLUX-3-Image Klein-BaseDPO24.0532.836.3591.42738.3923.8130.166.3591.31837.33
FLUX-3-Image Klein-BaseTPO24.3833.416.4551.48338.5224.1631.646.4361.42937.80
FLUX-3-Image Klein-BaseNTPO (Ours)24.7933.876.5351.53538.7024.5333.656.5581.60438.57

(Weakness 2) Comparison with Recent Methods:PNAPO

(Response 2): We thank the reviewer for pointing out this relevant work. We have cited PNAPO and expanded the discussion in Section 2 (Related Work) of the revised manuscript. We also provide an empirical comparison between our reproduced PNAPO baseline and NTPO in Table 2. Given our limited financial budget, we conduct this comparison using SD3.5-M and FLUX.1-schnell as representative backbones.

Table 2. Performance comparison on the HPSv2 and PartiPrompt benchmarks using SD3.5-M and FLUX.1-schnell.

HPSv2PartiPrompt
ModelMethodPS↑HPS↑AES↑IR↑CLIP↑PS↑HPS↑AES↑IR↑CLIP↑
SD3.5-MBase22.9528.886.0911.16537.0422.5829.086.0530.88734.01
SD3.5-MDPO23.0129.126.1261.18337.1022.6429.216.1370.99834.09
SD3.5-MPNAPO-rep23.2329.385.9621.20837.1322.7129.336.1441.13734.15
SD3.5-MTPO23.5930.056.1541.25737.3523.2730.236.2011.17934.64
SD3.5-MNTPO24.1730.876.2151.32337.6623.8931.106.2851.24735.22
FLUX.1-schnellBase23.0129.596.1031.21837.7823.1529.426.1351.05134.68
FLUX.1-schnellDPO23.1230.766.2171.23838.0523.2530.026.1861.10534.89
FLUX.1-schnellPNAPO-rep23.2631.686.3581.24938.2423.4830.276.2181.21635.06
FLUX.1-schnellTPO23.8532.136.4011.30538.4723.7430.856.2771.27735.42
FLUX.1-schnellNTPO24.3932.786.4871.35938.8524.1331.896.3581.33735.97

Following the suggestion, we have independently reproduced PNAPO and added an empirical comparison with NTPO. We address this coment from two perspectives: (i) the additional empirical comparison, and (ii) the methodological relationship between PNAPO and NTPO.

(1) Independent reproduction and additional comparison. As we could not locate a publicly available official implementation of PNAPO, we independently reproduced the method following the methodological details and experimental configurations reported in its paper. We have added a comparison with NTPO under the same evaluation protocol, as shown in Table 2. The results show that NTPO achieves better overall performance than our reproduced PNAPO baseline, further supporting its effectiveness against this recent method. We label the baseline as “PNAPO-rep” to distinguish our implementation from official results.

(2) Methodological relationship between PNAPO and NTPO. PNAPO and NTPO address distinct aspects of preference alignment. PNAPO leverages tracked generation-time prior noises to improve trajectory estimation and introduces dynamic regularization for rectified-flow optimization. In contrast, NTPO explicitly models ternary preference outcomes—win, loss, and tie—and corrects annotation-label noise. Thus, the “noise” considered by PNAPO refers to the initial noise used for image generation, whereas NTPO addresses corruption in preference labels. We therefore view the two approaches as potentially complementary.

(Weakness 3) Stronger Qualitative Comparisons

(Response 3): We thank the reviewer for this careful observation and constructive suggestion. The softer background in the first example is also observed in DPO, DMPO, α\alpha-DPO, and TPO, indicating that this behavior is not unique to NTPO and should not be attributed solely to its design. We agree that the current examples do not sufficiently highlight the differences among methods. In the revised manuscript, we will supplement the qualitative evaluation with clearer examples that show more pronounced visual differences among methods and better illustrate NTPO's advantages.

(Weakness 4) Visual Artifacts in Figure 2(b)

(Response 4): We thank the reviewer for identifying these artifacts and raising the concern about the figure-production process. We have checked the reported numerical values against the original experimental records and confirmed that they are accurate and have not been manually altered. We acknowledge that the original figure-preparation and post-processing workflow did not ensure sufficiently reliable graphical presentation. We have therefore redrawn Figure 2(b) using the original experimental results and verified that all bar heights and numerical labels are accurate.

Reviewer_fJpsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Noisy Ternary Preference Optimization (NTPO) for text-to-image alignment. Existing preference alignment frameworks (e.g., Diffusion-DPO) typically adopt two simplifying assumptions: pairwise preferences are strictly binary, and human annotations are clean/noise-free. However, real-world datasets (e.g., Pick-a-Pic) contain substantial tied comparisons (~11%) as well as subjective or inconsistent labeling noise.

To overcome these limitations:

  1. The authors replace the binary Bradley-Terry model with the ternary Rao-Kupper formulation to explicitly capture win, loss, and tie outcomes.

  2. They extend the RK model by introducing a label transition matrix Q parameterized by a noise flip rate η, proving that optimizing this objective is unbiased with respect to the underlying clean preference distribution.

  3. To address non-convexity issues in diffusion/flow-matching likelihood ratios, they derive a convex variational surrogate upper bound via a variational weight matrix R, extending it to timestep-level objectives for diffusion and flow matching models.

  4. Experiments on SD1.5, SDXL, SD3.5-M, FLUX.1-schnell, and AuraFlow-v0.3 across HPSv2 and PartiPrompts show that NTPO achieves superior alignment metrics compared to standard binary (DPO, DMPO, α-DPO) and noise-free ternary (TPO) baselines.

Critical strengths
  1. Well-Formulated Problem Grounding: Moving beyond binary preferences and clean-label assumptions directly addresses documented issues in crowdsourced visual preference datasets where ties and annotator disagreements are prevalent.

  2. Solid Theoretical Foundation and Variational Derivation: The paper demonstrates that standard ternary preference optimization under noise is biased, whereas NTPO recovers the true distribution. Furthermore, rather than ignoring the non-convexity of the mixture log-likelihood in diffusion/flow-matching derivations, the authors rigorously formulate a variational surrogate bound via Jensen's inequality and prove its tightness at the optimum.

  3. Broad Model Coverage (Diffusion + Flow Matching): The framework is adapted and validated across both classic diffusion backbones (SD1.5, SDXL) and modern rectified flow transformers (FLUX.1-schnell, SD3.5-M, AuraFlow-v0.3), showcasing versatility across paradigms.

  4. Comprehensive Ablations: The paper systematically dissects the individual contributions of tie modeling vs. noise modeling, evaluates sensitivity to the tie threshold α and noise parameter η, and investigates simplified vs. full transition matrices.

Critical weaknesses
  1. Highly Restrictive and Unrealistic Noise Transition Model: The simplified transition matrix assumes noise occurs strictly as flips between opposite preferences (x_1>x_2↔x_1<x_2) with probability η, while assuming that ties are completely noise-free (q_2,2=1, q_0,2=q_1,2=q_2,0=q_2,1=0). In practical human annotation, borderline samples are far more likely to be confused between a weak preference and a tie than a complete inversion of preference (i.e., flips between strict preference and tie are very common in human labeling).

  2. Flawed Synthetic Validation Pipeline / Circular Setup: To evaluate noise robustness, the authors use Cleanlab + HPSv3 to filter Pick-a-Pic v2 to create a "clean" dataset, and then synthetically inject symmetric label noise at η=0.2. This setup creates an artificial testbed that matches the exact mathematical assumptions of their simplified transition matrix Q. The paper does not thoroughly demonstrate whether NTPO's specific noise model handles real, naturally occurring crowdsourced noise without pre-filtering and synthetic corruption.

  3. Manual / Fixed Noise Rate Tuning (η): In practical applications, the true noise rate η is unknown and varies significantly across datasets and prompt categories. Although the paper provides a sensitivity analysis over η∈{0.1,0.2,0.4}, it assumes a static, dataset-wide noise rate rather than an adaptive, instance-dependent, or learnable transition estimation during training.

  4. Lack of Human Subject Evaluation: The evaluation relies entirely on automated metrics (PickScore, HPSv2, ImageReward, CLIP, AES). Given that automated scoring models (especially HPSv3/HPSv2) are also used in the data auditing pipeline, automated metrics alone can be prone to reward exploitation; a rigorous human side-by-side study is essential to confirm whether the modest score gains translate to discernible perceptual improvements.

Questions for authors

Please consider the following queries for rebuttal:

  1. How does NTPO perform if the noise corruption model includes transitions to and from the tie class? Can the authors report results where synthetic noise flips strict preferences into ties and vice versa, and does the variational surrogate derivation in Section 4.3 still hold under a transition matrix with non-zero tie off-diagonals?

  2. In the main experiments, was the model trained on raw Pick-a-Pic v2 with its organic human noise, or exclusively on the Cleanlab-audited dataset with synthetically injected noise? If the latter, what is the performance of NTPO compared to TPO and α-DPO on raw, uncurated crowdsourced data?

  3. In Section 5.4 / Appendix A.9, the authors compare the "Simplified Q" to a "General Q". How exactly is the General Q parameterized and estimated/optimized during training, and why does it require significantly more training time?

  4. Are the performance gains in Table 1 and Table 5 statistically significant across multiple seeds? Do the authors have human pairwise preference / win-rate evaluations comparing NTPO against TPO and α-DPO to ensure that the metric correspond to human-perceptible quality differences?

9 comments

Dear Reviewer fJps,

Thank you very much for your kind message, your understanding, and your willingness to engage in a staged discussion. We are also grateful for your patience: over the past few days we have been allocating compute and annotation resources to the additional experiments, which require considerable time to complete rigorously. We apologize for the delay in our response.

We are encouraged that you recognize the following strengths of our work:

  • Practical motivation: Jointly modeling ties and annotation noise addresses important challenges in crowdsourced human preference data.
  • Theoretical foundations: Our analysis establishes the distribution-recovery property of NTPO under the assumed noise model and derives a variational surrogate with a provable tightness condition.
  • Broad applicability: The framework is adapted and validated across both diffusion and flow-matching models.
  • Comprehensive ablations: Our experiments examine the contributions of tie and noise modeling, hyperparameter sensitivity, and alternative transition matrix designs.

We address your concerns and questions point by point below.

Question 1: How does NTPO perform if the noise corruption model includes transitions to and from the tie class? Can the authors report results where synthetic noise flips strict preferences into ties and vice versa, and does the variational surrogate derivation in Section 4.3 still hold under a transition matrix with non-zero tie off-diagonals?

Thank you for highlighting the practical prevalence of label confusion between weak preferences and ties (Strict↔Tie\text{Strict} \leftrightarrow \text{Tie}) in crowdsourced annotations. In fact, we have already included these experiments in Section 5.4 (Table 3) and Appendix A.9 (Tables 6 and 7) of the manuscript. Due to page-limit constraints, we did not provide a detailed explanation of their setup in the main text. Below, we clarify your questions in detail.

1. Empirical Clarification and Performance under Tie-Involving Noise Flips.

Specifically, in Section 5.4 (Table 3) and Appendix A.9 (Tables 6 and 7), General QQ explicitly incorporates bidirectional label flips between strict preferences and ties (q2,0,q2,1>0q_{2,0}, q_{2,1} > 0 and q0,2,q1,2>0q_{0,2}, q_{1,2} > 0), rather than assuming tie labels are noise-free. To evaluate this setting, we merged the clean strict-preference subset with the native tie subset and applied general ternary label flips across all three classes (y∈{0,1,2}y \in \{0, 1, 2\}). Thus, the noise model explicitly includes flips from strict preferences to ties (q2,0,q2,1>0q_{2,0}, q_{2,1} > 0) and from ties to strict preferences (q0,2,q1,2>0q_{0,2}, q_{1,2} > 0).

We consolidate the results from Tables 3, 6, and 7 of our manuscript into the table below. As shown in the Table R1, General QQ slightly outperforms Simplified QQ across almost all metrics, but incurs a longer training time. In contrast, Simplified QQ achieves comparable alignment performance with substantially shorter training time. This makes Simplified QQ a practical and scalable default in large-scale generative alignment.

Table R1. Comparison between General QQ and Simplified QQ under general ternary noise flips (including Strict↔Tie\text{Strict} \leftrightarrow \text{Tie} transitions) across diffusion and flow-matching backbones.

ModelTransition MatrixTraining Time (h)PS$\uparrow$HPSv2$\uparrow$AES$\uparrow$IR$\uparrow$CLIP$\uparrow$
SD1.5General QQ3522.3028.985.8250.83134.78
SD1.5Simplified QQ2922.2628.935.8120.83834.72
SDXLGeneral QQ9223.9230.376.4211.17338.54
SDXLSimplified QQ8523.8630.346.4471.18838.50
SD3.5-MGeneral QQ1523.4929.926.2011.25237.40
SD3.5-MSimplified QQ1023.4429.856.1971.24937.32
FLUX.1-schnellGeneral QQ5223.9130.566.3111.24537.71
FLUX.1-schnellSimplified QQ4423.8730.516.3051.23337.60
AuraFlow-v0.3General QQ2723.2029.396.2521.08337.30
AuraFlow-v0.3Simplified QQ2223.1629.356.2471.10237.23

2. Variational Surrogate Validity in Section 4.3 under Non-Zero Tie Off-Diagonals. We confirm that the variational surrogate derivation in Section 4.3 still holds without modification under a general 3×33\times 3 transition matrix QQ with non-zero tie off-diagonals. There are two reasons for this:

  • Global Convexity of Each Component (win, lose and tie): In Appendix A.3 (Eqs. (63)–(65) and Eqs. (71)–(78)), we have already proven via second-order derivatives that each individual negative log-likelihood term—namely −log⁡pθRK(y=0∣z)-\log p_{\theta}^{\mathrm{RK}}(y=0|z), −log⁡pθRK(y=1∣z)-\log p_{\theta}^{\mathrm{RK}}(y=1|z), and −log⁡pθRK(y=2∣z)-\log p_{\theta}^{\mathrm{RK}}(y=2|z)—is independently globally convex with respect to Δθ\Delta_\theta.
  • Surrogate Convexity and Tightness: Under a general 3×33 \times 3 transition matrix QQ and variational weight matrix R=(ry~,i)∈R3×3R = (r_{\tilde{y}, i}) \in \mathbb{R}^{3 \times 3} (where ry~,i≥0r_{\tilde{y}, i} \ge 0 and ∑i=02ry~,i=1\sum_{i=0}^{2} r_{\tilde{y}, i} = 1), the sample-wise variational surrogate loss for sample nn is given by: ℓVFC,n(θ,R)=−∑i=02ry~(n),ilog⁡qy~(n),ipθRK(y=i∣zn)ry~(n),i\ell_{\text{VFC}, n}(\theta, R) = -\sum_{i=0}^{2} r_{\tilde{y}(n), i} \log \frac{q_{\tilde{y}(n), i} p_{\theta}^{\text{RK}}(y=i \mid z_n)}{r_{\tilde{y}(n), i}} which is simply a non-negative linear combination of these three convex functions. Since non-negative linear combinations preserve convexity, ℓVFC,n(θ,R)\ell_{\mathrm{VFC}, n}(\theta, R) remains globally convex in Δθ\Delta_\theta, ensuring that the Jensen's inequality (Eq. (17)) holds.

Question 2: In the main experiments, was the model trained on raw Pick-a-Pic v2 with its organic human noise, or exclusively on the Cleanlab-audited dataset with synthetically injected noise? If the latter, what is the performance of NTPO compared to TPO and α\alpha-DPO on raw, uncurated crowdsourced data?

Thank you for raising this critical point regarding data provenance. To ensure known ground-truth noise rates for controlled theoretical verification, our main experiments used a semi-synthetic dataset that preserved all native ties while injecting controlled flip noise (η=0.2\eta=0.2) into the Cleanlab-filtered preference subset.

To directly evaluate robustness against organic crowdsourced noise without any filtering or synthetic corruption, we conducted new experiments on completely raw Pick-a-Pic v2 ( Table R2), where NTPO consistently outperforms both α\alpha-DPO and TPO.

Due to time and computational budget constraints during the rebuttal period, we report the results of SD1.5 on the HPSv2 and PartiPrompt benchmarks below. As shown in the Table R2, NTPO consistently outperforms both α\alpha-DPO and TPO.

Table R2. Performance comparison on the HPSv2 and PartiPrompt benchmarks using SD1.5 trained directly on the raw Pick-a-Pic v2 dataset. Best results are shown in bold.

ModelMethodHPSv2PartiPrompt
PS$\uparrow$HPS$\uparrow$AES$\uparrow$IR$\uparrow$CLIP$\uparrow$PS$\uparrow$HPS$\uparrow$AES$\uparrow$IR$\uparrow$CLIP$\uparrow$
SD1.5α\alpha-DPO21.6527.525.6570.60933.0821.2326.835.4360.41731.78
SD1.5TPO21.8127.735.6080.57033.8621.5827.185.6420.53532.19
SD1.5NTPO (Ours)22.0828.255.7340.65734.0521.8927.535.8350.60732.45
Reviewer_R4TPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes NTPO for text-to-image generation. It i) extends the traditional BT model to incorporate tied preferences and ii) introduces a noise transition matrix to realize robust optimization with noisy human feedback. Experiments across multiple models and metrics demonstrate consistent improvements in t2i preference learning.

Critical strengths
  1. Clear motivation and formulation. The neglect of tied pairs in human preference dataset and the noises in human feedback are practical issues.
  2. The proposed method, NTPO, is simple and generalizable. Plus, the mathematics is clear and looks solid.
  3. Experiments include both traditional unet-based backbones and modern mm-dits, which is good, showing the general applicability of NTPO.
Critical weaknesses
  1. The paper only reports evaluation results at fixed training steps. Reward curve on the eval set during the training process is needed.
  2. The authors do not clearly demonstrate how information regarding "ties" helps improve model performance. For example, it remains unclear whether a higher proportion of "tie" pairs in the dataset leads to better model performance, or whether, compared to DPO, training with TPO results in a ∣Δθ(x1,x2)∣|\Delta\theta(x_1, x_2)| value that is closer to 0 (for each pair (x1,x2)(x_1, x_2) labeled as a "tie") for the finetuned model.
  3. (minor) If possible, evaluation with newer metrics such as hpsv3 is welcome.
Questions for authors
  1. In the definition of Q (eq. 11), why does the author assume that only "win" or "loss" outcomes can be mislabeled, while a "tie" does not include noise? A clear justification for the would help to strengthen the formulation.
  2. Could the authors provide a comparison with DPO and TPO under different levels of injected label noise? While the current experiments showed sensitivity to the assumed noise ratio, such a comparison would be better to directly demonstrate whether NTPO offers stronger robustness as the actual noise level increases.
8 comments

Dear Reviewer R4TP,

Thank you very much for your thoughtful review, constructive feedback, and patience! We sincerely apologize for the delay in our response. Over the past few days, we have been dedicating our compute and evaluation resources to thoroughly conducting all the additional experiments you suggested, which took considerable time to complete rigorously.

We are deeply grateful for your positive evaluation and for recognizing the following key strengths of our work:

  • Clear motivation and practical formulation: Addressing the neglect of tied preference pairs and the presence of noisy human feedback in text-to-image preference datasets.
  • Simplicity, generalizability, and solid theoretical foundation: Providing a clean and mathematically sound formulation for NTPO.
  • Broad architectural validation: Demonstrating the general applicability of NTPO across both traditional UNet-based diffusion backbones and modern MMDiT flow-matching architectures.

Below, we address each of your insightful comments and questions point by point, supported by new experimental results and theoretical clarifications.

Question 1: In the definition of QQ (Eq. 11), why does the author assume that only "win" or "loss" outcomes can be mislabeled, while a "tie" does not include noise? A clear justification for this would help to strengthen the formulation.

Answer 1: Thank you very much for this insightful question. In fact, we have already included these experiments in Section 5.4 (Table 3) and Appendix A.9 (Tables 6 and 7) of the manuscript. Due to page-limit constraints, we did not provide a detailed explanation of their setup in the main text. Below, we clarify your questions in detail.

Specifically, in Section 5.4 (Table 3) and Appendix A.9 (Tables 6 and 7), General QQ explicitly incorporates bidirectional label flips between strict preferences and ties (q2,0,q2,1>0q_{2,0}, q_{2,1} > 0 and q0,2,q1,2>0q_{0,2}, q_{1,2} > 0), rather than assuming tie labels are noise-free. To evaluate this setting, we merged the clean strict-preference subset with the native tie subset and applied general ternary label flips across all three classes (y∈{0,1,2}y \in \{0, 1, 2\}). Thus, the noise model explicitly includes flips from strict preferences to ties (q2,0,q2,1>0q_{2,0}, q_{2,1} > 0) and from ties to strict preferences (q0,2,q1,2>0q_{0,2}, q_{1,2} > 0).

We consolidate the results from Tables 3, 6, and 7 of our manuscript into Table R1. As shown in Table R1, General QQ slightly outperforms Simplified QQ across almost all metrics, but incurs a longer training time. In contrast, Simplified QQ achieves comparable alignment performance with substantially shorter training time. This makes Simplified QQ a practical and scalable default in large-scale generative alignment.

Table R1: Comparison between General QQ and Simplified QQ under general ternary noise flips (including Strict↔Tie\text{Strict} \leftrightarrow \text{Tie} transitions) across diffusion and flow-matching backbones.

ModelTransition MatrixTraining Time (h)PS$\uparrow$HPSv2$\uparrow$AES$\uparrow$IR$\uparrow$CLIP$\uparrow$
SD1.5General QQ3522.3028.985.8250.83134.78
Simplified QQ2922.2628.935.8120.83834.72
SDXLGeneral QQ9223.9230.376.4211.17338.54
Simplified QQ8523.8630.346.4471.18838.50
SD3.5-MGeneral QQ1523.4929.926.2011.25237.40
Simplified QQ1023.4429.856.1971.24937.32
FLUX.1-schnellGeneral QQ5223.9130.566.3111.24537.71
Simplified QQ4423.8730.516.3051.23337.60
AuraFlow-v0.3General QQ2723.2029.396.2521.08337.30
Simplified QQ2223.1629.356.2471.10237.23

Question 2: Could the authors provide a comparison with DPO and TPO under different levels of injected label noise? While the current experiments showed sensitivity to the assumed noise ratio, such a comparison would be better to directly demonstrate whether NTPO offers stronger robustness as the actual noise level increases.

Answer 2: Thank you very much for this constructive suggestion. Following your advice, we conducted controlled experiments comparing DPO, TPO, and NTPO across varying true noise levels. Due to time and computational budget constraints during the rebuttal period, we performed these experiments on SD1.5 across three injected noise levels (ηtrue\eta_\text{true} ∈ {0%, 20%, 40%}) using the Cleanlab-filtered Pick-a-Pic v2 dataset (with original tie pairs retained), and evaluated all models on the HPSv2 benchmark.

The quantitative results are summarized in Table R2. As the injected noise level increases from 0% to 40%, both DPO and TPO suffer substantial performance degradation across all metrics, since their objectives treat corrupted labels as ground truth. In contrast, our NTPO exhibits only a slight performance decrease. These results directly demonstrate that NTPO maintains strong robustness under high levels of annotation noise.

Table R2: Robustness comparison of DPO, TPO, and NTPO on SD1.5 evaluated on the HPSv2 benchmark under different injected label noise levels ηtrue\eta_{\text{true}}∈ {0%, 20%, 40%} (Cleanlab-filtered Pick-a-Pic v2 dataset with ties retained). Best results within each noise level are highlighted in bold.

Injected Noise (ηtrue\eta_{\mathrm{true}})MethodPS$\uparrow$HPS$\uparrow$AES$\uparrow$IR$\uparrow$CLIP$\uparrow$
0% (Clean)DPO21.5727.885.7850.47933.97
TPO22.2528.935.8380.82734.73
NTPO (Ours)22.4929.245.9530.96834.96
20% NoiseDPO21.3327.535.5310.35333.67
TPO22.0328.785.6980.71134.56
NTPO (Ours)22.2628.935.8120.83834.72
40% NoiseDPO20.2624.755.1810.21632.51
TPO21.0826.365.4190.53333.07
NTPO (Ours)22.1028.805.7030.79434.53

Weakness 1: The paper only reports evaluation results at fixed training steps. Reward curve on the eval set during the training process is needed.

Answer 1: Thank you very much for pointing out this limitation and for this constructive suggestion. To demonstrate the training dynamics across different optimization stages, we report the evaluation results at intermediate checkpoints in the added tables. Specifically, Table R3 presents the results of SD1.5 and SDXL on the HPSv2 benchmark, and Table R4 reports the corresponding results of SD1.5 and SDXL on the PartiPrompt benchmark. Furthermore, Table R5 provides the results of flow-matching models (SD3.5-M, FLUX.1-schnell, and AuraFlow-v0.3) on the HPSv2 benchmark, while Table R6 reports the results of these flow-matching models on the PartiPrompt benchmark.

Table R3: Evaluation dynamics (PS, HPS, AES, IR, CLIP) across intermediate training checkpoints (Steps 500, 1000, 1500, 2000) on the HPSv2 benchmark for diffusion models (SD1.5 and SDXL).

ModelMethodStep 500Step 1000Step 1500Step 2000
PS↑HPS↑AES↑IR↑CLIP↑PS↑HPS↑AES↑IR↑CLIP↑PS↑HPS↑AES↑IR↑CLIP↑PS↑HPS↑AES↑IR↑CLIP↑
SD1.5DPO21.1126.935.4150.20533.1521.2027.255.4670.28533.2821.2527.455.5050.33133.4821.3327.535.5310.35333.67
DMPO20.2326.455.3460.51531.3820.9627.375.5080.60632.6221.5828.125.6160.65034.0121.9728.705.7080.70534.91
α\alpha-DPO21.3426.785.4530.53233.2721.6127.605.5760.60134.1121.9328.225.6220.64834.8922.0928.715.7090.69735.35
TPO21.5827.545.5810.54433.4921.7828.105.6410.61834.0121.9228.495.6680.67034.2422.0328.785.6980.71134.56
NTPO (Ours)21.7127.695.6180.61733.6121.9528.255.7160.72034.1222.1028.705.7650.78934.3922.2628.935.8120.83834.72
SDXLDPO23.1029.256.0700.70336.2023.2429.536.1230.84136.8123.3629.706.1570.93637.4523.4729.846.1791.00337.80
DMPO22.9028.335.9640.93737.1823.1829.066.0441.01437.6223.4129.746.1221.06437.9523.5330.196.1661.11538.22
α\alpha-DPO23.0528.545.9110.95137.2923.2229.126.0641.02837.8423.3629.626.1631.07238.2023.4329.826.2621.11238.43
TPO23.1728.855.9351.01937.4123.3729.416.0871.07937.7723.5529.726.2111.11037.9523.6530.146.2851.14738.13
NTPO (Ours)23.4029.126.0151.12637.6323.5929.736.2191.15538.0423.7630.066.3581.17138.2823.8630.346.4471.18838.50

Comments on the paper

1 comment

Summary Response

Dear Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

We sincerely thank you for your time and effort throughout the review process, with special thanks to reviewers R4TP, fJps, and 4TT5 for their insightful feedback.

Following our rebuttal, Reviewer R4TP indicated that our response had resolved their concerns and maintained their original score of 3. Reviewer fJps also confirmed that our response had addressed their concerns and increased their score from 2 to 3. Reviewer 4TT5 acknowledged that our response had resolved their concerns and increased their score from 1 to 3.

Below, we summarize the strengths recognized in the reviews and clarify the main points raised.

Summary of Recognized Strengths

  • Writing: Clear, Well-justified Motivation: All reviewers highlighted that jointly addressing tied preference pairs and human annotation noise targets practical bottlenecks in text-to-image preference alignment.
  • Theoretical Contributions: Solid Theoretical Grounding: All reviewers commended the clarity and mathematical rigor of our Rao–Kupper noisy transition formulation and variational surrogate derivations.
  • Performance: Superior to Baselines: All reviewers recognized that the reported comparisons with binary and ternary baselines support NTPO's effectiveness.
  • Evaluation: Comprehensive Experimental Validation: All reviewers acknowledged that our extensive experiments across multiple benchmarks and generative backbones effectively validate the proposed method.

We hope our clarifications and revisions address the reviewers' concerns. Thank you again for your time and for considering our submission and responses.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.