acceptodds
Withdrawn from ICLR 2027

Arena as Offline Reward: Efficient Fine-Grained Preference Optimization for Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning from human feedback (RLHF) effectively promotes preference alignment of text-to-image (T2I) diffusion models. To improve computational efficiency, direct preference optimization (DPO), which avoids explicit reward modeling, has been widely studied. However, its reliance on binary feedback limits it to coarse-grained modeling on chosen–rejected pairs, resulting in suboptimal optimization. In this paper, we propose ArenaPO, which leverages Arena scores as offline rewards to provide refined feedback, thus achieving efficient and fine-grained optimization without a reward model. This enables ArenaPO to benefit from both the rich rewards of traditional RLHF and the efficiency of DPO. Specifically, we first construct a model Arena in which each model’s capability is represented as a Gaussian distribution, and infer these capabilities by traversing the annotated pairwise preferences. Each output image is treated as a sample from the corresponding capability distribution. Then, for a image pair, conditioned on the two capability distributions and the observed pairwise preference, the absolute quality gap is estimated using latent-variable inference based on truncated normal distribution, which serves as fine-grained feedback during training. It does not require a reward model and can be computed offline, thus introducing no additional training overhead. We conduct ArenaPO training on Pick-a-Pic v2 and HPD v3 datasets, and the results on various T2I benchmarks demonstrate that ArenaPO consistently outperforms existing baselines.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Preference Optimization, Offline Reward, Model Arena, Diffusion Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wUufOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents an interesting approach to dynamic-margin DPO by exploiting an often-overlooked piece of metadata in large-scale preference datasets such as Pick-a-Pic v2 and HPD: the source model of each image.

ArenaPO first builds a model-level arena from pairwise preferences, representing each model's capability with a Gaussian distribution updated through Bayesian inference. It then estimates the expected quality gap between preferred and rejected images using a truncated Gaussian distribution, and incorporates this gap as a dynamic margin into Diffusion-DPO.

Experiments on SD 1.5 and SDXL show improvements over several existing preference optimization methods, supported by additional ablation studies.

I find the idea clever and inspiring, especially its use of existing metadata to extract richer supervision from binary preferences. However, its strong reliance on source-model identity also raises concerns about its generality.

Critical strengths

Simple yet interesting idea. The paper cleverly exploits an often-overlooked piece of metadata, the source model of each image, to construct a dynamic-margin DPO method. The idea is intuitive, easy to implement, and empirically effective.

Comprehensive experiments. The authors conduct training on two preference datasets (Pick-a-Pic v2 and HPD v3) with two diffusion backbones (SD 1.5 and SDXL), and evaluate on three benchmarks using multiple metrics. The results provide solid empirical support for the effectiveness of the method.

Critical weaknesses

Limited generality. The method heavily relies on source-model identity, which is not always available or informative in modern preference datasets. For example, the recently released HPDv3++ dataset constructs its main preference pairs main from Qwen-Image generations. In such a single-source setting, ArenaPO doesn't work, losing its core advantage of adaptive margins. Moreover, as the quality gap between frontier generative models narrows, it remains unclear how much useful information source-model identity alone can provide.

Questions for authors

Qualitative results for SDXL. I noticed that all qualitative comparisons, including those in the appendix, are based on SD 1.5. Could the authors also provide qualitative results for SDXL, ideally covering both training datasets?

Reviewer_Vm18OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Compared with the commonly used DPO, this paper proposes ArenaPO, which aims to leverage the Arena scores of generation models as offline rewards to provide refined feedback instead of using binary feedback only. Specifically, the proposed approach first constructs a model Arena and assumes that the model capacity follows Gaussian distribution. As such, given an image pair sampled from two models, the quality gap between is estimated by the latent-variable inference on truncated normal distribution. Experimental results on Pick-a-Pic v2 and HPD v3 datasets demonstrate that the proposed ArenaPO can improve several baselines, including Diffusion-DPO, MaPO, DSPO, and SDPO.

Critical strengths
  • S1. The proposed approach is reasonable, which leverages the information that is useful but has not been leveraged before to improve the model. The proposed approach does not require additional reward model and is compatible to widely-used DPO.
  • S2. The definition of quality gap estimator and margin-based objective is clear and easy to follow.
  • S3. Experimental results are conducted on two backbones and show consistent improvement.
Critical weaknesses
  • W1. The novelty on theoretical results are somewhat limited. Moreover, the improvement is relatively small, especially on SDXL. Although the improvement in this field is usually small, this paper leverages additional signals so it is expected to show more improvement. It is also suggested to conduct experiments on the shuffled margin control and uncertainty estimation using multi-seed.
  • W2. For ICLR 2027, current image generation applications usually use DiT-based backbones, such as SD3.5 and FLUX. It is suggested to conduct the experiments using new backbones.
  • W3. The proposed quality gap is obtained from the capability distributions of the source models. However, this paper does not compare against a simple baseline that assigns preferred and rejected images based on source-model rankings. It is also suggested to compare with the baseline that uses the difference between source-model capability means as a preference margin while retaining the original human labels.
  • W4. Although this paper emphasizes that ArenaPO introduces no additional training overhead, the paper does not report preprocessing time or total GPU hours. Constructing the model Arena and estimating quality gaps require additional offline computation. A breakdown of preprocessing and training costs would help the readers to verify the efficiency claims relative to standard Diffusion-DPO.
Questions for authors
  • Q1. It is suggested to compare the proposed ArenaPO with the baselines mentioned in W3. Moreover, results on SD3.5 or FLUX.1-dev are welcome.
  • Q2. It is suggested to analyze the correlation between the estimated gap and human preference-strength, together with the shuffled, source-only, and random margin controls.
Reviewer_jzWQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

ArenaPO augments Diffusion-DPO with a per-pair margin derived from the identity of the generators that produced the two images in each preference pair. Generator capability is modeled as a Gaussian updated with K-Sort-Arena-style Bayesian rules over the pairwise labels; each image is treated as a draw from its generator’s capability distribution, and the expected quality gap conditioned on the observed ordering is obtained in closed form from the truncated-normal mean. This gap, scaled by a coefficient, is inserted as a margin in the Diffusion-DPO logit, requiring no reward model and no extra training cost. Experiments on SD1.5 and SDXL trained on Pick-a-Pic v2 and HPD v3 report small, consistent gains over Diffusion-DPO, MaPO, DSPO, and SDPO on five automatic metrics across three prompt sets, with ablations on uniform margins, the scale coefficient, combination with DSPO, an explicit-reward comparison, and K-Sort Eval.

Critical strengths
  1. The method makes practical use of source-model metadata and computes margins offline, avoiding additional reward-model evaluations during fine-tuning.
  2. Evaluation spans two backbones and two training datasets. Fixed-margin controls and the ArenaPO-plus-DSPO experiment provide useful evidence that the proposed margins can improve existing objectives.
Critical weaknesses
  1. The quality-gap model conflates capability uncertainty with image-level variability. Appendix A explicitly distinguishes uncertainty in estimated capability, σ2\sigma^2, from single-instance performance variation, β2\beta^2. However, Eqs. (10)–(12) sample individual image qualities using only the capability distributions and set the difference variance to σw2+σl2\sigma_w^2+\sigma_l^2. Under the appendix’s independent Gaussian assumptions, predictive image-quality variance should also account for the β\beta terms. Greater certainty about average capability does not eliminate variation among a model’s outputs. This inconsistency directly affects the margins and undermines their posterior-quality interpretation. The capability/performance distinction is also standard in Bayesian skill-rating models.
  2. The claimed fine-grained supervision is not validated at the image-pair level. Once the Arena is fitted, Eq. (12) assigns the same margin to every pair with the same ordered source-model identities, regardless of prompt or image content. It therefore estimates a group-level conditional expectation, rather than the realized quality difference of an individual pair. No comparison against independently annotated preference strength establishes its reliability. The fixed-margin ablation does not distinguish the proposed latent model from simpler source-pair weighting or empirical win-rate margins. Moreover, the shifted logistic loss encourages exceeding a margin rather than matching a numerical quality gap.
  3. The evaluation does not establish the claimed improvement in human alignment. Figure 4’s 79.0% win rate averages five automated evaluators against the pretrained model; it is neither a human win rate nor a direct comparison against the strongest fine-tuned baseline. Several gains over strong baselines are small, without training-seed variability or paired confidence intervals.
Questions for authors
  1. Why are the single-instance β\beta terms omitted from Eq. (10), and how would corrected predictive-variance margins change the results?
  2. Does ArenaPO outperform source-pair win-rate margins under matched tuning, and do its estimated gaps correlate with independently measured preference strength?
  3. What is the direct human preference advantage over the strongest fine-tuned baseline, with uncertainty estimates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.