- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents an interesting approach to dynamic-margin DPO by exploiting an often-overlooked piece of metadata in large-scale preference datasets such as Pick-a-Pic v2 and HPD: the source model of each image.
ArenaPO first builds a model-level arena from pairwise preferences, representing each model's capability with a Gaussian distribution updated through Bayesian inference. It then estimates the expected quality gap between preferred and rejected images using a truncated Gaussian distribution, and incorporates this gap as a dynamic margin into Diffusion-DPO.
Experiments on SD 1.5 and SDXL show improvements over several existing preference optimization methods, supported by additional ablation studies.
I find the idea clever and inspiring, especially its use of existing metadata to extract richer supervision from binary preferences. However, its strong reliance on source-model identity also raises concerns about its generality.
Critical strengths
Simple yet interesting idea. The paper cleverly exploits an often-overlooked piece of metadata, the source model of each image, to construct a dynamic-margin DPO method. The idea is intuitive, easy to implement, and empirically effective.
Comprehensive experiments. The authors conduct training on two preference datasets (Pick-a-Pic v2 and HPD v3) with two diffusion backbones (SD 1.5 and SDXL), and evaluate on three benchmarks using multiple metrics. The results provide solid empirical support for the effectiveness of the method.
Critical weaknesses
Limited generality. The method heavily relies on source-model identity, which is not always available or informative in modern preference datasets. For example, the recently released HPDv3++ dataset constructs its main preference pairs main from Qwen-Image generations. In such a single-source setting, ArenaPO doesn't work, losing its core advantage of adaptive margins. Moreover, as the quality gap between frontier generative models narrows, it remains unclear how much useful information source-model identity alone can provide.
Questions for authors
Qualitative results for SDXL. I noticed that all qualitative comparisons, including those in the appendix, are based on SD 1.5. Could the authors also provide qualitative results for SDXL, ideally covering both training datasets?