acceptodds
Withdrawn from ICLR 2027

RatioDiff: Aligning Diffusion Models via Preference Ratio Matching

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion models have achieved remarkable success in text-to-image generation, producing diverse and visually compelling images from natural language prompts. Preference learning further aligns these models with human judgments of image quality and prompt adherence. However, effective preference alignment faces two challenges: weak corrective signals on strongly misranked pairs and difficulty balancing preference improvement with denoising preservation. We introduce RatioDiff, an offline framework centered on ratio diffusion preference learning. Our core contribution is a preference ratio formulation that explicitly accounts for distinct noisy diffusion states through a reference correction. This formulation leads to a tractable Bregman objective that sustains corrective pressure on misranked pairs, providing a principled foundation for diffusion preference optimization. To support this objective, we introduce learned pair weights guided by reference consistency, separating the strength of preference correction from each pair's contribution to training. We further develop adaptive regularization that controls reference proximity and preferred-image denoising relative to the preference update, limiting model drift while preserving denoising behavior. Together, these contributions combine preference ratio matching with explicit control over pair influence and regularization, without additional inference overhead. Experiments on multiple text-to-image backbones and preference benchmarks show that RatioDiff consistently improves human-preference alignment while preserving and enhancing visual quality and text–image consistency.

Primary area:
reinforcement learning
Keywords:
Diffusion Models, Preference Learning, Ratio Matching

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bSBiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RatioDiff, an offline method for aligning diffusion models with human preferences. It learns preference ratios between denoising transitions, with a reference correction that accounts for their different noisy states. And a Bregman objective is used to preserve the learning signal for pairs that the model ranks incorrectly with high confidence. The method also learns pair weights from the reference model and adjusts regularization during training to limit model drift and preserve denoising quality. Experiments on SD1.5 and SDXL across three prompt benchmarks show improvements in preference scores, visual quality, and text–image alignment.

Critical strengths
  1. The motivation is clear. The method addresses the need to correct incorrectly ranked pairs while preserving denoising quality. It separates the preference objective from pair weighting and regularization, allowing the contribution of each pair and the strength of regularization to be controlled separately.
  2. The paper provides extensive theoretical analysis with detailed derivations and proofs. The presentation is clear and easy to follow.
Critical weaknesses
  1. The baseline comparisons are not convincing enough. Comparisons with methods such as SPO, InPO, and IAPO are missing, making it difficult to assess how competitive RatioDiff is beyond established baselines. Furthermore, some gains over the reported baselines are also small. For example, compared with Diffusion-DPO on PartiPrompt with SDXL, ImageReward increases from 1.0788 to 1.0804 and CLIP from 0.3660 to 0.3671. Without results across multiple seeds or confidence intervals, these small margins provide limited evidence of a reliable advantage.
  2. The ablations do not show how much the proposed ratio objective contributes to the results. Table 3 only removes adaptive regularization or adaptive preference scaling. There is no ablation of the reference correction and no comparison with DPO using the same weighting and regularization. These comparisons are needed to distinguish the benefit of the core objective from that of the added training components.
Questions for authors
  1. Add comparisons with SPO, InPO, IAPO or any other baselines under comparable data and compute budgets.
  2. What does each component contribute to the reported gains? Provide an ablation of the reference correction, a comparison with DPO using the same auxiliary components, and separate tests of the two regularizers.
  3. Does RatioDiff retain its benefits on flow-based models such as SD3 and FLUX? Support this with experimental results and explain any changes needed to accommodate their prediction parameterizations.
Reviewer_g8uwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

RatioDiff ports Bregman density-ratio preference matching (BPO/TokenRatio) to diffusion denoising transitions. It derives a reference-corrected transition ratio with a KL correction for the two distinct noisy states, reduces it under Gaussian transitions to an inner-product margin between noise-prediction residuals, and optimizes a Basu-generator Bregman loss whose gradient does not vanish on misranked pairs. Two auxiliary mechanisms complete the method: a small confidence head that weights each noisy pair by predicted reference-consistency, and two regularizers (reference-KL anchoring and preferred-image denoising) whose coefficients are set per pair so that their gradient norms stay within a fixed fraction of the preference gradient. Experiments on SD1.5 and SDXL trained on Pick-a-Pic v2 report five automatic metrics on three prompt sets against SFT, Diffusion-DPO, KTO, DMPO, DSPO and MaPO, with two ablations.

Critical strengths
  1. The paper identifies useful distinctions between preference correction, pair weighting, and regularization. Its derivation of state-dependent reference corrections and analytic output-gradient norms is potentially informative.
  2. The method remains offline and introduces no inference-time components. Experiments cover two backbones and several prompt distributions, with particularly consistent improvements in HPSv2 and ImageReward.
Critical weaknesses
  1. The reported implementation differs materially from the proposed algorithm. Eqs. (23)–(27) and Appendix B describe independent forward noises, a posterior-averaged ratio, and per-pair output-gradient budgets. Appendix C.2 instead states that the implementation samples posterior actions, shares forward noise within pairs, and computes auxiliary coefficients from gradients aggregated across GPUs. These changes affect the stochastic objective and the applicability of Proposition 4’s local bound; they are not merely engineering details. Appendix C.3 also removes the winner auxiliary for SDXL. The paper therefore does not clearly establish which proposed guarantees and components its results validate.
  2. The link from image-level preferences to transition-level ratio learning is not established. Eq. (7) represents a trajectory preference probability as a product of local Bradley–Terry probabilities. Under the natural binary interpretation, two locally tied comparisons assign probability 1/41/4 to each global orientation, which does not normalize to one. Interpreting the product as the event that all local comparisons agree would require a different observation model from the supplied clean-image preferences. Appendix A.3 invokes ordered-comparison sampling without showing that noisy transitions generated from these image pairs satisfy it. Thus, the propositions do not yet justify the actual supervision scheme, even if their local identities hold under the stated idealizations.
Questions for authors
  1. Which exact objective and gradient-budget implementation produced each result, and can the main-text algorithm be evaluated separately?
  2. What normalized comparison model makes Eq. (7) and Appendix A.3 applicable to clean-image preference labels after forward noising?
  3. How much benefit remains over DPO when weights, initial update scale, auxiliary terms, and the controller are matched?
Reviewer_hCChOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RatioDiff, an offline preference alignment framework for text-to-image diffusion models. The method derives a reference-corrected preference ratio for transitions conditioned on different noisy states and optimizes it using a Bregman ratio-matching objective. It further introduces learned reference-consistency weights and adaptive regularization to control reference drift and preserve preferred-image denoising. Experiments on SD1.5 and SDXL show competitive results across several automated preference and image-quality metrics.

Critical strengths
  • Potentially meaningful ratio formulation. Explicitly accounting for the different conditioning states of preferred and rejected diffusion transitions is a technically motivated contribution. The reference correction provides a potentially useful extension of preference ratio matching to diffusion models.

  • Careful attention to optimization behavior. The framework distinguishes preference correction, pair influence, and denoising preservation. Controlling auxiliary gradients relative to preference gradients is a reasonable approach to balancing these objectives.

  • Competitive automated evaluation results. RatioDiff improves over the pretrained models across the reported metrics and performs particularly well on HPSv2, CLIP, and ImageReward. The reported ablations also support the usefulness of the complete framework.

Critical weaknesses
  • Evaluation is restricted to older backbones. Both SD1.5 and SDXL are earlier UNet-based diffusion models. Results on these backbones provide limited evidence that the method remains effective for stronger DiT-based and flow-matching models.

  • Insufficient differentiation from closely related work. Diffusion-SDPO [1] already identifies that increasing the preference margin can degrade preferred-image denoising and introduces an adaptive gradient safeguard. Although RatioDiff uses a different mechanism, its motivation around adaptive denoising preservation substantially overlaps with this work. The paper neither discusses nor directly compares against Diffusion-SDPO, leaving the additional value of its regularization mechanism unclear. The strongest potential novelty appears to lie in the reference-corrected ratio formulation; several other components are incremental combinations of established ideas.

  • Ablations do not isolate the central contributions. Removing adaptive regularization or preference scaling as a group does not establish the individual value of the state-dependent correction, Bregman objective, learned weights, and gradient-budget controller. Particularly important comparisons include removing the ratio correction, replacing learned weights with simple reference-error-based weights

  • Limited empirical gains over existing baselines. Although RatioDiff achieves the best scores on several metrics, the improvements over strong baselines are often modest and inconsistent across metrics. On SD1.5, SFT or Diffusion-KTO obtains higher PickScore, while several aesthetic scores are nearly tied. On SDXL, Diffusion-DPO consistently achieves higher PickScore, and MaPO consistently achieves higher aesthetic scores across all three benchmarks. Some reported gains are marginal—for example, ImageReward on PartiPrompt increases from 1.0788 with Diffusion-DPO to 1.0804 with RatioDiff. Without uncertainty estimates, these results provide limited evidence that the additional complexity yields a substantial and reliable advantage.

  • Limited evidence on robustness and human preference. The evaluation primarily relies on automated scores, without reported uncertainty estimates or direct human preference comparisons.

[1] Fu M, Wang GH, Cui T, Chen QG, Xu Z, Luo W, Zhang K. Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Models. InEuropean Conference on Computer Vision 2026 Sep 8 (pp. 468-484). Cham: Springer Nature Switzerland.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.