acceptodds
Under review as a conference paper at ICLR 2027

Relative Density Ratio Optimization for Stable and Statistically Consistent Model Alignment

Abstract

Aligning language models with human preferences is essential for ensuring their safety and reliability. Although most existing approaches assume specific human preference models such as the Bradley-Terry model, this assumption may fail to accurately capture true human preferences, and consequently, these methods lack statistical consistency, i.e., the guarantee that language models converge to the true human preference as the number of samples increases. In contrast, direct density ratio optimization (DDRO) achieves statistical consistency without assuming any human preference models. DDRO models the density ratio between preferred and non-preferred data distributions using the language model, and then optimizes it via density ratio estimation. However, this density ratio is unstable and often diverges, leading to training instability of DDRO. In this paper, we propose a novel alignment method that is both stable and statistically consistent. Our approach is based on the relative density ratio between the preferred data distribution and a mixture of the preferred and non-preferred data distributions. Our approach is stable since this relative density ratio is bounded above and does not diverge. Moreover, it is statistically consistent and yields significantly tighter convergence guarantees than DDRO. We experimentally show its effectiveness with Qwen 2.5 and Llama 3.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Alignment, Relative Density Ratio Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qcfwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RDRO, a pointwise LLM alignment method based on the density ratio between preferred responses and a preferred/non-preferred mixture. It derives a simple softplus objective with bounded per-sample gradient coefficients and provides analyses of consistency and finite-sample error. Experiments on four models and two preference datasets evaluate alignment quality, reasoning performance, parameter sensitivity, and training dynamics.

Critical strengths
  1. Clear motivation and substantial theoretical analysis. The relative-ratio formulation addresses a concrete limitation of DDRO. The loss derivation, conditional population-optimum result, and bounded-coefficient analysis provide useful theoretical support for the objective and its optimization behavior.
  2. A reasonably broad experimental evaluation. Four models, two datasets, and three training seeds provide useful evidence beyond a single setting. The gains on UF-G, competitive results on MIX-14K, and additional BBH, sensitivity, and training-dynamics analyses support the method's practical value.
Critical weaknesses

The finite-sample guarantee needs further justification for the implemented estimator. Appendix D's Eq.49 analyzes an empirical Bregman risk containing the unknown r∗r^*, while training minimizes the computable objective in Eq.17. Their population objectives agree up to a parameter-independent constant, but their finite-sample minimizers need not coincide. Clarifying how the proof covers Eq.17, or directly analyzing that estimator, would strengthen the consistency and convergence claims.

Questions for authors
  1. See the weakness above.
  2. Since α\alpha is central to the reference-mixture assumption, I am curious about the reported robustness to its value. Could you report results for α=0\alpha=0 and values close to or equal to 1, and discuss what these results reveal about α\alpha's role?
3 comments

Thank you for your insightful comments, which we shall address below.

Weaknesses: The finite-sample guarantee needs further justification for the implemented estimator. Appendix D's Eq.49 analyzes an empirical Bregman risk containing the unknown r∗r^{*}, while training minimizes the computable objective in Eq.17. Their population objectives agree up to a parameter-independent constant, but their finite-sample minimizers need not coincide. Clarifying how the proof covers Eq.17, or directly analyzing that estimator, would strengthen the consistency and convergence claims.

As you noted, the minimizers of Eqs. (17) and (49) need not coincide for finite samples. Following DDRO, we analyze the Bregman divergence for any strongly convex function ff in Appendix D. Substituting f(t)=tlog⁡t−(1+t)log⁡(1+t)f(t)=t\log t-(1+t)\log(1+t), which we use in our approach, into Eq. (48) gives Eq. (16) plus a constant independent of the parameters. We omit this constant and minimize Eq. (17), the empirical approximation of Eq. (16). Since the identity Epref[r∗t]=Ep+[t]\mathbb{E}_{p_{\mathrm{ref}}}[r^{\ast}t]=\mathbb{E}_{p^{+}}[t] used in this derivation does not generally hold for sample averages, substituting the same function into Eq. (49) does not generally give Eq. (17). We will clarify this distinction in the revised paper.

To address this, we directly analyze the minimizer of Eq. (17), using Eqs. (16) and (17) as the true risk and empirical risk, respectively. Following the empirical risk minimization and uniform deviation analysis in Appendix D, the estimation error is bounded above as follows:

\mathbb{E}_{\mathcal{D}^{\pm}}\left[\mathbb{E}_{p^{+}}\left[\left(p_{\hat{\theta}}-p^{+}\right)^{2}\right]\right]\leq\frac{4}{\alpha}\left[\inf_{\theta\in\Theta}\mathcal{L}_{\mathrm{RDRE}}(\theta)+4\left\\{\mathcal{R}_{N}^{+}(\mathcal{T})+(1-\alpha)\mathcal{R}_{M}^{-}(\mathcal{T})\right\\}\right].

Here, θ^\hat{\theta} minimizes Eq. (17), and \mathcal{T}=\left\\{T_{\theta}=\log p_{\theta}-\log p_{\mathrm{ref}}\mid\theta\in\Theta\right\\} is the corresponding model class. RN+\mathcal{R}_{N}^{+} and RM−\mathcal{R}_{M}^{-} are the Rademacher complexities of T\mathcal{T} under the preferred and non-preferred data distributions, respectively.

Please see Appendix J (Theorem J.1) for the complete theorem, assumptions, and proof. This yields a bound of the same form as Eq. (57), establishing statistical consistency for the minimizer of the actual training objective.

Questions 1: See the weakness above.

Please see our response to the weakness above.

Questions 2: Since α\alpha is central to the reference-mixture assumption, I am curious about the reported robustness to its value. Could you report results for α=0\alpha=0 and values close to or equal to 1, and discuss what these results reveal about α\alpha's role?

At α=0\alpha=0, the mixture model gives pref=p−p_{\mathrm{ref}}=p^{-}; at α=1\alpha=1, it gives pref=p+p_{\mathrm{ref}}=p^{+}. These endpoints are outside our problem setting, which assumes 0<α<10<\alpha<1. Figures 8(a) and 8(b) in Appendix I show the loss functions for these endpoints. For α=0\alpha=0, the loss for preferred samples has no finite minimizer and encourages an increase in Tθ=log⁡pθ(y∣x)−log⁡pref(y∣x)T_{\theta}=\log p_{\theta}(y|x)-\log p_{\mathrm{ref}}(y|x). For α=1\alpha=1, the loss for non-preferred samples vanishes, while the loss for preferred samples is minimized at pθ(y∣x)=pref(y∣x)p_{\theta}(y|x)=p_{\mathrm{ref}}(y|x).

Therefore, α\alpha can be interpreted as a regularization parameter that controls deviations from the reference policy. For 0<α<10<\alpha<1, the loss for preferred samples is minimized at Tθ=log⁡(1/α)T_{\theta}=\log(1/\alpha), and this minimum moves closer to the reference policy as α\alpha increases. At α=0\alpha=0, the finite minimum that provides this control disappears; at α=1\alpha=1, the minimum corresponds to the reference policy itself. Figures 1 and 2 further show that RDRO outperforms DDRO on UF-G and remains comparable on MIX-14K for α=0.1,…,0.9\alpha=0.1,\ldots,0.9, including values near both endpoints.

Thank you for the clarification and additional analyses. I will maintain my score.

Thank you for your response and for considering our additional analyses. We appreciate your feedback.

Reviewer_PqRpOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Relative Density Ratio Optimization (RDRO), an alignment method that achieves both training stability and statistical consistency by modeling the ratio between preferred data and a mixture of preferred and non-preferred distributions. Unlike Direct Density Ratio Optimization (DDRO), which suffers from instability due to potentially diverging density ratios, RDRO leverages a bounded relative density ratio to prevent gradient explosions and provide tighter convergence guarantees. Theoretical analysis demonstrates that RDRO maintains statistical consistency without assuming specific human preference models, while experiments on Qwen 2.5 and Llama 3 models show that it outperforms or matches existing methods like KTO and DDRO in terms of performance and robustness.

Critical strengths
  1. The writing is well-presented.
  2. The motivation is well-justified, and the proposed solution provides relevant theoretical guarantees.
Critical weaknesses
  1. The paper claims statistical consistency but lacks empirical validation of convergence behavior as dataset size increases. To strengthen this core theoretical motivation, the authors should include a scaling experiment showing how RDRO’s performance error changes with varying sample sizes (e.g., 10%, 50%, 100% of the data) and compare this trend against inconsistent methods like DPO to demonstrate its superior convergence properties in data-rich regimes.

  2. The method relies on the strong assumption that the reference policy is a mixture of preferred and non-preferred distributions with a constant α\alpha. The current experiments do not sufficiently address the impact of violating this assumption when the true preference probability varies significantly across prompts. The authors should provide a deeper analysis or ablation study on how α\alpha affects performance.

Questions for authors

Please refer to "Critical Weaknesses".

3 comments

Thank you for your insightful comments, which we shall address below.

Weaknesses 1: The paper claims statistical consistency but lacks empirical validation of convergence behavior as dataset size increases. To strengthen this core theoretical motivation, the authors should include a scaling experiment showing how RDRO’s performance error changes with varying sample sizes (e.g., 10%, 50%, 100% of the data) and compare this trend against inconsistent methods like DPO to demonstrate its superior convergence properties in data-rich regimes.

Following your suggestion, we conducted additional experiments using Qwen2.5-3B-Instruct on 10%, 50%, and 100% of UF-G. We compare RDRO with DDRO and KTO in the same pointwise setting. RDRO and DDRO guarantee statistical consistency, whereas KTO, which can be regarded as a generalization of DPO, does not. The subsets are nested and preserve the label proportions. Training settings follow Section 5.

The table reports AlpacaEval LC win rates (%, mean ±\pm standard deviation over three seeds). The paper uses weighted_alpaca_eval_gpt4_turbo, whose underlying model (gpt-4-1106-preview) has since been retired from the API. We therefore use GPT-4.1 for this additional experiment, retaining the same evaluation prompt and GPT-4 Turbo reference responses. These scores are not directly comparable to those reported in the paper because the judge has changed.

Method10%50%100%
RDRO13.93±0.4813.93\pm0.4814.55±0.3114.55\pm0.3114.71±0.2114.71\pm0.21
DDRO13.44±0.4713.44\pm0.4713.62±0.6913.62\pm0.6913.82±0.6813.82\pm0.68
KTO14.43±0.3114.43\pm0.319.90±0.109.90\pm0.1010.50±0.2310.50\pm0.23

RDRO improves as the dataset size increases and achieves a higher mean LC win rate than DDRO at every tested data size. In contrast, KTO does not guarantee statistical consistency and behaves differently as the dataset size increases. In this experiment, KTO achieves its highest mean LC win rate with 10% of the data and lower scores with 50% and 100%. With 100% of the data, RDRO achieves a higher mean LC win rate than KTO at all tested data sizes. These results complement the theoretical consistency and convergence analyses in Sections 3.3 and 3.4. We will include these results in the revised paper.

Weakness 2: The method relies on the strong assumption that the reference policy is a mixture of preferred and non-preferred distributions with a constant α\alpha. The current experiments do not sufficiently address the impact of violating this assumption when the true preference probability varies significantly across prompts. The authors should provide a deeper analysis or ablation study on how α\alpha affects performance.

Figures 9(a) and 9(b) in Appendix K show predictions of p(+∣x)p(+|x) for individual prompts in UF-G and DPO-MIX-7K (which shares the same set of unique prompts as MIX-14K) obtained using the OpenAI Decisions API. Although the predicted probabilities are generally high, they vary across prompts, suggesting that p(+∣x)p(+|x) may depend on the prompt. Hence, the assumption that p(+∣x)=αp(+|x)=\alpha for a constant α\alpha may not hold for these datasets.

Nevertheless, Tables 1 and 2 show that RDRO achieves strong performance using a fixed α\alpha on these datasets. Figures 1 and 2 further show that RDRO maintains good performance for various values of α\alpha.

Thank you for your reply, I will maintain my score.

Thank you for your response and for considering our additional analyses. We appreciate your feedback.

Reviewer_TZzeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose a relative density ratio optimization approach for stable and statistically consistent model alignment. The approach is based on existing relative density ratio estimation, which can incur more stable and consistent alignment. The authors conduct experimentation with Qwen 2.5 and Llama 3.

Critical strengths
  1. The unstable problem is important for LLM alignment.
  2. The idea of exploiting relative density ratio estimation is interesting.
  3. The theoretical analysis is interesting.
Critical weaknesses
  1. The authors have a strong assumption, i.e., f is strictly convex. However, it is complicated to ensure the convexity of LLMs.
  2. Then novelty of the paper is restricted. Relative density ratio estimation already exists.
  3. The advantage of RDRO is not obvious as shown in Table 2, in which DPO and SimPO may be better on Qwen.
  4. Some related stable alignment approaches are not analyzed.
Questions for authors
  1. The authors are encouraged to conduct experimentation on recent LLMs and large LLMs.
  2. I wonder how to ensure that f is strictly convex for LLMs.
  3. Some recent stable alignment approaches can be analyzed.
3 comments

Thank you for your insightful comments, which we shall address below.

Weaknesses 1: The authors have a strong assumption, i.e., f is strictly convex. However, it is complicated to ensure the convexity of LLMs.

We do not assume the convexity of LLMs. ff is the scalar function defining the Bregman divergence. Both DDRO (Section 2.2) and RDRO (Section 3.1) use f(t)=tlog⁡t−(1+t)log⁡(1+t)f(t)=t\log t-(1+t)\log(1+t), which is strictly convex because f′′(t)=1/[t(1+t)]>0f''(t)=1/[t(1+t)]>0 for t>0t>0. We will clarify this in the revised paper.

Weaknesses 2: Then novelty of the paper is restricted. Relative density ratio estimation already exists.

As you noted, relative density ratio estimation was introduced by Yamada et al. (2013). Our novelty lies not in the estimator itself but in its nontrivial application to LLM alignment by extending DDRO. Specifically, the novelty of our approach can be summarized as follows:

  • We propose a novel alignment method called relative density ratio optimization (RDRO), which is based on relative density ratio estimation. We derive a computable objective using the softplus function (Section 3.1) and show that the gradient coefficient for each sample is bounded, contributing to training stability (Section 3.2).
  • We theoretically show that RDRO is statistically consistent (Section 3.3) and yields significantly tighter convergence guarantees than DDRO (Section 3.4).

In addition, we demonstrate the effectiveness of RDRO with Qwen 2.5 and Llama 3. We will clarify this point in the revised paper.

Weaknesses 3: The advantage of RDRO is not obvious as shown in Table 2, in which DPO and SimPO may be better on Qwen.

As noted in lines 460–463 of Section 5.3, the methods in Table 2 are not directly comparable since DPO and SimPO use stronger (pairwise) supervision: (x,y+,y−)(x, y^{+}, y^{-}), while KTO, DDRO and RDRO use weaker (pointwise) supervision: (x,y+)(x, y^{+}) or (x,y−)(x, y^{-}). Note that we convert pairwise supervision into pointwise supervision for KTO, DDRO, and RDRO. Nevertheless, RDRO achieves performance comparable to or better than DPO and SimPO on Llama-3B and Llama-8B using only pointwise labels.

In practice, pointwise supervision is easier to obtain than pairwise supervision, as it can be collected through binary user feedback such as “good” or “bad.” Pairwise supervision can be converted into pointwise labels, whereas pointwise supervision alone does not generally determine pairwise preferences. Methods using pointwise supervision therefore apply to a broader range of feedback settings. As shown by the results on UF-G (Table 1), where pairwise supervision is unavailable, RDRO achieves strong performance in this realistic setting, highlighting its practical value.

Weaknesses 4: Some related stable alignment approaches are not analyzed.

RDRO is the only method that achieves both statistical consistency and training stability. DDRO also guarantees statistical consistency, and we compare RDRO with DDRO in terms of stability and convergence guarantees in Section 3.

In terms of training stability, other relevant approaches include IPO [1], which uses a squared loss; Cal-DPO [2], which calibrates the scale of implicit rewards; and SimPO (Meng et al., 2024), which introduces a target reward margin. More recently, Gate-DPO [3] weakens gradient updates that further suppress already low-probability rejected responses. Although these methods improve training stability, they do not guarantee statistical consistency. Specifically, they do not guarantee that the language model pθp_{\theta} converges to the true preference distribution p+p^{+} as the number of samples increases. We will include this discussion in Related Work.

[1] Azar, Mohammad Gheshlaghi, et al. "A general theoretical paradigm to understand learning from human preferences." International conference on artificial intelligence and statistics. PMLR, 2024.

[2] Xiao, Teng, et al. "Cal-dpo: Calibrated direct preference optimization for language model alignment." Advances in Neural Information Processing Systems 37 (2024): 114289-114320.

[3] Mouiche, Inoussa, and Abderaouf Bahi. "Gradient-gated dpo: Stabilizing preference optimization in language models." arXiv preprint arXiv:2605.02626 (2026).

Question 1: The authors are encouraged to conduct experimentation on recent LLMs and large LLMs.

Our experiments use Qwen 2.5 1.5B/3B and Llama 3 3B/8B, following standard practice in alignment research. Closely related studies, including DDRO (Higuchi and Suzuki, 2025) and Kim et al. (2025), also evaluate models of comparable sizes with similar release dates.

Question 2: I wonder how to ensure that f is strictly convex for LLMs.

Please see our response to Weakness 1 above.

Question 3: Some recent stable alignment approaches can be analyzed.

Please see our response to Weakness 4 above.

Reviewer_TZzeOpenReview ↗

Response

Thank you for your reply. Some points are clear. However, I will maintain my score.

Thank you for your response. We are glad that our response has helped clarify some of your concerns. Please let us know if you have any further questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Additional Analyses and Proof in Appendices I, J, and K

We thank the reviewers for their helpful comments. We have added Appendices I, J, and K to the revised manuscript to provide the figures and detailed proof discussed in our responses:

  • Appendix I: Analysis of the loss functions at α=0\alpha=0 and α=1\alpha=1 (Reviewer qcfw, Question 2).
  • Appendix J: A direct statistical consistency analysis of the training objective in Eq. (17) (Reviewer qcfw, Weakness).
  • Appendix K: Prompt-based predictions of p(+∣x)p(+|x) using the OpenAI Decisions API (Reviewer PqRp, Weakness 2).

We would appreciate it if you could review these additions together with our responses.

AuthorsOpenReview ↗

Rebuttal Summary

We thank the reviewers and Area Chair for their time and feedback. We summarize our responses to each reviewer below:

  • Reviewer TZze: We clarified the strict convexity of the scalar function ff defining the Bregman divergence, the novelty beyond existing relative density ratio estimation, and the practical advantages of RDRO over DPO and SimPO when only pointwise feedback is available. We also discussed related stable alignment methods and explained our choice of base models.
  • Reviewer PqRp: We provided additional experiments comparing RDRO, DDRO, and KTO using 10%, 50%, and 100% of UF-G. We also provided prompt-based predictions of preference probabilities p(+∣x)p(+|x) in Appendix K and discussed the empirical effectiveness of using a fixed α\alpha.
  • Reviewer qcfw: We directly analyzed the minimizer of the actual training objective in Eq. (17), providing a finite-sample bound and a consistency result in Appendix J. We also analyzed the loss functions at α=0\alpha=0 and α=1\alpha=1 in Appendix I and discussed the existing sensitivity results near both endpoints.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.