- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes RDRO, a pointwise LLM alignment method based on the density ratio between preferred responses and a preferred/non-preferred mixture. It derives a simple softplus objective with bounded per-sample gradient coefficients and provides analyses of consistency and finite-sample error. Experiments on four models and two preference datasets evaluate alignment quality, reasoning performance, parameter sensitivity, and training dynamics.
Critical strengths
- Clear motivation and substantial theoretical analysis. The relative-ratio formulation addresses a concrete limitation of DDRO. The loss derivation, conditional population-optimum result, and bounded-coefficient analysis provide useful theoretical support for the objective and its optimization behavior.
- A reasonably broad experimental evaluation. Four models, two datasets, and three training seeds provide useful evidence beyond a single setting. The gains on UF-G, competitive results on MIX-14K, and additional BBH, sensitivity, and training-dynamics analyses support the method's practical value.
Critical weaknesses
The finite-sample guarantee needs further justification for the implemented estimator. Appendix D's Eq.49 analyzes an empirical Bregman risk containing the unknown , while training minimizes the computable objective in Eq.17. Their population objectives agree up to a parameter-independent constant, but their finite-sample minimizers need not coincide. Clarifying how the proof covers Eq.17, or directly analyzing that estimator, would strengthen the consistency and convergence claims.
Questions for authors
- See the weakness above.
- Since is central to the reference-mixture assumption, I am curious about the reported robustness to its value. Could you report results for and values close to or equal to 1, and discuss what these results reveal about 's role?
3 comments
Thank you for the clarification and additional analyses. I will maintain my score.
Thank you for your response and for considering our additional analyses. We appreciate your feedback.
Thank you for your insightful comments, which we shall address below.
As you noted, the minimizers of Eqs. (17) and (49) need not coincide for finite samples. Following DDRO, we analyze the Bregman divergence for any strongly convex function f in Appendix D. Substituting f(t)=tlogt−(1+t)log(1+t), which we use in our approach, into Eq. (48) gives Eq. (16) plus a constant independent of the parameters. We omit this constant and minimize Eq. (17), the empirical approximation of Eq. (16). Since the identity Epref[r∗t]=Ep+[t] used in this derivation does not generally hold for sample averages, substituting the same function into Eq. (49) does not generally give Eq. (17). We will clarify this distinction in the revised paper.
To address this, we directly analyze the minimizer of Eq. (17), using Eqs. (16) and (17) as the true risk and empirical risk, respectively. Following the empirical risk minimization and uniform deviation analysis in Appendix D, the estimation error is bounded above as follows:
\mathbb{E}_{\mathcal{D}^{\pm}}\left[\mathbb{E}_{p^{+}}\left[\left(p_{\hat{\theta}}-p^{+}\right)^{2}\right]\right]\leq\frac{4}{\alpha}\left[\inf_{\theta\in\Theta}\mathcal{L}_{\mathrm{RDRE}}(\theta)+4\left\\{\mathcal{R}_{N}^{+}(\mathcal{T})+(1-\alpha)\mathcal{R}_{M}^{-}(\mathcal{T})\right\\}\right].
Here, θ^ minimizes Eq. (17), and \mathcal{T}=\left\\{T_{\theta}=\log p_{\theta}-\log p_{\mathrm{ref}}\mid\theta\in\Theta\right\\} is the corresponding model class. RN+ and RM− are the Rademacher complexities of T under the preferred and non-preferred data distributions, respectively.
Please see Appendix J (Theorem J.1) for the complete theorem, assumptions, and proof. This yields a bound of the same form as Eq. (57), establishing statistical consistency for the minimizer of the actual training objective.
Please see our response to the weakness above.
At α=0, the mixture model gives pref=p−; at α=1, it gives pref=p+. These endpoints are outside our problem setting, which assumes 0<α<1. Figures 8(a) and 8(b) in Appendix I show the loss functions for these endpoints. For α=0, the loss for preferred samples has no finite minimizer and encourages an increase in Tθ=logpθ(y∣x)−logpref(y∣x). For α=1, the loss for non-preferred samples vanishes, while the loss for preferred samples is minimized at pθ(y∣x)=pref(y∣x).
Therefore, α can be interpreted as a regularization parameter that controls deviations from the reference policy. For 0<α<1, the loss for preferred samples is minimized at Tθ=log(1/α), and this minimum moves closer to the reference policy as α increases. At α=0, the finite minimum that provides this control disappears; at α=1, the minimum corresponds to the reference policy itself. Figures 1 and 2 further show that RDRO outperforms DDRO on UF-G and remains comparable on MIX-14K for α=0.1,…,0.9, including values near both endpoints.