acceptodds
Under review as a conference paper at ICLR 2027

Disentangling Optimization Scale from Preference Scale in DPO

Abstract

Direct Preference Optimization (DPO) uses β both to control the sharpness of the preference likelihood and, through an explicit gradient prefactor, to rescale optimization. Under fixed-learning-rate, finite-budget training, this coupling can make policy movement non-monotone in β: our sweeps exhibit near-zero KL movement at very small β (validation KL below 0.05 nats per token), a peak at an intermediate value, and reduced movement at larger values. A scalar stopping-tolerance model explains this qualitative pattern. We also show that standard DPO losses are difficult to compare across β, because nearly identical loss curves can correspond to substantially different margins and KL divergences. We propose a normalized centered-softplus objective that is a positive affine transformation of DPO for every fixed β > 0, preserving its ordering and any attained minimizers while removing the explicit β gradient prefactor. This parameterization separates likelihood sharpness from first-order update scale and yields more informative loss trajectories in our experiments. It also has a continuous β → 0 limit corresponding to a linear preference-margin objective.

Primary area:
reinforcement learning
Keywords:
DPO, RLHF, β optimization dynamics, KL divergence, preference learning, hyperparameter tuning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UhzrOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper study how the DPO coefficient β affects both preference optimization and gradient magnitude, leading to non-monotonic policy deviation and incomparable loss values across β settings. The authors propose a normalized centered-softplus objective that removes the explicit β-dependent gradient scaling while preserving the original DPO optimum.

Critical strengths

The paper identifies an important issue in DPO: β controls not only the preference–regularization trade-off but also the gradient scale, which can lead to unexpected optimization behavior.

The paper provides a clear analysis of the non-monotonic relationship between β and policy deviation, as well as the incomparability of DPO losses across different β values.

The paper is based on human intelligence.

Critical weaknesses

The central observation that β directly scales the DPO gradient follows immediately from the standard gradient expression, while the proposed normalization is essentially a rescaling of the original loss by 1/β1/\beta. Novelty is relatively limited.

The gradient-scale coupling is most apparent under SGD, whereas adaptive optimizers such as AdamW can largely compensate for uniform gradient rescaling?

The authors should also discuss related work on DPO gradient dynamics, particularly Mitigating Mismatch within Reference-based Preference Optimization (Yuan et al., ICLR 2026).

Questions for authors

N/A

1 comment

The authors have posted their responses to your comments. Could you please take a look and let us know whether your concerns have been addressed? We encourage active discussion between reviewers and authors during this period.

Reviewer_1BkUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents an empirical study of the effect of (\beta) on Direct Preference Optimization (DPO). Notably, the authors identify regimes in which optimization can stagnate due to the coupling between (\beta) and the gradient scale. To address this issue, they propose a normalized centered-softplus objective that removes the explicit (\beta) multiplier from the gradient, potentially enabling more effective optimization and providing a more informative training-loss signal.

Critical strengths

The authors' observation of how (\beta) affects the DPO loss gradient is interesting, particularly their finding that very small (\beta) values can lead to weak gradients and stagnation during training. This highlights an important interaction between the preference-scale parameter and optimization dynamics that may be overlooked when interpreting (\beta) solely in terms of regularization.

Critical weaknesses

The proposed fix lacks sufficient theoretical justification. Although the authors demonstrate that the normalized objective removes the explicit (\beta) factor from the gradient, they do not establish theoretically that this reformulation provides a meaningful optimization advantage over standard DPO with appropriately tuned learning rates. Given that the two objectives are equivalent up to a positive affine transformation for fixed (\beta>0), the practical benefits of the proposed approach remain insufficiently justified.

Questions for authors

Given that the proposed normalized objective is a positive affine transformation of standard DPO for any fixed (\beta>0), can the authors provide a stronger theoretical justification for why this reformulation offers a meaningful optimization advantage beyond appropriately rescaling the learning rate?

Can the authors propose a concrete, adaptive procedure for selecting (\beta) using observable training signals? For example, could a short pilot run be used to monitor gradient magnitudes, validation KL divergence, and preference-pair NLL to identify a regime where optimization stagnates or policy movement becomes excessive, and then adjust (\beta) accordingly?

1 comment

The authors have posted their responses to your comments. Could you please take a look and let us know whether your concerns have been addressed? We encourage active discussion between reviewers and authors during this period.

Reviewer_KyY2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper examines two roles of β in DPO: controlling the preference-likelihood scale and explicitly scaling the gradient. It studies how this coupling affects training under a fixed learning rate and finite computational budget.

The experiments show that policy movement can vary non-monotonically with β, with little movement at very small values and a peak at an intermediate value. A scalar stopping-tolerance model provides a qualitative explanation.

The authors propose a normalized centered-softplus objective that removes the explicit β gradient prefactor. For fixed β>0, it preserves the ordering and attained minimizers of standard DPO, while admitting a continuous linear-margin limit as β approaches zero.

Experiments across several datasets and model backbones illustrate these effects, verify the expected learning-rate equivalence under SGD, and show that similar standard-DPO losses can correspond to substantially different KL divergences and downstream performance.

Critical strengths

The paper identifies a useful practical distinction. Separating β’s role in the preference likelihood from its effect on optimization helps explain why its behavior under finite-budget training can differ from the usual regularization intuition. The loss-monitoring observations are compelling. Figure 4 and Table J1 show that nearly overlapping DPO losses can accompany meaningful differences in both policy divergence and downstream performance. This is a useful caution for practitioners comparing runs across β.

The proposed normalization is simple and mathematically transparent. The affine relationship to standard DPO makes the change easy to understand, and the learning-rate-matched SGD comparisons provide an appropriate check of the analysis. The empirical coverage and discussion of limitations are valuable. The paper examines multiple datasets and backbones, includes repeated-seed experiments for several key comparisons, and explicitly acknowledges optimizer dependence and the approximate nature of the stopping model.

Critical weaknesses

The paper convincingly demonstrates equivalence under learning-rate rescaling, but provides less evidence that the new coordinates make tuning easier or more reliable. The main sweep estimates τ from the observed peak or fits it to the same measurements. Appendix M offers a useful independent diagnostic, but its coverage is limited, leaving uncertainty about how well the estimated threshold transfers across settings.

The normalized loss separates the reported curves, but this does not by itself establish a reliable criterion for selecting better policies. The monitoring benefit needs a more precise interpretation.

Questions for authors

Can τ estimated from an independent probe trajectory predict the peak or low-movement boundary at other β or learning-rate settings without refitting?

Do the existing experiments show a setting where the normalized parameterization makes hyperparameter selection more efficient or reliable?

Can the AdamW explanation be tested more directly through an ε ablation or measurements of the optimizer’s stored second moments?

The discussion of token-level preference optimization could be broadened to include recent related work, such as TIS-DPO, SparsePO, T-REG, Se-DPO and TI-DPO, etc.

3 comments

The authors have posted their responses to your comments. Could you please take a look and let us know whether your concerns have been addressed? We encourage active discussion between reviewers and authors during this period.

Dear Reviewer KyY2, a brief follow-up on our response above. We ran the AdamW ε ablation you suggested (Figs. B1–B2) and added two new probe-based estimates of τ at different learning rates in Appendix M. We’d be happy to clarify any of these results.

Thank you for the detailed responses and additional experiments, particularly the cross-learning-rate predictions and the AdamW ε ablation. These additions and the clarified limitations address several of my concerns, and I will raise my score accordingly.

Comments on the paper

1 comment

The authors have posted their rebuttal, and the discussion period will end soon. Could you please read it and reply at your earliest convenience, stating whether your concerns have been addressed and whether your rating changes? Thank you so much for your active participation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.