- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper study how the DPO coefficient β affects both preference optimization and gradient magnitude, leading to non-monotonic policy deviation and incomparable loss values across β settings. The authors propose a normalized centered-softplus objective that removes the explicit β-dependent gradient scaling while preserving the original DPO optimum.
Critical strengths
The paper identifies an important issue in DPO: β controls not only the preference–regularization trade-off but also the gradient scale, which can lead to unexpected optimization behavior.
The paper provides a clear analysis of the non-monotonic relationship between β and policy deviation, as well as the incomparability of DPO losses across different β values.
The paper is based on human intelligence.
Critical weaknesses
The central observation that β directly scales the DPO gradient follows immediately from the standard gradient expression, while the proposed normalization is essentially a rescaling of the original loss by . Novelty is relatively limited.
The gradient-scale coupling is most apparent under SGD, whereas adaptive optimizers such as AdamW can largely compensate for uniform gradient rescaling?
The authors should also discuss related work on DPO gradient dynamics, particularly Mitigating Mismatch within Reference-based Preference Optimization (Yuan et al., ICLR 2026).
Questions for authors
N/A
The authors have posted their responses to your comments. Could you please take a look and let us know whether your concerns have been addressed? We encourage active discussion between reviewers and authors during this period.