- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Plateau-DPO, a robust preference optimization method that aims to preserve learning from difficult but correctly labeled comparisons while reducing the effect of samples with extreme margins. The proposed loss assigns a constant gradient weight within a predefined margin interval and uses Gaussian tails to gradually reduce the weight outside this interval. The authors provide theoretical results on optimal minimum gradient allocation and conditional objective descent. Experiments on Anthropic HH and UltraFeedback show improvements under different levels of label noise.
Critical strengths
[S1] Simple and well-motivated method.
The paper addresses an important problem in robust preference optimization, reducing the effect of noisy labels without losing useful learning signals from difficult but correctly labeled comparisons. I think the proposed plateau-shaped gradient weighting is a simple and intuitive way to address this problem.
[S2] Strong experimental results with useful ablations.
The proposed Plateau-DPO consistently outperforms the evaluated baselines across different noise levels and two settings. Also, the matched-shape ablations and gradient analysis support the idea that covering a wider margin interval helps preserve useful learning signals. I believe these results show the practical value of the proposed method.
Critical weaknesses
[W1] Missing comparisons with relevant robust preference optimization methods.
Although the paper introduces Dr.DPO and follows the experimental setup, it does not include Dr.DPO as a baseline. Robustness is also tested only with artificially flipped labels, without experiments on naturally noisy preference annotations.
Questions for authors
[Q1] Comparison with Dr. DPO
Could the authors explain why Dr. DPO was not included and provide a comparison under the same experimental conditions? I believe it allows the paper more integrated.