- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces a Gate-DPO method to stabilize DPO by selectively attenuating gradient updates on rejected responses when they enter low-probability regions. The authors find that negative gradients on rejected responses concentrate probability mass on high-confidence outputs while suppressing other responses. Gate-DPO addresses this by computing a detached, probability-aware gate that modulates the rejected-response contribution based on its current probability geometry. Results show reductions in squeezing, improved chosen-response likelihood, and healthier probability-mass redistribution, with favorable pairwise win rates against ungated baselines.
Critical strengths
- The paper identifies a specific, localized pathology (rejected responses entering low-probability valleys) and proposes a targeted intervention. The detached gate is a clean design choice that avoids introducing additional optimization targets.
- The experiments cover four architectures, two datasets, five preference objectives, and both quantitative and pairwise evaluation. The consistency of the effect across configurations is convincing.
- The authors acknowledge honestly that human judgments and standardized benchmarks remain future work.
Critical weaknesses
- No systematic approach to select \tau a priori is provided, and no details are given on how to perform "model-dataset-specific tuning of \tau may yield further gains. This is important as the sequence-level default of \tau could fail (e.g., on Pythia-410M Extend).
- Pairwise evaluation uses 30 held-out prompts from UltraFeedback. The win rates are based on a small sample and may not generalize at scale. No AlpacaEval 2.0 or MT-Bench results are reported. The paper acknowledges that "broader human and standardized evaluation is left to future work."
- The selected baselines for comparison are limited. More recent SOTA methods should be included.
Questions for authors
- The paper notes that \tau is model-dataset sensitive, but provides no prescriptive method. This is critical in practical applications of the proposed method.
- The pairwise evaluation uses only 30 prompts. No confidence intervals or statistical significance tests for the win rates are provided. How stable is it across seeds?
- No comparison with recent methods like SimPO and ORPO.
The authors have posted their responses to your comments. Could you please take a look and let us know whether your concerns have been addressed? We encourage active discussion between reviewers and authors during this period.