acceptodds
Under review as a conference paper at ICLR 2027

Gradient-Gated DPO: Stabilizing Preference Optimization in Language Models

Abstract

Direct Preference Optimization (DPO) improves relative preference by increasing the margin between chosen and rejected responses, but this objective does not specify how probability mass should be redistributed to achieve that margin. Consequently, preference separation can improve even as probability mass collapses away from both preferred and unrelated responses, a phenomenon associated with the squeezing effect. We show that this pathology is localized: continued negative updates become destructive when rejected responses enter low-probability valleys, where further suppression induces harmful redistribution through the coupled softmax geometry. We introduce Gradient-Gated Preference Optimization (Gate-DPO), which uses the current probability geometry of each rejected response to selectively attenuate its contribution, limiting excessive rejected suppression while delaying preference-loss saturation. Gate-DPO is objective-agnostic and composes with existing preference methods. Across four architectures, two preference datasets, and multiple objectives, gating consistently reduces squeezing and improves chosen-response likelihood, while exhibiting robust reductions in rejected-response collapse across architectures and objectives. Mass-dynamics analysis further shows that large preference margins can conceal substantial suppression of unrelated responses, while gating achieves healthier redistribution. Together, our results show that preference margin alone is insufficient to characterize optimization health and that selectively controlling probability dynamics provides a simple, general mechanism for stabilizing preference optimization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Direct Preference Optimization, Preference Optimization, LLM Alignment, RLHF, Gradient Gating, Probability Dynamics, Post-Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hPgHOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces a Gate-DPO method to stabilize DPO by selectively attenuating gradient updates on rejected responses when they enter low-probability regions. The authors find that negative gradients on rejected responses concentrate probability mass on high-confidence outputs while suppressing other responses. Gate-DPO addresses this by computing a detached, probability-aware gate that modulates the rejected-response contribution based on its current probability geometry. Results show reductions in squeezing, improved chosen-response likelihood, and healthier probability-mass redistribution, with favorable pairwise win rates against ungated baselines.

Critical strengths
  1. The paper identifies a specific, localized pathology (rejected responses entering low-probability valleys) and proposes a targeted intervention. The detached gate is a clean design choice that avoids introducing additional optimization targets.
  2. The experiments cover four architectures, two datasets, five preference objectives, and both quantitative and pairwise evaluation. The consistency of the effect across configurations is convincing.
  3. The authors acknowledge honestly that human judgments and standardized benchmarks remain future work.
Critical weaknesses
  1. No systematic approach to select \tau a priori is provided, and no details are given on how to perform "model-dataset-specific tuning of \tau may yield further gains. This is important as the sequence-level default of \tau could fail (e.g., on Pythia-410M Extend).
  2. Pairwise evaluation uses 30 held-out prompts from UltraFeedback. The win rates are based on a small sample and may not generalize at scale. No AlpacaEval 2.0 or MT-Bench results are reported. The paper acknowledges that "broader human and standardized evaluation is left to future work."
  3. The selected baselines for comparison are limited. More recent SOTA methods should be included.
Questions for authors
  1. The paper notes that \tau is model-dataset sensitive, but provides no prescriptive method. This is critical in practical applications of the proposed method.
  2. The pairwise evaluation uses only 30 prompts. No confidence intervals or statistical significance tests for the win rates are provided. How stable is it across seeds?
  3. No comparison with recent methods like SimPO and ORPO.
1 comment

The authors have posted their responses to your comments. Could you please take a look and let us know whether your concerns have been addressed? We encourage active discussion between reviewers and authors during this period.

Reviewer_MirhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the squeezing pathology in direct preference optimization (DPO): the preference margin can improve while the likelihood of chosen, rejected, and unrelated responses decreases and probability concentrates near high-probability outputs. The authors propose Gate-DPO, which computes a detached gate from the current probability geometry of each rejected response. The gate uses either the geometric mean of teacher-forced token probabilities or a lower-tail q10 statistic, and attenuates the rejected log-ratio in the preference objective. The paper evaluates Gate-DPO across Pythia-410M, Qwen-0.5B, LLaMA-7B, and Qwen1.5-7B on Anthropic-HH and UltraFeedback, and composes the gate with DPO, IPO, Cal-DPO, DPO-Shift, and extended-SFT variants. The reported results show reduced rejected-response collapse and improved chosen-response likelihood, while a small pairwise pilot provides initial behavioral evidence.

Critical strengths

S1. The paper identifies an important distinction between preference-margin improvement and absolute probability dynamics. The gradient analysis and toy softmax experiment make the proposed failure mode easy to inspect, and the paper appropriately notes that negative updates to rejected responses are not inherently pathological. S2. Gate-DPO is simple and modular. The detached gate can be applied to several preference objectives without changing model architecture, and the experiments cover multiple model families and preference datasets.

Critical weaknesses

W1. The novelty relative to Bounded-DPO is insufficiently established. The paper describes Gate-DPO as the first probability-aware, gradient-level intervention. However, Bounded-DPO already makes the rejected-response contribution depend on a mixture of the current policy and reference-policy probabilities, producing a probability-dependent attenuation effect. Appendix D.2 only argues that the two methods are algebraically complementary and explicitly leaves empirical evaluation of their combination to future work. Without a direct comparison or a gradient-matched analysis, the paper does not establish that Gate-DPO offers a substantively different mechanism or practical advantage over this closely related method. W2. The paper aims to show that standard DPO causes unhealthy probability-mass concentration and that Gate-DPO produces healthier probability-mass redistribution across the output distribution. However, the main evidence is based on ∆Chosen, ∆Rejected, and ∆Others, which measure the average change in log-probability, relative to the reference policy, over a small set of manually selected responses. In particular, ∆Others does not measure the total probability mass assigned to the “other” response region; it only measures the average log-probability change of the sampled strings in that group. Therefore, a ∆Others value closer to zero indicates only that these selected responses are suppressed less severely. It cannot reveal where the removed probability mass is transferred, whether it becomes concentrated around a few high-probability outputs, or whether Gate-DPO improves redistribution over the full output distribution. Thus, the metric does not adequately support the paper’s broader conclusion about healthier probability-mass redistribution.

Questions for authors

Q1. Referring to W1, the novelty of Gate-DPO appears limited because Bounded-DPO also modulates the rejected-response contribution according to the current-policy and reference-policy probabilities, while Appendix D.2 provides no direct empirical comparison or gradient-matched analysis. What is the substantive gradient-level distinction between Gate-DPO and Bounded-DPO, and can the authors provide matched experiments demonstrating that Gate-DPO offers an advantage that cannot be obtained from Bounded-DPO? Please explain. Q2. Referring to W2, ∆Others measures the average log-probability change of a limited set of manually selected responses, rather than the total probability mass assigned to an output region. It therefore cannot establish where probability mass is transferred or whether Gate-DPO reduces concentration across the full output distribution. Should the paper instead use a metric that directly measures probability mass or distributional concentration, such as cumulative probability mass over defined output regions, entropy, or another appropriate distribution-level measure? Please answer. Q3. Table 1 reports a DPO ∆Chosen value of −11.90 and default sequence-level and q10 Gate-DPO values of +1.17 and +1.30, respectively. In contrast, Appendix E.2 and Figure 7 report −4.84 for DPO and +3.50 and +3.93 for the corresponding default Gate-DPO settings, even though Appendix E.2 states that the sweep uses the same training budget and is directly comparable to Table 1. Were these results obtained using different evaluation sets, random seeds, checkpoints, or metric-computation procedures? Please specify the exact setting underlying each result and explain why the reported values differ.

1 comment

The authors have posted their responses to your comments. Could you please take a look and let us know whether your concerns have been addressed? We encourage active discussion between reviewers and authors during this period.

Reviewer_upP1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Gate-DPO, which reduces the weight of rejected responses when their probabilities are already low. This aims to prevent preference training from also lowering the probability of preferred responses. Experiments across four models and two datasets show improved likelihood dynamics.

Critical strengths
  1. The motivation is clear: a larger preference margin does not necessarily mean better model behavior.
  2. The method is simple, adds little computational overhead, and can be combined with several existing preference objectives.
  3. The experiments provide useful evidence across different models, datasets, and objectives, supported by gradient analysis.
Critical weaknesses
  1. The response-quality evaluation is limited to 30 prompts. Judges assess similarity to a reference answer, which does not fully capture helpfulness, correctness, or fluency. Stronger evaluation is needed to establish practical benefits.
  2. The experiments do not clearly isolate the value of adapting the gate to each response. A constant gate with matched average strength would help establish whether the gains require probability-aware gating.
  3. The “healthier redistribution” claim is stronger than the evidence. The unrelated responses include random and corrupted text, so preserving their probabilities is not necessarily desirable.
  4. Some reported win rates need clarification. Appendix G.1 describes two runs of 30 binary judgments, but values such as 68.7% in Table 3 do not match the possible percentages under this protocol. Raw win counts would resolve this concern.
Questions for authors

See Weaknesses Above.

1 comment

The authors have posted their responses to your comments. Could you please take a look and let us know whether your concerns have been addressed? We encourage active discussion between reviewers and authors during this period.

Comments on the paper

1 comment

The authors have posted their rebuttal, and the discussion period will end soon. Could you please read it and reply at your earliest convenience, stating whether your concerns have been addressed and whether your rating changes? Thank you so much for your active participation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.