acceptodds
Under review as a conference paper at ICLR 2027

Plateau-DPO: Gradient-Envelope Allocation for Robust Preference Optimization

Abstract

Robust preference optimization must limit the influence of incorrect labels while preserving learning from valid preferences that the model has yet to learn. Both types of comparisons can have negative reference-relative margins, so downweighting comparisons based on their margins can also weaken useful supervision. For robust losses whose gradient weights concentrate around a single peak, valid comparisons away from that peak receive smaller weights even when they remain useful for learning. To address this issue, we introduce Plateau-DPO, which assigns a constant gradient weight to all comparisons within a prescribed margin interval and smoothly attenuates weights outside it. Comparisons within the interval are therefore not downweighted merely because their margins deviate from a peak, while extreme margins still receive reduced weights. We join the plateau with Gaussian tails and integrate the resulting weighting function to obtain a bounded symmetric preference loss. For a fixed interval, loss range, and integrated tail mass, we prove that the constant plateau maximizes the minimum gradient weight within the interval. Further analysis provides sufficient conditions for one-step descent of a target objective. Experiments under controlled label flips on Anthropic HH and UltraFeedback show improvements in preference accuracy and generation win rate. Gradient diagnostics and shape ablations show that the plateau retains larger normalized gradient weights on hard, unflipped comparisons, supporting the design of preserving learning signals across a finite margin interval. Code is available at https://anonymous.4open.science/r/Plateau-DPO-5B2F.

Primary area:
learning theory
Keywords:
Direct Preference Optimization, Robustness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4ZFLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Plateau-DPO, a robust preference optimization method that aims to preserve learning from difficult but correctly labeled comparisons while reducing the effect of samples with extreme margins. The proposed loss assigns a constant gradient weight within a predefined margin interval and uses Gaussian tails to gradually reduce the weight outside this interval. The authors provide theoretical results on optimal minimum gradient allocation and conditional objective descent. Experiments on Anthropic HH and UltraFeedback show improvements under different levels of label noise.

Critical strengths

[S1] Simple and well-motivated method.

The paper addresses an important problem in robust preference optimization, reducing the effect of noisy labels without losing useful learning signals from difficult but correctly labeled comparisons. I think the proposed plateau-shaped gradient weighting is a simple and intuitive way to address this problem.

[S2] Strong experimental results with useful ablations.

The proposed Plateau-DPO consistently outperforms the evaluated baselines across different noise levels and two settings. Also, the matched-shape ablations and gradient analysis support the idea that covering a wider margin interval helps preserve useful learning signals. I believe these results show the practical value of the proposed method.

Critical weaknesses

[W1] Missing comparisons with relevant robust preference optimization methods.

Although the paper introduces Dr.DPO and follows the experimental setup, it does not include Dr.DPO as a baseline. Robustness is also tested only with artificially flipped labels, without experiments on naturally noisy preference annotations.

Questions for authors

[Q1] Comparison with Dr. DPO

Could the authors explain why Dr. DPO was not included and provide a comparison under the same experimental conditions? I believe it allows the paper more integrated.

Reviewer_5ipJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Plateau-DPO, a bounded preference loss whose gradient coefficient is constant over a prescribed reference-relative margin interval and decays smoothly through Gaussian tails outside that interval. The motivation is to suppress comparisons with extremely negative margins, which may contain corrupted labels, without excessively downweighting valid but unresolved comparisons whose margins are moderately negative. The paper interprets differentiable preference objectives through their scalar gradient envelopes. Given an active half-width BB and integrated tail mass ϵ\epsilon, Plateau-DPO allocates weight (1−ϵ)/(2B)(1-\epsilon)/(2B) uniformly over −B,B-B,B and uses Gaussian tails whose scale is determined by the normalization constraints. The resulting loss is bounded, smooth, and symmetric, satisfying ℓ(z)+ℓ(−z)=1\ell(z)+\ell(-z)=1. The theoretical results show that vanishing negative-tail weights bound the contribution of extreme-margin examples, that Plateau-DPO yields one-step descent under an explicit active-gradient alignment condition, and that the constant plateau maximizes the minimum coefficient over the active interval at fixed interval width and tail mass. An appendix also observes that, under independent uniform label flips, the corrupted population risk is a positive affine transformation of the clean risk. Experiments use Pythia-2.8B on Anthropic HH with 10–40% uniform label flips and Mistral-7B on UltraFeedback with synthetically constructed instance-dependent corruption. Plateau-DPO obtains the best reported preference accuracy and generation win rate in these settings. Gradient diagnostics and matched-envelope ablations suggest that broad central coverage, rather than the particular Gaussian tail, is associated with the observed gains.

Critical strengths

Separating extreme-margin attenuation from gradient allocation within an active interval is a useful perspective. Plateau-DPO directly addresses the tension between suppressing potentially corrupted comparisons and preserving updates from valid but unresolved examples.

Critical weaknesses
  1. The novelty relative to symmetric-loss preference optimization is unclear. Plateau-DPO is a bounded symmetric loss satisfying ℓ(z)+ℓ(−z)=1\ell(z)+\ell(-z)=1 and approaches a smoothed ramp loss as the tail mass vanishes. Nishimori et al. (TMLR 2026) already develop SymPO from the symmetric-loss perspective, analyze robustness to noisy preferences and policy improvement, and explicitly discuss sigmoid and ramp losses. This work is neither cited nor compared experimentally. The appendix’s uniform-flip identity is also a standard consequence of loss symmetry and is weaker than the existing SymPO analysis. The paper should therefore explain which theoretical or algorithmic property is genuinely new beyond selecting a particular smooth symmetric envelope.

  2. The empirical evidence is too narrow to establish superiority over existing robust preference methods. Both noise processes are artificially injected; the study covers only two model–dataset configurations; the Pythia results use three training seeds; and the number of independent Mistral training runs is unclear. The reported baselines omit SymPO and do not empirically compare against cited robust approaches such as ROPO or Dr. DPO. In addition, the exact hyperparameter search spaces and per-method selections are not reported. These omissions are consequential because the Pythia gains and the matched-shape ablation differences are often on the order of one percentage point.

Questions for authors
  1. How does Plateau-DPO differ theoretically and empirically from SymPO and, more specifically, from using a ramp or smoothed-ramp symmetric loss for offline preference optimization? Since Plateau-DPO satisfies ℓ(z)+ℓ(−z)=1\ell(z)+\ell(-z)=1, could the authors include SymPO as a direct baseline and clarify what is new beyond the choice of gradient-envelope shape?

  2. How many independent policy-training runs were used for the Mistral-7B/UltraFeedback results in Table 2? Could the authors report per-seed results or confidence intervals that include training randomness, rather than only repeated evaluation subsets? For both experimental settings, could the authors also provide the exact hyperparameter grids, selected values, and whether tuning was performed separately at each corruption level?

  3. Can the authors quantify the terms in the sufficient condition of Theorem 5.2 during actual training, particularly the active-region alignment term and the resulting value of qq? As written, the theorem establishes descent after assuming that active examples already align sufficiently with the target direction. Evidence that Plateau-DPO satisfies this condition more reliably than competing symmetric envelopes would materially strengthen the theoretical interpretation.

Reviewer_JAqgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Plateau-DPO, a preference loss that assigns a constant gradient weight to pairs whose margin falls within a selected interval and reduces the weight of pairs outside the interval. The authors provide theoretical results that bound the contributions of pairs with extreme margins, establish a descent direction under stated conditions, and prove that a constant weight maximizes the minimum weight within the interval when both the total area under the weighting function and the area outside the interval are fixed. On Anthropic HH and UltraFeedback datasets with a controlled fraction of flipped labels, the authors report higher preference accuracy and higher win rates for generated responses.

Critical strengths

Strengths

  1. The paper addresses an important tradeoff in preference learning: suppressing incorrect labels can also suppress useful supervision from difficult comparisons. Plateau-DPO and its theoretical results, which appear correct, give a concrete framework for studying this tradeoff. This strength carries limited weight in my score, because the paper's presentation makes the framework hard to understand.

  2. The controlled experiments support the design of Plateau-DPO. Plateau-DPO improves preference accuracy and win rates under two procedures for corrupting labels. Comparisons with other weighting functions support the benefit of constant weighting within the interval. These results strengthen the empirical case, but their weight in my score is limited because the paper does not fully specify the diagnostics, the evaluated policies, the intervals, or the hyperparameter search.

Critical weaknesses

Weaknesses

  1. The presentation prevents a clear understanding of the contributions. The paper explains terminology and notation late. Dense jargon and abrupt transitions obscure how the motivating problem, the method, and the theoretical results connect. Even after careful reading, the reader must reconstruct how the contributions address the motivating problem. This presentation problem is the main reason for my score, because communicating the paper’s significance will require substantial revision.

  2. The paper does not clearly connect its robustness results to the incorrect preferences that motivate the work. The introduction invokes “incorrect preferences” without specifying whose preferences define a correct label or how legitimate disagreement differs from annotation error. The experiments flip existing labels artificially and use the original labels as the reference for measuring added corruption. The theory also establishes robustness under a model of randomly flipped labels (Section 5.3). These results support robustness under specified corruption models, but the paper does not explain how those models represent the ambiguity and heterogeneous preferences invoked in its motivation. This gap leaves the scope and significance of the robustness claim unclear.

Questions for authors

Questions

  1. Could the authors state the motivating problem in plain language and explain how Plateau-DPO and each theoretical result address it?

  2. Could the authors specify what makes a preference label incorrect? Whose preferences define a correct label, and how does the paper distinguish annotation error from legitimate disagreement?

  3. Could the authors state the scope of the robustness claimed in the title? Which parts of this robustness do the conditional descent theorem, the result for randomly flipped labels, and the experiments each establish? Does the claim extend beyond flipped labels?

  4. Could the authors identify the target objective FF in Theorem 5.2? Does FF represent the original preferences, the observed preferences with flipped labels, or another target? Once FF is fixed, what evidence shows that the alignment assumptions of Theorem 5.2 hold in the experiments? As written, neither point is clear.

  5. How did the authors choose the interval width and the area under the weighting function outside the interval? These choices affect the tradeoff between suppressing pairs with flipped labels and retaining useful supervision from difficult pairs with correct labels. What do the reported diagnostics establish about both sides of this tradeoff?

  6. Could the authors clarify what each diagnostic in Figure 3 and Table 3 measures and how it is computed? Please specify the target direction, the normalization, the evaluated policies, and the frozen predictor that defines the difficult pairs with correct labels.

  7. Could the authors explain why maximizing the minimum weight within the interval is an appropriate criterion for learning from unreliable preferences? Theorem 5.3 establishes optimality for this criterion, but how does that result support improved preference learning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.