acceptodds
Under review as a conference paper at ICLR 2027

LCPO: Towards Addressing Reward Degeneration in Direct Preference Optimization

Abstract

Direct Preference Optimization (DPO) and its variants are widely used for preference alignment in large language models. However, prior work has shown that DPO often suffers from reward degeneration during training, where the implicit rewards of both chosen and rejected responses drift into extremely negative regions. In this paper, we analyze this phenomenon through the lens of reward margin and reward level. We show that DPO enlarges the reward margin but provides no lower-bound control along the reward-level direction. Consequently, once optimization enters a low-level region, vanilla DPO has no intrinsic mechanism to restore the reward level. We further show that pairwise preference data do not identify the absolute reward value, so correcting reward degeneration necessarily requires introducing additional priors. This motivates us to formulate degeneration correction as a prior design problem. Under this view, we propose three design principles for such priors: level-active, target-preserving, and calibration-free. Guided by these principles, we introduce Level-Constrained Preference Optimization (LCPO), a simple and general solution that imposes a relational constraint between chosen and rejected rewards to counteract reward-level collapse. Empirical results on representative benchmarks show that LCPO effectively mitigates reward degeneration and consistently outperforms existing baselines. Code is provided in the supplementary material.

Primary area:
foundation or frontier models, including LLMs
Keywords:
direct preference optimization, reward degeneration, preference alignment, large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_A8TdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies reward degeneration in DPO: the implicit rewards of both preferred and dispreferred responses can decrease even as their preference margin grows. It analyzes this behavior by separating reward margin from reward level and proposes three principles for corrective priors: level-active, target-preserving, and calibration-free.

The proposed method, LCPO, adds a batch-level hinge penalty that constrains the relationship between preferred and dispreferred rewards. When the constraint is satisfied, the penalty vanishes and the objective reduces to standard DPO.

Experiments on UltraFeedback/AlpacaEval-2.0, TL;DR, and Anthropic-HH report improved alignment performance and less reward degeneration across several model families and sizes. The paper also examines training dynamics, hyperparameter sensitivity, and batch-wise versus sample-wise constraints.

Critical strengths

LCPO operates on rewards already computed during training and requires only an additional hinge penalty, making it a practical modification to an existing pipeline.

On Qwen2.5-7B, LCPO reaches a 68.4% win rate against the chosen TL;DR responses, compared with 44.5% for DPO and 60.5% for the strongest competing baseline.

Results include Qwen models from 1.5B to 14B, Llama and Mistral backbones, and three preference-alignment datasets. This provides useful evidence that the method is not tied to a single model or task.

Reward trajectories, hyperparameter sweeps, and the comparison between batch-wise and sample-wise constraints illustrate the trade-off between limiting reward decline and allowing preference optimization.

Critical weaknesses

The theoretical motivation needs a clearer distinction between underlying and implicit rewards. Underlying rewards are identifiable only up to an additive shift, but the policy-based implicit reward has its offset fixed by normalization. LCPO’s constraint is also sensitive to a common reward shift when γ≠1\gamma\neq1.

The experiments do not isolate the proposed mechanism. Appendix B shows that the active penalty adds preferred-response likelihood and dispreferred-response unlikelihood updates. Without a matched comparison that applies the same auxiliary update continuously, it is difficult to tell how much of the gain comes from the proposed activation boundary.

Questions for authors

How do you distinguish the additive ambiguity of underlying rewards from the normalization-fixed implicit rewards? In particular, what precisely does “calibration-free” mean given that Equation 11 is sensitive to a common reward shift?

Have you compared LCPO with a version that applies the same likelihood/unlikelihood auxiliary update at every step, with its weight tuned separately? This would help establish whether the activation boundary itself accounts for the improvement.

Can you provide confidence intervals for the main win-rate comparisons and results across multiple training seeds, particularly for the AlpacaEval gains over the strongest baselines?

6 comments

Response to Reviewer A8Td

We thank the reviewer for the rigorous and insightful feedback! We address the concerns below.


Response to W1 & Q1: Calibration-Free vs. Shift-Invariance.

We agree with the reviewer's derivation: because γ≠1\gamma \neq 1, the LCPO constraint is mathematically sensitive to a constant shift CC in the implicit reward space, i.e., it introduces a (γ−1)C(\gamma-1)C term. However, we would like to clarify that this observation concerns shift-invariance, which is conceptually distinct from the calibration-free principle we proposed.

(1) Distinguishing shift-invariance from calibration-free.

"Shift-invariance" refers to whether a constraint remains unchanged under r→r+Cr \rightarrow r+C. In contrast, in our paper, "calibration-free" is explicitly defined as avoiding reliance on externally specified absolute baselines or fixed target values for the implicit rewards (lines 211–213 of the original submission). For example, methods that enforce conditions such as rw≥0r_w \geq 0, or drive rwr_w and rlr_l toward fixed constants, introduce an external absolute calibration. By contrast, LCPO does not impose such externally specified absolute targets; it constrains only the relation between the chosen and rejected implicit rewards. Therefore, our claim of being calibration-free was not intended to mean invariance to arbitrary common shifts, but rather freedom from external human-defined absolute calibration.

(2) Shift-invariance cannot provide level-active control.

We further argue that common-shift invariance would make a corrective term blind to the reward-level direction. In the (u,v)(u,v) decomposition, a common shift leaves the margin uu unchanged while changing the level vv. Therefore, a strictly common-shift-invariant correction cannot remain directly sensitive to vv, and thus cannot provide an explicit restoring force along the reward-level direction. Under such a formulation, the correction can only act through the margin-related component, rather than directly regulating the reward level itself. In this sense, the shift sensitivity of LCPO is precisely what enables the method to be level-active and correct reward degeneration. That is, common-shift sensitivity is not in conflict with our design goal; it is a necessary feature for explicit level control in the (u,v)(u,v) view.

(3) Revisions to the manuscript.

To prevent ambiguity, we have revised the description of "calibration-free" in Section 3.3 (lines 214–233 of the latest submission) to further clarify that it means free from externally specified absolute calibration. We have also added a new discussion in Appendix F.1 that distinguishes calibration-free from shift-invariance, and explains why common-shift sensitivity is necessary for a correction that is explicitly level-active.


Response to W2 & Q2: Compare LCPO with a matched version that applies the same auxiliary update continuously.

We thank the reviewer for this helpful suggestion and agree that such a matched control is important for isolating the role of LCPO's activation boundary. In response, we added a new ablation study in appendix section D.5 comparing LCPO against a matched continuous auxiliary baseline (Cont-Aux) that applies the same auxiliary update direction at every training step, instead of activating it only when the LCPO constraint is violated. As suggested by the reviewer, we tune the weight of this baseline separately.

This additional comparison is informative. Cont-Aux consistently improves over vanilla DPO, which confirms that the auxiliary update direction itself is indeed helpful. At the same time, it still consistently underperforms LCPO. On UltraFeedback, it achieves 38.5% WR and 38.3% LC, compared with 41.4% and 41.8% for LCPO. On TL;DR, it reaches 62.5% against the chosen responses and 68.1% against SFT, compared with 68.4% and 74.8% for LCPO.

Taken together, these results suggest that LCPO's improvement is not explained solely by the auxiliary update direction. Rather, the conditional activation boundary itself contributes additional value: applying the level correction only when needed performs better than keeping the same correction active throughout training. We have added this experiment and the corresponding discussion in the revised manuscript Appendix D.5.

Response to Q3: Provide multi-seed results.

We thank the reviewer for raising this important point. To address the concern about result stability, we added a new appendix section D.6 reporting multi-seed evaluation for the main results. Specifically, we reran all methods with 3 different random seeds on both UltraFeedback and TL;DR, and report the mean$\pm$standard deviation of the win rates. In particular, this directly addresses the robustness of the AlpacaEval-2.0 gains over the strongest baselines.

The multi-seed results show that the seed variance is consistently small, while the ranking of methods remains stable. On UltraFeedback, the standard deviations are within about $\pm$0.6%, and LCPO achieves 41.2$\pm$0.4% WR and 41.9$\pm$0.3% LC, compared with 39.3$\pm$0.4% and 38.9$\pm$0.5% for the strongest baseline DPOP. On TL;DR, the standard deviations are within about $\pm$0.9%, and LCPO reaches 68.2$\pm$0.6% against the chosen responses and 74.5$\pm$0.6% against SFT, compared with 60.1$\pm$0.4% and 65.6$\pm$0.5% for the strongest baseline APO-zero.

Overall, the performance gaps of LCPO over the strongest baselines are substantially larger than the observed seed-to-seed variation, indicating that the gains are robust and not driven by a favorable random initialization. We have added these multi-seed results and discussion in the revised manuscript Appendix D.6.


We hope our responses clarify the points raised. please let us know if you have any further questions. Thank you for your time and feedback!

AuthorsOpenReview ↗

Further Discussion with Reviewer A8Td

Dear Reviewer A8Td,

Thank you again for taking the time to review our paper and for your thoughtful and constructive comments. We have carefully considered your feedback and provided detailed responses to each of your questions:

  1. For W1/Q1, we clarified the distinction between the notion of calibration-free used in our paper and the shift-invariance interpretation raised in your comment. We also explained why shift-invariance cannot provide a restoring force along the reward-level direction, and therefore why LCPO is not intended to be shift-invariant.

  2. For W2/Q2, we added the comparison experiment you suggested in the revision and verified the contribution of the conditional activation boundary.

  3. For Q3, we added multi-seed experiments in the revision and confirmed that the performance gains of LCPO over the baselines are substantially larger than the variation caused by random seeds.

As the author–reviewer discussion phase is drawing to a close, we would greatly appreciate knowing whether our responses have addressed your concerns. We would also be very happy to clarify any remaining questions if needed.

Thank you again for your careful review.

Best regards,

Authors

Reviewer_d6GxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies reward degeneration during DPO training, in which the implicit rewards of both chosen and rejected responses decrease substantially at the same time. The authors analyze this phenomenon in terms of the reward margin and overall reward level, frame degeneration correction as a prior design problem, and propose LCPO: a conditionally activated hinge penalty based on the relationship between batch-averaged rewards is added to the original DPO loss to limit excessive downward drift in the level relative to the margin. Through experiments across multiple models and tasks, together with training trajectory analysis and hyperparameter ablations, the authors report mitigation of reward degeneration and improvements in downstream preference win rates.

Critical strengths
  • LCPO uses a relatively simple modification to the objective and is reported to deliver good and consistent performance improvements across multiple tasks and model scales, indicating practical value.
  • In the main experiments, LCPO achieves a length-controlled win rate of 41.8% on UltraFeedback, compared with the highest value of 39.1% among the other baselines; its win rate against SFT on TL;DR is 74.8%, compared with the highest baseline value of 65.2%. The appendix reports similar advantages across different model scales, model families, and additional tasks. These results suggest that the method makes an empirical contribution worth considering, with benefits extending beyond a single experimental setting.
  • The paper is well organized and easy to follow.
Critical weaknesses
  • The paper's explanation of how LCPO corrects reward degeneration remains insufficient. The proposed relative constraint allows the rewards of chosen and rejected responses to continue decreasing together while the penalty remains zero. The current analysis therefore does not adequately support the paper's stronger explanations concerning restoration and collapse prevention.

  • The proposed constraint does not itself exclude the failure mode motivating the paper. The constraint can remain satisfied while both chosen and rejected rewards become arbitrarily negative, provided the rejected reward decreases sufficiently faster; therefore, feasibility alone does not prevent reward degeneration as defined in the paper.

Questions for authors
  • The paper motivates LCPO using the non-identifiability of the underlying reward, but LCPO constrains policy-induced implicit rewards whose values are fixed once the policy and reference model are specified. How does the former justify imposing the proposed constraint on the latter? In what precise sense is LCPO calibration-free, given that its constraint depends on the reward origin?
  • Both chosen and rejected rewards can continue decreasing while satisfying LCPO’s constraint; under what conditions does the method actually prevent this degeneration? Does “target-preserving” mean preserving DPO’s optimal policy, or only recovering the DPO loss when the penalty is inactive?
5 comments

Response to Reviewer d6Gx

We thank the reviewer for the rigorous and insightful feedback! We address the concerns below.


Response to W1 & W2: Constraint allows simultaneous decrease / Does not exclude failure mechanism.

We thank the reviewer for identifying this important distinction. We agree that the LCPO constraint imposes no hard absolute lower bound on the implicit rewards and permits jointly decreasing trajectories. LCPO mitigates the severe reward-level collapse observed in DPO training by coupling reward level to the achieved reward margin and supplying a corrective update when that relation is violated.

(1) The relational constraint in LCPO is intentional.

LCPO allows some joint decreases in the chosen and rejected rewards because it does not impose an absolute reward floor. This is a deliberate design choice: pairwise preference data do not specify a universal absolute value to which implicit rewards should be restored. Enforcing such a value, for example by setting a zero baseline for the chosen reward, would introduce an external calibration assumption beyond what those data justify. LCPO therefore uses a relational constraint to mitigate severe reward-level collapse without prescribing an absolute target.

(2) Algebraic feasibility does not imply uncontrolled collapse during training.

The reviewer's extreme trajectory cannot be sustained merely by decreasing both rewards. To keep the LCPO penalty inactive as they fall, optimization must continually produce enough additional preference margin to compensate for the declining reward level. Let bars denote means over a minibatch BB, and suppose rˉw=−a<0\bar r_w=-a<0. Writing uˉ=rˉw−rˉl\bar u=\bar r_w-\bar r_l, the zero-penalty condition for 0<γ<10<\gamma<1 is

=\gamma\bar r_l-\bar r_w =(1-\gamma)a-\gamma\bar u \leq 0 \quad\Longleftrightarrow\quad \bar u\geq\frac{1-\gamma}{\gamma}a.$$ At our default $\gamma=0.2$, an unpenalized chosen-reward decline to depth $a$ requires a mean margin of at least $4a$. Joint reward decline at a fixed margin therefore activates LCPO; the reviewer's construction requires the optimizer to keep expanding the margin as the rewards fall. This requirement becomes increasingly demanding for the DPO scalar margin signal. For pair $i$, let $u_i$ be its margin and $w_i=(1+e^{u_i})^{-1}$ the magnitude of the DPO loss derivative with respect to $u_i$. By concavity of $\log w_i$, an inactive-penalty minibatch satisfies $$\left(\prod_{i\in B}w_i\right)^{1/|B|} \leq \frac{1}{1+e^{\bar u}} \leq \frac{1}{1+\exp\left(\frac{1-\gamma}{\gamma}a\right)} \leq \exp\left(-\frac{1-\gamma}{\gamma}a\right).$$ Thus, at $\gamma=0.2$, maintaining an unpenalized decline to greater depth requires continued margin growth while the geometric mean of the corresponding DPO scalar weights is at most $e^{-4a}$. **Deep joint decline is therefore not a freely available downward direction under training:** remaining within the LCPO boundary requires sustained compensating margin growth precisely as these scalar learning signals become saturated. If reward decline outpaces margin growth, the penalty activates and, under gradient flow at the same parameters and on the same minibatch, reduces the instantaneous growth rate of $\Delta$ relative to DPO by $\lambda\|\nabla_\theta\Delta\|_2^2$. Fig. 2 shows the resulting substantial mitigation of severe reward decline, while Fig. 3(c) shows how $\gamma$ controls this behavior in practice. **(3) The choice of $\gamma$ balances protection and preference performance.** At the endpoint $\gamma=0$, the zero-penalty condition becomes $\bar r_w\geq0$. This algebraically excludes the reviewer's extreme trajectory in which both rewards tend to $-\infty$ while the penalty remains inactive. Achieving this stricter protection, however, requires anchoring the chosen reward at zero. As Fig. 3 shows, our choice of $\gamma=0.2$ allows a mild, margin-supported reward decrease while achieving better preference performance than $\gamma=0$. We therefore choose an appropriate $\gamma$ in practice to achieve the best trade-off between reward-level protection and preference performance. If the sole objective were to exclude the reviewer's zero-penalty collapse construction algebraically, setting $\gamma=0$ would suffice; our results show that effective mitigation in practice does not require this extreme endpoint.

Response to Q1: How is imposing a constraint on implicit rewards justified, and in what sense is LCPO calibration-free?

We agree that once the policy πθ\pi_\theta and reference model πref\pi_{\mathrm{ref}} are specified, the implicit reward rθr_\theta is fixed. The key point is that reward degeneration in our paper is a pathology of the implicit reward, and the role of non-identifiability is to explain why correcting this pathology cannot rely on assigning a uniquely correct absolute target to that implicit reward.

(1) Why does non-identifiability motivate a prior on implicit rewards?

The logic is the following. First, the pathology we study is that DPO explicitly drives the reward margin but does not control the reward level, so the implicit reward can drift into a deep negative region. A natural ideal solution would be: if we knew the correct absolute target value of the implicit reward, we could simply pull it back to that target. However, under the optimal policy, rθ∗(x,y)=r(x,y)−βlog⁡Z(x)r_\theta^*(x,y)=r(x,y)-\beta\log Z(x), so the absolute value of the optimal implicit reward depends on both the underlying reward r(x,y)r(x,y) and the prompt-dependent offset βlog⁡Z(x)\beta\log Z(x). While an additive shift in the underlying reward is canceled by the corresponding change in βlog⁡Z(x)\beta\log Z(x), pairwise preference data do not specify a universal absolute target to which the resulting implicit rewards should be restored. This is the role of the non-identifiability argument: not that the implicit reward itself is unidentifiable once πθ\pi_\theta and πref\pi_{\mathrm{ref}} are fixed, but that pairwise preference data do not specify a universal, externally chosen absolute target to which implicit rewards should be calibrated. Therefore, if we want to correct degeneration of the implicit reward, we should introduce an additional prior rather than rely on absolute calibration. LCPO is exactly such a prior: instead of specifying a fixed absolute target for implicit rewards, it imposes a relational constraint between chosen and rejected implicit rewards.

We have revised the corresponding discussion in the manuscript (lines 177–190 of the latest submission) to distinguish the additive ambiguity of the underlying reward from the reference-defined origin of the implicit reward, and to clarify why LCPO uses a relational prior rather than an externally specified absolute reward target.

(2) In what sense is LCPO calibration-free?

We agree that LCPO's constraint depends on the reward origin, but "calibration-free" should not be interpreted as invariance to arbitrary additive shifts of the implicit reward. Rather, by calibration-free we mean that LCPO does not impose an externally specified absolute baselines or fixed target values on the implicit reward (lines 211–213 of the original submission). Instead, it only constrains the internal relation between chosen and rejected implicit rewards. The relevant reward origin here is the one already induced by the DPO parameterization through the reference model πref\pi_{\mathrm{ref}}; this is an intrinsic geometric property of the implicit-reward representation, not an additional externally chosen calibration target introduced by LCPO.


Response to Q2: Under what conditions does LCPO prevent degeneration, and what does target-preserving mean?

(1) Under what conditions does LCPO prevent degeneration?

As clarified in our response to W1/W2, LCPO is not intended to forbid every trajectory in which both chosen and rejected rewards decrease simultaneously. The degeneration we target is the collapse of reward level relative to the achieved reward margin, i.e., when rewards drift into an excessively low-level region not supported by sufficient margin. LCPO becomes corrective exactly when this relational constraint is violated.

(2) What does "target-preserving" mean?

By "target-preserving", we mean preserving the original DPO target when no correction is needed, i.e., when the LCPO constraint is not active. Equivalently, if a DPO optimum already satisfies the LCPO constraint, LCPO leaves that optimum unchanged. We do not mean preserving every unconstrained DPO optimum regardless of feasibility; when the DPO optimum violates the constraint, LCPO intentionally modifies the target by introducing the additional prior needed to prevent harmful reward-level collapse.

We have revised the explanation of "target-preserving" in the manuscript (lines 205–213 of the latest submission) for clarity.


We hope our responses clarify the points raised. please let us know if you have any further questions. Thank you for your time and feedback!

AuthorsOpenReview ↗

Further Discussion with Reviewer d6Gx

Dear Reviewer d6Gx,

Thank you again for taking the time to review our paper and for your thoughtful and constructive comments. We have carefully considered your feedback and provided detailed responses to each of your questions:

  1. For W1/W2, we clarified that LCPO is not intended to impose a hard absolute lower bound or exclude every joint decrease of chosen and rejected rewards. Instead, it is designed to mitigate severe reward-level collapse by activating a correction when the reward level falls too much relative to the achieved margin.

  2. For Q1, we clarified that the role of non-identifiability is not that the implicit reward is undefined once the policy and reference are fixed, but that pairwise preference data do not specify a universal absolute target for implicit rewards. Accordingly, LCPO uses a relational prior rather than an externally specified absolute calibration target.

  3. For Q2, we clarified both the scope of the degeneration LCPO is meant to prevent and the meaning of "target-preserving", namely that LCPO preserves the original DPO objective whenever the constraint is inactive.

As the author–reviewer discussion phase is drawing to a close, we would greatly appreciate knowing whether our responses have addressed your concerns. We would also be very happy to clarify any remaining questions if needed.

Thank you again for your careful review.

Best regards,

Authors

Reviewer_p95wOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies reward degeneration in DPO, where the implicit rewards of both chosen and rejected responses drift to extremely negative values while the margin keeps growing. Through a margin–level decomposition, the authors show that DPO constrains only the margin and provides no lower-bound protection on the level (Prop 3.1 gives the structural coupling v≤−u+C(x,yw)v \le -u + C(x, y_w)); furthermore, the absolute reward offset (βlog⁡Z(x)\beta \log Z(x)) is non-identifiable from pairwise data, so any correction must introduce a prior. They distill three design principles — level-active, target-preserving, calibration-free — and instantiate them as LCPO: a hinge penalty λ⋅max⁡(0,E[γrl−rw])\lambda \cdot \max\big(0, \mathbb{E}[\gamma r_l - r_w]\big) added to the DPO loss, with γ=k−1k+1\gamma = \frac{k-1}{k+1}, k>1k > 1 justified by feasibility geometry in the (u,v)(u, v) plane. Experiments across Qwen2.5-{1.5B,7B,14B}, Llama-3-8B, and Mistral-7B on UltraFeedback, TL;DR, and Anthropic-HH show LCPO consistently achieves the best or near-best downstream win rates while keeping implicit rewards stable.

Critical strengths
  1. Clear and useful analytical lens. The margin–level decomposition is transparent; Prop 3.1 is correct (one-line algebra) and the non-identifiability argument (Eq. 9) properly motivates the "prior design" framing. Recasting NLL regularization, zero-anchoring, and calibrated objectives as different priors over the non-identifiable level direction (Table 1) is a helpful conceptual organization, with the appropriate caveat that violating a principle does not invalidate a method.
  2. Minimal, nearly free method. A single hinge term; the objective reduces exactly to DPO when inactive; the gradient analysis (Appendix B) cleanly interprets the active penalty as a chosen-side likelihood update plus a γ\gamma-weighted rejected-side unlikelihood update. The exclusion of k≤1k \le 1 is logically argued.
  3. Broad, consistent experiments: 3 scales ×\times 3 architectures ×\times 3 datasets; batch-vs-sample constraint ablation (D.4); full γ\gamma/λ\lambda sweeps; evaluation prompts disclosed (Appendix E).
  4. A persuasive detail: LCPO achieves the best results even with slightly negative chosen rewards, consistent with its rejection of forced reward positivity.
Critical weaknesses
  1. Limited novelty; the central claim lacks a decisive ablation. The active penalty is essentially a gated combination of a chosen-side likelihood push and a rejected-side unlikelihood push. What distinguishes LCPO from, e.g., a "target-preserving DPOP" is (i) hinge gating and (ii) relational rather than zero-anchored formulation — but Table 1's taxonomy is conceptual only. An absolute-hinge baseline (e.g., λ⋅max⁡(0,−rw)\lambda \cdot \max(0, -r_w)) is needed to demonstrate that the relational form, not merely "adding level pressure," drives the gains.
  2. No statistical uncertainty whatsoever. Every table reports single runs without std/CI. The $\sim$8-point TL;DR advantage over APO-zero cannot be assessed without seed variance, which matters greatly for DPO-family training.
  3. Judge-dependent evaluation. TL;DR and Anthropic-HH results rest entirely on GPT-4o pairwise judging, with no human-agreement validation, no second judge, and no reported judge variance. AlpacaEval-2.0 is standard, but half the headline evidence is not.
  4. Missing baselines: SimPO, KTO, β\beta-DPO, and the simple alternative in Weakness 1.
Questions for authors
  1. Can you add an absolute-hinge ablation (λ⋅max⁡(0,−rw)\lambda \cdot \max(0, -r_w)) to separate the contributions of relational constraint vs. hinge gating?
  2. Can you report mean ±\pm std over ≥3\geq 3 seeds, at least for the main table? Does the +8+8-point TL;DR gap survive seed variance?
  3. Batch-level constraints permit unbounded per-sample level drops as long as the batch mean holds. Do you observe per-pair pathologies? Would a quantile-based constraint be a better middle ground?
  4. What are the response-length statistics per method on AlpacaEval? Please substantiate the LC metric.
4 comments

Response to Reviewer p95w

We thank the reviewer for the rigorous and insightful feedback! We address the concerns below.


Response to W1 & Q1: Add an absolute-hinge ablation.

Thank you for this helpful suggestion. We agree that a controlled comparison against an absolute-hinge alternative is important for isolating whether the gain comes from the relational formulation rather than hinge gating alone.

This comparison is already included in our γ\gamma-ablation in Sec 4.2: under our formulation, the case γ=0\gamma=0 corresponds to a absolute-hinge / chosen-reward zero-anchoring baseline, i.e., it removes the relational coupling to the rejected reward while preserving the same hinge-gated correction mechanism. In the revised manuscript, we have made this explicit in the ablation text by stating that "when γ=0.0\gamma=0.0, LCPO reduces to a batch-wise absolute-hinge baseline, namely zero-anchoring the chosen reward without relational coupling to the rejected reward." (lines 460–462 of the latest submission)

Empirically, γ=0\gamma=0 improves over vanilla DPO, confirming that hinge-gated level correction alone is beneficial. However, it is not optimal: the best performance is achieved at γ=0.2\gamma=0.2, not γ=0\gamma=0. Thus, the gain of LCPO is not explained by hinge gating alone; compared with absolute zero-anchoring, the relational formulation avoids imposing an external calibration target on the non-identifiable reward level, making the correction more flexible while still preventing collapse.


Response to W2 & Q2: Provide multi-seed results.

We thank the reviewer for raising this important point. In the revised manuscript, we added a new multi-seed evaluation subsection (Appendix D.6) and report mean$\pm$std over 3 random seeds for all methods on both UltraFeedback and TL;DR.

The main conclusion is that the results are highly stable across seeds: the standard deviations are small (mostly within $\pm$0.6% on UltraFeedback and $\pm$0.9% on TL;DR), and the method ranking remains unchanged. Importantly, the reviewer's highlighted TL;DR gap clearly survives seed variance: LCPO achieves 68.2$\pm$0.6% against the chosen responses, compared with 60.1$\pm$0.4% for APO-zero, i.e., an ∼8\sim 8-point advantage that is far larger than the observed seed variation. On UltraFeedback, LCPO also remains best with 41.2$\pm$0.4% WR versus 39.3$\pm$0.4% for DPOP.

Thus, the gains of LCPO are robust across seeds and are unlikely to be explained by a lucky run.


Response to W3: Judge-dependent evaluation.

Thank you for raising this concern. First, our original LLM-as-a-Judge protocol is standardized and controlled across methods: for TL;DR and Anthropic-HH, we use the same judge, the same prompt template and decoding configuration, randomized candidate order, explicit instructions against position/length bias, and tie-allowed scoring. These details are provided in Appendix E.

Second, to further reduce concern about dependence on a single judge, we added a cross-judge evaluation using Gemini-3.5-Flash on both TL;DR and Anthropic-HH in the revised manuscript Appendix E.3. Although the absolute win-rate scales differ somewhat across judges, the overall ranking pattern is preserved: LCPO remains the best-performing method, and its advantage over strong baselines is consistent under both GPT and Gemini. Thus, our main conclusions are robust to the choice of LLM judge.


Response to W4: Missing baselines.

We agree that SimPO, KTO, and β\beta-DPO are important general preference optimization baselines. However, our main experimental comparisons were intentionally centered on methods explicitly proposed to mitigate or analyze reward degeneration. Since SimPO, KTO, and β\beta-DPO were not primarily motivated by degeneration correction, we did not include them in the original main table.

To address the reviewer's concern, we additionally evaluated these methods on UltraFeedback (Qwen2.5-7B-Instruct backbone). The AlpacaEval-2.0 results are as follows:

MethodWR (%)LC (%)
SimPO40.639.4
KTO39.840.3
β\beta-DPO37.337.1
LCPO41.441.8

Thus, LCPO remains competitive with these additional general baselines and achieves the best overall performance in this comparison, suggesting that our conclusions do not depend on a narrow choice of degeneration-specific baselines.

Response to Q3: Batch-level constraints and per-pair pathologies.

We agree that a batch-level mean constraint does not strictly bound the worst-case per-sample reward drop. To directly test whether such averaging masks per-pair pathologies, we analyzed the per-sample implicit reward distributions on the UltraFeedback test set. As shown below, LCPO substantially improves the lower tail and yields a tighter distribution than DPO, which does not indicate severe hidden per-pair collapse:

MethodMetricMeanP10P90Span
DPOrwr_w-12.58-19.47-5.8813.59
rlr_l-19.12-27.59-11.4216.17
LCPOrwr_w-0.46-4.564.118.67
rlr_l-4.32-9.370.519.88

Moreover, Appendix D.4 already compares the batch-wise and sample-wise variants directly, and the batch-wise constraint performs better in practice. This suggests that a stricter per-sample formulation is not necessarily a better choice empirically, even though it is tighter by construction.

We agree that a quantile-based constraint is an interesting middle ground, as it would control tail behavior more directly than a pure batch mean. However, such objectives require non-trivial differentiable approximations and introduce additional optimization complexity. Given both the tail-distribution evidence above and the empirical comparison in Appendix D.4, we chose the simpler and more effective batch-level formulation in this work.


Response to Q4: Response-length statistics on AlpacaEval.

We computed the average generated response length for each method for the AlpacaEval-2.0 evaluations reported in Table 2:

MethodAvg. length
SFT1928
DPO2183
DPO+NLL2186
DPOP2113
IPO2247
APO-zero2414
Cal-DPO2092
LCPO2164

These statistics show that response lengths differ substantially across methods, confirming that verbosity is a potential confounding factor on AlpacaEval-2.0. This is precisely why we report the LC metric in addition to the raw win rate. Notably, LCPO is not the longest-response method and it still achieves the best LC result. This supports that LCPO's gains are not simply an artifact of generating longer responses, and also substantiates the usefulness of LC as a length-aware evaluation metric.


We hope our responses clarify the points raised. please let us know if you have any further questions. Thank you for your time and feedback!

AuthorsOpenReview ↗

Further Discussion with Reviewer p95w

Dear Reviewer p95w,

Thank you again for taking the time to review our paper and for your thoughtful and constructive comments. We have carefully considered your feedback and provided detailed responses to each of your questions:

  1. For W1/Q1, we clarified that the ablation in the main paper already includes the absolute-hinge comparison, and we made this more explicit in the revision.

  2. For W2/Q2, we added multi-seed experiments in the revision and confirmed that the performance gains of LCPO over the baselines are substantially larger than the variation caused by random seeds.

  3. For W3/W4/Q3/Q4, we added additional experiments and analyses to address each of these concerns, including cross-judge evaluation, additional baselines, per-sample reward distribution analysis, and response-length statistics.

As the author–reviewer discussion phase is drawing to a close, we would greatly appreciate knowing whether our responses have addressed your concerns. We would also be very happy to clarify any remaining questions if needed.

Thank you again for your careful review.

Best regards,

Authors

AuthorsOpenReview ↗

Follow-up to Reviewer p95w

Dear Reviewer p95w,

Thank you again for your thoughtful review. We have addressed each of your concerns in our rebuttal. With about 24 hours remaining in the discussion period, we would greatly appreciate any further comments or questions you may have, and we would be happy to clarify any remaining points. We hope our responses have been helpful in addressing your concerns.

Thank you for your time and consideration.

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers and area chairs for a brief summary

Dear reviewers and area chairs,

We sincerely thank all reviewers and area chairs for their valuable time and thoughtful comments. As the author discussion period is coming to an end, we would like to provide a brief summary.

We are pleased that:

  1. All reviewers recognize the importance of the problem studied in this paper, namely reward degeneration in DPO, and acknowledge the value of addressing it for preference optimization.
  2. All reviewers recognize that LCPO is a simple and practical method, requiring only a lightweight additional hinge penalty on rewards already computed during training, making it easy to integrate into existing pipelines.
  3. All reviewers acknowledge the strength of the empirical results, including strong downstream performance on representative benchmarks and consistent improvements beyond a single model or task setting.
  4. All reviewers recognize the breadth of the experimental study, including evaluations across multiple model scales, model families, and preference-alignment datasets, together with training-dynamics analysis and ablation studies.

We have provided detailed responses to each reviewer and would like to offer a summary:

  1. We clarify the role of the proposed relational constraint and make explicit that the γ=0\gamma=0 case in our ablation corresponds to an absolute-hinge baseline, which isolates the benefit of relational coupling beyond hinge-gated correction alone.
  2. We add a matched continuous auxiliary baseline (Cont-Aux) that applies the same auxiliary update direction at every training step. This comparison shows that this auxiliary direction is itself helpful, while the conditional activation boundary in LCPO provides additional improvement.
  3. We add a multi-seed evaluation on both UltraFeedback and TL;DR. The results show small seed variance and stable ranking across methods, supporting the robustness of the reported gains.
  4. We add a cross-judge evaluation using Gemini-3.5-Flash in addition to GPT-based judging. The results show that the overall ranking pattern is preserved.
  5. We compare with additional general preference-optimization baselines, including SimPO, KTO, and β\beta-DPO, and LCPO remains best overall in this broader comparison on UltraFeedback.
  6. We further analyze the batch-wise constraint by examining per-sample implicit reward distributions. The results show that LCPO improves not only the mean reward level but also the lower tail.
  7. We provide response-length statistics for AlpacaEval-2.0 to address possible verbosity confounds. LCPO is not the longest-response method, yet it still achieves the best length-controlled result.
  8. We revise the discussion of calibration-free to clarify that it refers to avoiding externally specified absolute baselines or fixed reward targets, rather than common-shift invariance.
  9. We revise the explanation of target-preserving to clarify that LCPO preserves the original DPO objective when no correction is active, rather than preserving every unconstrained DPO optimum regardless of feasibility.
  10. We further clarify the intended interpretation of LCPO: it is designed to mitigate reward-level collapse relative to the achieved reward margin, rather than imposing a hard absolute lower bound on the implicit rewards.

We again thank all reviewers and area chairs for their thoughtful feedback and careful consideration.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.