- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies reward degeneration in DPO: the implicit rewards of both preferred and dispreferred responses can decrease even as their preference margin grows. It analyzes this behavior by separating reward margin from reward level and proposes three principles for corrective priors: level-active, target-preserving, and calibration-free.
The proposed method, LCPO, adds a batch-level hinge penalty that constrains the relationship between preferred and dispreferred rewards. When the constraint is satisfied, the penalty vanishes and the objective reduces to standard DPO.
Experiments on UltraFeedback/AlpacaEval-2.0, TL;DR, and Anthropic-HH report improved alignment performance and less reward degeneration across several model families and sizes. The paper also examines training dynamics, hyperparameter sensitivity, and batch-wise versus sample-wise constraints.
Critical strengths
LCPO operates on rewards already computed during training and requires only an additional hinge penalty, making it a practical modification to an existing pipeline.
On Qwen2.5-7B, LCPO reaches a 68.4% win rate against the chosen TL;DR responses, compared with 44.5% for DPO and 60.5% for the strongest competing baseline.
Results include Qwen models from 1.5B to 14B, Llama and Mistral backbones, and three preference-alignment datasets. This provides useful evidence that the method is not tied to a single model or task.
Reward trajectories, hyperparameter sweeps, and the comparison between batch-wise and sample-wise constraints illustrate the trade-off between limiting reward decline and allowing preference optimization.
Critical weaknesses
The theoretical motivation needs a clearer distinction between underlying and implicit rewards. Underlying rewards are identifiable only up to an additive shift, but the policy-based implicit reward has its offset fixed by normalization. LCPO’s constraint is also sensitive to a common reward shift when .
The experiments do not isolate the proposed mechanism. Appendix B shows that the active penalty adds preferred-response likelihood and dispreferred-response unlikelihood updates. Without a matched comparison that applies the same auxiliary update continuously, it is difficult to tell how much of the gain comes from the proposed activation boundary.
Questions for authors
How do you distinguish the additive ambiguity of underlying rewards from the normalization-fixed implicit rewards? In particular, what precisely does “calibration-free” mean given that Equation 11 is sensitive to a common reward shift?
Have you compared LCPO with a version that applies the same likelihood/unlikelihood auxiliary update at every step, with its weight tuned separately? This would help establish whether the activation boundary itself accounts for the improvement.
Can you provide confidence intervals for the main win-rate comparisons and results across multiple training seeds, particularly for the AlpacaEval gains over the strongest baselines?
6 comments
Response to Q3: Provide multi-seed results.
We thank the reviewer for raising this important point. To address the concern about result stability, we added a new appendix section D.6 reporting multi-seed evaluation for the main results. Specifically, we reran all methods with 3 different random seeds on both UltraFeedback and TL;DR, and report the mean$\pm$standard deviation of the win rates. In particular, this directly addresses the robustness of the AlpacaEval-2.0 gains over the strongest baselines.
The multi-seed results show that the seed variance is consistently small, while the ranking of methods remains stable. On UltraFeedback, the standard deviations are within about $\pm$0.6%, and LCPO achieves 41.2$\pm$0.4% WR and 41.9$\pm$0.3% LC, compared with 39.3$\pm$0.4% and 38.9$\pm$0.5% for the strongest baseline DPOP. On TL;DR, the standard deviations are within about $\pm$0.9%, and LCPO reaches 68.2$\pm$0.6% against the chosen responses and 74.5$\pm$0.6% against SFT, compared with 60.1$\pm$0.4% and 65.6$\pm$0.5% for the strongest baseline APO-zero.
Overall, the performance gaps of LCPO over the strongest baselines are substantially larger than the observed seed-to-seed variation, indicating that the gains are robust and not driven by a favorable random initialization. We have added these multi-seed results and discussion in the revised manuscript Appendix D.6.
We hope our responses clarify the points raised. please let us know if you have any further questions. Thank you for your time and feedback!
Further Discussion with Reviewer A8Td
Dear Reviewer A8Td,
Thank you again for taking the time to review our paper and for your thoughtful and constructive comments. We have carefully considered your feedback and provided detailed responses to each of your questions:
-
For W1/Q1, we clarified the distinction between the notion of calibration-free used in our paper and the shift-invariance interpretation raised in your comment. We also explained why shift-invariance cannot provide a restoring force along the reward-level direction, and therefore why LCPO is not intended to be shift-invariant.
-
For W2/Q2, we added the comparison experiment you suggested in the revision and verified the contribution of the conditional activation boundary.
-
For Q3, we added multi-seed experiments in the revision and confirmed that the performance gains of LCPO over the baselines are substantially larger than the variation caused by random seeds.
As the author–reviewer discussion phase is drawing to a close, we would greatly appreciate knowing whether our responses have addressed your concerns. We would also be very happy to clarify any remaining questions if needed.
Thank you again for your careful review.
Best regards,
Authors
Response to Reviewer A8Td
We thank the reviewer for the rigorous and insightful feedback! We address the concerns below.
Response to W1 & Q1: Calibration-Free vs. Shift-Invariance.
We agree with the reviewer's derivation: because γ=1, the LCPO constraint is mathematically sensitive to a constant shift C in the implicit reward space, i.e., it introduces a (γ−1)C term. However, we would like to clarify that this observation concerns shift-invariance, which is conceptually distinct from the calibration-free principle we proposed.
(1) Distinguishing shift-invariance from calibration-free.
"Shift-invariance" refers to whether a constraint remains unchanged under r→r+C. In contrast, in our paper, "calibration-free" is explicitly defined as avoiding reliance on externally specified absolute baselines or fixed target values for the implicit rewards (lines 211–213 of the original submission). For example, methods that enforce conditions such as rw≥0, or drive rw and rl toward fixed constants, introduce an external absolute calibration. By contrast, LCPO does not impose such externally specified absolute targets; it constrains only the relation between the chosen and rejected implicit rewards. Therefore, our claim of being calibration-free was not intended to mean invariance to arbitrary common shifts, but rather freedom from external human-defined absolute calibration.
(2) Shift-invariance cannot provide level-active control.
We further argue that common-shift invariance would make a corrective term blind to the reward-level direction. In the (u,v) decomposition, a common shift leaves the margin u unchanged while changing the level v. Therefore, a strictly common-shift-invariant correction cannot remain directly sensitive to v, and thus cannot provide an explicit restoring force along the reward-level direction. Under such a formulation, the correction can only act through the margin-related component, rather than directly regulating the reward level itself. In this sense, the shift sensitivity of LCPO is precisely what enables the method to be level-active and correct reward degeneration. That is, common-shift sensitivity is not in conflict with our design goal; it is a necessary feature for explicit level control in the (u,v) view.
(3) Revisions to the manuscript.
To prevent ambiguity, we have revised the description of "calibration-free" in Section 3.3 (lines 214–233 of the latest submission) to further clarify that it means free from externally specified absolute calibration. We have also added a new discussion in Appendix F.1 that distinguishes calibration-free from shift-invariance, and explains why common-shift sensitivity is necessary for a correction that is explicitly level-active.
Response to W2 & Q2: Compare LCPO with a matched version that applies the same auxiliary update continuously.
We thank the reviewer for this helpful suggestion and agree that such a matched control is important for isolating the role of LCPO's activation boundary. In response, we added a new ablation study in appendix section D.5 comparing LCPO against a matched continuous auxiliary baseline (Cont-Aux) that applies the same auxiliary update direction at every training step, instead of activating it only when the LCPO constraint is violated. As suggested by the reviewer, we tune the weight of this baseline separately.
This additional comparison is informative. Cont-Aux consistently improves over vanilla DPO, which confirms that the auxiliary update direction itself is indeed helpful. At the same time, it still consistently underperforms LCPO. On UltraFeedback, it achieves 38.5% WR and 38.3% LC, compared with 41.4% and 41.8% for LCPO. On TL;DR, it reaches 62.5% against the chosen responses and 68.1% against SFT, compared with 68.4% and 74.8% for LCPO.
Taken together, these results suggest that LCPO's improvement is not explained solely by the auxiliary update direction. Rather, the conditional activation boundary itself contributes additional value: applying the level correction only when needed performs better than keeping the same correction active throughout training. We have added this experiment and the corresponding discussion in the revised manuscript Appendix D.5.