acceptodds
Under review as a conference paper at ICLR 2027

LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization

Abstract

Direct Preference Optimization (DPO) has become a standard reward-model-free approach for aligning language models with preference data. However, as the scaled preference margin grows during training, the logistic DPO loss becomes progressively less sensitive to further changes. We study DPO from a loss-level geometric perspective and identify the sigmoid factor as a learning signal that characterizes the local sensitivity of the objective. Based on this view, we propose Learning-Signal-Controlled Direct Preference Optimization (LSC-DPO), which dynamically regulates the learning signal near a target regime. A log-space analysis establishes conditions for stable tracking of the target learning-signal regime. Experiments on AlpacaEval 2, MT-Bench, and Anthropic-HH show that LSC-DPO consistently improves over DPO and strong preference-optimization baselines. We further observe that learning-signal dynamics are systematically related to the initial coefficient. We then investigate this relationship and derive a signal-budget compensation rule that substantially reduces performance variation across coefficient initializations.

Primary area:
reinforcement learning
Keywords:
Language Model Alignment, Direct Preference Optimization, Training Dynamics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dSHrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the training dynamics of Direct Preference Optimization (DPO) from a loss-level sensitivity perspective and proposes Learning-Signal-Controlled Direct Preference Optimization (LSC-DPO). The method identifies the sigmoid factor as a learning signal and dynamically adjusts the DPO coefficient through a feedback control mechanism to maintain the signal near a target level. The authors further provide a local stability analysis of the controller and introduce a signal-budget compensation rule to reduce performance variation across different coefficient initializations. Experiments with Llama3-8B and Qwen3-8B on multiple alignment benchmarks demonstrate that LSC-DPO outperforms standard DPO and several existing preference-optimization baselines on the main evaluation metrics.

Critical strengths
  1. The paper proposes a learning-signal-controlled update mechanism based on the local sensitivity of the DPO loss. By dynamically adjusting the DPO coefficient through feedback control, LSC-DPO regulates the learning signal without introducing additional model components. The authors also provide a local stability analysis of the controller, offering theoretical support for the proposed update rule.
  2. The paper evaluates LSC-DPO on two 8B language models across multiple alignment benchmarks. Comparisons with existing preference-optimization baselines, analyses of learning-signal dynamics, and experiments on coefficient initialization provide evidence of the method's effectiveness.
Critical weaknesses

The decay of the sigmoid factor as the preference margin increases may reflect normal preference learning rather than an optimization issue. Although LSC-DPO improves performance, the paper does not sufficiently explain why this decay needs to be regulated or how doing so improves preference optimization.

Questions for authors
  1. The paper defines the sigmoid factor (\sigma(-z)) as a learning signal and dynamically adjusts (\beta) to maintain it near a target level. However, the decrease in gradient weighting as the preference margin increases may also reflect normal preference learning. Could the authors further explain why this natural decay needs to be regulated and how maintaining the learning signal near a target level improves preference optimization?
  2. LSC-DPO uses the batch-mean preference margin (m_k) to compute the feedback signal. Since (\sigma(-\beta\mathbb{E}[m])) generally differs from (\mathbb{E}[\sigma(-\beta m)]), could the authors clarify the rationale for this choice? A comparison with feedback based on the mean per-example learning signal would help assess its effectiveness.
  3. Although the paper compares LSC-DPO with several preference-optimization baselines, most were introduced between 2023 and 2025. Could the authors include comparisons with relevant recent methods, particularly those proposed in 2026 that address preference-optimization dynamics, or explain why these methods were not considered?
  4. LSC-DPO dynamically adjusts (\beta) without introducing additional model components. Could the authors provide a comparison of training time with standard DPO to quantify the computational overhead of the feedback control mechanism?
  5. The target scaled margin (z^) is a key parameter of LSC-DPO, and Appendix J shows that its value substantially affects performance. Could the authors clarify how (z^) should be selected for new models or datasets and whether suitable target values transfer across different settings?
  6. Table 1 shows that LSC-DPO achieves strong results on AlpacaEval 2 but does not consistently outperform other methods on MT-Bench. For example, its MT-Bench score of 6.9 on Llama3-8B matches those of ε-DPO and SimPO. Could the authors further discuss the differences in relative performance across these benchmarks and identify the settings in which LSC-DPO offers the greatest benefits?
Reviewer_25R6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes LSC-DPO, which adjusts DPO’s coefficient during training to keep a learning signal near a target level. It also introduces a compensation rule to reduce sensitivity to the initial β\beta. Results show improvements across standard alignment benchmarks.

Critical strengths
  1. The method is simple and easy to integrate into existing DPO training.
  2. Results are consistent across both model families. AlpacaEval 2 length-controlled win rates improve from 19.2 to 22.7 for Llama3-8B and from 17.2 to 22.1 for Qwen3-8B.
  3. The paper includes useful comparisons with fixed β schedules, parameter ablations, and checks of general model capabilities.
Critical weaknesses
  1. The explanation for the gains remains incomplete. The controlled signal uses the average margin and does not directly capture individual examples’ gradients or the full model update.
  2. The theory establishes local stability of the β controller under positive, slowly changing margins. It does not establish convergence of the overall training process or explain improved alignment.
  3. Parameter sensitivity remains substantial. For Llama3-8B, changing the target margin from 0.40 to 0.30 reduces the length-controlled win rate from 22.7 to 14.7.
  4. Experiments cover only two 8B models, and variation across training seeds is not reported. Broader testing would strengthen confidence in the results.
Questions for authors

See Weaknesses Above.

1 comment

The authors have posted their responses to your comments. Could you please take a look and let us know whether your concerns have been addressed? We encourage active discussion between reviewers and authors during this period.

Reviewer_RY4fOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LSC-DPO, a feedback-based variant of Direct Preference Optimization that dynamically adjusts the DPO coefficient to keep the sigmoid of the scaled batch-mean preference margin near a target level. The authors interpret this quantity as a learning signal and analyze the controller’s local stability and tracking behavior. They also introduce signal-budget compensation, which adjusts the target to offset differences in cumulative signal arising from coefficient initialization. Experiments on Llama3-8B and Qwen3-8B report gains on AlpacaEval 2, MT-Bench, and Anthropic-HH, alongside reduced initialization sensitivity and broadly preserved downstream capabilities.

Critical strengths
  • The method is simple and could be incorporated into existing DPO training with little additional machinery.
  • The reported gains across two model families and the initialization study suggest practical value, while the downstream evaluations show broadly preserved capabilities.
Critical weaknesses
  1. The terminology makes the central idea difficult to follow. The abstract and introduction repeatedly use “learning signal,” “loss-level geometry,” and “local sensitivity” before explaining their relationship. Section 3.1 identifies the sigmoid factor as the learning signal, but the discussion does not clearly distinguish sensitivity to the scaled margin from sensitivity to model parameters. This distinction matters because changing the coefficient also changes the gradient beyond its sigmoid factor. The paper should introduce these quantities more directly and state precisely what the controller regulates.

  2. The motivation for maintaining sensitivity is not sufficiently established. At a fixed coefficient, a small sigmoid factor can indicate that a training pair already has a large positive preference margin relative to the reference model. Reducing its gradient contribution is then an expected consequence of successful fitting. The paper does not convincingly distinguish this situation from premature saturation that prevents useful learning. Although the benchmark gains support the method’s usefulness, they do not establish this explanation. The stability analysis shows that the controller can track its target under stated assumptions, not that maintaining the target improves preference learning.

  3. The open-ended evaluation leaves a relevant comparison unresolved. Appendix E.2 and Table 9 evaluate baseline checkpoints on Arena-Hard with an updated judge, but omit LSC-DPO. The unavailable original judge explains why historical scores cannot be reproduced directly; it does not prevent comparing the proposed method and baselines under the same updated protocol. Such a comparison would strengthen a future version, particularly given the changed rankings reported in the appendix.

Questions for authors

What evidence distinguishes harmful premature saturation from successful fitting of training preferences, and supports maintaining the proposed signal as the reason for improved performance?

1 comment

The authors have posted their responses to your comments. Could you please take a look and let us know whether your concerns have been addressed? We encourage active discussion between reviewers and authors during this period.

Comments on the paper

2 comments

The authors have posted their rebuttal, and the discussion period will end soon. Could you please read it and reply at your earliest convenience, stating whether your concerns have been addressed and whether your rating changes? Thank you so much for your active participation.

Thank you for coordinating the discussion and encouraging reviewer follow-up. We have uploaded a revised manuscript incorporating the clarifications and additional evidence presented during the rebuttal.

The revision clarifies the learning-signal interpretation and the scope of our theoretical claims, and incorporates mechanistic diagnostics and controlled intervention experiments. It also includes updated Arena-Hard evaluations, comparisons with five recent methods on both model families, combination experiments with HyPO and RC, and three-seed comparisons. We have posted a summary of the revisions in each reviewer thread to facilitate checking the updated manuscript.

We sincerely appreciate the time and effort you and the reviewers have devoted to our submission. We hope that the revised manuscript and additional evidence address the concerns raised during the discussion and help inform your assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.