- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the training dynamics of Direct Preference Optimization (DPO) from a loss-level sensitivity perspective and proposes Learning-Signal-Controlled Direct Preference Optimization (LSC-DPO). The method identifies the sigmoid factor as a learning signal and dynamically adjusts the DPO coefficient through a feedback control mechanism to maintain the signal near a target level. The authors further provide a local stability analysis of the controller and introduce a signal-budget compensation rule to reduce performance variation across different coefficient initializations. Experiments with Llama3-8B and Qwen3-8B on multiple alignment benchmarks demonstrate that LSC-DPO outperforms standard DPO and several existing preference-optimization baselines on the main evaluation metrics.
Critical strengths
- The paper proposes a learning-signal-controlled update mechanism based on the local sensitivity of the DPO loss. By dynamically adjusting the DPO coefficient through feedback control, LSC-DPO regulates the learning signal without introducing additional model components. The authors also provide a local stability analysis of the controller, offering theoretical support for the proposed update rule.
- The paper evaluates LSC-DPO on two 8B language models across multiple alignment benchmarks. Comparisons with existing preference-optimization baselines, analyses of learning-signal dynamics, and experiments on coefficient initialization provide evidence of the method's effectiveness.
Critical weaknesses
The decay of the sigmoid factor as the preference margin increases may reflect normal preference learning rather than an optimization issue. Although LSC-DPO improves performance, the paper does not sufficiently explain why this decay needs to be regulated or how doing so improves preference optimization.
Questions for authors
- The paper defines the sigmoid factor (\sigma(-z)) as a learning signal and dynamically adjusts (\beta) to maintain it near a target level. However, the decrease in gradient weighting as the preference margin increases may also reflect normal preference learning. Could the authors further explain why this natural decay needs to be regulated and how maintaining the learning signal near a target level improves preference optimization?
- LSC-DPO uses the batch-mean preference margin (m_k) to compute the feedback signal. Since (\sigma(-\beta\mathbb{E}[m])) generally differs from (\mathbb{E}[\sigma(-\beta m)]), could the authors clarify the rationale for this choice? A comparison with feedback based on the mean per-example learning signal would help assess its effectiveness.
- Although the paper compares LSC-DPO with several preference-optimization baselines, most were introduced between 2023 and 2025. Could the authors include comparisons with relevant recent methods, particularly those proposed in 2026 that address preference-optimization dynamics, or explain why these methods were not considered?
- LSC-DPO dynamically adjusts (\beta) without introducing additional model components. Could the authors provide a comparison of training time with standard DPO to quantify the computational overhead of the feedback control mechanism?
- The target scaled margin (z^) is a key parameter of LSC-DPO, and Appendix J shows that its value substantially affects performance. Could the authors clarify how (z^) should be selected for new models or datasets and whether suitable target values transfer across different settings?
- Table 1 shows that LSC-DPO achieves strong results on AlpacaEval 2 but does not consistently outperform other methods on MT-Bench. For example, its MT-Bench score of 6.9 on Llama3-8B matches those of ε-DPO and SimPO. Could the authors further discuss the differences in relative performance across these benchmarks and identify the settings in which LSC-DPO offers the greatest benefits?
The authors have posted their responses to your comments. Could you please take a look and let us know whether your concerns have been addressed? We encourage active discussion between reviewers and authors during this period.