- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates prompt-specific regularization in Direct Preference Optimization by deriving optimal coefficients from per-prompt KL constraints using KKT conditions. The authors establish that , where is the reference-policy reward standard deviation and depends on the standardized reward distribution and KL budget. Under shared standardized reward distributions, optimal coefficients are proportional to . Based on this result, KKT-DPO estimates reward dispersion from policy-to-reference log-ratios using reference samples and a lagged policy, with median anchoring. Experiments on Mistral-7B and Llama-3-8B demonstrate improvements over DPO and dynamic- baselines on AlpacaEval 2 and Arena-Hard, alongside gains in reasoning, mathematics, coding, and instruction following.
Critical strengths
The paper provides a principled theoretical foundation for prompt-specific through KKT conditions and reward-variance decomposition. Experiments demonstrate consistent improvements across models and benchmarks, supported by analyses of realized KL divergence.
Critical weaknesses
The practical estimator has not been shown to closely approximate the theoretically optimal multipliers, particularly given the assumption of similar standardized reward distributions across prompts and the use of a lagged policy.
Evaluations on newer model families, such as Qwen3.5, are lacking.
Questions for authors
- How closely do the estimated values approximate the KKT-optimal multipliers in a controlled setting with known rewards and KL budgets?
- Does KKT-DPO maintain its improvements on newer model families, such as Qwen3.5?
5 comments
Response for Reviewer PDwj (Part2/2)
W2 and Q2: Does KKT-DPO maintain its improvements on newer model families such as Qwen3.5?
A2: Thank you for your suggestion. We extend our evaluation to Qwen3.5-4B, testing KKT-DPO and all baselines on AlpacaEval 2 and Arena-Hard with thinking mode disabled.
We use HuggingFaceH4/ultrafeedback_binarized[1], training on61,135 preference pairs and validating on 2,000 pairs. For DPO and KKT-DPO, we use and learning rate , following the Zephyr DPO recipe in alignment-handbook[2]. Training uses cosine scheduling, a warmup ratio of 0.1, one epoch, and effective batch size 128. We retain maximum sequence/prompt lengths of 2048/1800, consistent with our Llama and Mistral experiments. Other baselines use the configurations reported in their original papers. The complete results are below.
Qwen3.5-4B
| Method | AlpacaEval 2 LC (%) | AlpacaEval 2 WR (%) | Arena-Hard SC (%) | Arena-Hard WR (%) |
|---|---|---|---|---|
| SFT | 41.91 | 50.43 | 73.7 | 63.3 |
| DPO | 44.32 | 51.35 | 77.7 | 68.5 |
| IPO | 44.26 | 51.79 | 76.3 | 67.2 |
| SimPO | 46.88 | 53.68 | 78.5 | 68.9 |
| β-DPO | 46.10 | 52.47 | 76.6 | 66.1 |
| ε-DPO | 46.03 | 51.63 | 77.9 | 69.1 |
| MallowsPO-θ | 46.95 | 53.25 | 78.9 | 69.5 |
| KKT-DPO | 47.13 | 55.75 | 79.4 | 70.3 |
KKT-DPO achieves the highest point estimates across all four metrics, improving over DPO by 2.81 pp in AlpacaEval 2 LC and 1.70 pp in Arena-Hard SC. These results support the extension of its improvements to the newer Qwen3.5 model family.
We have added the Qwen3.5-4B experiment in Appendix E.5.
References
[1] Cui et al. (2023). UltraFeedback: Boosting Language Models with High-quality Feedback.
[2] Tunstall et al. (2023). Zephyr: Direct Distillation of LM Alignment.
Thank you again for your time, effort, and valuable feedback. If you have any questions or suggestions, please let us know. We would be grateful to hear them.
Further discussion with Reviewer PDwj
Dear Reviewer PDwj,
Thank you for taking the time to review our paper and for your thoughtful and constructive comments. We have carefully considered your comments and provided detailed responses to your questions:
- We conduct a controlled known-reward experiment that compares the estimated coefficients directly with exact KKT-optimal multipliers.
- We extend the evaluation to Qwen3.5-4B, comparing KKT-DPO with all baselines on AlpacaEval v2.0 and Arena-Hard v0.1.
As the author–reviewer discussion phase is drawing to a close, we would appreciate knowing whether our responses have addressed your concerns, and we are happy to clarify any remaining points.
Thank you again for your careful review.
Best regards,
Authors
Response for Reviewer PDwj (Part1/2)
We thank the reviewer for recognizing the principled theoretical foundation, consistent empirical improvements, and realized-KL analyses.
W1 and Q1: How closely do the estimated coefficients approximate the KKT-optimal multipliers in a controlled setting?A1: Thank you for suggesting. We conducted a controlled preference-learning experiment with known rewards and reference distributions, allowing us to compare the estimated coefficients directly with the exact KKT-optimal multipliers.Data and reward construction. The experiment contains 38,400 training prompts, each with 64 candidate responses. For each prompt, the full-support reference distribution is categorical, with probabilities obtained by applying softmax to 64 independently sampled N(0,1) logits. Rewards follow a shared learnable mapping and are centered and scaled under each prompt’s reference distribution, so their reference mean and standard deviation are exactly μi and σi. We sample μi∼N(0,1) and generate σi through an exponentiated fixed projection of the prompt features; the training reward scales have median 1 and log-scale standard deviation 0.6. This construction introduces prompt-dependent reward scales without explicitly forcing identical standardized reward distributions.
Preference sampling and training. Preference pairs contain two distinct candidates sampled from the reference distribution, with labels sampled according to Bradley–Terry probabilities determined by their known reward differences. Separately, we cache n=5 reference samples per prompt for scale estimation; the 64-candidate support is used for exact distributional calculations. The policy takes the form πθ∝πrefexp(gθ), where gθ is a two-hidden-layer ReLU network with width 128 and a zero-initialized output layer, making the initial policy equal to the reference. We train with Adam at learning rate 0.001, batch size 128, one epoch, and one seed: 30 common-coefficient warmup steps followed by five pools of 54 steps each. Coefficient estimation follows our LLM procedure, using lagged snapshots, pool-wise median anchoring, and default shrinkage α=2/3; the synthetic refresh interval is N=54.
Exact multipliers. We numerically solve the full-distribution KL equality for each βi∗. The common budget is K=0.458155, calibrated so that the median exact training multiplier equals βanchor=1. Thus, absolute comparisons use an anchor-matched budget.
Metric. Spearman correlation measures ranking agreement. We report its mean across pools and its pooled value. Fold discrepancy is max(a/b,b/a); values closer to 1 indicate closer magnitudes. Relative fold compares independently pool-median-normalized coefficients, whereas absolute fold compares unnormalized coefficients under the anchor-matched budget. P90 denotes the 90th percentile.
Results.
The α=0 row is computed post hoc from the same lagged estimates, not from separate training. The high correlations and relative fold agreement support approximate recovery of KKT-optimal coefficient ordering and magnitude in this controlled setting. Default shrinkage preserves within-pool ordering.
These results show that, in this controlled experiment, the estimated coefficients βi correlate strongly with the exact KKT-optimal multipliers βi∗.
We have included the experiment in Appendix H.2, with the training configuration and results reported in Tables H.2–H.3 (highlighted in blue).