acceptodds
Under review as a conference paper at ICLR 2027

KKT-DPO: Direct Preference Optimization with Optimality-Derived Per-Prompt

19 views

Abstract

Direct Preference Optimization (DPO) aligns large language models (LLMs) with human preferences using a coefficient that controls deviation from a reference model but is fixed across prompts. Existing dynamic methods adjust across batches, pairs, or prompts using method-specific signals, rather than deriving it from the optimization problem. In this paper, we study a per-prompt KL-constrained formulation, in which each prompt has its own multiplier , and characterize the optimal through the KKT conditions. We show that the optimal multiplier decomposes into the product of the reward standard deviation under the reference model and a dimensionless factor determined by the standardized reward distribution and the KL budget. When prompts share the same standardized reward distribution, their optimal multipliers are proportional to their reward standard deviations. This suggests that the reward standard deviations determine how the optimal multipliers differ across prompts, and that an anchor sets the overall level. Based on this, we propose KKT-DPO, which approximates the relative reward standard deviations using responses sampled from the reference policy and scored by policy-to-reference log-ratios under a lagged policy snapshot. It then uses a coefficient tuned for vanilla DPO as a practical anchor to set the absolute multiplier. Experiments show that KKT-DPO attains the best performance among the baselines on the AlpacaEval and Arena-Hard and improves over DPO on reasoning, mathematics, coding, and instruction-following tasks, demonstrating that our per-prompt coefficients derived from the optimality conditions are effective.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Direct Preference Optimization, Reinforcement Learning from Human Feedback, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_PDwjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates prompt-specific regularization in Direct Preference Optimization by deriving optimal coefficients βi∗\beta_i^{*} from per-prompt KL constraints using KKT conditions. The authors establish that βi∗=σiΦi\beta_i^* = \sigma_i \Phi_i, where σi\sigma_i is the reference-policy reward standard deviation and Φi\Phi_i depends on the standardized reward distribution and KL budget. Under shared standardized reward distributions, optimal coefficients are proportional to σi\sigma_i. Based on this result, KKT-DPO estimates reward dispersion from policy-to-reference log-ratios using reference samples and a lagged policy, with median anchoring. Experiments on Mistral-7B and Llama-3-8B demonstrate improvements over DPO and dynamic-β\beta baselines on AlpacaEval 2 and Arena-Hard, alongside gains in reasoning, mathematics, coding, and instruction following.

Critical strengths

The paper provides a principled theoretical foundation for prompt-specific β\beta through KKT conditions and reward-variance decomposition. Experiments demonstrate consistent improvements across models and benchmarks, supported by analyses of realized KL divergence.

Critical weaknesses

The practical β\beta estimator has not been shown to closely approximate the theoretically optimal multipliers, particularly given the assumption of similar standardized reward distributions across prompts and the use of a lagged policy.

Evaluations on newer model families, such as Qwen3.5, are lacking.

Questions for authors
  1. How closely do the estimated β\beta values approximate the KKT-optimal multipliers in a controlled setting with known rewards and KL budgets?
  2. Does KKT-DPO maintain its improvements on newer model families, such as Qwen3.5?
5 comments
AuthorsOpenReview ↗

Response for Reviewer PDwj (Part1/2)

We thank the reviewer for recognizing the principled theoretical foundation, consistent empirical improvements, and realized-KL analyses.


W1 and Q1: How closely do the estimated coefficients approximate the KKT-optimal multipliers in a controlled setting?

A1: Thank you for suggesting. We conducted a controlled preference-learning experiment with known rewards and reference distributions, allowing us to compare the estimated coefficients directly with the exact KKT-optimal multipliers.

Data and reward construction. The experiment contains 38,400 training prompts, each with 64 candidate responses. For each prompt, the full-support reference distribution is categorical, with probabilities obtained by applying softmax to 64 independently sampled N(0,1)\mathcal{N}(0,1) logits. Rewards follow a shared learnable mapping and are centered and scaled under each prompt’s reference distribution, so their reference mean and standard deviation are exactly μi\mu_i and σi\sigma_i. We sample μi∼N(0,1)\mu_i\sim\mathcal{N}(0,1) and generate σi\sigma_i through an exponentiated fixed projection of the prompt features; the training reward scales have median 1 and log-scale standard deviation 0.6. This construction introduces prompt-dependent reward scales without explicitly forcing identical standardized reward distributions.

Preference sampling and training. Preference pairs contain two distinct candidates sampled from the reference distribution, with labels sampled according to Bradley–Terry probabilities determined by their known reward differences. Separately, we cache n=5n=5 reference samples per prompt for scale estimation; the 64-candidate support is used for exact distributional calculations. The policy takes the form πθ∝πrefexp⁡(gθ)\pi_\theta\propto\pi_{\mathrm{ref}}\exp(g_\theta), where gθg_\theta is a two-hidden-layer ReLU network with width 128 and a zero-initialized output layer, making the initial policy equal to the reference. We train with Adam at learning rate 0.001, batch size 128, one epoch, and one seed: 30 common-coefficient warmup steps followed by five pools of 54 steps each. Coefficient estimation follows our LLM procedure, using lagged snapshots, pool-wise median anchoring, and default shrinkage α=2/3\alpha=2/3; the synthetic refresh interval is N=54N=54.

Exact multipliers. We numerically solve the full-distribution KL equality for each βi∗\beta_i^*. The common budget is K=0.458155K=0.458155, calibrated so that the median exact training multiplier equals βanchor=1\beta_{\mathrm{anchor}}=1. Thus, absolute comparisons use an anchor-matched budget.

Metric. Spearman correlation measures ranking agreement. We report its mean across pools and its pooled value. Fold discrepancy is max⁡(a/b,b/a)\max(a/b,b/a); values closer to 1 indicate closer magnitudes. Relative fold compares independently pool-median-normalized coefficients, whereas absolute fold compares unnormalized coefficients under the anchor-matched budget. P90 denotes the 90th percentile.

Results.

AllocationMean within-pool SpearmanPooled SpearmanRelative fold: median / P90Absolute fold: median / P90
α=0\alpha=00.9770.9781.123 / 1.3781.125 / 1.278
α=2/3\alpha=2/30.9770.9781.295 / 1.3601.196 / 1.361

The α=0\alpha=0 row is computed post hoc from the same lagged estimates, not from separate training. The high correlations and relative fold agreement support approximate recovery of KKT-optimal coefficient ordering and magnitude in this controlled setting. Default shrinkage preserves within-pool ordering.

These results show that, in this controlled experiment, the estimated coefficients βi\beta_i correlate strongly with the exact KKT-optimal multipliers βi∗\beta_i^*.

We have included the experiment in Appendix H.2, with the training configuration and results reported in Tables H.2–H.3 (highlighted in blue).

AuthorsOpenReview ↗

Response for Reviewer PDwj (Part2/2)

W2 and Q2: Does KKT-DPO maintain its improvements on newer model families such as Qwen3.5?

A2: Thank you for your suggestion. We extend our evaluation to Qwen3.5-4B, testing KKT-DPO and all baselines on AlpacaEval 2 and Arena-Hard with thinking mode disabled.

We use HuggingFaceH4/ultrafeedback_binarized[1], training on61,135 preference pairs and validating on 2,000 pairs. For DPO and KKT-DPO, we use β=βanchor=0.01\beta=\beta_{\mathrm{anchor}}=0.01 and learning rate 5×10−75\times10^{-7}, following the Zephyr DPO recipe in alignment-handbook[2]. Training uses cosine scheduling, a warmup ratio of 0.1, one epoch, and effective batch size 128. We retain maximum sequence/prompt lengths of 2048/1800, consistent with our Llama and Mistral experiments. Other baselines use the configurations reported in their original papers. The complete results are below.

Qwen3.5-4B

MethodAlpacaEval 2 LC (%)AlpacaEval 2 WR (%)Arena-Hard SC (%)Arena-Hard WR (%)
SFT41.9150.4373.763.3
DPO44.3251.3577.768.5
IPO44.2651.7976.367.2
SimPO46.8853.6878.568.9
β-DPO46.1052.4776.666.1
ε-DPO46.0351.6377.969.1
MallowsPO-θ46.9553.2578.969.5
KKT-DPO47.1355.7579.470.3

KKT-DPO achieves the highest point estimates across all four metrics, improving over DPO by 2.81 pp in AlpacaEval 2 LC and 1.70 pp in Arena-Hard SC. These results support the extension of its improvements to the newer Qwen3.5 model family.

We have added the Qwen3.5-4B experiment in Appendix E.5.

References

[1] Cui et al. (2023). UltraFeedback: Boosting Language Models with High-quality Feedback.

[2] Tunstall et al. (2023). Zephyr: Direct Distillation of LM Alignment.


Thank you again for your time, effort, and valuable feedback. If you have any questions or suggestions, please let us know. We would be grateful to hear them.

AuthorsOpenReview ↗

Further discussion with Reviewer PDwj

Dear Reviewer PDwj,

Thank you for taking the time to review our paper and for your thoughtful and constructive comments. We have carefully considered your comments and provided detailed responses to your questions:

  1. We conduct a controlled known-reward experiment that compares the estimated coefficients directly with exact KKT-optimal multipliers.
  2. We extend the evaluation to Qwen3.5-4B, comparing KKT-DPO with all baselines on AlpacaEval v2.0 and Arena-Hard v0.1.

As the author–reviewer discussion phase is drawing to a close, we would appreciate knowing whether our responses have addressed your concerns, and we are happy to clarify any remaining points.

Thank you again for your careful review.

Best regards,

Authors

Reviewer_esfPOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary: This paper interprets the β of the DPO as a Lagrange multiplier under prompt-by-prompt KL constraints, obtaining a unique optimal multiplier from the KKT conditions. Then, utilizing the translation invariance and homogeneity of the reward, we obtain β_i* = σ_iΦ_i. Based on this, we propose KKT-DPO: estimating the relative σ_i using the standard deviation of the policy-to-logarithm ratio on the reference model sample, and using the original DPO's β as the median anchor. On open-ended and inference benchmarks for both models, it outperforms the DPO and existing dynamic β methods.

Critical strengths

Strengths

The problem is clear, and the derivations are correct, consistent, and clean. The decomposition of β* = σΦ intuitively explains "why β should be tuned according to the reward scale."

The practical design is reasonable: median anchoring allows the hyperparameter β to be directly reused from DPO; no additional RM is required; coefficients do not backpropagate gradients, simplifying implementation.

The experiments are relatively complete: seven baselines, two types of evaluations, plus ablation, hyperparameter sensitivity, KL mechanism analysis (Figure 3), empirical testing of key hypotheses (Appendix D), and a truthful report of computational costs.

Critical weaknesses

Weaknesses

"Optimal" depends on the model chosen by the author.

A strategy that satisfies "KL ≤ K for each prompt" must also satisfy "average KL ≤ K," meaning the feasible set of the former is a subset of the latter.

Therefore, in the sense of "maximizing expected reward," the optimal value of the prompt-by-prompt constraint will not exceed the average constraint; and the KKT solution of the average constraint is precisely a shared β, i.e., vanilla DPO.

The paper needs to positively demonstrate "why each prompt should have the same KL budget" (e.g., robustness to miscalculation of rewards). Otherwise, the "optimality-derived" in the title is misleading.

The estimator is inconsistent with its own fixed point.

Equation (14) requires the snapshot strategy to share the same τ on all prompts for the ratio to cancel out.

However, after the first pool, the snapshots are trained with different β_i. If the training is close to the optimal solution of the DPO on that prompt, then sd[log π/π_ref] ≈ σ_i/β_i. Substituting β_i ∝ σ_i, the estimator tends to a constant, and the next round pulls β_i back to the anchor point, forming a self-feedback mechanism.

This may explain two phenomena: the LC for N=1 (refreshing the snapshot every step) is 40.39, lower than DPO's 41.78; and why such a strong contraction of α = 2/3 is needed.

The theoretical prescription itself may not be the source of gain.

In Table F.1, the LC for α=0 (directly using the theoretical ratio, no contraction) is 40.88, lower than DPO's 41.78.

N=1 and N=50 also do not exceed DPO; only the default combination (α=2/3, N=100, n=5) is significantly ahead. These hyperparameters appear to have been selected on the AlpacaEval benchmark.

On Mistral, the deliberately misclassified estimators, "missampling" and "misscoring," achieved an LC improvement of approximately 84–86% relative to DPO (25.48 → 32.8/33.0, compared to 34.17 for the full method). On Llama, however, these misclassified estimators were on par with or even slightly lower than DPO, indicating inconsistency between the two models.

This suggests that the gain may primarily stem from "arbitrary, sequential heterogeneous β," or from an increased average β due to a right-skewed distribution, rather than from a correct estimation of σ.

Additional information needed: DPO with the same average β as KKT-DPO; and a control with randomly shuffled β_i allocation.

The testing of key hypotheses is weak.

Appendix D uses a split-half of 32 samples as the noise baseline, but with 32 samples, the W1 distance itself is highly noisy, making it difficult to detect shape differences.

More importantly, Φ is related to K_max = −log Q* (where Q* is the probabilistic quality of the highest reward answer) and depends on the upper tail of the distribution. W1, however, primarily reflects the main body of the distribution.

Statistical significance is insufficient.

All results are from a single training iteration. The standard error of the raw win rate in AlpacaEval is approximately 1.5pp.

On Mistral, the LC difference between KKT-DPO and SimPO is only 1.1; on Llama, the confidence intervals of Arena-Hard and MallowsPO overlap. The statement "best in all" is overly strong.

For inference tasks, comparisons are only made with SFT and DPO. Simply reallocating β improves MBPP by 9.8 points and IFEval by 6.7 points; such a significant improvement requires multiple validation iterations.

Questions for authors

See Weaknesses

6 comments
AuthorsOpenReview ↗

Response for Reviewer esfP (Part1/3)

We thank you for recognizing our clear and consistent theoretical derivations, practical design with straightforward DPO integration, and comprehensive experimental evaluation; we address your concerns below.


W1: Does the feasible-set relationship imply weaker performance than DPO, and why impose a common per-prompt KL budget?

A1: We clarify that our optimality claim is with respect to Eq. (5), where we choose a common per-prompt budget to control each prompt’s departure from the reference policy individually. We agree that the average-constrained problem admits at least as large an optimal expected reward. However, its shared β\beta is the exact multiplier calibrated to the specified KK, whereas standard DPO uses a preset β\beta. Therefore, the feasible-set relationship alone does not establish a performance ordering between KKT-DPO and standard DPO.

An average KL constraint allows some prompts to exceed KK when smaller divergences elsewhere compensate. We adopt separate constraints to prevent this compensation, and use a common upper bound when task-specific divergence tolerances are unspecified. This gives each prompt an independent, equal divergence allowance relative to its own reference distribution, while still permitting different optimal policies and coefficients. Thus, common KK serves uniform prompt-level divergence control without assuming identical task or reward structures.

We have clarified themotivation in Section 3.1 and the scope of constrained optimality in Appendix B.5 (highlighted in blue).

W2: Training with estimated coefficients may create a self-reinforcing estimator feedback loop.

A2: We clarify that our training order excludes the recurrent prompt self-feedback loop you raised. Algorithm C.1 estimates each upcoming pool’s coefficients using a frozen snapshot before training those prompts. Training uses one epoch without repeated prompts, so each post-warmup prompt is scored before its only training visit and is never subsequently re-estimated. The first snapshot follows common-coefficient warmup. The relation si≈σi/βis_i\approx\sigma_i/\beta_i describes a prompt at its own ideal optimum; it does not automatically characterize a snapshot on a new, untrained prompt. Thus, a prompt’s allocated coefficient is not fed back into a subsequent estimate for that same prompt.

Moreover, we have discussed both observations in Section 4.3 and Appendix F.1–F.2; below, we clarify their connection to estimation stability. For snapshot refresh, NN controls both the refresh interval and the pool size for median normalization. At N=1N=1, normalization uses one batch and the snapshot refreshes every step, removing the multi-step lag. The LC increase from 40.39 at N=1N=1 to 44.65 at N=100N=100 in Appendix F.2 supports less frequent lagged-policy refresh. For shrinkage, Section 4.3 identifies finite-nn sampling variation as its motivation. Appendix F.1 shows that without shrinkage, this variation enters the coefficients directly and AE2 LC falls 3.77 points below the default, supporting shrinkage for reducing sensitivity to noise from only five reference samples.

We have expanded Appendix F.1–F.2 to clarify the effect of shrinkage and the joint role of NN.

AuthorsOpenReview ↗

Response for Reviewer esfP (Par2/3)

W3: The gains may depend on selected hyperparameters or the coefficient distribution, while the backbone-dependent ablations raise questions about the scaling signal.

A3: Thank you for this suggestion. We address this concern with a within-pool coefficient-shuffling experiment and an analysis of the existing approximately mean-matched DPO comparison.

(1) β\beta-shuffling experiment. We performed a within-pool β\beta-shuffling experiment, randomly permuted the per-prompt β\beta values obtained from KKT-DPO with Llama-3-8B-Instruct on UltraFeedback within each pool, reassigned them to the training examples, and performed one additional training run using the DPO loss. The permutation preserves the coefficient distribution within each pool, and all other training and evaluation settings remained unchanged.

MethodAlpacaEval 2 LC (%)AlpacaEval 2 WR (%)Arena-Hard SC (%)Arena-Hard WR (%)
DPO41.7840.9124.828.3
KKT-DPO44.6543.9127.030.3
shuffled β\beta39.02 ↓\downarrow38.66 ↓\downarrow23.2 ↓\downarrow25.6 ↓\downarrow

Arrows indicate scores below fixed-β\beta DPO.

(2) DPO with approximately the same average β\beta. In the Llama-3 UltraFeedback experiment, the mean KKT-DPO coefficient is 0.010509, 5.09% above DPO’s fixed 0.01, making the existing baseline approximately mean-matched.

KKT-DPO outperforms this approximately mean-matched baseline. More importantly, shuffling preserves the original coefficient mean and distribution within each pool but lowers all four metrics below DPO. These results support the importance of the prompt-to-coefficient assignment: preserving the coefficient distribution alone is insufficient to retain the gain.

Moreover, the observations you raised concern estimation stability and the contribution of individual components. α=0\alpha=0 uses estimated scales without shrinkage, and its lower LC is consistent with greater sensitivity to sampling noise. Moreover, the claim that only the default configuration exceeds DPO is incorrect: α=1/3\alpha=1/3, N=50N=50, and n=3n=3 achieve LC scores of 42.30, 41.92, and 42.54, respectively, versus DPO’s 41.78. The shared defaults also yield gains on Arena-Hard and all ten downstream tasks on both backbones, supporting their applicability beyond AlpacaEval.

The mis-specified estimators show a consistent direction relative to full KKT-DPO. Mis-specified sampling and scoring achieve LC scores of 32.77 and 32.99, respectively, versus 34.17 on Mistral, and 41.19 and 41.32 versus 44.65 on Llama; both substitutions lower all four open-ended metrics on both backbones. The consistent degradation supports the contribution of reference sampling and policy-to-reference log-ratio scoring, while the shuffle control directly demonstrates the importance of prompt-specific allocation.

W4: Sampling noise and limited sensitivity to rare upper-tail differences may weaken the standardized-shape validation.

A4: We clarify that our validation explicitly measures sampling noise and includes observed upper-tail information.

(1) Appendix D compares shape differences against a sampling-matched baseline. Both same-prompt and cross-prompt comparisons use 32 independently standardized responses:

Reward modelSame-prompt W1Cross-prompt W1ExcessRatio
Skywork Reward V20.1930.2200.0271.14×
InternLM2 Reward0.1920.2290.0371.19×

Across 256 prompts, observed cross-prompt distances are only modestly above the finite-sample baseline for both reward models. The 100 random split repetitions check sensitivity to the partition; Figure D.1(a–b) additionally compares quantile functions using all 64 responses per prompt. Thus, our conclusion is grounded in noise-calibrated comparisons and full-sample quantile plots, not small W1 values alone.

(2) The validation includes the observed upper tail. Eq. (55) compares every sampled order statistic, including the largest rewards. Reference sampling uses temperature 1.0, top-p 1.0, and no top-k truncation, while duplicates are retained to preserve empirical probability mass. The quantile plots also display the observed upper reward quantiles.

Moreover, Q⋆Q^\star is the total reference probability mass of all reward-maximizing responses, and Kmax⁡=−log⁡Q⋆K_{\max}=-\log Q^\star characterizes the limiting divergence. At finite budgets, Φi\Phi_i depends on the full reward cumulant-generating function, as derived in Appendix B.1. Our known-reward experiment therefore complements the shape comparisons by directly comparing estimated coefficients with exact multipliers computed from the complete 64-response distributions.

AuthorsOpenReview ↗

Response for Reviewer esfP (Part3/3)

W5: Small open-ended margins require stronger statistical support, and the large downstream gains require repeated training and broader baseline comparisons.

A5: Thank you for your suggestion. We provide paired-bootstrap comparisons for the small open-ended margins, an additional dynamic-β\beta baseline for downstream tasks, and three-seed validation of the two highlighted gains.

(1) Statistical support for the open-ended margins. We jointly resample the shared evaluation prompts and recompute each method’s controlled metric to obtain paired-bootstrap confidence intervals:

ComparisonMetricImprovement (pp)Paired-bootstrap 95% CI (pp)
Mistral: KKT-DPO − SimPOAlpacaEval 2 LC+1.12[0.42, 1.62]
Llama: KKT-DPO − MallowsPO-θArena-Hard SC+1.50[0.10, 3.10]

Both confidence intervals exclude zero, providing statistical support for these small pairwise gains. As shown in Table 1 and Figure 2, KKT-DPO also improves open-ended performance and all ten downstream tasks on both backbones. The consistent cross-task improvements and statistically supported differences make the gains convincing even where the margins are small.

(2) Additional downstream comparison with a dynamic-β\beta baseline. We evaluate MallowsPO-θ on all ten downstream tasks using both backbones. Scores are percentages; IFEval reports prompt-level strict accuracy.

Llama-3-8B-Instruct

TaskDPOMallowsPO-θ (NEW)KKT-DPO
MATH24.7026.0628.38
TheoremQA17.2516.3819.25
SVAMP78.2078.4079.00
ASDiv83.3385.0086.08
GSM-Plus54.6654.9655.05
BBH72.1472.2773.61
HumanEval54.2753.0458.54
LeetCode11.678.8916.11
MBPP51.1351.3760.90
IFEval62.6662.8564.14
Average51.0050.9254.11

Mistral-7B-Instruct-v0.2

TaskDPOMallowsPO-θ (NEW)KKT-DPO
MATH11.1010.6211.62
TheoremQA11.5010.7511.63
SVAMP62.4062.9063.70
ASDiv67.6466.9969.58
GSM-Plus32.0930.8532.44
BBH63.4063.4165.23
HumanEval29.8831.0931.71
LeetCode3.334.447.22
MBPP32.8334.0834.84
IFEval41.7742.7048.43
Average35.5935.7837.64

KKT-DPO exceeds both baselines on every task for both backbones, extending the downstream evidence beyond comparison with fixed-β\beta DPO.

(3) Three-seed validation of the highlighted gains. We added two independent training seeds (43 and 44) and compare DPO and KKT-DPO using matched seeds:

ComparisonDPO (%)KKT-DPO (%)Matched-seed improvement (pp)
Llama-3 / MBPP51.29 ± 1.2660.82 ± 0.63+9.52 ± 1.89
Mistral / IFEval P-strict41.90 ± 0.3848.49 ± 0.47+6.59 ± 0.83

Values are means ± sample standard deviations over three seeds, computed from unrounded per-seed accuracies. Improvements are positive in every matched-seed comparison, showing that the highlighted MBPP and IFEval gains persist across repeated training runs.


Thank you again for your time, effort, and valuable feedback. If you have any questions or suggestions, please let us know. We would be grateful to hear them.

Reviewer_a9J1OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Standard Direct Preference Optimization (DPO) fixes the regularization coefficient β across all prompts, ignoring variations in optimal reward-divergence trade-offs. To resolve this, the authors formulate alignment under per-prompt KL constraints and derive the optimal multipliers β_i* via KKT conditions. Theoretically, β_i* decomposes into the reference model's reward standard deviation σ_i and a dimensionless factor, proving that optimal coefficients scale with reward dispersion. Leveraging this insight, the authors propose KKT-DPO, which estimates relative reward standard deviations via policy-to-reference log-ratios under a lagged policy snapshot and anchors the median prompt to a tuned DPO coefficient. Across Llama-3-8B and Mistral-7B, KKT-DPO consistently outperforms vanilla DPO and existing dynamic baselines on open-ended generation, reasoning, mathematics, coding, and instruction-following benchmarks.

Critical strengths
  1. Unlike existing heuristic-driven dynamic methods, this work establishes a rigorous optimization perspective by formulating a per-prompt KL-constrained problem and deriving multiplier optimality conditions via KKT, providing clear mathematical grounding.
  2. The method requires no auxiliary reward model training or online rollouts. By merely computing log-ratio standard deviations with policy snapshots, it easily integrates into standard DPO infrastructure with minimal pipeline disruption.
  3. The approach yields consistent gains across two LLM backbones on reasoning, math, code, and chat tasks. Furthermore, realized KL analysis demonstrates that improvements originate from fairer cross-prompt budget reallocation rather than extra divergence.
Critical weaknesses
  1. The practical algorithm relies on unproven heuristics: assuming identical standardized reward distributions across distinct prompts, approximating the optimal policy using lagged snapshots, and setting the median multiplier via an ungrounded anchor rather than KKT derivation.
  2. Enforcing an identical divergence budget ‭KK‬ for all prompts alters the original alignment objective. Tasks varying from deterministic reasoning to open-ended creativity naturally demand heterogeneous policy deviations, making a rigid per-prompt equality poorly justified theoretically and practically.
  3. The experiments demonstrate that dynamic scaling improves performance, but they fail to prove whether the estimated multipliers truly approximate the KKT-optimal coefficients, rather than merely acting as an empirical entropy- or uncertainty-based heuristic.
Questions for authors
  1. Could you evaluate KKT-DPO in a controlled synthetic environment with a known reward function, verifying whether the estimated coefficients ‭βi\beta_i‬ correlate strongly with the exact KKT-optimal multipliers ‭βi∗\beta_i^*‬?
  2. Why should diverse tasks share an identical KL budget ‭KK‬? Have you compared this equal per-prompt constraint against adaptive per-prompt budgets or the standard global expectation constraint?
  3. How does KKT-DPO compare against simpler per-prompt scaling baselines, such as scaling coefficients proportionally to reference model output entropy or prompt reward margins, to isolate the benefit of the theoretical decomposition?
8 comments
AuthorsOpenReview ↗

Response for Reviewer a9J1 (Part1/4)

We thank you for recognizing our rigorous optimization foundation, straightforward integration into standard DPO, and consistent gains across backbones and tasks supported by the KL analysis. We address your concerns below.


W1: The practical algorithm relies on three unproven heuristics.

A1: We clarify the theoretical motivation and empirical support for these three design choices below.

(1) Identical standardized reward distributions provide a sufficient condition for proportionality, with empirical support for approximate similarity. Proposition 3.3 gives the exact decomposition βi⋆=σiΦi\beta_i^\star=\sigma_i\Phi_i. Corollary 3.4 further proves that identical standardized reward distributions yield a shared Φi\Phi_i at a common budget, making optimal coefficients exactly proportional to reward standard deviations. We adopt approximate proportionality in practice and assess this condition in Appendix D: median cross-prompt Wasserstein distances are 1.14× and 1.19× the corresponding same-prompt split-half baselines. These results support treating the standardized-distribution factor as approximately shared in the studied setting.

(2) Lagged snapshots estimate a theoretically derived statistic using a target-network construction. Eq. 14 establishes the exact relationship between policy-to-reference log-ratio standard deviations and reward standard deviations under the ideal common-coefficient policy. Since the true reward and this ideal policy are unavailable, we estimate the same statistic using a lagged snapshot in Sections 4.2–4.3. Following the target-network construction in reinforcement learning, we refresh the snapshot once per pool and hold it fixed during scale estimation, separating estimation from subsequent policy updates.

(3) Median anchoring sets the overall scale through a global hyperparameter. The proportionality result in Section 4.1 determines coefficient ratios, while their overall scale depends on the KL budget, which is unspecified in practice. We therefore use βanchor\beta_{\mathrm{anchor}} as a global hyperparameter, reuse the coefficient selected for vanilla DPO, and assign it to the median-scale prompt through median normalization. Table F.1 reports Llama-3 AlpacaEval 2 LC scores of 42.44, 44.65, and 38.86 for anchors of 0.005, 0.01, and 0.05, respectively. The best tested anchor, 0.01, coincides with the coefficient selected for vanilla DPO. These results support the practical calibration choice.

W2 and Q2-1: Why should diverse tasks share an identical per-prompt KL budget?

A2-1: Thank you for raising this modeling question. We clarify that we adopt a common per-prompt budget to control each prompt’s departure from its reference policy independently under a uniform divergence tolerance. In contrast, an average constraint permits divergence at some prompts to exceed KK through compensation elsewhere. When task-specific tolerances are unspecified, a common upper bound provides a task-agnostic default. We agree that deterministic reasoning and open-ended creativity have different reward structures and may benefit from different divergence budgets. However, task-type differences alone do not determine the appropriate KiK_i for each task. A common budget still permits prompt-dependent reward scales and optimal regularization coefficients. We further clarify that Eq. (5) imposes a common KL upper bound. In practice, KKT-DPO uses the theoretically derived relative coefficient allocation with median anchoring but does not explicitly enforce the per-prompt KL constraints; therefore, realized KL varies across prompts.

AuthorsOpenReview ↗

Response for Reviewer a9J1 (Part2/4)

Q2-2: Have you compared this equal per-prompt constraint against adaptive per-prompt budgets or the standard global expectation constraint?

A2-2: Thank you for your suggestion. We further compare the three formulations theoretically according to their divergence-control objectives.

(1) A global average budget allows cross-prompt compensation. Some prompts can exceed KK if others use less budget. This formulation maximizes expected reward under an overall budget but does not provide a common prompt-level divergence bound.

(2) Adaptive per-prompt budgets express task-specific divergence tolerances. They allow some prompts to depart further from their reference policies than others, requiring a justified budget-allocation rule. However, KL measures departure from the reference distribution, so differences in response determinism or diversity alone do not determine the appropriate KiK_i.

(3) A common per-prompt budget provides uniform, independent divergence bounds. Each prompt retains its own reference distribution and reward structure; therefore, the same KK still permits different optimal policies and coefficients. A common bound controls departure from the reference without assuming identical tasks.

Thus, we adopt a common budget as a task-agnostic default when task-specific divergence tolerances are unspecified. Designing and evaluating heterogeneous budgets based on specified task requirements remains a direction for further research.

We have clarified the motivation in Section 3.1 and added the comparison of budget formulations in Appendix B.5 (highlighted in blue).

AuthorsOpenReview ↗

Response for Reviewer a9J1 (Part3/4)

W3-1 and Q1: Do the estimated coefficients approximate the exact KKT-optimal multipliers?

A3-1: Thank you for suggesting. We conducted a controlled preference-learning experiment with known rewards and reference distributions, allowing us to compare the estimated coefficients directly with the exact KKT-optimal multipliers.

Data and reward construction. The experiment contains 38,400 training prompts, each with 64 candidate responses. For each prompt, the full-support reference distribution is categorical, with probabilities obtained by applying softmax to 64 independently sampled N(0,1)\mathcal{N}(0,1) logits. Rewards follow a shared learnable mapping and are centered and scaled under each prompt’s reference distribution, so their reference mean and standard deviation are exactly μi\mu_i and σi\sigma_i. We sample μi∼N(0,1)\mu_i\sim\mathcal{N}(0,1) and generate σi\sigma_i through an exponentiated fixed projection of the prompt features; the training reward scales have median 1 and log-scale standard deviation 0.6. This construction introduces prompt-dependent reward scales without explicitly forcing identical standardized reward distributions.

Preference sampling and training. Preference pairs contain two distinct candidates sampled from the reference distribution, with labels sampled according to Bradley–Terry probabilities determined by their known reward differences. Separately, we cache n=5n=5 reference samples per prompt for scale estimation; the 64-candidate support is used for exact distributional calculations. The policy takes the form πθ∝πrefexp⁡(gθ)\pi_\theta\propto\pi_{\mathrm{ref}}\exp(g_\theta), where gθg_\theta is a two-hidden-layer ReLU network with width 128 and a zero-initialized output layer, making the initial policy equal to the reference. We train with Adam at learning rate 0.001, batch size 128, one epoch, and one seed: 30 common-coefficient warmup steps followed by five pools of 54 steps each. Coefficient estimation follows our LLM procedure, using lagged snapshots, pool-wise median anchoring, and default shrinkage α=2/3\alpha=2/3; the synthetic refresh interval is N=54N=54.

Exact multipliers. We numerically solve the full-distribution KL equality for each βi∗\beta_i^*. The common budget is K=0.458155K=0.458155, calibrated so that the median exact training multiplier equals βanchor=1\beta_{\mathrm{anchor}}=1. Thus, absolute comparisons use an anchor-matched budget.

Metric. Spearman correlation measures ranking agreement. We report its mean across pools and its pooled value. Fold discrepancy is max⁡(a/b,b/a)\max(a/b,b/a); values closer to 1 indicate closer magnitudes. Relative fold compares independently pool-median-normalized coefficients, whereas absolute fold compares unnormalized coefficients under the anchor-matched budget. P90 denotes the 90th percentile.

Results.

AllocationMean within-pool SpearmanPooled SpearmanRelative fold: median / P90Absolute fold: median / P90
α=0\alpha=00.9770.9781.123 / 1.3781.125 / 1.278
α=2/3\alpha=2/30.9770.9781.295 / 1.3601.196 / 1.361

The α=0\alpha=0 row is computed post hoc from the same lagged estimates, not from separate training. The high correlations and relative fold agreement support approximate recovery of KKT-optimal coefficient ordering and magnitude in this controlled setting. Default shrinkage preserves within-pool ordering.

These results show that, in this controlled experiment, the estimated coefficients βi\beta_i correlate strongly with the exact KKT-optimal multipliers βi∗\beta_i^*.

We have included the experiment in Appendix H.2, with the training configuration and results reported in Tables H.2–H.3.

Reviewer_3iYNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper characterizes optimal DPO coefficients under a common per-prompt KL budget. The coefficients are proportional to reward standard deviations when standardized reward distributions match across prompts. KKT-DPO uses policy snapshots to estimate these scales and reports gains on two model families.

Critical strengths
  1. The analysis separates the effects of reward scale, distribution shape, and KL budget on the optimal coefficient. This gives a useful interpretation of per-prompt regularization.

  2. KKT-DPO has the highest reported point estimates in Table 1 and improves over DPO on all ten downstream tasks for both models. These results support its empirical promise.

Critical weaknesses
  1. Eq. (14) recovers reward scales from an optimal policy with a shared coefficient. The paper does not establish whether Eq. (15) preserves these relative scales when later snapshots are trained with prompt-dependent coefficients.

  2. Section 5.2 compares only with SFT and DPO. These results show gains over fixed-β DPO but do not establish an advantage over existing dynamic-β methods on these tasks.

Questions for authors
  1. What supports the shared-scale approximation in Eq. (15) after training with prompt-dependent β? How do the estimates compare with those from a fixed common-β DPO policy?

  2. What motivates replacing the average KL constraint in Eq. (1) with the same KL budget for every prompt in Eq. (5)? What benefit is this change intended to provide?

  3. Does shuffling β across prompts within each pool reduce performance? This would test whether the prompt-to-coefficient assignment matters beyond the coefficient distribution itself.

  4. Can you compare with at least one dynamic-β baseline on the downstream tasks under a matched tuning budget?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3iYN (Part1/2)

We thank the reviewer for recognizing the useful interpretation of per-prompt regularization, and for highlighting KKT-DPO’s leading point estimates and consistent downstream improvements across both model families. We address the following concerns below.


W1 and Q1-1: What supports the shared-scale approximation in Eq. 15 after training with prompt-dependent βi\beta_i?

A1-1: Under a prompt-specific exponential-tilt approximation, Eq. (14) gives siτisjτj≈σiσj\frac{s_i\tau_i}{s_j\tau_j}\approx\frac{\sigma_i}{\sigma_j}. Thus, using si/sjs_i/s_j to estimate σi/σj\sigma_i/\sigma_j introduces a scale error when τi/τj\tau_i/\tau_j deviates from unity. However, our implementation implicitly assumes an approximately common effective scale within each pool, which cancels under median normalization. Our training order motivates this assumption: βi\beta_i is the preference-loss coefficient, whereas τi\tau_i describes the current snapshot’s effective scaling relative to the true reward. In our one-epoch setting with no repeated prompts, each pool is scored before its prompts are trained under their own βi\beta_i; therefore, different assigned coefficients do not imply that the snapshot has already acquired corresponding prompt-specific effective scales. We provide empirical support through two complementary experiments.

(1) Real-LLM scale estimates agree with a common-coefficient checkpoint (Experiment details are in Q1-2). We compare median-normalized scale estimates from KKT-DPO and a common-β\beta DPO checkpoint. The estimates have Spearman correlation 0.9765 and median/P90 larger-to-smaller ratios of 1.108/1.287. This agreement in ordering and normalized magnitude supports approximate preservation of relative scales under prompt-dependent training.

(2) Known-reward validation directly measures relative KKT multiplier recovery (Experiment details are in Reviewer a9J1 W3-1 and Q1). We compare estimated coefficients with computed exact KKT multipliers in a controlled experiment with known rewards. The mean within-pool and pooled Spearman correlations are 0.977 and 0.978, while the median/P90 pool-median-normalized fold discrepancies are 1.123/1.378 before shrinkage and 1.295/1.360 with default shrinkage, using the same lagged estimates. These results support approximate recovery of the exact multipliers’ relative ordering and magnitude, providing a ground-truth check of the approximation’s practical consequence.

We have added the analysis to the Section 4.3 and Appendix H (highlighted in blue).

Q1-2: How do the estimates compare with those from a fixed common-β\beta DPO policy?

A1-2: Thank you for your suggestion. We compare the step-400 KKT-DPO snapshot (Llama-3-8B-Instruct) used for the final scale estimation with a step-400 DPO checkpoint trained with β=0.01\beta=0.01. Both runs use the same training data, hyperparameters, and learning-rate schedule. We use the 256 held-out prompts from the llama3-ultrafeedback test split in Appendix D, with the same five reference responses per prompt and identical numerical settings. We compute sis_i using Eq. 15 and normalize each method’s estimates by its own median.

We report the Spearman correlation, which measures agreement in prompt ordering, with a value of 1 indicating identical rankings. For each prompt, we compute the ratio of the larger normalized estimate to the smaller and report the median and 90th percentile (P90) of these ratios. Ratios closer to 1 indicate closer agreement in normalized magnitude. The results are below.

MetricValue
Spearman correlation0.9765
Ratio between normalized estimates (larger / smaller): median / P901.108 / 1.287

These results support close agreement in both ordering and normalized magnitude with the common-β\beta.

AuthorsOpenReview ↗

Response to Reviewer 3iYN (Part2/2)

Q2: What motivates replacing the average KL constraint in Eq. (1) with the same KL budget for every prompt in Eq. (5)? What benefit is this change intended to provide?

A2: (1) Motivation: As stated in Section 3.1, the average KL constraint couples prompts through a shared divergence budget. Increasing divergence at one prompt leaves less budget for the others. We adopt equal per-prompt KL budgets as a modeling choice to remove this coupling, making the objective and constraints separable into independent per-prompt subproblems. This also assigns the same prompt-level trust region relative to the reference model, rather than allowing divergence to concentrate on a subset of prompts.

(2) Benefit: This enables us to characterize each prompt’s optimal Lagrange multiplier through the KKT conditions and derive βi⋆=σiΦi\beta_i^\star=\sigma_i\Phi_i, providing an optimality-based foundation for per-prompt β\beta. It also yields a per-prompt coefficient allocation rule. Our mechanism analysis is consistent with this interpretation: KKT-DPO flattens the dependence of realized KL on the estimated reward scale and attains higher AlpacaEval 2 LC at lower mean realized KL in Figure 3(b)-(d).

We have clarified the motivation in Section 3.1.

Q3: Does shuffling β\beta across prompts within each pool reduce performance?

A3: Thank you for this suggestion. We performed a within-pool β\beta-shuffling experiment, randomly permuted the per-prompt β\beta values obtained from KKT-DPO with Llama-3-8B-Instruct on UltraFeedback within each pool, reassigned them to the training examples, and performed one additional training run using the DPO loss. The permutation preserves the coefficient distribution within each pool, and all other training and evaluation settings remained unchanged.

MethodAlpacaEval 2 LC (%)AlpacaEval 2 WR (%)Arena-Hard SC (%)Arena-Hard WR (%)
DPO41.7840.9124.828.3
KKT-DPO44.6543.9127.030.3
shuffled β\beta39.02 ↓\downarrow38.66 ↓\downarrow23.2 ↓\downarrow25.6 ↓\downarrow

Arrows indicate scores below fixed-β\beta DPO.

These results show that shuffling β\beta reduces performance, falling below fixed-β\beta DPO, and support the importance of the prompt-to-coefficient assignment beyond the coefficient distribution alone.

We have added this ablation to Section 5.

W2 and Q4: Comparison with dynamic-β\beta baselines on downstream tasks.

A4: Thank you for your suggestion. We have added MallowsPO-θ\theta as a dynamic-β\beta baseline. We recalibrate MallowsPO-θ\theta’s method-specific hyperparameter ϕ⋆\phi^\star on the training split of the UltraInteract–UltraFeedback mixture, following the MallowsPO paper. The remaining training hyperparameters follow the SimPO configuration.

Llama-3-8B-Instruct

TaskDPOMallowsPO-θ\theta (NEW)KKT-DPO
MATH24.7026.0628.38
TheoremQA17.2516.3819.25
SVAMP78.2078.4079.00
ASDiv83.3385.0086.08
GSM-Plus54.6654.9655.05
BBH72.1472.2773.61
HumanEval54.2753.0458.54
LeetCode11.678.8916.11
MBPP51.1351.3760.90
IFEval62.6662.8564.14
Avg51.0050.9254.11

Mistral-7B-Instruct-v0.2

TaskDPOMallowsPO-θ\theta (NEW)KKT-DPO
MATH11.1010.6211.62
TheoremQA11.5010.7511.63
SVAMP62.4062.9063.70
ASDiv67.6466.9969.58
GSM-Plus32.0930.8532.44
BBH63.4063.4165.23
HumanEval29.8831.0931.71
LeetCode3.334.447.22
MBPP32.8334.0834.84
IFEval41.7742.7048.43
Avg35.5935.7837.64

These results extend the evidence for KKT-DPO’s downstream advantages beyond fixed-β\beta DPO to a dynamic-β\beta baseline recalibrated for the current training setting.

We have added the baseline to Section 5.2, with the detailed results included in Appendix E.1.


Thank you again for your time, effort, and valuable feedback. If you have any questions or suggestions, please let us know. We would be grateful to hear them.

AuthorsOpenReview ↗

Further discussion with Reviewer 3iYN

Dear Reviewer 3iYN,

Thank you for taking the time to review our paper and for your thoughtful and constructive comments. We have carefully considered your comments and provided detailed responses to your questions:

  1. We clarify that Eq. 15 relies on the assumption of an approximately common effective scale within each pool, and we support this assumption with two complementary experiments: agreement with a common-coefficient DPO checkpoint and comparison with exact KKT multipliers under known rewards.
  2. We clarify the motivation and benefit for the common per-prompt KL budget.
  3. We conduct a within-pool β\beta-shuffling experiment, supporting the importance of our prompt-to-coefficient assignment.
  4. We add MallowsPO-θ\theta as a dynamic-β\beta baseline on all ten downstream tasks for both backbones.

As the author–reviewer discussion phase is drawing to a close, we would appreciate knowing whether our responses have addressed your concerns, and we are happy to clarify any remaining points.

Thank you again for your careful review.

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers and Area Chairs: Summary of Rebuttal

Dear Area Chair and Reviewers,

We sincerely thank all reviewers and area chairs for their time and feedback, and appreciate the positive assessments from Reviewers 3iYN, esfP, and PDwj following our rebuttal. We also thank Reviewer a9J1 for their follow-up shortly before the discussion deadline.

We are pleased that:

  1. All four reviewers recognize the value of our KKT-based derivations and reward-scale decomposition.

  2. 3iYN and esfP highlight the useful and intuitive interpretation of per-prompt regularization.

  3. a9J1 and esfP appreciate the straightforward integration into DPO without auxiliary reward-model training; a9J1 also notes that the method requires no online rollouts, while esfP highlights the reuse of the DPO coefficient through median anchoring.

  4. 3iYN, a9J1, and PDwj acknowledge the consistent improvements across backbones and tasks.

  5. a9J1 and PDwj highlight the realized-KL analysis and a9J1 further notes that it supports fairer cross-prompt budget reallocation.

  6. esfP appreciates the clean derivations and comprehensive empirical evaluation, including ablations, sensitivity analysis, hypothesis validation, and transparent computational-cost reporting.

We have provided detailed responses to each reviewer and would like to offer a summary:

  1. We clarify how the KKT analysis motivates practical coefficient estimation, including the shared-scale approximation, standardized-distribution assumption, and median anchoring.

  2. We provide complementary real-LLM and known-reward validations to assess relative-scale preservation and approximate recovery of exact KKT multipliers.

  3. We clarify the motivation for common per-prompt KL budgets, the scope of optimality under Eq. (5), and the theoretical comparison with alternative budget formulations.

  4. We explain why the training order excludes recurrent same-prompt feedback and clarify the roles of snapshot refresh and shrinkage.

  5. We examine the source of gains through coefficient-shuffling controls, approximately mean-matched DPO, scoring ablations, and comparisons with simpler scaling baselines.

  6. We provide paired-bootstrap confidence intervals for the highlighted open-ended comparisons and three-seed validation for the highlighted MBPP and IFEval gains, strengthening statistical support and robustness.

  7. We broaden evaluation by adding MallowsPO-θ\theta on downstream tasks and extending comparisons to Qwen3.5-4B.

We sincerely thank all reviewers and area chairs again for their constructive feedback and careful consideration.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.