acceptodds
Under review as a conference paper at ICLR 2027

Uncertainty-Normalized Margins for Direct Preference Optimization

Abstract

Direct preference optimization (DPO) models binary preferences through a Bradley–Terry model with a common noise scale, without explicitly accounting for preference strength or prompt-dependent uncertainty from human feedback. We introduce uncertainty-normalized margin DPO (UNM-DPO), which combines strength-dependent margins with a learned prompt scale. Motivated by a heteroskedastic Bradley–Terry model, we develop two training objectives. Both compare the implicit rewards of preferred and rejected responses, derived from response log-probability ratios to a reference policy. Advantage-only (AO) divides this reward difference by the prompt scale before subtracting the margin; whole-residual (WR) subtracts the margin before dividing by the scale. For the WR comparison model, we establish a necessary and sufficient condition under which known margins make the prompt scale identifiable. We introduce a practical procedure for learning the scale. Building on WR, we introduce ULNM-DPO-WR, which normalizes each response’s implicit reward by its length. We evaluate our methods against DPO and related baselines on HelpSteer2 and HelpSteer3, using the Skywork reward model as a judge. With Llama-3.1-8B-Instruct, ULNM-DPO-WR achieves tie-adjusted win rates against matched DPO of 68.00% and 65.31% on evaluation panels, with higher mean rewards and shorter responses on average. On AlpacaEval with a GPT-4.1 judge and GPT-4-Turbo reference answers, the same 8B policy achieves a length-controlled win rate of 21.62%, compared with 16.39% for DPO and 15.30% for SimPO. These results demonstrate the potential of combining preference-strength margins, learned prompt scales, and length normalization for policy optimization.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Direct preference optimization, Preference strength, Margin-based optimization, Length normalization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Q2DNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

本文探讨了直接偏好优化(DPO)未明确利用偏好强度及不同提示间不确定性差异的问题,并提出了不确定性归一化边际方法(UNM-DPO)。该方法将偏好强度映射为边际,并引入了一个可学习的提示相关尺度,从而构建了两个优化目标AO和WR,其中边际分别在尺度归一化之前或之后进行减法运算。为了将尺度学习与策略优化分离,作者利用五折辅助策略生成的折外奖励差异来拟合尺度网络,随后冻结该尺度并训练最终策略。在此基础上,ULNM-DPO-WR 在尺度拟合和策略训练中均一致地采用按答案长度进行归一化处理,以减轻答案长度对奖励差异和尺度估计的影响。理论部分给出了在已知 WR 比较分数条件下尺度可识别性的必要且充分条件,并通过受控构造分析了长度变化对尺度拟合的影响。实验在 HelpSteer2、HelpSteer3 和 AlpacaEval 数据集上进行,ULNM-DPO-WR 表现最佳,且与固定尺度版本的直接对比验证了在所评估场景中学习尺度的有效性。

Critical strengths
  1. Combine preference intensity with prompt-related scaling and clearly distinguish their roles in the optimization goal. The paper doesn’t just use the direction of preferences; it also adjusts the expected reward gap through intensity-related margins and introduces prompt-related scaling to regulate the sensitivity of the loss. The WR design, in particular, places the margin inside the normalization so that changes in scale don’t alter the threshold the reward gap needs to exceed. This separates, at the target level, 'how big a preference gap is needed' from 'how to penalize deviations from that gap.' This design is clearly interpretable and provides a valuable optimization framework for combining fine-grained preference information with prompt differences.

  2. Scale fitting and policy training are done in stages, with length normalization used consistently in both stages. The method uses out-of-fold predictions from an auxiliary policy to fit the scale, then freezes the scale to train the final policy, reducing the chance that simultaneous updates of policy and scale could compensate for each other and lower the loss by adjusting the scale. ULNM-DPO-WR further uses consistent per-answer length normalization in both scale fitting and policy optimization so that the reward gaps have a consistent measurement across stages, alleviating the interference of sequence length on scale learning. This process balances reasonable training design with deployment convenience, as only the final policy is needed during inference—no auxiliary model or scale network is required.

Critical weaknesses

尺度学习可能会将辅助模型能力的不足解读为降低训练影响的信号。该尺度是基于辅助策略的奖励差距与偏好强度阈值之间的残差进行拟合的。当奖励差距尚未达到阈值时,增加尺度可以降低拟合损失。但这种情况可能是由于标注不可靠,也可能是辅助模型尚未掌握如何处理此类问题。在后一种情况下,后续训练实际上可能会削弱对难度较高但标注可靠的样本的学习效果。反事实预测可以降低样本内拟合的影响,但无法消除辅助模型能力中固有的偏差;缩放边界和正则化也无法区分这两种来源。尽管作者承认缩放可能包含模型误差,但他们仍需解释这种误差将如何影响最终策略,以及是否存在避免持续削弱高难度样本的方法。

Questions for authors

1.Appendix E.5 mentions that in the 8B experiment, AO uses the scale fitted by the 8B auxiliary model, while WR uses the scale transferred from the 1B model. Could you explain the reasoning for using scales from different sources, and whether this difference might affect how we interpret AO and WR results?

2.The length-normalized coefficient is set based on the median length of the training answers. For data with large differences in answer lengths, is this uniform calibration still appropriate? It would be great if the authors could further explain the rationale and scope of this choice.

3 comments

We thank the reviewer for their valuable comments and address them below:

Scale Learning and its interpretation...

Thank you for identifying this concern. The learned scale is intended to capture several sources of residual variation. As stated in Section 3.2, these include annotation ambiguity, task difficulty, and limitations in the pilot's capabilities or optimization. For example, a pilot may fit its training folds but fail to achieve the required preference margins on held-out prompts. Out-of-fold scoring reveals this mismatch, but does not identify its cause.

How pilot errors affect final-policy training. Your concern is valid: when the pilot falls below the target margin, scale fitting can favor a larger scale (Appendix B.1). Let bb denote the final policy's reward gap and c=τm(k)c=\tau m(k) its target margin. The WR loss satisfies ∂ℓWR∂b=−1qσ(c−bq)\frac{\partial \ell_{\mathrm{WR}}}{\partial b}=-\frac{1}{q}\sigma\left(\frac{c-b}{q}\right), where σ\sigma is the sigmoid function. For b≤cb\leq c, increasing qq weakens the learning signal. Thus, a pilot's capability limitations can reduce the influence of difficult but reliable examples. However, the signal still encourages increasing the preferred response's reward gap, and the target margin remains unchanged (Section 3.1).

What the existing safeguards provide. Section 3.2 introduces complementary controls:

  • Frozen scales: The scales are fitted once and frozen before final-policy training. They cannot keep increasing in response to the final policy's difficulties.

  • Geometric-mean constraint: The geometric mean of the scales across unique training prompts is fixed at one. Larger scales for some prompts must therefore be balanced by smaller scales elsewhere, preventing global scale inflation.

  • Bounds and regularization: The bounds q∈[0.5,2]q\in[0.5,2] limit attenuation, while regularization penalizes departures of log⁡q\log q from zero. For an example below its target margin, the bounds imply

    \frac{\partial \ell_{\mathrm{WR}}}{\partial b} \right| \geq \frac{1}{4}.$$ This bounds the loss sensitivity to the reward gap, not the full parameter-gradient magnitude or eventual learning outcome.

These controls constrain the adaptation, but do not distinguish annotation ambiguity from pilot capability errors. In particular, a difficult example can still receive persistently reduced emphasis.

Evidence and a possible additional safeguard. The constant-scale ablation supports the practical usefulness of this adaptation: ULNM-DPO-WR achieves a direct win rate of 57.69% [54.31, 61.00] against its q=1q=1 control on 800 prompts (Appendix F.5). This supports an aggregate benefit in the evaluated setting, rather than a benefit for every difficult subset.

One possible safeguard against persistent attenuation is to gradually return the frozen scales toward q=1q=1, restoring the constant-scale objective at the end of training. This is an untested extension, not part of our reported experiments. We will clarify the scale's broader interpretation, its effect on the learning signal, and the scope of the existing safeguards.


Appendix E.5 mentions that in the 8B experiment...

Thank you for highlighting this distinction. The different scale sources are configuration choices, not requirements of AO or WR. As documented in Appendix E.5, the sequence-level 8B AO policy uses a scale fitted from 8B pilots, whereas the sequence-level 8B WR policy reuses the existing frozen 1B scale. This reuse is possible because the scale supplies a scalar value for each prompt; the transferred estimator retains its original 1B encoder and feature mapping (Appendix E.3).

This difference does affect interpretation. Changing the scale source changes the training signal, so the 8B AO--WR comparison does not isolate the placement of the margin. We will explicitly label scale provenance in the main table and avoid attributing their performance difference solely to the objectives.

A comparison using the same scale is available in the 1B experiments, where AO and WR share one frozen scale (Appendix E.3). Their win rates against the common DPO reference are 58.25% versus 58.75% on 400 prompts, and 56.94% versus 55.88% on 800 prompts (Table 1). These results do not establish a consistent ordering between AO and WR.

Finally, the strongest reported method, ULNM-DPO-WR, uses a newly fitted scale derived from length-normalized out-of-fold scores of the 8B pilots (Appendices E.4--E.5). Its results therefore do not use the transferred 1B scale. We will make this distinction more prominent.

The length-normalized coefficient is set based on the median length...

Actually, we normalize each response by its own length (Equation (13), and the median is only used to rescale the global KL regularization coefficient to adapt the change from a sequence-level units to a token-level units (Appendix E.4) Thus, the procedure does not assume that all responses have similar lengths.

As specified in Appendix E.4, we use βLN=β0Lref\beta_{\mathrm{LN}}=\beta_0 L_{\mathrm{ref}}, where Lref=median⁡i∈I[ni++ni−2]L_{\mathrm{ref}}=\operatorname{median}_{i\in\mathcal{I}}\left[\frac{n_i^+ + n_i^-}{2}\right],

where I\mathcal I contains eligible training comparisons. Dividing log-probability ratios by response length changes their numerical scale; multiplying by LrefL_{\mathrm{ref}} compensates at a typical training length. In particular, the normalized and sequence-level reward gaps coincide when both responses have length LrefL_{\mathrm{ref}}. We use the median because it is less sensitive to unusually long responses than the mean, and compute it entirely from training data. For HelpSteer3, Lref=364.5L_{\mathrm{ref}}=364.5 and βLN=18.225\beta_{\mathrm{LN}}=18.225.

For widely varying lengths, this remains a global magnitude calibration, not an exact equivalence to the sequence-level objective. Per-response normalization still uses each actual length, but one coefficient is not guaranteed to be optimal across all length ranges or to preserve an equivalent KL penalty, as already noted in Appendix E.4. Our experiments support this calibration in the evaluated settings; they do not establish its optimality for arbitrary length distributions. We will clarify this rationale and scope in Section 3.3.

We hope the above explanation addresses your concern, and we sincerely appreciate your time and consideration.

I appreciate the authors’ response. I will keep my score at 3.

Reviewer_uEM1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a method to introduce preference strength information and prompt-dependent uncertainty into direct preference optimization. It presents a new objective using a map to represent margins and a prompt-dependent uncertainty scale fitted via out-of-fold training compatible with standard DPO. They also introduce a length-normalized variant of their approach. The theoretical analysis gives a condition under which known margins identify the prompt scale in the WR model, and shows that response-length variability can distort scale fitting. Finally, they evaluate their method on HelpSteer 2 and 3, and compare against baseline approaches.

Critical strengths
  • The empirical results (at least on the limited selection of models and datasets) seem convincing, i.e., ULNM-DPO-WR performs well against all baselines
  • The paper combines conceptualization, theoretical analysis, and experimental results
  • A lot of details are given (including hashes, seeds, etc.), enabling reproducibility
Critical weaknesses
  • I was somewhat confused by splitting the two pretty similar formulations into two seemingly separate conditions (AO and WR); i.e., it sounded like a significant architecture difference instead of just a minor tweak, which somewhat oversells the contribution. E.g., then in 3.2 q is set to 1, which means both formulations are equivalent anyways here. AO is then largely ignored for the theoretical analysis, and (while benchmarked in section 5), only briefly discussed and without notable results. I think I would treat it as an ablation condition instead.
  • I feel the results from Theorem 4.1 are immediately hedged in the remark; i.e., the practical impact of the proven result is not clear based on the remark. How much of it still transfers over to the realistic scenario/experiments?
  • I feel the response length bias is known in general, and the specific insight related to the formulation and construction, so Proposition 4.3 is a solid result to motivate length normalization, but not a generalizable result outside of the proposed method.
  • The authors make a solid effort to showcase their method, and the results are convincing; however, they are mostly done on just two datasets and one model family (AlpacaEval is discussed, but detailed results are a bit buried). As the paper wants to introduce a new method, I think the empirical evidence could be stronger. Also as far as i understand, training is done on just a single seed (for confidence intervals, boostrapping of eval prompts is used).
  • I was pretty confused by how tau and m(k) were actually used, i.e., how strength information actually enters the learning process and how they are extracted from a dataset. I think they are basically hyperparameters, set over sweeps, but details are buried in the appendices and not clear from reading the main text.
  • A length-normalized DPO baseline with q=1 and no margin seems to be missing; it would be a natural comparison to understand the effect of margins vs. margin scale fitting.
  • It is not properly evaluated whether the learned scale q really captures uncertainty; i.e., I am missing an analysis of correlation with disagreement.

Minor:

  • I feel that y for the prompt and x for the responses is a bit counter-intuitive; I think the other way around is way more common (e.g., in the original DPO paper)
  • CountSketch is not ablated/verified? I am not familiar with this technique used in such a context.
  • The margin scaling coefficient tau is mentioned twice in section 2 (e.g., L120), but it’s never shown how it is used until section 3.1
  • Can you add a reference to section 2.3 - it reads like drawing from existing work („consider... “), so this is properly acknowledged, but I think pointing to related work would be good for readers.
  • The related work section is tucked away to the very last appendix, and (i think) not even referenced in the main text. I would appreciate some mention of it, and it being placed a bit earlier, e.g., as Appendix A if there is no space in the main text.
Questions for authors
  • I do not understand how tau and m(k), i.e., the margins, are actually fitted. I feel there is no mention of preference strength in the experiment section. It is described how q_psi is fitted in 3.2, but i am missing a discussion of tau.
  • Why is the scale network output bounded in [1/2,2]? I feel this is not explained.
  • Don’t the choices of beta0, tau, and m have large effects on psi when fitting the prompt scale? How are these chosen?
  • Is the connection to the Rao-Kupper extension really justified? I feel the tie threshold is something else.
  • Why does beta-DPO collapse in this way?
5 comments

We thank the reviewer for their valuable comments and address them below:

I was somewhat confused by splitting the two pretty similar formulations

Thank you for this suggestion. We agree that AO and WR should be described as two variants of the same loss, rather than different architectures. We will present WR as the main formulation and AO as a margin-placement ablation.

Three points clarify why we included both:

  • They differ in how uncertainty scaling affects the target (Section 3.1; Appendix A). In AO, increasing the scale also increases the reward gap required to exceed the strength margin. In WR, the target gap stays fixed; the scale adjusts the loss's sensitivity around that target. This fixed-margin structure also motivates our theoretical focus on WR: under Theorem 4.1's conditions, known margins allow the scale to be distinguished from reward differences.

  • They coincide only during the q=1q=1 pilot stage (Section 3.2). After training the pilots, we learn a prompt-dependent scale from their held-out predictions. The final AO and WR policies use this learned scale, so their objectives generally differ.

  • AO is empirically informative (Table 1). On HelpSteer3 with the 1B model and 800 prompts, AO achieves 56.94% [53.56, 60.31] against DPO, while WR achieves 55.88% [52.44, 59.25]. Both improve over DPO under the reported prompt-bootstrap intervals, supporting AO's inclusion as an ablation without establishing WR's superiority.

We will clarify these points and reduce the prominence of AO accordingly.


Proposition 4.3 is a solid result to motivate length normalization, but not a generalizable result outside

We agree that Proposition 4.3 motivates length normalization within our formulation. Its specific contribution is showing that length variability alone can produce unequal learned scales. Length normalization with regularization removes this effect under the stated construction (Section 4.2; Appendix B.2).

We will clarify that this is a mechanism-specific result about residual-based scale fitting, without claiming a general characterization of length bias or guaranteed performance improvements.


The authors make a solid effort to showcase their method...

AlpacaEval in the main text. We will move the results from Appendix H into the main experimental discussion.

Additional model family. We have initiated a comparison using Qwen2.5-7B-Instruct, covering DPO, SimPO, and ULNM-WR on HelpSteer3. We will report the results once evaluation and verification are complete (reported below).

Dataset selection. We selected HelpSteer2-Preference and HelpSteer3-Preference because both provide human preference-strength annotations, enabling direct evaluation of our strength-dependent margins. Related work also uses a small number of preference-training datasets: MMPO (Kim et al., 2024) uses two, while SimPO (Meng et al., 2024, Section 3) constructs its preference-training variants from one underlying prompt dataset. This provides context for our training-data scope, although broader evaluation remains valuable.


length-normalized DPO baseline with q=1 and no margin

We thank the reviewer for suggesting this ablation. We have now evaluated length-normalized DPO with q=1q=1 and no margin (τ=0\tau=0), using the same protocol as Table 2. On the 400-prompt panel, this baseline achieves a 58.50% win rate against DPO, close to fixed-margin DPO’s 58.13%. Retaining strength-dependent margins with q=1q=1 yields 62.00%, while the full ULNM-DPO-WR method with learned scaling achieves 68.00%. These point estimates support the contribution of margins and learned scaling and indicate that length normalization alone does not account for the full method’s observed improvement.

Method400 prompts: win rate (%)
Fixed-margin DPO58.13 [53.25, 62.88]
Length-normalized DPO, no margin, q=1q=158.50 [53.63, 63.13]
Length normalization + strength-dependent margins, q=1q=162.00 [57.25, 66.63]
Full ULNM-DPO-WR, learned qq68.00 [63.38, 72.38]

All win rates use the same DPO reference and count ties as half a win. Brackets report 95% prompt-bootstrap confidence intervals; these single-seed results do not measure variability across training runs.

Training seeds and computational constraints

We agree that independent training runs capture variability that prompt-bootstrap intervals cannot. Given the preliminary seed-sensitivity results below and the substantial cost of repeating billion-parameter policy training and the auxiliary-policy pipeline, we used a fixed seed for the main experiments.

Similar trade-offs appear in related work: Furuta et al. (NeurIPS 2024, Appendix B) report one seed per setting, and Yang et al. (ICLR 2025, Appendix E.1) report single-run DPO and SimPO experiments because of computational cost. Further examples include single-run results in DiaTool-DPO (Jung et al., SIGDIAL 2025), a fixed preference-optimization seed for real-world LLM experiments in Won et al. (2025, Appendix J.2), and one-seed results in ASTPrompter (Hardy et al., Findings of EMNLP 2025, Table 1). SimPO does not specify multi-seed averaging of its main results, and its released Llama-3-8B-Instruct configuration fixes seed 42.

Appendix F.3 evaluates three earlier configurations on HelpSteer3 with the 1B model using seeds 17 and 42, each against a same-seed DPO baseline. On the same 800-prompt panel, tie-adjusted win rates were:

ConfigurationSeed 42Seed 17
EMA-joint prompt AO54.25%54.25%
EMA-joint prompt WR53.06%53.19%
Entropy-CF prompt WR54.44%55.13%

All three retain point estimates above 50%, differing by at most 0.69 percentage points between seeds. These results provide preliminary evidence of consistency for those configurations, but do not establish stability of final ULNM. We will clarify this scope and distinguish prompt-bootstrap uncertainty from training-run variability.


I feel that y for the prompt and x for the responses is a bit counter-intuitive

Thank you. We agree that following the conventional DPO notation would improve readability. We will consistently use xx for prompts and yy for responses throughout the text, equations, and appendices.


CountSketch is not ablated/verified?

Thank you. We chose CountSketch to keep scale learning lightweight. The frozen prompt embeddings have 2,048 or 4,096 dimensions; CountSketch compresses them into a 64-dimensional representation using fixed signed hashing (Appendix E.3). This reduces the input dimension and parameter count of the small scale network without requiring an additional learned projection. The fixed hash mapping also makes feature construction reproducible.

CountSketch serves as an implementation choice for computational efficiency and controlling the scale predictor's capacity. We will clarify this distinction and the compression rationale, and cite Charikar et al. (2002) and the related feature-hashing literature.


*The margin scaling coefficient tau is mentioned twice in section 2 *

Thank you for pointing out that this information could be more explicit. We will define the operational margin immediately in Section 2 as c(k)=τm(k)c(k)=\tau m(k), explain that m(k)=km(k)=k maps annotated strength to a target gap, and identify τ\tau as its global multiplier. A forward reference to the AO/WR objectives in Section 3.1 will make its subsequent use explicit.


Can you add a reference to section 2.

Thank you. We will add a direct reference to the established random-utility derivation of logistic choice probabilities, such as Train (2009, Chapter 3), alongside the Bradley--Terry citation. We will clarify that independent Gumbel utility errors yield the logistic comparison model, and treating the noise scale (i.e. Gumbel scale parameter) as prompt-dependent, makes the Bradely-Terry model heteroscedastic over the prompts.


The related work section is tucked away

Thank you. Space constraints led us to place the extended related-work discussion in the appendix, but we agree that it should be easier to find. We will add a concise discussion to the main text, explicitly reference the extended discussion, and move it to the first appendix.

Interpretation of the learned scale

The scale-fitting objective (Equation 12) can be high for several reasons: the out-of-fold implicit reward difference (B~i\widetilde B_i) may poorly approximate the latent ground-truth utility difference because the pilot’s implicit reward function is misspecified or generalizes poorly to held-out prompts; the margin (τm(ki)\tau m(k_i)) may be an imperfect proxy for that utility difference; or the annotations may contain noise or reflect genuine ambiguity. In the motivating random-utility model, noise that is sufficiently large relative to the latent utility gap can make binary choices approach a fair coin flip, rather than consistently favoring the response with higher latent utility. The ground-truth utility difference is not observed, so the fitting objective measures the signed margin discrepancy between the pilot score and the strength-dependent margin rather than error against ground truth. Thus, the fitted scale (q(y)q(y)) may reflect several sources of mismatch and uncertainty that our procedure cannot readily distinguish. We show empirically that using this scale in policy optimization is beneficial in the evaluated configurations (Section 5 and Appendix F.5). Intuitively, larger values temper WR gradients for comparisons below the target margin.

We will expand the discussion in Section 3.2 to clarify these points, which are currently addressed only briefly. The appendix already examines these aspects empirically through three complementary analyses:

  • Do strength labels contain information about annotation reliability? (Appendix D). We withhold one annotator's judgment, compute preference strength from the remaining annotators, and test whether the withheld annotator agrees with their preferred answer. Across 7,165 eligible held-out judgments, agreement rises from 96.35% in the low-strength bin to 100% in the high-strength bin—a 3.65-percentage-point difference [95% CI: 2.98, 4.32]. The analysis includes only cases where the remaining annotators agree on direction. Because it uses human judgments without pilot outputs, this result independently supports the relationship between preference strength and annotation reliability.

  • Can scales be learned directly from annotator disagreement? (Appendix F.3). Our alternative Entropy-CF estimator constructs training targets from variation in repeated human judgments about both the preferred answer and preference strength. It learns to predict this annotation dispersion from prompt or pair features, without training or using auxiliary pilot policies or their reward-gap predictions. Policies using its prompt-based scale achieve 54.44% and 55.13% win rates against same-seed DPO baselines on 800 prompts with policy seeds 42 and 17. However, its held-out predictions of annotation dispersion are weak. Thus, scales can be constructed directly from human-annotation supervision, but although policies using these scales improve over DPO, the scale predictions correlate only weakly with observed annotation dispersion. Therefore, the policy improvements should not be interpreted as evidence that the scales accurately predict annotator disagreement..

  • Does the final learned scale improve policy performance? (Appendix F.5). The (q=1)(q=1) control removes prompt-dependent scaling while retaining length normalization and strength margins. ULNM-DPO-WR achieves a direct win rate of 57.69% [54.31%, 61.00%] against this control on 800 prompts. On AlpacaEval, the learned-scale policy also has a higher length-controlled win-rate point estimate: 21.62% versus 17.83%, against the common reference (Appendix H). Together, these results support the practical benefit of learned prompt scaling over a constant scale in the evaluated configurations.

Therefore, as stated in Section 3.2, our main proposed scale learning can reflect annotation uncertainty, task difficulty, unclear task specification, and other sources of pilot error. It is fitted to the discrepancy between pilot reward gaps and preference-strength margins (Equation 12; Appendix B.1), and adjusts the WR loss’s sensitivity while preserving its target-margin threshold (Section 3.1).

The evidence supports the reliability information in strength labels (Appendix D) and the usefulness of learned scaling (Appendix F.5). Separating annotation uncertainty from pilot difficulty in the final fitted scale remains an open diagnostic question. We will make these distinctions and the supporting appendix results more prominent.

We hope the above explanation addresses your concern, and we sincerely appreciate your time and consideration.

Reviewer_w3a8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Uncertainty-Normalized Margins for Direct Preference Optimization, combining preference-strength-dependent margins with learned prompt-specific scales. These scales are fitted using out-of-fold auxiliary-policy log-ratio differences and frozen during final policy training. This paper also considers advantage-only and whole-residual normalization, and introduces a length-normalized variant, ULNM-DPO-WR. The theoretical analysis characterizes scale identifiability from given comparison scores and illustrates how response length can confound scale fitting. Experiments on HelpSteer2, HelpSteer3, and AlpacaEval report improvements over several preference-optimization baselines, with a matched constant-scale control supporting the contribution of learned scaling.

Critical strengths
  1. The empirical evaluation includes a meaningful control for learned scaling. The constant-scale variant retains length normalization and strength-dependent margins, allowing a more targeted assessment than comparison with vanilla DPO alone.

  2. The paper provides useful analysis and carefully states its limitations. The distinction between advantage-only and whole-residual normalization clarifies how scale placement changes the objective.

Critical weaknesses
  1. The comparative evaluation does not sufficiently justify the additional training complexity. The method requires five auxiliary-policy training runs, while the main comparisons match only final-policy updates. Consequently, the results do not establish whether the gains remain competitive when simpler baselines receive comparable total training or tuning budgets. This paper lacks some necessary related work and latest baselines, e.g.,[R1-R3].

  2. The robustness and interpretation of the central contribution remain incompletely validated. The final proposed methods are evaluated with a single training seed; the additional seed experiments concern earlier estimators rather than the main ULNM configuration. Prompt-bootstrap intervals therefore do not establish stability across training runs. The severe degradation of the reproduced beta-DPO baseline further warrants targeted stability and tuning checks. In addition, the learned scales are fitted to pilot residuals, without independent evidence that they capture uncertainty rather than pilot-model difficulty.

References:

[R1]Cao X, Xu Z, Guang M, et al. RE-PO: Robust enhanced policy optimization as a general framework for LLM alignment[C]//International Conference on Learning Representations. 2026, 2026: 109970-109991.

[R2] Yang Z, Zhang Y, Xue W, et al. Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment[J]. arXiv preprint arXiv:2605.20834, 2026.

[R3] Deng R, Feng D, Lei W. AMaPO: Adaptive Margin-attached Preference Optimization for Language Model Alignment[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2026, 40(44): 37341-37349.

Questions for authors
  1. How does ULNM-DPO-WR compare with the strongest baselines when auxiliary-policy training and tuning costs are included? Can the improvement be retained with fewer pilots or smaller pilot models?

  2. How do strength-dependent margins and residual normalization relate to CPO [R1] correction of reference-induced preference misalignment? Do the proposed objectives ensure correct policy-level ordering under a misaligned reference, or primarily alter optimization weights and target gaps? A controlled comparison with CPO [R1] would help establish the additional contribution.

  3. Do the gains over the constant-scale control and SimPO [R2] persist across independent training seeds for the final ULNM configuration? What explains the substantial degradation of beta-DPO [R3], and does it persist after targeted tuning?

  4. Can the authors provide an independent diagnostic separating annotator disagreement from pilot-model difficulty or residual magnitude? This would help determine whether the uncertainty interpretation offers explanatory value beyond adaptive scaling.

References:

[R1] Yang Z, Zhang Y, Xue W, et al. Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment[J]. arXiv preprint arXiv:2605.20834, 2026.

[R2] Meng Y, Xia M, Chen D. Simpo: Simple preference optimization with a reference-free reward[J]. Advances in Neural Information Processing Systems, 2024, 37: 124198-124235.

[R3] Wu J, Xie Y, Yang Z, et al. $\beta $-DPO: Direct Preference Optimization with Dynamic β\beta[J]. Advances in Neural Information Processing Systems, 2024, 37: 129944-129966.

3 comments

We thank the reviewer for their valuable comments and address them below:

More comparsion required...

Thank you for highlighting the additional computational cost and suggesting these relevant baselines. Within our available computational resources, we conducted the following additional experiments.

1. Increasing SimPO’s training budget

Our earlier training-duration experiments also showed that additional updates did not consistently improve performance. Nevertheless, to investigate whether our strongest existing baseline benefits from additional training, we extended the Table 2 SimPO run on HelpSteer3 with Llama-3.1-8B-Instruct. We evaluated the results at two different checkpoints preserving the original evaluation panels, decoding settings, and judge.

SimPO training budget400 prompts: win rate (%)800 prompts: win rate (%)
150 updates61.00 [56.13, 65.63]58.94 [55.56, 62.38]
300 updates45.63 [40.88, 50.50]40.38 [37.06, 43.75]
AMaPO training budget400 prompts: win rate (%)800 prompts: win rate (%)
150 updates61.88 [57.00, 66.63]57.19 [53.75, 60.50]
300 updates29.25 [24.75, 33.88]31.38 [28.19, 34.63]

Increasing the final-policy update budget did not improve SimPO and AMaPO’s performance under this configuration. Instead, performance declined on both panels, consistent with overfitting or over-optimization.

We even went further with Simpo and its performance degraded further. For SimPO at 600 updates:

Evaluation panelWin rate (%)95% confidence interval
400 prompts25.13[20.88, 29.38]
800 prompts24.75[21.81, 27.75]

These results show that simply extending the existing SimPO run does not close the observed performance gap. We report this as an increased final-policy training-budget experiment.

2. Comparisons with the recommended recent methods

We additionally evaluated RE-PO’s RE-DPO formulation [R1], CPO [R2], and AMaPO [R3] on HelpSteer3 with Llama-3.1-8B-Instruct and evaluated under the original Table 2 protocol. We implemented the published objective equations and used method-specific settings; unspecified settings followed our documented Table 2 configuration.

Method400 prompts: win rate (%)800 prompts: win rate (%)
RE-PO / RE-DPO — new56.25 [51.38, 61.13]47.88 [44.50, 51.38]
CPO — new*14.75 [11.38, 18.25]14.81 [12.38, 17.31]
AMaPO — new61.88 [57.00, 66.63]57.19 [53.75, 60.50]
SimPO — original Table 261.00 [56.13, 65.63]58.94 [55.56, 62.38]
ULNM-DPO-WR — original Table 268.00 [63.38, 72.38]65.31 [62.00, 68.56]

AMaPO achieves the highest point estimates among the three newly evaluated baselines. On the 400-prompt panel, it slightly exceeds SimPO’s point estimate (61.88% versus 61.00%), while ULNM-DPO-WR achieves 68.00%. On the 800-prompt panel, ULNM-DPO-WR achieves 65.31%, compared with 57.19% for AMaPO and 58.94% for SimPO. ULNM-DPO-WR therefore retains the highest point estimate on both panels after adding these baselines.

CPO’s published equations differ from its released demonstration code; we followed the equations. Although we did not find discrepancy in the tested loss and gradient calculations, this configuration exhibited severe optimization instability. We therefore report its performance under the stated settings without claiming that it represents CPO’s code's best achievable performance.

We will incorporate these results, the relevant related work, and the implementation and hyperparameter details into the revised manuscript.

Robustness and Interpretation

Training seeds and computational constraints. We agree that independent training runs provide information that prompt-bootstrap intervals cannot capture. Given the preliminary seed-sensitivity results described below and the substantial computational cost of repeating the final billion-parameter models and their auxiliary-policy pipelines, we continued the main experiments with a fixed training seed.

Similar computational trade-offs are explicitly reported in closely related work. Furuta et al. (NeurIPS 2024, Appendix B) report one seed per setting for preference optimization, while Yang et al. (ICLR 2025, Appendix E.1) report single-run experiments involving DPO and SimPO because of computational cost. Additional examples include DiaTool-DPO (Jung et al., SIGDIAL 2025, Limitations), which reports single-run experimental results; Won et al. (2025, Appendix J.2), which fixes the preference-optimization seed to 1 for its real-world LLM experiments; and ASTPrompter (Hardy et al., Findings of EMNLP 2025, Table 1), which explicitly reports one-seed results for that comparison. For our closest baseline, SimPO (Meng et al., NeurIPS 2024), the paper does not specify averaging its main results across independent training seeds, and its released Llama-3-8B-Instruct configuration fixes the seed to 42.

We did investigate training-seed sensitivity during the earlier stages of this work. Specifically, Appendix F.3 evaluates three earlier entropy-supervised and EMA-centered configurations on HelpSteer3 with the 1B model using seeds 17 and 42, each against a DPO baseline trained with the corresponding seed.

On the same 800-prompt evaluation panel, the tie-adjusted win rates were:

ConfigurationSeed 42Seed 17
EMA-joint prompt AO54.25%54.25%
EMA-joint prompt WR53.06%53.19%
Entropy-CF prompt WR54.44%55.13%

These experiments showed the same qualitative trend across both seeds: all three configurations retained win-rate point estimates above 50%, with differences of at most 0.69 percentage points between seeds. This provided encouraging preliminary evidence of consistency for those configurations. Nevertheless, we agree that these earlier seed experiments do not directly establish the stability of the final ULNM configuration. We will clarify their supporting scope and explicitly distinguish the reported prompt-bootstrap uncertainty from variability across independent training runs.

The reproduced β\beta-DPO baseline. We audited the code and examined the training behavior. Our preliminary assessment is that training instability may have contributed to the unusually severe degradation; we do not currently attribute its magnitude solely to the chosen hyperparameter settings. We will investigate this further, including the interaction between those settings and optimization stability.


How do strength-dependent margins and residual normalization relate to CPO

CPO and our method address different aspects of preference optimization. CPO uses a reference-dependent correction to address preference misalignment, with ordering guarantees under its stated conditions. Our margins encode annotated preference strength, while residual normalization adjusts optimization sensitivity across prompts (Section 3.1).

Our objectives do not generally guarantee correct policy-level ordering under a misaligned reference. For sequence-level WR, writing δθ\delta_\theta and δref\delta_{\mathrm{ref}} for the preferred-minus-rejected log-probability gaps,

\iff \delta_\theta>\delta_{\mathrm{ref}}+\frac{\tau m(k)}{\beta_0}.$$ This threshold can remain negative when the reference is sufficiently misaligned. Strength margins therefore change the target gap, while positive residual normalization changes gradient magnitudes and sigmoid saturation without changing WR’s threshold. The length-normalized extension likewise provides no general guarantee of full-sequence probability ordering. As discussed above, CPO exhibited optimization instability, so its results should not be interpreted as its best achievable performance. We will add this comparison and clarify that our contribution concerns strength-aware targets and learned scaling.

Interpretation of the learned scale

The scale-fitting objective (Equation 12) can be high for several reasons: the out-of-fold implicit reward difference (B~i\widetilde B_i) may poorly approximate the latent ground-truth utility difference because the pilot’s implicit reward function is misspecified or generalizes poorly to held-out prompts; the margin (τm(ki)\tau m(k_i)) may be an imperfect proxy for that utility difference; or the annotations may contain noise or reflect genuine ambiguity. In the motivating random-utility model, noise that is sufficiently large relative to the latent utility gap can make binary choices approach a fair coin flip, rather than consistently favoring the response with higher latent utility. The ground-truth utility difference is not observed, so the fitting objective measures the signed margin discrepancy between the pilot score and the strength-dependent margin rather than error against ground truth. Thus, the fitted scale (q(y)q(y)) may reflect several sources of mismatch and uncertainty that our procedure cannot readily distinguish. We show empirically that using this scale in policy optimization is beneficial in the evaluated configurations (Section 5 and Appendix F.5). Intuitively, larger values temper WR gradients for comparisons below the target margin.

We will expand the discussion in Section 3.2 to clarify these points, which are currently addressed only briefly. The appendix already examines these aspects empirically through three complementary analyses:

  • Do strength labels contain information about annotation reliability? (Appendix D). We withhold one annotator's judgment, compute preference strength from the remaining annotators, and test whether the withheld annotator agrees with their preferred answer. Across 7,165 eligible held-out judgments, agreement rises from 96.35% in the low-strength bin to 100% in the high-strength bin—a 3.65-percentage-point difference [95% CI: 2.98, 4.32]. The analysis includes only cases where the remaining annotators agree on direction. Because it uses human judgments without pilot outputs, this result independently supports the relationship between preference strength and annotation reliability.

  • Can scales be learned directly from annotator disagreement? (Appendix F.3). Our alternative Entropy-CF estimator constructs training targets from variation in repeated human judgments about both the preferred answer and preference strength. It learns to predict this annotation dispersion from prompt or pair features, without training or using auxiliary pilot policies or their reward-gap predictions. Policies using its prompt-based scale achieve 54.44% and 55.13% win rates against same-seed DPO baselines on 800 prompts with policy seeds 42 and 17. However, its held-out predictions of annotation dispersion are weak. Thus, scales can be constructed directly from human-annotation supervision, but although policies using these scales improve over DPO, the scale predictions correlate only weakly with observed annotation dispersion. Therefore, the policy improvements should not be interpreted as evidence that the scales accurately predict annotator disagreement..

  • Does the final learned scale improve policy performance? (Appendix F.5). The (q=1)(q=1) control removes prompt-dependent scaling while retaining length normalization and strength margins. ULNM-DPO-WR achieves a direct win rate of 57.69% [54.31%, 61.00%] against this control on 800 prompts. On AlpacaEval, the learned-scale policy also has a higher length-controlled win-rate point estimate: 21.62% versus 17.83%, against the common reference (Appendix H). Together, these results support the practical benefit of learned prompt scaling over a constant scale in the evaluated configurations.

Therefore, as stated in Section 3.2, our main proposed scale learning can reflect annotation uncertainty, task difficulty, unclear task specification, and other sources of pilot error. It is fitted to the discrepancy between pilot reward gaps and preference-strength margins (Equation 12; Appendix B.1), and adjusts the WR loss’s sensitivity while preserving its target-margin threshold (Section 3.1).

The evidence supports the reliability information in strength labels (Appendix D) and the usefulness of learned scaling (Appendix F.5). Separating annotation uncertainty from pilot difficulty in the final fitted scale remains an open diagnostic question. We will make these distinctions and the supporting appendix results more prominent.

We hope the above explanation addresses your concern, and we sincerely appreciate your time and consideration.

Comments on the paper

1 comment

Dear All,

Thank you again for taking the time to review our paper. As the discussion period approaches its end, we thank the reviewers for their time, effort, and constructive comments, which helped improve the soundness of the paper. To address reviewer comments, we conducted a series of new experiments. Summary of key experiments:

  • Three new baselines: We evaluated RE-DPO [R1], CPO [R2], and AMaPO [R3] on HelpSteer3 with Llama-3.1-8B-Instruct under the original Table 2 protocol. AMaPO turned out to be a strong baseline, achieving the highest performance among the three, but still below our ULNM-DPO-WR (win rates in 400/800-prompt panels: 61.88% vs 68.00% and 57.19% vs 65.31%).
  • New model family: We ran additional experiments with Qwen2.5-7B-Instruct on HelpSteer3 to evaluate the performance of our strongest baseline SimPO against ULNM-DPO-WR. On both original 400- and 800-prompt panels, ULNM-DPO-WR achieved a higher win rate against the matched DPO reference. These results extend our evidence of ULNM-DPO-WR performance beyond the Llama family.
  • Increased computational cost for baselines: To address the question about baseline performance under the same computational budget with ULNM-DPO, we studied whether we can improve our strongest baselines (SimPO and AMaPO) performance by training them more. We evaluated the results under Table 2 protocol at two different checkpoints. Increasing the final-policy update budget did not improve SimPO nor AMaPO’s performance under this configuration.

We believe that these experiments, together with the ablations and clarifications below, including the interpretation of the learned scale, address most of the reviewer comments, while we understand that more exhaustive experiments would provide even stronger evidence. We tried to do our best within our available computational resources.

References:

[R1] Cao X, Xu Z, Guang M, et al. RE-PO. Robust enhanced policy optimization as a general framework for LLM alignment. International Conference on Learning Representations. 2026

[R2] Yang Z, Zhang Y, Xue W, et al. Conditional Equivalence of DPO and RLHF: Implicit Assumption, Failure Modes, and Provable Alignment. International Conference on Machine Learning. 2026.

[R3] Deng R, Feng D, Lei W. AMaPO: Adaptive Margin-attached Preference Optimization for Language Model Alignment. Proceedings of the AAAI Conference on Artificial Intelligence. 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.