- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
本文探讨了直接偏好优化(DPO)未明确利用偏好强度及不同提示间不确定性差异的问题,并提出了不确定性归一化边际方法(UNM-DPO)。该方法将偏好强度映射为边际,并引入了一个可学习的提示相关尺度,从而构建了两个优化目标AO和WR,其中边际分别在尺度归一化之前或之后进行减法运算。为了将尺度学习与策略优化分离,作者利用五折辅助策略生成的折外奖励差异来拟合尺度网络,随后冻结该尺度并训练最终策略。在此基础上,ULNM-DPO-WR 在尺度拟合和策略训练中均一致地采用按答案长度进行归一化处理,以减轻答案长度对奖励差异和尺度估计的影响。理论部分给出了在已知 WR 比较分数条件下尺度可识别性的必要且充分条件,并通过受控构造分析了长度变化对尺度拟合的影响。实验在 HelpSteer2、HelpSteer3 和 AlpacaEval 数据集上进行,ULNM-DPO-WR 表现最佳,且与固定尺度版本的直接对比验证了在所评估场景中学习尺度的有效性。
Critical strengths
-
Combine preference intensity with prompt-related scaling and clearly distinguish their roles in the optimization goal. The paper doesn’t just use the direction of preferences; it also adjusts the expected reward gap through intensity-related margins and introduces prompt-related scaling to regulate the sensitivity of the loss. The WR design, in particular, places the margin inside the normalization so that changes in scale don’t alter the threshold the reward gap needs to exceed. This separates, at the target level, 'how big a preference gap is needed' from 'how to penalize deviations from that gap.' This design is clearly interpretable and provides a valuable optimization framework for combining fine-grained preference information with prompt differences.
-
Scale fitting and policy training are done in stages, with length normalization used consistently in both stages. The method uses out-of-fold predictions from an auxiliary policy to fit the scale, then freezes the scale to train the final policy, reducing the chance that simultaneous updates of policy and scale could compensate for each other and lower the loss by adjusting the scale. ULNM-DPO-WR further uses consistent per-answer length normalization in both scale fitting and policy optimization so that the reward gaps have a consistent measurement across stages, alleviating the interference of sequence length on scale learning. This process balances reasonable training design with deployment convenience, as only the final policy is needed during inference—no auxiliary model or scale network is required.
Critical weaknesses
尺度学习可能会将辅助模型能力的不足解读为降低训练影响的信号。该尺度是基于辅助策略的奖励差距与偏好强度阈值之间的残差进行拟合的。当奖励差距尚未达到阈值时,增加尺度可以降低拟合损失。但这种情况可能是由于标注不可靠,也可能是辅助模型尚未掌握如何处理此类问题。在后一种情况下,后续训练实际上可能会削弱对难度较高但标注可靠的样本的学习效果。反事实预测可以降低样本内拟合的影响,但无法消除辅助模型能力中固有的偏差;缩放边界和正则化也无法区分这两种来源。尽管作者承认缩放可能包含模型误差,但他们仍需解释这种误差将如何影响最终策略,以及是否存在避免持续削弱高难度样本的方法。
Questions for authors
1.Appendix E.5 mentions that in the 8B experiment, AO uses the scale fitted by the 8B auxiliary model, while WR uses the scale transferred from the 1B model. Could you explain the reasoning for using scales from different sources, and whether this difference might affect how we interpret AO and WR results?
2.The length-normalized coefficient is set based on the median length of the training answers. For data with large differences in answer lengths, is this uniform calibration still appropriate? It would be great if the authors could further explain the rationale and scope of this choice.
3 comments
The length-normalized coefficient is set based on the median length...
Actually, we normalize each response by its own length (Equation (13), and the median is only used to rescale the global KL regularization coefficient to adapt the change from a sequence-level units to a token-level units (Appendix E.4) Thus, the procedure does not assume that all responses have similar lengths.
As specified in Appendix E.4, we use , where ,
where contains eligible training comparisons. Dividing log-probability ratios by response length changes their numerical scale; multiplying by compensates at a typical training length. In particular, the normalized and sequence-level reward gaps coincide when both responses have length . We use the median because it is less sensitive to unusually long responses than the mean, and compute it entirely from training data. For HelpSteer3, and .
For widely varying lengths, this remains a global magnitude calibration, not an exact equivalence to the sequence-level objective. Per-response normalization still uses each actual length, but one coefficient is not guaranteed to be optimal across all length ranges or to preserve an equivalent KL penalty, as already noted in Appendix E.4. Our experiments support this calibration in the evaluated settings; they do not establish its optimality for arbitrary length distributions. We will clarify this rationale and scope in Section 3.3.
We hope the above explanation addresses your concern, and we sincerely appreciate your time and consideration.
I appreciate the authors’ response. I will keep my score at 3.
We thank the reviewer for their valuable comments and address them below:
Thank you for identifying this concern. The learned scale is intended to capture several sources of residual variation. As stated in Section 3.2, these include annotation ambiguity, task difficulty, and limitations in the pilot's capabilities or optimization. For example, a pilot may fit its training folds but fail to achieve the required preference margins on held-out prompts. Out-of-fold scoring reveals this mismatch, but does not identify its cause.
How pilot errors affect final-policy training. Your concern is valid: when the pilot falls below the target margin, scale fitting can favor a larger scale (Appendix B.1). Let b denote the final policy's reward gap and c=τm(k) its target margin. The WR loss satisfies ∂b∂ℓWR=−q1σ(qc−b), where σ is the sigmoid function. For b≤c, increasing q weakens the learning signal. Thus, a pilot's capability limitations can reduce the influence of difficult but reliable examples. However, the signal still encourages increasing the preferred response's reward gap, and the target margin remains unchanged (Section 3.1).
What the existing safeguards provide. Section 3.2 introduces complementary controls:
Frozen scales: The scales are fitted once and frozen before final-policy training. They cannot keep increasing in response to the final policy's difficulties.
Geometric-mean constraint: The geometric mean of the scales across unique training prompts is fixed at one. Larger scales for some prompts must therefore be balanced by smaller scales elsewhere, preventing global scale inflation.
Bounds and regularization: The bounds q∈[0.5,2] limit attenuation, while regularization penalizes departures of logq from zero. For an example below its target margin, the bounds imply
\frac{\partial \ell_{\mathrm{WR}}}{\partial b} \right| \geq \frac{1}{4}.$$ This bounds the loss sensitivity to the reward gap, not the full parameter-gradient magnitude or eventual learning outcome.These controls constrain the adaptation, but do not distinguish annotation ambiguity from pilot capability errors. In particular, a difficult example can still receive persistently reduced emphasis.
Evidence and a possible additional safeguard. The constant-scale ablation supports the practical usefulness of this adaptation: ULNM-DPO-WR achieves a direct win rate of 57.69% [54.31, 61.00] against its q=1 control on 800 prompts (Appendix F.5). This supports an aggregate benefit in the evaluated setting, rather than a benefit for every difficult subset.
One possible safeguard against persistent attenuation is to gradually return the frozen scales toward q=1, restoring the constant-scale objective at the end of training. This is an untested extension, not part of our reported experiments. We will clarify the scale's broader interpretation, its effect on the learning signal, and the scope of the existing safeguards.
Thank you for highlighting this distinction. The different scale sources are configuration choices, not requirements of AO or WR. As documented in Appendix E.5, the sequence-level 8B AO policy uses a scale fitted from 8B pilots, whereas the sequence-level 8B WR policy reuses the existing frozen 1B scale. This reuse is possible because the scale supplies a scalar value for each prompt; the transferred estimator retains its original 1B encoder and feature mapping (Appendix E.3).
This difference does affect interpretation. Changing the scale source changes the training signal, so the 8B AO--WR comparison does not isolate the placement of the margin. We will explicitly label scale provenance in the main table and avoid attributing their performance difference solely to the objectives.
A comparison using the same scale is available in the 1B experiments, where AO and WR share one frozen scale (Appendix E.3). Their win rates against the common DPO reference are 58.25% versus 58.75% on 400 prompts, and 56.94% versus 55.88% on 800 prompts (Table 1). These results do not establish a consistent ordering between AO and WR.
Finally, the strongest reported method, ULNM-DPO-WR, uses a newly fitted scale derived from length-normalized out-of-fold scores of the 8B pilots (Appendices E.4--E.5). Its results therefore do not use the transferred 1B scale. We will make this distinction more prominent.