acceptodds
Under review as a conference paper at ICLR 2027

A Unified Random-Utility View of Heterogeneity in Direct Preference Optimization

Abstract

Direct Preference Optimization (DPO) assumes a single comparison scale, although in practice some response pairs are easy to distinguish while others remain ambiguous at similar utility gaps. Recent DPO-like variants increasingly introduce pair-adaptive temperatures, margins, confidence signals, or latent structure to address heterogeneity, yet similar language is often used for methods that model different forms of heterogeneity at different stages of preference modeling. To disentangle these differences, we present a unified random-utility view of heterogeneity in DPO and use it to organize DPO-like methods into a branch-level taxonomy. This taxonomy isolates a statistically distinct DPO-side error-layer branch, comparison-specific observation-scale heterogeneity, whose minimal instantiation is Heteroscedastic Direct Preference Optimization (H-DPO), a heteroscedastic-logit generalization of DPO that preserves the standard utility-gap parameterization while introducing comparison-specific scale inside the logit. In this branch, homoscedastic DPO is semantically misspecified: to match the correct preference probabilities, its score must absorb comparison-specific scale, converging to a scale-distorted quantity rather than the latent utility gap itself. Under restricted smooth families, the corresponding score-scale decomposition is locally identifiable up to global scaling from binary preference data. Synthetic experiments confirm the predicted score distortion, and real-world benchmarks provide empirical support for the comparison-scale formulation and its preference-discrimination benefits.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Direct Preference Optimization, Random Utility Models, Preference Alignment, Heterogeneity Modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VNJ3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper casts DPO variants as random-utility models and sorts them by where heterogeneity enters: utility, error/scale, outcome, time, or loss. It singles out comparison-specific observation scale as a distinct branch. Its instantiation, H-DPO, models P(yw≻yl)=σ(βΔπ/τϕ(x,yw,yl))P(y_w\succ y_l)=\sigma(\beta\Delta_\pi/\tau_\phi(x,y_w,y_l)) with a learned log-scale head. On the theory side, homoscedastic DPO under heteroscedastic truth recovers \Delta^/\tau^, not \Delta^ (Thm. 1, Cor. 1). Positive loss reweighting cannot fix this (Prop. 2). The decomposition is locally identifiable up to global scale in restricted smooth families (Thm. 2). In synthetic experiments, H-DPO recovers \Delta^ (ρ 0.95 vs. 0.78 for DPO at α=1). On UltraFeedback with three 7–9B models, H-DPO gains about 1–2 points on RewardBench and AlpacaEval/Arena-Hard over DPO, and most generation gains are not significant.

Critical strengths
  1. Separating error-layer scale from loss-layer reweighting is a useful distinction. The synthetic study shows it cleanly: oracle W-DPO follows DPO, while H-DPO recovers the true gap (Table 2).

  2. The baseline set is broad and matched to the taxonomy (L-X, L-V and L-T methods), and the ablation isolates "division inside the logit" from capacity.

Critical weaknesses
  1. The theory adds little and does not address the question that matters for DPO. Heteroscedastic logit and its bias under homoscedastic fitting are classical in discrete choice, as the paper notes (Fiebig et al., Hess & Train). Thm. 1 and Cor. 1 follow almost directly: if \tau^ varies, \Delta^/\tau^ is not proportional to \Delta^. Identifiability holds only locally in restricted families, which a neural τϕ(x,yw,yl)\tau_\phi(x,y_w,y_l) does not satisfy. More importantly, DPO's implicit reward is tied to the KL-regularized RLHF optimum. With a pair-dependent τ\tau, it is unclear what policy objective H-DPO optimizes, or why recovering \Delta^ rather than Δ/τ∗\Delta^/\tau^* gives a better policy. The paper's main claim is about semantic faithfulness, but it does not show that faithfulness affects downstream behavior.

  2. The real-data evidence is thin. There is one training set (UltraFeedback), and gains are around 1 point: RB v1 +1.0/+1.4/+0.1, and generation gains are mostly not significant. On Gemma, H-DPO trails ε-DPO, IPO and SimPO on AlpacaEval. The capacity-matched additive-head ablation comes within 0.2–1.3 points of full H-DPO on most metrics. The "ambiguity" interpretation of τϕ\tau_\phi rests on one partial correlation (ρ=0.30\rho=0.30) with a proxy.

Questions for authors
  1. What KL-regularized objective does the H-DPO policy optimize when τ\tau depends on the pair? Is the learned π\pi still the optimal policy for any well-defined reward?

  2. Can you show that the improved recovery of Δ∗\Delta^* seen in the synthetic setting produces better policies (e.g., in a synthetic setup where the true reward is known)?

  3. With a flexible neural τϕ\tau_\phi, what prevents it from absorbing part of the utility signal? Do you observe drift in the learned Δπ\Delta_\pi scale across seeds?

4 comments
AuthorsOpenReview ↗

Response to Reviewer VNJ3

Thank you for the careful critique. We agree the original submission did not close the loop tightly enough between branch-level semantics and policy-level relevance on real models. Scope up front: the policy interpretation we add (Corollary 2, App. N) is restricted and population-level, not a guarantee for the full neural head.

W1. "Theory adds little / does not address what matters." We agree with the reviewer that heteroscedastic logit, and the bias of homoscedastic fitting under heteroscedastic truth, are classical (Hess & Train 2017; Fiebig et al. 2010). Our contribution is not the econometric fact but its DPO-specific consequence: (1) homoscedastic DPO recovers the distorted target Δ⋆/τ⋆\Delta^\star/\tau^\star, not Δ⋆\Delta^\star (Theorem 1 / Corollary 1); (2) loss-layer reweighting does not fix this (Proposition 2 / Proposition 4); (3) the distortion is largely ranking-invariant but magnitude-visible synthetically (App. F.8); (4) new: a restricted population-level policy interpretation (Corollary 2). We also agree identifiability is only local and restricted-family, not a neural-head theorem.

W2. "Real-data evidence is thin." We agree the gains are modest and benchmark-specific, and have narrowed the claim: RewardBench discrimination is the primary real-world result, generation gains are descriptive, and the new best-of-N result (App. M) is the strongest behavioral evidence beyond pairwise discrimination. We retain your caveats: one training dataset (UltraFeedback); on Gemma AlpacaEval 2 LC, H-DPO (48.79%) is above DPO (47.62%) but below ε\varepsilon-DPO (54.74%), SimPO (52.35%) and IPO (51.92%) (Table 11); the additive-head control is partial rather than null; and RewardBench is significant on Llama/Gemma, null on Mistral.

Q1. What policy objective does H-DPO optimize when τ\tau is pair-dependent? A pair-dependent τ\tau in the training-time comparison channel does not, in general, preserve the standard DPO$\to$RLHF derivation for the full neural model. What we add is narrower: under Corollary 2's restricted recovery conditions, the corrected pairwise gaps recover a single response-level reward up to a common factor, so global rescaling to the target KL budget yields the KL-regularized optimum for that reward; for any same-budget comparator πD\pi_D, U(πH)−U(πD)=βK⋅ExDKL(πD∥πH)≥0U(\pi_H)-U(\pi_D)=\beta_K\cdot\mathbb E_x D_{\mathrm{KL}}(\pi_D\Vert\pi_H)\ge 0. We do not claim this for the deployed neural head; there we rely on empirical policy-value evidence (App. N, M). On whether the learned policy is optimal for some reward: any policy with the reference's support is optimal for its own implicit reward λlog⁡(π/πref)+c(x)\lambda\log(\pi/\pi_{\mathrm{ref}})+c(x) (Gibbs reparameterization, Rafailov et al. 2023); the substantive question is when those differences also match the label-generating reward, which App. B.4 gives — with a counterexample showing scale variation alone need not force a strict gain (exact-fit is needed).

Q2. Does better Δ⋆\Delta^\star recovery produce a better policy? Yes — with known synthetic reward it yields a better same-KL policy (App. N), and the same direction appears behaviorally on a real model. Direct synthetic answer (matched KL): at α=1\alpha=1, mean KL 0.10 nat, H-DPO beats DPO in all 5/5 seeds — DPO 0.1716, H-DPO 0.2013, oracle 0.2113 — paired gain +0.0297±0.0029, closing ~75% of DPO's shortfall; in KL to the oracle H-DPO is ~4× closer (0.01045 vs 0.04151). Real-model analogue: on 1,602 RewardBench v2 prompts (App. M), H-DPO selects a benchmark-chosen answer +3.06 pts more often (p=5×10−5p=5\times10^{-5}) and scores +0.239 higher under an external RM. The real-model result is behavioral response-selection evidence, not cardinal-utility identification.

Q3. Does a flexible neural τϕ\tau_\phi absorb utility? Seed drift? We do not claim a structural guarantee, but the new diagnostics do not support the failure mode that τϕ\tau_\phi simply absorbs utility. Synthetic ground-truth (App. F.9): at α=1\alpha=1, binned by true log-scale, the within-bin correlation of learned log-scale with true ∣Δ⋆∣|\Delta^\star| is only −0.123±0.039, while scale recovery vs true log-scale is 0.949 (Table 2). Real held-out (App. H.4): ρ(log⁡τ,∣Δ^∣)=+0.058\rho(\log\tau,|\hat\Delta|)=+0.058, and the difficulty association is unchanged after controlling for ∣Δ^∣|\hat\Delta| (−0.710 → −0.708). Cross-seed drift (App. L; 1,596 pairs, seeds 42/43/44): the RMS of Δπ\Delta_\pi differs by ~14%; after unit-RMS normalization the pairwise RMSE is 0.111–0.141, while log-scale rankings correlate 0.998–0.999 — high reproducibility with moderate nonzero variation, not exact invariance (RewardBench uses only the gap's sign).

We hope these additions narrow the gap you identified: the revision claims no full neural guarantee but shows (i) a restricted KL-policy interpretation, (ii) better Δ⋆\Delta^\star recovery improving decisions under known synthetic reward, and (iii) the same direction behaviorally on a real model.

Follow-up on Q1/Q2: the direct policy-value evidence

We don't want our point-by-point reply to be missed. Our full response to your W1/W2 and Q1–Q3 is posted above; here we surface the one result most directly tied to your core concern — whether better Δ⋆\Delta^\star recovery actually yields a better policy (Q2), and what objective H-DPO targets (Q1).

Matched-KL policy value with known reward (App. N, Table 29). A two-action policy built from the learned scores, evaluated at a common KL budget against the true synthetic reward (an instantiation of Corollary 2):

αmean KLU(DPO)U(H-DPO)U(oracle)U(H)−U(D)seeds
0.00.100+0.2087+0.2087+0.2113-0.0000±0.00044/5
1.00.100+0.1716+0.2013+0.2113+0.0297±0.00295/5

At the strongest heterogeneity (α=1) H-DPO earns higher true reward than DPO at the same KL budget in all 5/5 seeds, closing ~75% of DPO's gap to the oracle; in KL to the oracle policy it is ~4× closer (0.01045 vs 0.04151, App. N Table 30). At α=0 (no heterogeneity) the two are equivalent, exactly as the theory predicts. This is the direct known-reward test of your Q2; on a real model the same direction appears behaviorally (App. M best-of-N: +3.06 pts selection accuracy, p=5×10−5p=5\times10^{-5}), which we read as response-selection evidence, not cardinal identification.

We do not claim a neural-head guarantee: Corollary 2 is restricted and population-level, and App. B.4 includes a counterexample showing scale variation alone need not force a strict gain. But we hope this closes the policy-relevance loop you identified.

We would be grateful for any indication of whether these additions address your concerns and whether they might support reconsidering your assessment; we are happy to answer anything further.

Thank you for your response and for addressing my concerns on the questions. However, my concerns on the weaknesses remain. I will remain my score at weak reject.

Dear Reviewer VNJ3,

Thank you for the follow-up and for acknowledging that the questions are addressed. We respect that your concerns on the weaknesses remain. In the revision we have narrowed our claims to what the evidence supports — the real-world result is scoped to RewardBench discrimination (generation descriptive), and the theory is stated as a restricted, population-level result rather than a neural-head guarantee. We recognize the remaining difference is one of assessment about the significance of the contribution, and we respect your view. Thank you again for the rigor and time you brought to the review.

Best regards, Authors

Reviewer_e1PJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies a unified framework to describe the heterogeneity captured in various DPO frameworks. It provides a comparison-scale version of DPO and provides encouraging empirical evidence. It shows that the correct probability prediction does not necessarily lead to the right recovery of the latent utility.

Critical strengths
  1. Recognition of the identifiable issue for predicting preference probabilities and obtaining latent utility gaps.
  2. Provide an intuitive extension to the standard DPO framework and possible resolutions to the recovery of latent utilities.
Critical weaknesses
  1. The framework has a quite similar formulation to existing work and requires further justification of the novelty
  2. The theory does not establish utility recovery for the implemented approach, the neural H-DPO model.
Questions for authors
  1. Could you clarify the novelty of the proposed DPO scaling method better against the existing work, such as MallowsPO (prompt-dependent dispersion) and Ada-DPO (per-pair scales)?

  2. Could you verify to some extent that the identification assumptions required in the theoretical result are satisfied by the proposed H-DPO model?

  3. Could you clarify further how the experiments establish utility–noise separation, given that the main synthetic model lacks DPO’s response-difference constraint, “best-of-N” selects comparison gaps rather than responses, the symmetric additive control may violate preference complementarity, and the learned-weight control falls outside the fixed-weight theorem?

1 comment
AuthorsOpenReview ↗

Response to Reviewer e1PJ

Thank you for the constructive comments and positive assessment. Your main concerns were (i) novelty relative to existing scaled formulations, (ii) the scope of the theory for the neural implementation, and (iii) whether the experiments really separate utility from noise. We clarified the first two in the text (§3.2 + App. J; §4.2 / App. B / App. N) and added new evidence for the third (Apps. H.4, K, L).

W1 / Q1. Novelty relative to MallowsPO and Ada-DPO. We agree that "learning a scale" alone is not the novelty claim, and that heteroscedastic logit itself is not new. Our claim is narrower and branch-level: we isolate comparison-specific observation scale as a distinct DPO-side error-layer branch, show its semantic consequence for DPO — under pair-dependent observation scale a homoscedastic fit recovers the distorted target Δ⋆/τ⋆\Delta^\star/\tau^\star, not the latent utility gap Δ⋆\Delta^\star (Theorem 1 / Corollary 1) — show this is not equivalent to loss-layer adaptation (Proposition 2 / Proposition 4), and separate it from prompt-level and loss-layer adaptations that can look superficially similar. Two crisp distinctions, now explicit in §3.2 and App. J (side-by-side in Table 21): MallowsPO uses a prompt-level dispersion ϕ(x)\phi(x), not a comparison-specific scale — empirically, 55.7% of the learned log-scale variance is within prompt, which a prompt-level dispersion cannot represent; Ada-DPO/APS uses a training-time per-pair DRO variable τi\tau_i, not a persistent, feature-conditioned observation-scale function, so its deployed model is still a single homoscedastic score.

W2 / Q2. Theory does not establish utility recovery for the neural H-DPO. We agree, and the revised paper states this more explicitly: our theory is a restricted-family, population-level statement, not a theorem-level guarantee of utility recovery for the full neural implementation; it is scoped to restricted smooth heteroscedastic-logit families, local identifiability up to global scaling, and population-level consistency under realizability. We therefore view the theory as clarifying the branch-level target and identifiability question, while the new synthetic and held-out diagnostics test whether the deployed neural model empirically behaves in that regime. Against the most immediate failure mode — the scale simply absorbing utility magnitude — we added: App. H.4, log⁡τ\log\tau has only a small held-out correlation with ∣Δ^∣|\hat\Delta| (+0.058) and its difficulty association is essentially unchanged after partialling out ∣Δ^∣|\hat\Delta|; App. H.3, 55.7% of log-scale variance is within prompt; App. H.2, the scale aligns with an external ambiguity proxy after controls; App. F.9, on the synthetic branch where τ⋆\tau^\star is known, the learned scale tracks true scale strongly with only weak residual association with ∣Δ⋆∣|\Delta^\star|. We present these as empirical support against the main failure mode, not theorem-level verification.

Q3. How do the experiments establish utility–noise separation? We agree that no single experiment proves full utility–noise separation in the neural model; the revised claim is narrower — the observed H-DPO advantage is not well explained by the simpler alternatives raised in the review — and the revision triangulates this with complementary evidence rather than one experiment. On your four caveats: (1) Synthetic vs response-difference constraint — agreed; the synthetic study predicts the gap directly to isolate the confound cleanly, while the real-world experiments do use the actual DPO difference form r(x,yw)−r(x,yl)=βΔπr(x,y_w)-r(x,y_l)=\beta\Delta_\pi. (2) Best-of-N selects responses rather than gaps — agreed; that is why we added a real-model response-level best-of-N (App. M), read as response-selection evidence rather than cardinal-utility identification. (3) Symmetric additive control — agreed; it is a capacity control, not a normative preference model. (4) Learned-weight control outside the fixed-weight theorem — agreed; we rely on experiment, and the learned-weight ablation is substantially worse than full H-DPO while the oracle weighted control tracks DPO. Finally, App. K tests the noisy-preference alternative directly: cDPO/rDPO modestly improve over DPO but do not match H-DPO on utility-gap recovery or best-of-N under the synthetic heteroscedastic DGP; and on real benchmarks we added ε\varepsilon-DPO and R-DPO across all three bases (App. K, Table 23), where no single loss-layer robust variant matches H-DPO across bases and benchmarks.

We hope these clarifications and additions address your concerns.

Reviewer_Wt87OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies heterogeneity in Direct Preference Optimization. Standard DPO assumes a single global comparison scale: once the latent utility gap between two responses is fixed, the sharpness of the observed preference is fixed too. The authors argue that real preference pairs differ in how clear-cut they are, so the same utility gap can produce very different preference probabilities depending on the pair.

Contributions.

  • A random-utility taxonomy of DPO-like methods, organized by where heterogeneity enters the model: utility layer, error/observation-scale layer, outcome layer, temporal layer, or loss layer only.
  • Isolation of one branch within this taxonomy, comparison-specific observation-scale heterogeneity, and a minimal instantiation called H-DPO. H-DPO keeps the DPO log-ratio score but divides the logit by a learned, pair-specific scale τ, predicted by a small symmetric head on pooled representations.
  • Theory for this branch:
    • Homoscedastic DPO is "semantically misspecified": any exact probability fit recovers Δ*/τ* rather than the true utility gap Δ*.
    • Positive loss reweighting does not change this population target.
    • The heteroscedastic model is locally identifiable up to global scaling under restricted smooth families, illustrated with a one-feature worked example.

Validation.

  • Synthetic experiments. Data are generated from the heteroscedastic model with known Δ* and τ*.
    • As heterogeneity grows, DPO drifts toward the distorted target, while H-DPO keeps high correlation with Δ* and recovers τ*, at essentially identical test NLL.
    • An oracle loss-weighted DPO baseline tracks DPO, not H-DPO.
    • Pairwise ranking accuracy is tied across methods, but best-of-N selection clearly favors H-DPO.
  • Real-world experiments. Llama-3.1-8B, Gemma-2-9B, and Mistral-7B are trained on UltraFeedback.
    • H-DPO improves RewardBench v1/v2 over DPO by about 1–1.8 points on Llama and Gemma, and ties on Mistral.
    • AlpacaEval 2.0 and Arena-Hard results are mostly favorable but largely not significant.
  • Ablations (Llama). Using the same head as a loss weight instead of a logit divisor hurts substantially. Feature-masking variants have smaller effects.
  • Analyses of the learned scale. τ correlates with an external response-similarity proxy after controls. It varies substantially within prompts. It is modestly associated with positional instability of the Arena-Hard judge.
Critical strengths

None

Critical weaknesses

1. A clean, well-controlled demonstration of the core phenomenon. The synthetic study is the most convincing part of the paper. Because the true utility gap and scale are known, the authors can show directly what their theory predicts: DPO and H-DPO fit the labels equally well (identical test NLL), yet DPO's learned score drifts toward Δ*/τ* as heterogeneity grows, while H-DPO stays aligned with Δ*. The oracle W-DPO control is particularly effective. Even reweighting by the true 1/τ* tracks DPO rather than H-DPO, which makes the "reweighting can't fix this" claim hard to argue with. The best-of-N results then show the distortion has a practical cost, not just a semantic one. This matters for my assessment because it establishes that the problem the paper identifies is real and not an artifact of the formalism.

2. A useful conceptual map of DPO variants. The taxonomy separating utility-, error-, outcome-, temporal-, and loss-layer heterogeneity brings real clarity to a crowded literature where similar terms describe statistically different things. The distinction between putting a pair-specific quantity inside the likelihood and using it only as a loss weight is especially useful, and both the theory and the ablations back it up. This weighs in the paper's favor because, even if the empirical gains are modest, the framing could shape how future DPO variants are positioned and compared.

Questions for authors

Questions that could change my assessment

  1. Real-model magnitude evaluation. Your theory says the distortion affects score magnitudes, not pairwise ranking, yet the real-world evidence is RewardBench accuracy. Can you report a magnitude-sensitive evaluation on at least one real model, such as best-of-N selection with an external reward model or judge, or calibration of the implicit reward? A clear H-DPO advantage there would substantially strengthen the link between theory and practice.

  2. Noise-robust baselines. You attribute the real-world gains to a finite-sample effect in which ambiguous, label-noisy pairs receive smaller updates. If that is the mechanism, how does H-DPO compare to methods designed for noisy preferences, such as conservative/label-smoothed DPO, rDPO, or ROPO? If these match H-DPO, the gains would seem to come from noise robustness rather than from modeling comparison-specific scale.

  3. Is τ learning ambiguity or |Δ|? The scale head receives the absolute frozen-reference log-probability margin, which is closely tied to score magnitude. How strongly does the learned τ correlate with |Δ| (or |r_ref|) on held-out data? If τ is largely a function of |Δ|, the division acts more like a learned link function than a per-pair noise model. Partialling out |Δ| in the Appendix H analyses would help here.

  4. Seeds and tuning budget. Can you report results over multiple seeds for at least one base model? Can you also comment on tuning fairness? λτ was searched over up to 10 values per base, while DPO's β had 3. A comparison with matched tuning budgets would make the ~1-point RewardBench gains easier to trust.

Clarifications

  1. Relation to Hong et al. (2024). That work also learns per-pair scaling. What specifically places it in the loss layer rather than the error layer, and how would it perform in your setup?

  2. Mistral results. Do you have an explanation for why H-DPO shows no RewardBench gain on Mistral-7B? For example, does the learned scale vary less there?

Suggestions for future work (not affecting my score)

  1. Extending the identifiability analysis beyond the one-feature toy family toward the richer feature-dependent heads used in practice.
  2. Exploring whether a pairwise scoring rule that uses τ at evaluation time gives additional benefits beyond the standard implicit reward.
5 comments
AuthorsOpenReview ↗

Response to Reviewer Wt87

Thank you for highlighting the synthetic validation and taxonomy as strongest. Your review identified the key theory-to-practice gaps: (i) a real-model magnitude-sensitive evaluation, (ii) evidence against a pure noise-robustness explanation, and (iii) a check that the learned scale is not merely score magnitude. We add direct evidence for all three, plus three-seed Llama runs (Apps. M, K, H.4, L).

Q1. Real-model magnitude-sensitive evaluation. Yes. We added the requested evaluation: best-of-NN response selection on the main-text Llama-3.1-8B-Instruct DPO/H-DPO checkpoints over the official RewardBench v2 candidate pool (App. M, Tables 26–28), and H-DPO is better under both criteria. Held-out chosen labels: 57.62% → 60.67% (+3.06 pts, 95% CI [+1.62,+4.49]; paired permutation and McNemar p=5×10−5p=5\times10^{-5}). External RM (INF-ORM-Llama3.1-70B): selected-answer score 7.434 → 7.674 (+0.239, CI [+0.119,+0.360]). This speaks directly to your theory-to-practice concern: it ranks multiple candidates rather than one pairwise sign, and the gain exceeds the same pair's RewardBench discrimination gain (+1.06 RB v1, +1.78 RB v2), consistent with the distortion being ranking-invariant within a pair but mattering for multi-candidate selection. Scope: behavioral response-selection evidence, not cardinal-utility identification — an argmax over the single-response reward (monotone-invariant), on the benchmark's pool (not self-samples), one base/seed, with a not-proven-independent external RM.

Q2. Noise-robust baselines. We added a dedicated appendix (App. K); the evidence argues against a purely generic noise-robustness explanation. cDPO/rDPO retain the homoscedastic link σ(βΔ)\sigma(\beta\Delta), whereas H-DPO changes the model itself to σ(βΔ/τϕ)\sigma(\beta\Delta/\tau_\phi); consistent with Theorem 1, methods retaining the homoscedastic link stay tied to the homoscedastic target in population. New synthetic head-to-head (App. K, Table 22), α=1\alpha=1: ρΔ\rho_\Delta H-DPO 0.946 / cDPO 0.820 / rDPO 0.785 / DPO 0.776; GdistG_{\mathrm{dist}} H-DPO −0.205 (only negative) / +0.033 / +0.081 / +0.107; BoN-8 H-DPO 0.742 / 0.571 / 0.576 / 0.556. So the tested noisy-preference losses modestly regularize DPO but do not reproduce H-DPO's branch-specific behavior. On real data, the learned-weight ablation — the most direct "down-weight ambiguous pairs" control — is the only variant below plain DPO on both RewardBench versions and significantly worse than full H-DPO. The label-noise variants cDPO/rDPO were run synthetically only; we did also run two real-world robust-DPO variants (ε\varepsilon-DPO, R-DPO) across all three bases (App. K, Table 23), where no single loss-layer variant consistently matches H-DPO. The synthetic head-to-head thus tests your mechanism hypothesis directly, and does not bear it out.

Q3. Is τ\tau learning ambiguity or ∣Δ∣|\Delta|? We added the requested held-out analysis (App. H.4, Table 15); it argues against "τ\tau is largely a function of score magnitude." On held-out RewardBench v2: ρ(log⁡τ,∣Δ^∣)=+0.058\rho(\log\tau,|\hat\Delta|)=+0.058 [+0.008,+0.107] — only a weak correlation with the model's own magnitude; and the difficulty association is unchanged after controlling for ∣Δ^∣|\hat\Delta| (−0.710 → −0.708). The raw frozen-reference margin is strongly correlated with log⁡τ\log\tau (−0.970) but heavily length-confounded here; after per-token normalization it drops to −0.064. We present this as empirical support against that failure mode, not a theorem-level identification result.

Q4. Seeds and tuning budget. Three-seed Llama runs (App. L, Tables 24–25): H-DPO exceeds DPO on RewardBench in all three seeds — RB v1 75.63±0.15 → 76.70±0.17 (+1.07), RB v2 46.23±0.12 → 48.00±0.17 (+1.77); not a single-seed artifact. We did not run a matched-budget hyperparameter study, and now state that explicitly. The signal we ask you to rely on is primarily the RB v2 gain: positive for every tested λτ\lambda_\tau in the full Llama sweep and replicated across all three seeds. We do not ask you to read RB v1 as a matched-budget win.

Q5/Q6. Hong et al. (2024); Mistral. Ada-DPO/APS (§3.2, App. J) differs structurally, not syntactically: its τi\tau_i is a free per-training-pair DRO variable that does not generalize to unseen pairs and is discarded after training, whereas H-DPO learns a persistent, feature-conditioned comparison-specific scale. No head-to-head was run, so the oracle W-DPO control is a structural reference, not a performance guarantee. On Mistral, RewardBench is a genuine null case (AlpacaEval positive), so the benefit is base/benchmark-dependent; a flatter learned scale (closer to DPO by Proposition 1) is one unverified hypothesis.

Q7/Q8. Both future-work suggestions are now in the revised Conclusion (richer feature-dependent identifiability; test-time rules consuming τϕ\tau_\phi).

We hope these additions address your main concerns and would support reconsideration of the score.

Following up: complete tables for the four experiments behind your concerns (Q1 best-of-N, Q2 noise-robust, Q3 scale-vs-magnitude, Q4 three-seed), identical to the revised paper (Apps. M, K, H.4, L). On all four the evidence favors H-DPO.

1. Real-model best-of-N (App. M, Table 28). Llama-3.1-8B-Instruct DPO vs H-DPO, RewardBench v2 pool (1,602 paired prompts); last column = Δ\Delta (H-DPO −- DPO), 95% CI. Ext. RM = INF-ORM-Llama3.1-70B.

MetricDPOH-DPOΔ [95% CI]
Selection accuracy (%), chosen labels57.6260.67+3.06 [+1.62, +4.49]
Reference score of selected (ext. RM)7.4347.674+0.239 [+0.119, +0.360]
Regret (oracle - selected), ext. RM2.3802.141-0.239 [-0.360, -0.119]
Head-to-head (win/loss/tie), ext. RM123/85/1394net +2.37% [+0.62, +4.12]

Every CI excludes zero; selection accuracy is significant at p=5×10−5p=5\times10^{-5}, and +3.06 exceeds the pair's RewardBench discrimination gain (+1.06/+1.78). We read it as response-selection evidence, not cardinal utility recovery.

2. Noise-robust head-to-head (App. K, Table 22). Synthetic branch, mean±sd over five seeds; cDPO/rDPO keep the homoscedastic link σ(βΔ)\sigma(\beta\Delta), differing only in the loss; W-DPO = oracle weight 1/τ⋆1/\tau^\star. ρΔ = utility-gap recovery; G-dist<0 = toward true Δ⋆\Delta^\star; BoN-8 = magnitude-sensitive selection.

αMethodρΔG-distrank-accBoN-8
0.50DPO0.908±0.008+0.071±0.0100.952±0.0040.679±0.008
0.50H-DPO0.972±0.003-0.038±0.0080.957±0.0020.804±0.008
0.50W-DPO (oracle)0.897±0.008+0.083±0.0100.952±0.0030.677±0.020
0.50cDPO0.914±0.006+0.059±0.0060.947±0.0100.689±0.011
0.50rDPO0.896±0.012+0.085±0.0140.951±0.0030.685±0.020
0.75DPO0.839±0.012+0.108±0.0180.941±0.0090.611±0.013
0.75H-DPO0.954±0.011-0.094±0.0280.957±0.0050.756±0.036
0.75W-DPO (oracle)0.823±0.008+0.133±0.0080.952±0.0080.615±0.015
0.75cDPO0.868±0.008+0.064±0.0120.945±0.0060.624±0.015
0.75rDPO0.829±0.023+0.117±0.0360.941±0.0080.618±0.039
1.00DPO0.776±0.014+0.107±0.0340.937±0.0080.556±0.011
1.00H-DPO0.946±0.018-0.205±0.0450.954±0.0080.742±0.035
1.00W-DPO (oracle)0.748±0.020+0.156±0.0380.941±0.0130.567±0.009
1.00cDPO0.820±0.009+0.033±0.0180.937±0.0060.571±0.010
1.00rDPO0.785±0.036+0.081±0.0640.939±0.0060.576±0.025

Only H-DPO flips G-dist negative and dominates BoN-8 at every α>0\alpha>0; cDPO/rDPO improve ρΔ over DPO but fall well short, and rank-acc barely moves (sign-blind). So robustifying the homoscedastic loss does not reproduce H-DPO's behavior.

3. Is the learned scale just score magnitude? (App. H.4, Table 15). Held-out RewardBench v2 subset (n=1596n=1596); Spearman correlations of the learned log⁡τϕ\log\tau_\phi. Model-margin magnitude = ∣Δ^∣|\hat\Delta|; per-token-normalized margin = ∣gapnorm∣|\mathrm{gap}_{\mathrm{norm}}| (smaller = more ambiguous).

AnalysisEstimate95% CI
Spearman: learned log-scale vs model-margin magnitude+0.058[+0.008, +0.107]
Spearman: vs raw reference-margin magnitude-0.970[-0.975, -0.965]
Spearman: vs per-token-normalized margin, marginal-0.064[-0.120, -0.008]
Spearman: vs per-token-normalized margin, within-subset-0.302[-0.352, -0.249]
Partial: vs normalized margin, given lengths, subset-0.710-
Partial: vs normalized margin, given lengths and model-margin magnitude-0.708[-0.740, -0.673]

Weakly tied to score magnitude (+0.058); the difficulty association survives controlling for ∣Δ^∣|\hat\Delta| (−0.710 → −0.708); the raw-margin correlation (−0.970) is a length confound (normalized: −0.064). So τ\tau is not explained by magnitude alone (empirical, not theorem-level).

4. Three-seed stability and tuning budget (App. L, Table 25). Llama-3.1-8B-Instruct, mean±sd over seeds 42/43/44 (seed 42 = main-text run); RB = RewardBench.

MethodAE2-LCAE2-WRAH-SCAH-WRRB v1RB v2
DPO23.60±0.1026.70±0.1037.87±0.2138.17±0.2575.63±0.1546.23±0.12
H-DPO24.10±0.3027.63±0.4040.87±0.3140.47±0.0676.70±0.1748.00±0.17

H-DPO beats DPO on RewardBench in all three seeds (RB v1 +1.07, RB v2 +1.77, each larger than the seed sd), so it is not a single-seed artifact. No matched-budget tuning study was run; rely on the RB v2 gain (positive for every tested λτ\lambda_\tau), not RB v1.

We would be grateful to know whether these results address your concerns and, if so, whether they would support reconsidering your assessment; we are glad to run further analyses if useful.

AuthorsOpenReview ↗

Further Discussion with Reviewer Wt87

Dear Reviewer Wt87,

Thank you again for taking the time to review our paper and for your thoughtful, constructive comments. We have carefully considered your feedback and provided detailed responses to each of your questions:

For Q1 (real-model magnitude-sensitive evaluation), we added a real-model best-of-NN response-selection evaluation on the main-text Llama-3.1-8B-Instruct DPO/H-DPO checkpoints over the RewardBench v2 candidate pool (App. M, Table 28): H-DPO selects higher-quality answers under both held-out chosen labels (+3.06+3.06 pts, p=5×10−5p=5\times10^{-5}) and an external reward model (+0.239+0.239).

For Q2 (noise-robust baselines), we added a synthetic head-to-head against cDPO/rDPO (App. K, Table 22): they modestly regularize DPO but do not reproduce H-DPO's utility-gap recovery or best-of-NN behavior; we also added two real-world robust-DPO variants (ε\varepsilon-DPO, R-DPO) across all three base models (App. K, Table 23).

For Q3 (is the learned scale merely score magnitude?), we added a held-out analysis on RewardBench v2 (App. H.4, Table 15): the learned scale has only a weak correlation with the model's own score magnitude (+0.058+0.058), and its association with comparison difficulty is essentially unchanged after additionally controlling for ∣Δ^∣|\hat\Delta|.

For Q4 (seeds and tuning budget), we added three-seed Llama runs (App. L, Tables 24–25): H-DPO exceeds DPO on RewardBench in all three seeds (RB v1 +1.07+1.07, RB v2 +1.77+1.77), so the gain is not a single-seed artifact; we also now state explicitly that the tuning budget was not matched.

For Q5/Q6 (relation to Hong et al.; Mistral), we clarified that Ada-DPO/APS differs structurally rather than syntactically (§3.2, App. J), and we present Mistral RewardBench as a genuine null case (its AlpacaEval result remains positive).

For Q7/Q8 (future work), both suggestions are now reflected in the revised Conclusion.

As the author–reviewer discussion phase is drawing to a close, we would greatly appreciate knowing whether our responses have addressed your concerns. We would also be very happy to clarify any remaining questions if needed.

Thank you again for your careful review.

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global response

We thank all three reviewers for the careful reading and constructive feedback. We are encouraged that the reviews converged on the paper's strongest parts: (i) the clean synthetic demonstration of the distortion phenomenon, including the oracle W-DPO control; (ii) the branch-level taxonomy separating error-layer observation scale from loss-layer reweighting; (iii) the identifiability framing; and (iv) the broad baseline set together with the mechanism-isolating ablation.

The reviews also converged on four key gaps: (1) whether this branch has a policy interpretation beyond score semantics; (2) whether faithful Δ⋆\Delta^\star recovery improves downstream decisions; (3) whether the learned τ\tau is genuinely comparison-specific rather than merely score magnitude; and (4) whether the real-world gains could instead be explained by generic noise robustness, seed variance, or tuning budget. The revision adds direct evidence for each:

Restricted policy interpretation (VNJ3 W1/Q1, e1PJ W2): we add a restricted population-level KL-policy interpretation (Corollary 2; App. N), making explicit that it does not certify the full neural implementation and does not restore the standard homoscedastic DPO/RLHF equivalence theorem for a flexible neural head.

Decision relevance of faithful recovery (Wt87 Q1, VNJ3 Q2): we add a real-model best-of-N response-selection evaluation (App. M) and a synthetic matched-KL policy-value test with known reward (App. N), and retain the synthetic BoN regret analysis (App. F.8); all favor H-DPO. The real-model result is response-selection evidence — an argmax over the single-response implicit reward — rather than a direct identification of cardinal utility recovery.

Is τ\tau just score magnitude? (Wt87 Q3, VNJ3 Q3, e1PJ Q2): we add held-out real-data analyses (App. H.4), a synthetic ground-truth absorption check (App. F.9), and three-seed stability diagnostics (App. L); these do not support the failure mode that τ\tau simply absorbs utility magnitude.

Noise robustness / seeds / tuning (Wt87 Q2/Q4): we add synthetic cDPO/rDPO comparisons (App. K), three-seed Llama runs (App. L), and now state explicitly that the tuning budget was not matched.

We also tightened scope throughout. The theory is now stated explicitly as restricted-family and population-level; the real-world gains are described as moderate and benchmark-specific (significant on Llama/Gemma RewardBench, null on Mistral RewardBench); and the additive-head control is presented as partial rather than null evidence. Point-by-point responses follow below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.