- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper casts DPO variants as random-utility models and sorts them by where heterogeneity enters: utility, error/scale, outcome, time, or loss. It singles out comparison-specific observation scale as a distinct branch. Its instantiation, H-DPO, models with a learned log-scale head. On the theory side, homoscedastic DPO under heteroscedastic truth recovers \Delta^/\tau^, not \Delta^ (Thm. 1, Cor. 1). Positive loss reweighting cannot fix this (Prop. 2). The decomposition is locally identifiable up to global scale in restricted smooth families (Thm. 2). In synthetic experiments, H-DPO recovers \Delta^ (ρ 0.95 vs. 0.78 for DPO at α=1). On UltraFeedback with three 7–9B models, H-DPO gains about 1–2 points on RewardBench and AlpacaEval/Arena-Hard over DPO, and most generation gains are not significant.
Critical strengths
-
Separating error-layer scale from loss-layer reweighting is a useful distinction. The synthetic study shows it cleanly: oracle W-DPO follows DPO, while H-DPO recovers the true gap (Table 2).
-
The baseline set is broad and matched to the taxonomy (L-X, L-V and L-T methods), and the ablation isolates "division inside the logit" from capacity.
Critical weaknesses
-
The theory adds little and does not address the question that matters for DPO. Heteroscedastic logit and its bias under homoscedastic fitting are classical in discrete choice, as the paper notes (Fiebig et al., Hess & Train). Thm. 1 and Cor. 1 follow almost directly: if \tau^ varies, \Delta^/\tau^ is not proportional to \Delta^. Identifiability holds only locally in restricted families, which a neural does not satisfy. More importantly, DPO's implicit reward is tied to the KL-regularized RLHF optimum. With a pair-dependent , it is unclear what policy objective H-DPO optimizes, or why recovering \Delta^ rather than gives a better policy. The paper's main claim is about semantic faithfulness, but it does not show that faithfulness affects downstream behavior.
-
The real-data evidence is thin. There is one training set (UltraFeedback), and gains are around 1 point: RB v1 +1.0/+1.4/+0.1, and generation gains are mostly not significant. On Gemma, H-DPO trails ε-DPO, IPO and SimPO on AlpacaEval. The capacity-matched additive-head ablation comes within 0.2–1.3 points of full H-DPO on most metrics. The "ambiguity" interpretation of rests on one partial correlation () with a proxy.
Questions for authors
-
What KL-regularized objective does the H-DPO policy optimize when depends on the pair? Is the learned still the optimal policy for any well-defined reward?
-
Can you show that the improved recovery of seen in the synthetic setting produces better policies (e.g., in a synthetic setup where the true reward is known)?
-
With a flexible neural , what prevents it from absorbing part of the utility signal? Do you observe drift in the learned scale across seeds?
4 comments
Follow-up on Q1/Q2: the direct policy-value evidence
We don't want our point-by-point reply to be missed. Our full response to your W1/W2 and Q1–Q3 is posted above; here we surface the one result most directly tied to your core concern — whether better recovery actually yields a better policy (Q2), and what objective H-DPO targets (Q1).
Matched-KL policy value with known reward (App. N, Table 29). A two-action policy built from the learned scores, evaluated at a common KL budget against the true synthetic reward (an instantiation of Corollary 2):
| α | mean KL | U(DPO) | U(H-DPO) | U(oracle) | U(H)−U(D) | seeds |
|---|---|---|---|---|---|---|
| 0.0 | 0.100 | +0.2087 | +0.2087 | +0.2113 | -0.0000±0.0004 | 4/5 |
| 1.0 | 0.100 | +0.1716 | +0.2013 | +0.2113 | +0.0297±0.0029 | 5/5 |
At the strongest heterogeneity (α=1) H-DPO earns higher true reward than DPO at the same KL budget in all 5/5 seeds, closing ~75% of DPO's gap to the oracle; in KL to the oracle policy it is ~4× closer (0.01045 vs 0.04151, App. N Table 30). At α=0 (no heterogeneity) the two are equivalent, exactly as the theory predicts. This is the direct known-reward test of your Q2; on a real model the same direction appears behaviorally (App. M best-of-N: +3.06 pts selection accuracy, ), which we read as response-selection evidence, not cardinal identification.
We do not claim a neural-head guarantee: Corollary 2 is restricted and population-level, and App. B.4 includes a counterexample showing scale variation alone need not force a strict gain. But we hope this closes the policy-relevance loop you identified.
We would be grateful for any indication of whether these additions address your concerns and whether they might support reconsidering your assessment; we are happy to answer anything further.
Thank you for your response and for addressing my concerns on the questions. However, my concerns on the weaknesses remain. I will remain my score at weak reject.
Dear Reviewer VNJ3,
Thank you for the follow-up and for acknowledging that the questions are addressed. We respect that your concerns on the weaknesses remain. In the revision we have narrowed our claims to what the evidence supports — the real-world result is scoped to RewardBench discrimination (generation descriptive), and the theory is stated as a restricted, population-level result rather than a neural-head guarantee. We recognize the remaining difference is one of assessment about the significance of the contribution, and we respect your view. Thank you again for the rigor and time you brought to the review.
Best regards, Authors
Response to Reviewer VNJ3
Thank you for the careful critique. We agree the original submission did not close the loop tightly enough between branch-level semantics and policy-level relevance on real models. Scope up front: the policy interpretation we add (Corollary 2, App. N) is restricted and population-level, not a guarantee for the full neural head.
W1. "Theory adds little / does not address what matters." We agree with the reviewer that heteroscedastic logit, and the bias of homoscedastic fitting under heteroscedastic truth, are classical (Hess & Train 2017; Fiebig et al. 2010). Our contribution is not the econometric fact but its DPO-specific consequence: (1) homoscedastic DPO recovers the distorted target Δ⋆/τ⋆, not Δ⋆ (Theorem 1 / Corollary 1); (2) loss-layer reweighting does not fix this (Proposition 2 / Proposition 4); (3) the distortion is largely ranking-invariant but magnitude-visible synthetically (App. F.8); (4) new: a restricted population-level policy interpretation (Corollary 2). We also agree identifiability is only local and restricted-family, not a neural-head theorem.
W2. "Real-data evidence is thin." We agree the gains are modest and benchmark-specific, and have narrowed the claim: RewardBench discrimination is the primary real-world result, generation gains are descriptive, and the new best-of-N result (App. M) is the strongest behavioral evidence beyond pairwise discrimination. We retain your caveats: one training dataset (UltraFeedback); on Gemma AlpacaEval 2 LC, H-DPO (48.79%) is above DPO (47.62%) but below ε-DPO (54.74%), SimPO (52.35%) and IPO (51.92%) (Table 11); the additive-head control is partial rather than null; and RewardBench is significant on Llama/Gemma, null on Mistral.
Q1. What policy objective does H-DPO optimize when τ is pair-dependent? A pair-dependent τ in the training-time comparison channel does not, in general, preserve the standard DPO$\to$RLHF derivation for the full neural model. What we add is narrower: under Corollary 2's restricted recovery conditions, the corrected pairwise gaps recover a single response-level reward up to a common factor, so global rescaling to the target KL budget yields the KL-regularized optimum for that reward; for any same-budget comparator πD, U(πH)−U(πD)=βK⋅ExDKL(πD∥πH)≥0. We do not claim this for the deployed neural head; there we rely on empirical policy-value evidence (App. N, M). On whether the learned policy is optimal for some reward: any policy with the reference's support is optimal for its own implicit reward λlog(π/πref)+c(x) (Gibbs reparameterization, Rafailov et al. 2023); the substantive question is when those differences also match the label-generating reward, which App. B.4 gives — with a counterexample showing scale variation alone need not force a strict gain (exact-fit is needed).
Q2. Does better Δ⋆ recovery produce a better policy? Yes — with known synthetic reward it yields a better same-KL policy (App. N), and the same direction appears behaviorally on a real model. Direct synthetic answer (matched KL): at α=1, mean KL 0.10 nat, H-DPO beats DPO in all 5/5 seeds — DPO 0.1716, H-DPO 0.2013, oracle 0.2113 — paired gain +0.0297±0.0029, closing ~75% of DPO's shortfall; in KL to the oracle H-DPO is ~4× closer (0.01045 vs 0.04151). Real-model analogue: on 1,602 RewardBench v2 prompts (App. M), H-DPO selects a benchmark-chosen answer +3.06 pts more often (p=5×10−5) and scores +0.239 higher under an external RM. The real-model result is behavioral response-selection evidence, not cardinal-utility identification.
Q3. Does a flexible neural τϕ absorb utility? Seed drift? We do not claim a structural guarantee, but the new diagnostics do not support the failure mode that τϕ simply absorbs utility. Synthetic ground-truth (App. F.9): at α=1, binned by true log-scale, the within-bin correlation of learned log-scale with true ∣Δ⋆∣ is only −0.123±0.039, while scale recovery vs true log-scale is 0.949 (Table 2). Real held-out (App. H.4): ρ(logτ,∣Δ^∣)=+0.058, and the difficulty association is unchanged after controlling for ∣Δ^∣ (−0.710 → −0.708). Cross-seed drift (App. L; 1,596 pairs, seeds 42/43/44): the RMS of Δπ differs by ~14%; after unit-RMS normalization the pairwise RMSE is 0.111–0.141, while log-scale rankings correlate 0.998–0.999 — high reproducibility with moderate nonzero variation, not exact invariance (RewardBench uses only the gap's sign).
We hope these additions narrow the gap you identified: the revision claims no full neural guarantee but shows (i) a restricted KL-policy interpretation, (ii) better Δ⋆ recovery improving decisions under known synthetic reward, and (iii) the same direction behaviorally on a real model.