acceptodds
Under review as a conference paper at ICLR 2027

Robust Preference Alignment under Shifting Preference Distributions

Abstract

Current preference-based alignment techniques for large language models typically learn from fixed preference labels that order candidate responses for a given prompt. However, human judgments are multidimensional and context-dependent, so different but valid evaluation priorities may reorder the same responses. Standard objectives treat the observed order as a fixed target and therefore do not account for this variation. Consequently, when deployment priorities differ from those represented during training, a policy fitted to the training labels may not remain aligned. Existing robust methods mainly interpret deviations from observed labels as unreliable feedback at sample-level. We instead study robustness to conditional ranking shifts and propose Kendall Robust Preference Optimization (KRPO). KRPO captures the empirical structure of these shifts through a utility-gap-weighted Kendall ambiguity set, making reversals between similarly scored responses less costly than those between clearly separated responses. An entropy-regularized adversary then emphasizes difficult but plausible alternative rankings. Because KRPO changes only the ranking supplied to the base loss, it applies to both pairwise and listwise objectives. Experiments with seven baseline objectives on Llama-3.1-8B and Qwen3-1.7B show that KRPO consistently improves cross-benchmark generalization and surpasses sample-level robust baselines on identical, unperturbed data. KRPO thus shows that robustness to shifting evaluation priorities is achieved by modeling where a ranking is likely to change and it offers a general mechanism for extending existing preference objectives to criteria absent from their supervision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Preference Alignment, Robustness, Distribution Shift

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7ZsJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies robust LLM preference alignment under shifts in evaluation priorities, where different but valid criteria may induce different rankings over the same candidate responses. The authors propose Kendall Robust Preference Optimization (KRPO), which models conditional ranking uncertainty using a utility-weighted Kendall ambiguity set. The reversal costs are estimated from policy-derived utility gaps, allowing small-gap preference relations to change more easily while preserving strongly separated comparisons. An entropy-regularized adversary then assigns greater weight to plausible alternative rankings that incur higher training losses. The resulting robustification can be applied to existing pairwise and listwise preference optimization objectives.

Experiments on UltraFeedback-trained Qwen3-1.7B and Llama-3.1-8B evaluate KRPO across seven preference objectives and five downstream benchmarks. The method improves macro-average win rates across all 14 objective-model combinations. Additional comparisons with sample-level and ranking-level robust optimization methods, controlled preference-shift experiments, and geometry ablations provide evidence for the effectiveness of structured ranking uncertainty. The paper also provides conditional theoretical bounds relating the ambiguity set to deployment risk.

Critical strengths
  1. A well-motivated and structured approach to ranking-level preference uncertainty. The paper distinguishes valid changes in evaluation priorities from unreliable preference annotations. The proposed utility-weighted Kendall neighborhood provides a meaningful way to concentrate robustness on potentially reversible preference relations instead of treating all alternative rankings equally. The combination with an entropy-regularized adversary is mathematically coherent and compatible with multiple preference objectives.

  2. Broad empirical validation with targeted mechanism analysis. The evaluation covers seven preference objectives and two model families, with improvements in all 14 macro-average comparisons. The controlled preference-shift experiment in Section 5.4 is particularly valuable because it isolates changes in evaluation criteria while keeping prompts and candidate responses fixed. The uniform-distance ablation, radius sensitivity analysis, and comparisons with existing robust objectives provide additional evidence that the structured ambiguity set contributes to the observed improvements.

Critical weaknesses
  1. The evidence for robustness to realistic preference distribution shifts remains limited.

    The paper's central claim concerns robustness when deployment preferences differ from those represented in training. However, most of the experimental evidence in Table 1 comes from cross-benchmark generalization, where changes in task domains, prompts, response distributions, and evaluation criteria occur simultaneously. These improvements are encouraging but cannot independently establish robustness to conditional ranking shifts.

    The controlled experiment in Section 5.4 addresses this issue more directly, but its evaluation remains limited to three alternative annotation dimensions from UltraFeedback, a single Qwen3-1.7B model, and changes in a DPO-based surrogate loss over fixed candidate responses. It does not directly demonstrate improved generated-response quality under independently collected deployment preferences.

    Furthermore, the theoretical guarantees depend on assumptions about the locality of preference changes and the accuracy of policy-derived utility gaps. These assumptions are not directly verifiable from a single observed ranking, and the chosen robustness radius is not calibrated to an estimated deployment distribution. Consequently, the theory establishes conditional coverage rather than a practically verifiable guarantee of robustness.

  2. The incremental novelty and empirical reliability relative to existing ranking-level robust optimization require stronger support.

    Distributionally Robust Listwise Preference Optimization (Wu et al., 2026) already considers conditional uncertainty over ranking labels for a fixed candidate set. KRPO provides a meaningful extension through utility-weighted Kendall geometry and entropic adversarial weighting, but its broader conceptual formulation substantially overlaps with this prior setting. The novelty therefore primarily concerns the choice of ambiguity geometry and robust aggregation mechanism, rather than introducing ranking-level DRO itself.

    While the empirical results are positive, many individual improvements are relatively modest or negative, and the main training experiments use only one random seed. The reported bootstrap confidence interval for the aggregate gain captures evaluation-prompt variability but does not account for training randomness. The direct comparison with Robust PL in Appendix I is useful, but is restricted to the Plackett–Luce objective and two tested robustness radii.

    These limitations make it difficult to determine how consistently the proposed mechanism outperforms existing alternatives across repeated training runs and different hyperparameter configurations.

References

[1] Wu, X., Qian, J., Liu, P., Aggarwal, V., & Chen, J. (2026). Distributionally Robust Listwise Preference Optimization. arXiv:2607.01715. https://arxiv.org/abs/2607.01715

[2] Xu, Z., Vemuri, S., Panaganti, K., Kalathil, D., Jain, R., & Ramachandran, D. (2025). Robust LLM Alignment via Distributionally Robust Direct Preference Optimization. NeurIPS 2025. https://arxiv.org/abs/2502.01930

Questions for authors
  1. Utility-gap reliability: Figure 2 motivates the method using gaps from an external reward model, whereas KRPO constructs its ranking geometry using the current policy's implicit utilities. How well do these policy-derived gaps predict actual criterion-induced reversals throughout training? Is the relationship evaluated beyond the single diagnostic checkpoint?

  2. Hyperparameter selection: How were the robustness radii and prior parameters selected across the seven objectives? In particular, were the parameters chosen independently of the reported evaluation benchmarks? How sensitive are the improvements to this choice?

  3. Comparison with Robust PL: Appendix I compares KRPO with Robust PL using two total-variation radii. Were these configurations selected with comparable validation or hyperparameter-search budgets? Could the authors clarify how much of the improvement comes from the proposed geometry rather than differences in the robustness strength?

  4. Deployment relevance: The controlled-shift analysis measures changes in a DPO surrogate loss over fixed responses. Is there existing evidence that this reduction corresponds to improved generated-response preferences under the target evaluation criteria?

Reviewer_KssxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies preference shifts in which different evaluation criteria reorder the same candidate responses. It proposes Kendall Robust Preference Optimization (KRPO), combining a utility-gap-weighted Kendall neighborhood with an anchored, entropy-regularized adversary. Experiments cover two model families, seven objectives, five generation benchmarks, and a controlled fixed-candidate shift study.

The problem is meaningful and the method is clearly motivated. The results are promising, but numerical inconsistencies and incomplete mechanism validation limit the conclusions about deployment robustness.

Critical strengths
  • The formulation distinguishes valid criterion changes from annotation noise.
  • The weighted geometry, local prior, and entropic adversary have clear, interpretable roles.
  • All 14 model-objective combinations have positive macro-average point estimates in Table 1.
  • The controlled experiment directly tests criterion changes while keeping prompts and candidates fixed.
Critical weaknesses
  1. The reported aggregate improvement is inconsistent with Table 1. The text reports +1.36 percentage points, whereas averaging the 14 displayed macro improvements gives 1.1643, and averaging all 70 benchmark changes gives 1.1714. Rounding cannot explain this difference. Recompute the mean, bootstrap SE, and confidence interval from the same unrounded results. This does not establish that the improvement is absent, but the central statistic requires reconciliation.

  2. Lower shifted DPO loss does not necessarily imply better target-criterion ordering. Shrinking margins on conflicting pairs can reduce loss without correcting their predicted order. Add target pairwise accuracy or Kendall agreement, alongside matched label-smoothing or margin-shrinkage baselines.

  3. The practical utility proxy needs stronger validation. Figure 2 uses an external reward model, while training uses EMA policy scores. The theoretical reward-gap identity requires an ideal optimum or an approximation assumption. Figure 4c uses one Qwen LiPO checkpoint without EMA, so it does not directly validate the main training geometry.

  4. Reproducibility and stability evidence is incomplete. Full per-combination hyperparameters, validation procedures, baseline tuning budgets, and judging details are missing. Prompt-bootstrap intervals exclude training randomness, and the reported setup uses one training seed. Qwen KRPO step times also differ between Table 7 (4.53 seconds) and Appendix G.4 (7.56 seconds) without explanation.

  5. Disagreement and tie handling need sensitivity checks. The motivating analysis combines strict conflicts with all-ties, while the controlled study breaks ties by original candidate order. Separate these events, use prompt-cluster uncertainty intervals, and test tie-aware evaluation.

Questions for authors

Please see weaknesses.

Reviewer_HBz4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a failure mode of preference-based alignment that is orthogonal to label noise: even when every training annotation is reliable, a change in evaluation priorities (users, tasks, criteria) can reorder the same candidate responses within a record. The authors formalize this as a shift in the conditional ranking distribution given a prompt and candidate set, and propose Kendall Robust Preference Optimization (KRPO). KRPO (i) defines a utility-gap-weighted Kendall ambiguity set in which reversing closely scored responses is cheap while reversing clearly separated ones is expensive, (ii) places a distance-decayed prior anchored at the observed ranking over this set, and (iii) optimizes an entropy-regularized (KL-DRO) adversary with a closed-form solution. Because uncertainty enters only through the ranking argument of the base loss, KRPO is modular across pairwise and listwise objectives. Experiments on UltraFeedback with seven base objectives and two backbones (Qwen3-1.7B, Llama-3.1-8B) show consistent macro-average improvements (+1.36+1.36 points on average across 14 combinations; bootstrap CI [+0.88,+1.85][+0.88, +1.85]), superiority over sample-level robust baselines (KLDPO, DRDPO), a controlled attribute-shift experiment isolating the mechanism, and ablations supporting the weighted geometry and radius choice.

Critical strengths
  1. Well-motivated, clearly isolated problem. The distinction between valid criterion-induced reordering and label corruption is crisp, and Figure 2 provides direct empirical grounding: only 2.1%2.1\% (UltraFeedback) / 4.6%4.6\% (HelpSteer) of prompts receive one common weak order across annotation dimensions, and disagreement concentrates in small utility-gap pairs. This elevates the paper above "another robust DPO variant."
  2. Principled and modular method. The utility-weighted Kendall geometry gives the radius ρ\rho an interpretable meaning (a budget of "average reversals"), the anchored prior plus entropic DRO avoids the over-conservatism of a hard max, and the EMA stop-gradient geometry stabilizes the pair costs. Since only the ranking label changes, the construction composes with seven quite different base losses without modifying them — a genuine design strength demonstrated empirically.
  3. Unusually comprehensive and honest evaluation. Beyond the 14-combination grid with paired-prompt bootstrap, the paper (i) compares against both sample-level robust methods and the concurrent listwise-DRO method (Robust PL, Appendix I), (ii) adds a controlled fixed-candidate shift experiment (Section 5.4) that cleanly isolates the formalized risk, (iii) ablates geometry (uniform vs. weighted), radius, entropic-temperature limits (Appendix H), and support approximation fidelity (Appendix G), (iv) runs a pretraining-contamination audit with an explicit caveat for AlpacaEval (Appendix B.1), and (v) provides careful complexity accounting. The limitations section is candid (single-seed caveats, descriptive interpretations, capacity/size confound).
  4. Theory is appropriately scoped. The coverage result (Thm 1/Prop 1) and conditional deployment-risk bound (Thm 2) are stated with explicit acknowledgment that δ\delta and QAQ_A are unobservable, i.e., a conditional guarantee rather than a certificate. Proofs are complete and correct as far as I can verify.
Critical weaknesses
  1. Small and heterogeneous effect sizes; single training seed. The headline +1.36+1.36 macro-average gain rests on one seed (42) per configuration. Prompt-level bootstrap captures evaluation uncertainty but not training variance (data order, DoRA init), which for gains of +0.3+0.3 to +2.5+2.5 points could be of the same order. Moreover, several individual cells are notably negative (e.g., LPO4Rec-Llama MT-Bench −1.2-1.2, PL-Llama TL;DR −2.7-2.7, SimPO-Llama AlpacaEval −2.6-2.6), and the authors themselves concede that per-benchmark deltas do not survive multiplicity correction. The abstract's "consistently improves" is accurate only at the macro-average level.
  2. Non-standard judging protocol with a thematic circularity risk. All win rates use GPT-4.1-mini as judge with partially self-generated references (Mistral-family for HH/MT-Bench/TL;DR). Given that the paper's central thesis is precisely that evaluation criteria are unstable and multidimensional, relying on a single unaudited LLM judge — with no reported human-agreement validation — weakens the headline evidence. Absolute numbers are also not comparable to published results under standard protocols.
  3. Scalability beyond K=4K=4 is asserted but not demonstrated. All end-to-end training uses four-response rankings (2424 permutations). Appendix G's synthetic analysis shows the required support budget grows quickly (K=8K=8: budget 106106 at fixed ρ\rho, >128>128 when ρ\rho scales with pair count), and the end-to-end timing benefit of top-8 is admittedly modest. Since listwise supervision is most attractive precisely for longer lists, the practical scope of the method remains short-list post-training.
  4. The utility-gap proxy is validated only indirectly. The motivating diagnosis (Fig 2b) uses an external reward model (Skywork-Reward-V2-8B), whereas training-time reversal costs use the policy's own implicit rewards; Prop 2's identification holds only at the KL-regularized optimum. The coverage diagnostic (Fig 4c) is a nice partial bridge but is computed from a single LiPO diagnostic checkpoint. A direct measurement of agreement between "small policy-gap pairs" and "criterion-flippable pairs" at training time is missing.
Questions for authors
  1. Can you report mean ±\pm std over ≥3\geq 3 training seeds for 2–3 representative combinations (e.g., LiPO-Qwen, DPO-Llama)? Does the +1.36+1.36 aggregate survive seed variance?
  2. Did you validate GPT-4.1-mini judgments against human labels or a second judge on a subset? Given the paper's theme, judge robustness seems essential rather than optional.
  3. How correlated are training-time policy utility gaps with the external-RM gaps used in the Figure 2 diagnosis (e.g., Spearman correlation or top-kk overlap of "flippable" pairs)?
  4. Can the controlled shift experiment be replicated on Llama-3.1-8B? If your capacity-based explanation of the KLDPO/DRDPO contrast is correct, what would you predict?
  5. For K>4K>4 deployments, what configuration do you recommend, and what is the honest computational envelope (Table 11 suggests budgets >100>100 at K=8K=8)?
  6. Is the small-α\alpha degradation in Fig 3a present for other base objectives/backbones? Should practitioners avoid KRPO when the deployment criterion is expected to be close to the training criterion?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.