acceptodds
Under review as a conference paper at ICLR 2027

Robust Nash Learning from Human Feedback against Preference Uncertainty

Abstract

Nash learning from human feedback (NLHF) models preference alignment as a two-player zero-sum game under general pairwise preferences directly, bypassing the restrictive latent reward assumption. But prior works typically assume this preference is unique and fixed. Yet in practice, such preference can be uncertain or pluralistic. We thus study robust NLHF, seeking a policy that is robust to all plausible preferences within some ambiguity set through optimizing the worst-case performance under the opponent policy and adversarial preference. However, the resulting double minimization does not generally preserve the convex-concave structure of standard NLHF. In this paper, we show that this difficulty can be removed exactly by lifting the adversary to a joint scenario-response law with conditional KL regularization, from which we restore the convex-concave structure and preserve the original robust value. We first consider the case of finite ambiguity sets, showing our reformulation yields a closed-form joint entropic Mirror-Prox algorithm with an convergence guarantee. We then extend the framework beyond finite scenario families to finite-dimensional compact-convex uncertainty sets with affine preference kernels. Although the natural continuous initialization has infinite KL radius against atomic worst cases, we use a localization argument to recover an rate. We further evaluate the performance and robustness of our framework. Our studies thus provide a unified optimization framework for robust alignment under uncertain preference beyond latent rewards.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
alignment, Nash learning, preference learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_563iOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Robust Nash Learning from Human Feedback (NLHF), which accounts for uncertainty in preference models. The key technique is to reformulate the objective as a convex-concave saddle-point problem by introducing a joint distribution over preference scenarios and opponent responses. The authors develop a Mirror-Prox-based algorithm and prove convergence rates of O(1/T)O(1/T) for finite uncertainty sets and O(log⁡T/T)O(\log T/T) for continuous uncertainty sets. Experiments demonstrate improvements over a nominal NLHF baseline in settings where distributions over fixed candidate responses are optimized.

Critical strengths
  • The paper is well-written: reading introduction made me grasp the key contributions and the structure easily.
  • The proposed reformulation of the robust NLHF into convex-concave problem by introducing joint distribution is simple and generalizable technique.
  • The theoretical analysis is solid, providing convergence guarantees for both finite and continuous uncertainty sets.
Critical weaknesses

This is a theory-oriented paper, so the following points are not major weaknesses. However, I have several concerns about the gap between theory and practice.

  • The convergence analysis for continuous uncertainty sets assumes exact integration, whereas the experiments rely on numerical approximation. The impact of approximation errors on the convergence guarantees is not discussed.
  • The paper assumes that the ambiguity set is given. Although the authors suggest constructing it from multiple fitted preference models, the experiments do not investigate how to estimate a reliable ambiguity set from finite and noisy preference data.
Questions for authors
  • Does authors have some insights on how to estimate the reliable ambiguity set in the LLM experiment setting? For example, how to determine the number of preference models to fit and how to ensure that the resulting ambiguity set adequately captures the underlying preference uncertainty?
  • Could the authors give some guarantee or insights on the effect to the convergence rate O(logT/T) if the integration over the uncertainty parameter is approximated by finite quadrature points?
6 comments

Thank you for the positive assessment and for identifying these two gaps between the theory and the implementation. We have added a bound that separates the optimization error on fixed quadrature nodes from the approximation of the continuous uncertainty set, together with a grid-refinement study (Remark 4, Appendix D.2.6). In the new Appendix E, we construct the ambiguity set from comparison data and distinguish coverage of the true kernel from coverage of its projection onto a specified model family.

Weakness 1 & Question 2: Approximate integration and convergence

A fixed grid gives an O(1/T)O(1/T) optimization bound plus an approximation term that does not shrink with TT; refining the grid with the target horizon restores O(log⁡T/T)O(\log T/T). Let H⊂UH\subset\mathcal U be the nodes, let ωh>0\omega_h > 0 be their normalized weights, and initialize the joint law at ω⊗πref\omega\otimes\pi_{\rm ref}. The quadrature implementation is then Algorithm 1 on this finite scenario set. For the affine mixture family and exact node-game updates with fixed 0<η≤10 < \eta\le1, we show in Remark 4 and Appendix D.2.6 that

0≤VU⋆−gU(πˉT)≤GapH(πˉT,QˉT)+DH(πˉT)≤2R+log⁡(1/ωmin⁡)ηT+δH,0\le V^\star_{\mathcal U}-g_{\mathcal U}(\bar\pi_T) \le \mathrm{Gap}_H(\bar\pi_T,\bar Q_T)+D_H(\bar\pi_T) \le \frac{2R+\log(1/\omega_{\min})}{\eta T}+\delta_H,

where R=log⁡(1/πref,min)R=\log(1/\pi_{\rm ref,min}), DH(π)=gH(π)−gU(π)D_H(\pi)=g_H(\pi)-g_{\mathcal U}(\pi), and

δH=12sup⁡w∈Udist⁡1(w,conv⁡H).\delta_H=\frac12\sup_{w\in\mathcal U}\operatorname{dist}_1(w,\operatorname{conv}H).

Convergence on a fixed grid therefore does not by itself imply convergence to the continuous optimum.

The proof separates the two errors. A joint law supported on HH is also feasible for the continuous adversary, which gives the exact identity GapU=GapH+DH\mathrm{Gap}_{\mathcal U}=\mathrm{Gap}_H+D_H. Theorem 4.2 bounds the first term with initial comparison radius 2R+log⁡(1/ωmin⁡)2R+\log(1/\omega_{\min}). For the second, the Gibbs-eliminated payoff ϕπ(w)\phi_\pi(w) is concave and satisfies ∂ϕπ/∂wk=π⊤A(k)νw∈[0,1]\partial\phi_\pi/\partial w_k=\pi^\top A^{(k)}\nu_w\in[0,1]. Since differences of simplex weights sum to zero, ϕπ\phi_\pi is 12\tfrac12-Lipschitz in ℓ1\ell_1. Its minimum over conv⁡H\operatorname{conv}H equals its minimum over HH, which yields 0≤DH≤δH0\le D_H\le\delta_H.

Refining the grid. To recover the rate for a target horizon TT, we choose a grid HTH_T with δHT≤1/T\delta_{H_T}\le1/T, keep it fixed during the run, and use uniform node weights. A grid with O(TdU)O(T^{d_{\mathcal U}}) nodes suffices, giving an O(log⁡T/T)O(\log T/T) bound and O(TdU+1m2)O(T^{d_{\mathcal U}+1}m^2) Mirror-Prox arithmetic operations. Concavity permits a smaller construction: for dU≥2d_{\mathcal U}\ge2, suitably placed boundary nodes give δH=O(∣H∣−2/(dU−1))\delta_H=O(|H|^{-2/(d_{\mathcal U}-1)}) by convex-body approximation (Bronshtein and Ivanov, 1975). Hence O(T(dU−1)/2)O(T^{(d_{\mathcal U}-1)/2}) nodes suffice, which reduces the arithmetic cost to O(T(dU+1)/2m2)O(T^{(d_{\mathcal U}+1)/2}m^2). These costs exclude constructing the nodes and evaluating numerical certificates. For the two-dimensional IMDb uncertainty set, the boundary-node count is O(T)O(\sqrt T).

This guarantee covers a family of fixed-grid runs indexed by their target horizon; it does not automatically extend to a run that changes grids. Such an output can instead be assessed directly with its continuous saddle-gap certificate (next comment).

Weakness 1 & Question 2 (continued)

Refinement study. To measure the approximation error on its own, we ran EJMP on six grids at T=2048T=2048 on 48,000 problems: 4,000 prompts, two references and six radii (Table 11, Appendix D.2.6). Selected rows follow; all errors are means in units of 10−310^{-3}.

GridNodesNode-game gapGrid overestimate DHD_HA priori bound on δH\delta_H
12×212\times2244.546.8130.91
48×648\times6 (submitted)2886.100.804.00
96×1096\times109606.510.301.40
192×12192\times122,3046.610.200.79

At the submitted 48×648\times6 grid, the grid overestimate is already smaller than the node-game gap on average and in nearly every problem. Refinement reduces the overestimate substantially, but the continuous gap changes only from 6.89×10−36.89\times10^{-3} to 6.81×10−36.81\times10^{-3}, because the node-game gap at fixed TT rises as DHD_H falls. This is consistent with the larger initial radius log⁡(1/ωmin⁡)\log(1/\omega_{\min}) of finer grids. Solving the 48×648\times6 node games to a certified gap of 10−810^{-8} leaves a mean robust-value loss of 0.175×10−30.175\times10^{-3}, against 1.674×10−31.674\times10^{-3} for the fixed-budget runs. At T=2048T=2048, most of the error therefore comes from optimization rather than quadrature.

The numerical comparison uses interval enclosures of the continuous worst case: outer-polygon vertices give a lower bound by concavity, and feasible scenarios give an upper bound. For any feasible output (π,Q)(\pi,Q), a certified lower bound g‾U(π)\underline g_{\mathcal U}(\pi) and upper bound h‾(Q)\overline h(Q) give the stopping test h‾(Q)−g‾U(π)≤ϵ\overline h(Q)-\underline g_{\mathcal U}(\pi)\le\epsilon, which certifies a robust-value error of at most ϵ\epsilon regardless of how the output was obtained. We use this test with ϵ=10−6\epsilon=10^{-6} for the main results of the revision.

Weakness 2 & Question 1: Constructing a reliable ambiguity set from data

We thank you for this question. Constructing ambiguity set through concentration inequality is standard in statistical learning, and we now added Appendix E, which builds the set from the comparison likelihood as a split likelihood-ratio confidence region with finite-sample coverage, and fits our model. The number of components is chosen by held-out likelihood on a separate data split, and a transfer bound states what the set guarantees for the true preferences. Consider reciprocal component kernels A(1),…,A(K)A^{(1)},\ldots,A^{(K)} with entries in (0,1)(0,1) and their mixtures Ax(w)=∑kwkAx(k)A_x(w)=\sum_k w_kA_x^{(k)}, w∈ΔKw\in\Delta_K. No Bradley–Terry representation is required. Once the components are supplied, their mixture weights can be estimated from binary comparisons without evaluator-group labels.

We split the data into a pilot sample D1D_1 and an independent validation sample D0D_0. All component fitting, the selection of KK and the pilot estimate w^1\hat w_1 use D1D_1 only, and these choices are frozen before D0D_0 is examined. For comparison tt, let at,k=Axt(k)(yt≻yt′)a_{t,k}=A^{(k)}_{x_t}(y_t\succ y'_t) and pt(w)=∑kwkat,kzt(1−at,k)1−ztp_t(w)=\sum_k w_k a_{t,k}^{z_t}(1-a_{t,k})^{1-z_t}. For 0<δ<10 < \delta < 1, define

\widehat{\mathcal U}=\left\\{w\in\Delta_K: \sum_{t\in D_0}\log\frac{p_t(\hat w_1)}{p_t(w)} \le\log\frac1\delta\right\\}.

This is a split likelihood-ratio set (Wasserman, Ramdas and Balakrishnan, 2020). It contains w^1\hat w_1, and it is compact and convex because each validation log-likelihood is continuous and concave in ww.

Weakness 2 & Question 1 (continued)

Coverage. What the set covers depends on whether the family contains the truth. If P⋆=A(w⋆)P^\star=A(w^\star), the set contains w⋆w^\star with probability at least 1−δ1-\delta for every sample size. Conditional on the pilot sample and an exogenous query design, the validation likelihood ratio has expectation one, and independence and Markov's inequality give coverage.

Under misspecification, suppose the validation observations are independent and identically distributed, with queried pairs drawn from μ\mu, and that some mixture has finite risk

Rμ(w)=Eμ,kl(P⋆∣A(w))<∞.R_\mu(w)=\mathbb E_\mu\\,\mathrm{kl}(P^\star\\|A(w)) < \infty.

Then the set covers any fixed minimizer w†w^\dagger of this risk with probability at least 1−δ1-\delta: convexity of the mixture family gives the likelihood-ratio expectation bound that the same argument needs. Finite risk is automatic for a finite query space with positive component probabilities, or when the components are uniformly bounded away from zero and one. Appendix E gives the proof.

What the set implies for the true preferences. The second guarantee covers the KL projection, not an arbitrary true kernel outside the family. Define

\varepsilon &= \sup_{x,y,y'} |P^\star_x(y\succ y')-A_x(w^\dagger)(y\succ y')|, \\\\ \Delta_A(\widehat{\mathcal U}) &= \sup_{w,v\in\widehat{\mathcal U}}\sup_{x,y,y'} |A_x(w)(y\succ y')-A_x(v)(y\succ y')|. \end{aligned}$$ On the coverage event, for each evaluation prompt, the returned policy satisfies $$\begin{aligned} g^\star(\pi) &\ge V^\star_{\widehat{\mathcal U}}-\mathrm{Gap}(\pi,Q)-\varepsilon, \\\\ \max_{\pi'}g^\star(\pi')-g^\star(\pi) &\le \Delta_A(\widehat{\mathcal U})+\mathrm{Gap}(\pi,Q)+2\varepsilon \\\\ &\le \tfrac12\operatorname{diam}_1(\widehat{\mathcal U})+\mathrm{Gap}(\pi,Q)+2\varepsilon. \end{aligned}$$ Proposition E.1 states the last form, and Appendix E also states the middle one, which follows because each payoff averages kernel entries with weights $\pi_y\nu_{y'}$. The three terms separate uncertainty among plausible kernels, optimization error and approximation error of the chosen family. The validation data do not estimate the uniform error $\varepsilon$, especially on unobserved pairs; under correct specification, $\varepsilon=0$. **How many models.** We select $K$ and the components on $D_1$ by held-out likelihood, leaving $D_0$ untouched, so that coverage holds conditionally on the selected family. A component is worth adding only if it raises the held-out likelihood. A component outside the convex hull of the others enlarges the family and can reduce $\varepsilon$, but it adds a free weight, which can widen $\widehat{\mathcal U}$ and make the policy more conservative. A component inside the hull adds no new kernel. Anonymous comparisons alone generally cannot identify latent evaluator components, so the components come from side information, such as group labels, annotator identities or candidate models like our profiles, and the comparisons estimate their weights. The guarantee does not certify that the chosen $K$ is sufficient; the remaining misspecification enters only through $\varepsilon$. **Deployment shift.** We treat deployment shift as a modeling choice: it can be represented by $\overline{\mathcal U}_\rho=\\{w:\mathrm{KL}(w\\|v)\le\rho\text{ for some }v\in\widehat{\mathcal U}\\}$, which is compact and convex. On the event $w^\star\in\widehat{\mathcal U}$, it contains every deployment mixture within KL distance $\rho$ of $w^\star$. Training comparisons do not determine this budget. **In the LLM setting you ask about**, the pipeline is: fit candidate component kernels and choose $K$ on $D_1$; form $\widehat{\mathcal U}$ on $D_0$; enlarge it to $\overline{\mathcal U}_\rho$ if deployment shift is expected; run EJMP on the result and report its certificate. Our IMDb experiments prescribe the profiles, nominal weights and radius, so Appendix E is a statistical construction that we have not validated empirically. Its sets satisfy the structural assumptions of our optimization results; for a continuous set, exact integration or the fixed-node bound of Remark 4 is still required. ### References E. M. Bronshtein and L. D. Ivanov. The approximation of convex sets by polyhedra. Siberian Mathematical Journal, 16(5):852–853, 1975. L. Wasserman, A. Ramdas and S. Balakrishnan. Universal inference. Proceedings of the National Academy of Sciences, 117(29):16880–16890, 2020.
Reviewer_SqX6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses the problem of LM alignment under a general pairwise comparison preference model, potentially allowing for intransitive preferences in contrast to scalar reward models. In particular, it address the problem of preference model uncertainty (due to few preference examples, or contradictory examples coming from diverse annotators) within the framework of Nash learning from human feedback (NLHF) that models the alignment problem as a two-player zero-sum game. They start with a robust extension of the NLHF problem that comprises an additional minimization layer over an ambiguity set representing the possible preference models. From there, their contributions are : - In order to bypass the potentially non convex-concave structure of the robust NLHF alignment problem (due to the bilinearity in the preference parameter and the adversarial policy), they provide a new formulation, that the authors claim to have the advantage of being exact compared to previous work (Th 3.1). The idea is to consider the preference parameter and the adversarial policy as a joint parameter. - They then provide two optimization algorithm corresponding to two types of ambiguity sets (Section 4: discrete, Section5:continuous), each being associated with saddle-gap bounds. - Finally, they show numerical results on two preference datasets (synthetic and a real-world).

As I am not familiar with certain concepts, I am unable to properly assess the theoretical soundness of the theorems. I also find that the paper is written in a rather jargon-heavy style, which does not facilitate understanding for non-experts and makes the reading challenging in general.

Critical strengths
  • The paper addresses an interesting, challenging, and underexplored problem: allowing for more expressive preference models than scalarized reward models in the alignment problem, while accounting for uncertainty in these preference models. The combination of those two difficulties (general preference models and uncertainty) appears to have remained largely unexplored in the literature, and raise interesting optimization challenges. The proposed solutions seem to have an ambitious scope: optimization algorithms with theoretical guarantees, applicable to different settings and practical to implement.
Critical weaknesses
  • The paper misses a more thourough motivation of why we would need to consider general preference models and uncertainty modelization in the first place. The research on alignment with general preference model being indeed still in its early-stages (seminal paper (Munos, 2024), illustrations on examples and detailed motivations are in my opinion needed to convince the reader. The paper right now only provides an example of preference cycle, in the end of page 3, without making the link with the alignment objective. In addition, the numerical experiments do not further help to convince the reader regarding the interest of (robust)-NLHF, as no baseline based on the standard RLHF is included.
  • The accessibility of the papers seems somewhat limited by their jargon-heavy style, particularly from Section 3 onwards. However, this assessment should be interpreted in light of my relatively low confidence score.
Questions for authors
  • The paper could benefit from improvements in readability, with less jargon-heavy language and illustrative examples to help convey the underlying motivations. As this may require some rewriting, I am not sure to what extent it could be addressed during the rebuttal phase.
  • Regarding the experiments, would it make sense to include a baseline with RLHF (and potentially a robust version), with a reward model learned to approximate the preferences induced by the kernels?

################# Miscalleneous L.52: the notion of preference kernel should be probably defined before using it.

L.42: denote r(.) for a policy is confusing as it is associated to reward models. For 'representing the pre-alignment model from SFT' using \pi_{ref} seems more standard.

L.108 Nash-MD is undefined

L. 150: Isn't Assumption 1 the standard 'antisymmetry' property? Please use standard terminology from decision theory

L.222: Would it makes sense tp give the Gibbs variational identity? Also, it is not clear whether Eq.5 has already been provided in Gupta et al,2025.

5 comments

Thank you for the detailed comments. In the revision, we add a worked example that links general preferences and preference uncertainty to the alignment objective, rewrite Sections 2–5 in plainer language, and add RLHF and robust RLHF baselines (Appendix D.2.7). We also clarify the relationship to Gupta et al. (2025): their work already contains the pessimistic Nash objective and the elimination of the competing policy. Our contribution is the exact joint-law concave–convex formulation of that objective, with algorithms and certificates for optimizing it.

Weakness 1: Why general preferences, and why preference uncertainty?

We use general preferences because they keep pairwise information that a scalar reward model can lose. Preference uncertainty is a separate issue: the estimated comparison probabilities depend on limited data and on which evaluators are represented. The two arise together when evaluators disagree and their proportions at deployment are uncertain. We develop this motivation beyond the cycle example in Section 1 and the new Appendix A.1.

An example without a cycle. A fitted reward can select a response that a majority rejects even when there is no cycle. Suppose 55% of evaluators rank A≻B≻CA\succ B\succ C and 45% rank B≻C≻AB\succ C\succ A, and each reports its ranking on every pair with probability 0.9. Then P(A≻B)=P(A≻C)=0.54P(A\succ B)=P(A\succ C)=0.54 and P(B≻C)=0.90P(B\succ C)=0.90: A beats every alternative, and the majority relation has no cycle. With all pairs compared equally often and unrestricted response-level scores, the population Bradley–Terry (BT) fit ranks responses by their average win rates (Siththaranjan et al., 2024, Theorem 3.1): 0.68 for B, 0.54 for A and 0.28 for C. Reward maximization with a uniform reference and KL regularization therefore puts its largest probability on B, which loses to A with probability 0.54, whereas the Nash policy of the unregularized game selects A. The example is not an isolated case: maximum-likelihood fits of BT-type models need not follow the pairwise majority (Noothigattu et al., 2020; Ge et al., 2024).

What the alignment objective changes. Unregularized Nash alignment seeks a response distribution that no alternative beats in expectation; in social-choice terminology, this is a maximal lottery (Fishburn, 1984; Maura-Rivero et al., 2025). Our KL-regularized formulation also controls deviation from the reference policy, and it keeps the general pairwise kernel instead of first replacing it by a scalar reward. Which aggregation rule suits an application is a normative choice (Conitzer et al., 2024); a scalar reward makes that choice implicitly.

Cycles. Cycles are a second reason to keep the pairwise kernel. They can arise from pooling different rankings, within individual judgments that trade off several attributes (May, 1954; Tversky, 1969), and in LLM judges (Xu et al., 2025a). At language-model scale, a general preference model outperforms a BT reward model on RewardBench and represents cyclic preferences on which any BT model is at chance (Zhang et al., 2025). In our diagnostic experiment (Question 2 below), we use cyclic majority preferences to separate reward-based and Nash alignment.

Why preference uncertainty. The uncertainty has two sources. First, comparisons are noisy: agreement was about 63% between researchers and crowdworkers in Bai et al. (2022) and 72.6% among the training labelers of Ouyang et al. (2022), and optimizing a fitted model can exploit its errors (Gao et al., 2023; Gupta et al., 2025). Second, evaluators disagree: Santurkar et al. (2023) and the PRISM dataset (Kirk et al., 2024) document substantial variation across populations, and Son et al. (2025) study preference drift. A model fitted to one rater pool therefore need not describe the population encountered at deployment.

Robust NLHF addresses the two issues jointly. Its objective is the largest KL-regularized preference value that a policy can guarantee against every competing response distribution and every kernel in the prescribed ambiguity set. For evaluator mixtures, the uncertainty is in the mixture weights, while each component remains a general pairwise kernel. In our IMDb experiments we instantiate this setting with three scoring profiles, so the evidence is about fixed-candidate selection under that uncertainty family, not every population shift.

Weakness 2 & Question 1: Readability and jargon

We revised the abstract, the contribution list and the exposition of Sections 2–5. We now introduce the preference kernel and Nash-MD at first use, connect the cycle example to the alignment objective, and state the Gibbs variational identity before Eq. (5). In Section 3.1 we first introduce the joint law as a finite probability table and leave the measure-theoretic details to Appendix B.1. We also follow Theorem 3.1 with its proof idea and Proposition 4.1 with an explanation of the update, and mark all changes in blue in the revised PDF.

The construction can be read in four steps:

  1. Objective. The policy is evaluated against an adversary that chooses both a preference scenario and a competing response distribution (Eq. (4)).
  2. Joint choice. Treated as separate variables, these two choices need not give a convex inner problem. We represent them by one probability table over scenario–response pairs; each row gives the probability of a scenario and the competing policy used in it.
  3. Exactness. Charging the competitor's KL penalty within each row leaves the robust value of every policy unchanged. The resulting game is concave–convex, and its saddle gap bounds the returned policy's robust-value error (Theorem 3.1 and Eq. (9)).
  4. Optimization. The joint proximal step has a closed form. Mirror-Prox then gives an O(1/T)O(1/T) gap bound for finite scenarios and an O(log⁡T/T)O(\log T/T) bound for compact convex affine families under exact integration. Remark 4 separately bounds the error of fixed quadrature nodes.

Question 2 (and the baseline point in Weakness 1): RLHF baselines

We agree that RLHF is a necessary baseline and have added RLHF and robust RLHF on all 4,000 IMDb prompts (Table 12, Appendix D.2.7). When the nominal preferences are close to a BT model, RLHF and NLHF are nearly tied. In a diagnostic population with cyclic majority preferences, NLHF is better than RLHF on every metric. Under uncertainty, robust NLHF (EJMP) has the lowest worst-case exploitability in all three panels.

Baselines. RLHF fits a BT reward to the nominal pairwise probabilities and optimizes that reward with KL regularization. We give robust RLHF each profile's exact score, which favors it, and it maximizes the KL-regularized worst-case reward gain over πref\pi_{\rm ref} on U0.15\mathcal U_{0.15}, following the pessimistic reward objective of Fisch et al. (2024). To keep the comparison fair, we cap each reward baseline's KL divergence from πref\pi_{\rm ref}, prompt by prompt, at that of its Nash counterpart: NLHF for RLHF and EJMP for robust RLHF. All methods select among the same fixed candidates, and both Nash methods are solved to a certified gap below 10−610^{-6}.

Populations. In the main IMDb setting (panel A), a BT model fits the nominal pairwise probabilities with a mean absolute error of 0.74 percentage points, so the preference is nearly a BT model. The diagnostic population (panel B) gives the three profiles equal weight and sharpens each profile's comparisons (β\beta from 5 to 50); we fixed it from its preference structure before running any method. On 528 of its 4,000 prompts (panel C), every response loses a majority comparison to another response, so there is no Condorcet winner. Neither B nor C is a BT model.

Table R4 (Table 12 of the revision). W0W_0 and WUW_{\mathcal U} are the win probabilities (%) of the best single alternative against the policy, at the nominal mixture and at the worst mixture in U0.15\mathcal U_{0.15}; E=12−g0.15E=\tfrac12-g_{0.15} is worst-case exploitability. Lower is better in every column; bold marks the column minimum.

MethodA: W0W_0A: WUW_{\mathcal U}A: EEB: W0W_0B: WUW_{\mathcal U}B: EEC: W0W_0C: WUW_{\mathcal U}C: EE
RLHF50.1956.890.069353.3668.060.183355.8171.380.2036
NLHF50.1656.920.069550.0763.260.136850.2867.140.1618
Robust RLHF55.7465.170.142859.7575.290.244358.5275.440.2428
EJMP (ours)51.9155.640.047753.6859.940.091353.0662.950.1178

Results. RLHF and NLHF are nearly tied in panel A. In panels B and C, NLHF is better than RLHF on all three metrics. The nominal gap is larger where no Condorcet winner exists: there, the best alternative beats RLHF's response 5.53 percentage points more often than NLHF's, against 2.95 points on the remaining prompts. EJMP has the lowest worst-case values, WUW_{\mathcal U} and EE, in all three panels. Therefore, when the general preferences are considered together with uncertainty, RLHF and its robust variant can be applied, but will be outperformed by NLHF and our EJMP.

Miscellaneous comments

  • L.52: define the preference kernel. It is now defined at first use as P(y≻y′∣x)P(y\succ y'\mid x), the probability that response yy is preferred to y′y' for prompt xx.
  • L.42: use πref\pi_{\rm ref} instead of rr. The reference policy is now denoted πref\pi_{\rm ref} throughout.
  • L.108: define Nash-MD. Nash Mirror Descent (Munos et al., 2024) is now spelled out and cited at first use, and Appendix D.2.3 states its update. We also corrected the nominal IMDb comparator. The submission used an optimistic self-play variant that had not converged; we now use standard Nash-MD with a 10−610^{-6} optimality certificate (Appendix D.2.3; details in our reply to Reviewer bsMF).
  • L.150: antisymmetry. Yes. P(y≻y′)+P(y′≻y)=1P(y\succ y')+P(y'\succ y)=1 is called antisymmetry by Munos et al. (2024) and reciprocity in the literature on [0,1][0,1]-valued preference relations (Bezdek et al., 1978; De Baets et al., 2006). We now give both names.
  • L.222: Gibbs identity. It is now stated immediately before Eq. (5).
Reviewer_bsMFOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies robust Nash learning from human feedback, where an adversary chooses both a preference scenario and an opponent response distribution. The authors replace these separate choices with a joint scenario-response distribution and use a conditional KL penalty to preserve the original robust value. This gives a convex-concave formulation. They derive an explicit entropic Mirror-Prox update with an O(1/T) gap for finite scenarios. For compact convex uncertainty sets with affine kernels, a localization argument gives an O(log(T)/T) gap under exact integral access. Experiments evaluate fixed-candidate selection on PersonalLLM and IMDb.

Critical strengths
  1. The conditional KL term is the part of the formulation I find most useful. A KL penalty against a fixed joint reference would also penalize the scenario distribution and change the problem. Here, Theorem 3.1 preserves the robust value for each policy. The resulting gap then has a direct meaning: it bounds the policy's robust-value error and its distance to the optimal leader.

  2. The localization argument handles a genuine difficulty in the continuous case: a worst-case scenario can be an atom, with infinite KL from the initialization. Keeping the conditional opponent fixed avoids an extra regularization error. I also found the PersonalLLM checks in Appendix D.1 reassuring. Comparing against independently solved optima and reporting primal-dual gaps tests the optimization claim more directly than the headline IMDb metric.

Critical weaknesses
  1. It is unclear how accurately the implemented solver solves the continuous problem. Theorem 5.1 assumes exact integrals, but IMDb uses fixed 48-by-6 quadrature. I could not find a refinement study or a bound on the resulting error. The method may converge on that grid while still missing the worst case in the continuous set. The exact-access theorem is useful, but its guarantee cannot yet be attached to the implementation.

  2. I am less convinced by the IMDb results. EJMP gets 2,048 iterations, Nash-MD gets 200, and the best-response control gets 300 outer iterations (Table 6). Table 7 shows that EJMP itself benefits substantially from longer runs, so I would want to see that the baselines have also converged. The size of the gain also deserves more attention. WinFrac counts how often the preference probability is above 0.5, even if only slightly. The average probabilities are 0.552 and 0.556 with the SFT reference, and about 0.51 with the uniform reference. These numbers are reported in the appendix, but they give a more modest picture than the near-one fractions in the main text.

Questions for authors
  1. How do the continuous robust value, policy, and evaluation results change when the quadrature resolution is increased? Can you estimate the approximation error against a finer reference computation for the current low-dimensional uncertainty set?

  2. Are Nash-MD and the mixture-KL best-response control converged at the reported budgets? Please provide their objective or gap trajectories, or show that larger budgets do not materially change the comparisons.

  3. Are the three IMDb scoring profiles fully specified by reproducible formulas? Appendix D.2.2 names the profiles and gives the transformation from scores to pairwise probabilities, but I could not find the actual definitions of those scores. These definitions determine the uncertainty family being tested.

5 comments

Thank you for the careful review, which found two overlooked parts in our submission. First, we now bound and measure the quadrature error (Remark 4, Appendix D.2.6), and every EJMP policy in the revised Section 6 is certified within 10−610^{-6} of the continuous robust optimum. Second, we adopted an early stopping for submission, and the baselines in the submission did not converge yet w.r.t. their own objectives. We now compare against standard Nash-MD that converges to 10−610^{-6}-error. Against this baseline, EJMP still wins when the preference is uncertain, and its also gains in worst-case exploitability (−34.7% and −31.3%) and under evaluator shifts toward the profile under which the nominal policy is most exploitable. We also report MeanWin next to WinFrac in Section 6, and we specify the scoring profiles completely in Appendix D.2.2.

Weakness 1 & Question 1: Accuracy of quadrature in the continuous case

We agree that the exact-integration theorem did not cover the submitted implementation, and we did not justify the implementation. We have now added an a priori bound for fixed nodes (Remark 4) and a posteriori certificates for every reported policy (Appendix D.2.6).

Fixed-node bound. Quadrature replaces the integral by a weighted sum over a finite grid H⊂UρH\subset\mathcal U_\rho with strictly positive normalized weights ω\omega. The implemented EJMP then reduces to Algorithm 1 on the scenario set HH, initialized at ω⊗πref\omega\otimes\pi_{\rm ref}. Let R=log⁡(1/πref,min)R=\log(1/\pi_{\rm ref,min}), and let DH(π):=gH(π)−gU(π)≥0D_H(\pi):=g_H(\pi)-g_{\mathcal U}(\pi)\ge0 be the grid's overestimate of the worst case of π\pi. For the affine mixture family and exact node-game updates with fixed 0<η≤10 < \eta\le1, the averaged output satisfies

0 &\le V^\star_{\mathcal U}-g_{\mathcal U}(\bar\pi_T) \le \mathrm{Gap}_H(\bar\pi_T,\bar Q_T)+D_H(\bar\pi_T) \le \frac{2R+\log(1/\omega_{\min})}{\eta T}+\delta_H, \quad \delta_H := \tfrac12\sup_{w\in\mathcal U_\rho}\mathrm{dist}_1(w,\mathrm{conv}\\,H). \end{aligned}$$ Sketch of proof: 1. A joint adversary supported on $H$ is feasible for the continuous game and has the same leader best-response value $h(Q)$. Hence $\mathrm{Gap}_{\mathcal U}=\mathrm{Gap}_H+D_H$. 2. Theorem 4.2 bounds $\mathrm{Gap}_H$, with initial radius $2R+\log(1/\omega_{\min})$. 3. After the competing policy is eliminated (Eq. (5)), the payoff $\phi_\pi(w)$ is concave in $w$, with partial derivatives $\pi^\top A^{(k)}\nu_w\in[0,1]$. Mixture-weight increments sum to zero, so $\phi_\pi$ is $\tfrac12$-Lipschitz in $\ell_1$. Concavity gives $\min_{\mathrm{conv}\\,H}\phi_\pi=\min_H\phi_\pi$, hence $D_H\le\delta_H$. The grid therefore adds only the approximation term $\delta_H$, which vanishes as the grid is refined.

Empirical evaluations. For each output of our experiments, we now bound the continuous worst case, through a sandwich argument from both sides in interval arithmetic: from below by the minimum over the vertices of a polygon containing Uρ\mathcal U_\rho, which is valid by concavity, and from above by feasible scenarios. For certified bounds g‾(π)≤gU(π)\underline g(\pi)\le g_{\mathcal U}(\pi) and h(Q)≤h‾(Q)h(Q)\le\overline h(Q),

VU⋆−gU(π)≤h(Q)−gU(π)≤h‾(Q)−g‾(π).V^\star_{\mathcal U}-g_{\mathcal U}(\pi) \le h(Q)-g_{\mathcal U}(\pi) \le\overline h(Q)-\underline g(\pi).

A right-hand side of at most 10−610^{-6} certifies that π\pi is within 10−610^{-6} of the continuous robust optimum, however π\pi was computed. To see how large each error is in practice, we applied these enclosures to EJMP runs on six grids (Table R1).

Table R1. EJMP at T=2048T=2048; means over 48,000 problems (4,000 prompts, two references, six radii), in units of 10−310^{-3}. The a priori bound is averaged over the six radii. Value loss is VU⋆−gU(π)V^\star_{\mathcal U}-g_{\mathcal U}(\pi), given as a certified bracket (Table 11).

GridNodesNode-game gap GapH\mathrm{Gap}_HDHD_HA priori bound on δH\delta_HContinuous gap GapU\mathrm{Gap}_{\mathcal U}Value loss
12×2244.546.8130.9111.352.408–2.409
24×4965.542.0210.547.551.719–1.720
48×6 (submission)2886.100.804.006.891.674–1.675
72×85766.370.462.196.821.667–1.668
96×109606.510.301.406.811.665–1.666
192×122,3046.610.200.796.811.665–1.666

At the submitted 48×6 grid, DHD_H averages 0.80×10−30.80\times10^{-3} against a mean node-game gap of 6.10×10−36.10\times10^{-3}, and it is smaller than the node-game gap in nearly every problem. It never exceeds the a priori bound; its largest certified value for any grid is at most 78% of the bound. At T=2048T=2048, most of the error therefore comes from optimization, not quadrature: when we solve the 48×6 node games to a certified gap of 10−810^{-8}, the mean value loss drops from 1.674×10−31.674\times10^{-3} to 0.175×10−30.175\times10^{-3}.

The requested refinement comparisons are:

  • Robust value: from 48×6 to 192×12, the mean continuous robust value changes by less than 10−510^{-5}.
  • Policy: when we retrain on 96×10, each policy moves by at most 0.0510.051 in ℓ1\ell_1, and its robust value by less than 0.000940.00094.
  • Evaluation: under the submission's evaluation protocol (its comparator and stress tests), MeanWin changes by at most 0.020.02 percentage points from 48×6 to 96×10.

Main results. In the revised Section 6, we run EJMP on boundary nodes of Uρ\mathcal U_\rho (512, then 2,048) and stop at the first checkpoint whose certificate above is at most 10−610^{-6} (Appendix D.2.3). The main results therefore do not depend on a fixed grid, and the continuous-set guarantee applies to every reported policy.

Weakness 2 & Question 2: Convergence of the baselines

Thank you for pointing this out. We did not mention how the training steps are chosen in our submission. We have included this study in our revision at Figure 3 (Appendix D.2.3): we did an early stopping test for the Nash-MD baseline, computing the robust value of the output policies under different checkpoints, and found that early stopping can enhance its robustness against preference uncertainty, thus we use the baseline at 200 steps to compare the robustness.

However, the baselines have not yet converged on their own objectives due to early stopping. The near-one WinFrac values and mean preference probabilities in the submission came from two random stress tests against the unconverged comparator. The revised comparisons in Section 6 and Appendix D.2.5 now run both methods to a certified accuracy of 10−610^{-6} on their own objectives, and the following Table R2 provides an overview of the results. Against standard Nash-MD, the EJMP policies are below 50% in both MeanWin and WinFrac when dd is small (so test preferences are similar to the nominal one). This is as expected since Nash-MD is optimized for the nominal preference and our EJMP is more conservative here. However, EJMP lowers the worst-case exploitability 12−g0.15\tfrac12-g_{0.15}, the largest regularized advantage that any competitor obtains under the least favorable mixture in U0.15\mathcal U_{0.15}, by 34.7% (SFT-derived reference) and 31.3% (uniform reference). For head-to-head comparisons, EJMP also beats the baseline under preference shift, which still verifies our theory on the enhanced robustness. These results thus still validate our theory and claims, that our EJMP can enhance robustness more than the baselines.

Table R2. Certified EJMP (ρtrain=0.15\rho_{\rm train}=0.15) against certified Nash-MD with the same reference, all 4,000 prompts. Entries are percentages; 50% is a tie.

ReferenceMetricd=0.10d=0.100.200.300.400.50
SFT-derivedWinFrac29.8852.5060.9464.5966.47
MeanWin49.5050.0950.5350.9051.22
UniformWinFrac42.6370.4380.0784.1086.14
MeanWin49.9750.7051.2651.7252.12

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.