acceptodds
Under review as a conference paper at ICLR 2027

Risk-Sensitive Preference Games: Learning from Strategic Interaction & Uncertainty

Abstract

A growing line of work reframes preference-based fine-tuning of large language models game-theoretically: Nash Learning from Human Feedback (NLHF) recasts the problem as a zero-sum game over policies. However, optimization is over expected pairwise payoffs, thereby conflating policies with similar win rates but different tail behavior. As such, these methods are agnostic to where in the data distribution they succeed or fail: strong average performance can mask systematic failure across prompts, annotators, or safety-critical strata. Risk-sensitive preference games, introduced here, address this gap: players optimize convex risk measures of their preference loss, exploiting structure in preference uncertainty. We introduce risk-sensitive preference games, in which players optimize convex risk measures of their preference loss, exploiting structure in preference uncertainty. While risk transforms generally break zero-sum structure at the certainty-equivalent level, we show that translation invariance ensures constant-sum structure is preserved at the level of risk-adjusted payoff operators. This is sufficient to retain monotonicity, yielding fast convergence of sample-efficient self-play methods. The harder challenge is robustness. We establish algorithmic stability and offline sample complexity bounds that scale with risk, requiring simultaneous control of structural bias from nonlinear risk transformations, statistical bias in risk estimation, and concentration tailored to the risk-sensitive setting. To address statistical bias, we introduce a hierarchical game formulation and a two-timescale extragradient algorithm with bias correction that converges to the Stackelberg equilibrium and is especially effective in low-sample regimes. Empirically, risk-adjusted policies are robust across data strata, stable across risk choices, and match or exceed risk-neutral performance thereby achieving robustness without a performance tax.

Primary area:
optimization
Keywords:
game theory, nash learning from human feedback, reinforcement learning from human feedback, preference learning, preference optimization, risk-sensitive optimization, stochastic optimization, distributional robustness, self-play, algorithmic stability, sample complexity, convex risk measures

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yXo4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets the problem that standard preference-learning techniques are optimising for point-wise estimates of the population's preference distribution neglecting the tails.

The main contributions of the paper are

  1. The authors propose Risk-Sensitive Preference Games (RSPG) as a modelling framework to address this shortcoming of the previous works. Instead of optimising for the expected preference signal, this framework focuses convex risk measures.
  2. The authors address the two main limitations of optimising risk measures instead of expected payoff, namely, non-convexity and biased sample estimation. They propose solutions for both and efficient learning algorithms that converge to the equilibrium in RSPG.
  3. Their empirical results show pareto-dominance over previous baselines.
Critical strengths
  1. Clean, unifying formalization. The operator-level constant-sum observation lets the single-policy self-play machinery of NLHF carry over unchanged. λ‾R<β\overline{\lambda}_R < \beta gives an explicit, interpretable condition on how much risk distortion the KL regularization can absorb. This matters because it makes risk sensitivity a principled extension of NLHF (R=ER = E recovers it exactly) rather than an ad-hoc reweighting.
  2. Bias identified as the binding constraint, and corrected for free. The paper identifies estimator bias, rather than variance, as what limits training at the small rollout budgets used for LLMs, and corrects it with no additional samples. This is backed by direct measurement showing that bias dominates up to K ≈ 150, and jackknife tracking removes 56–76% of measured bias at K = 4.
Critical weaknesses
  1. The headline claims (“Pareto-dominance”, “robustness without a performance tax”) rest on 100–136 prompts per stratum, with confidence intervals bootstrapped over prompts only. The controlled study uses a single training seed, and I could not find seed variation for the main comparisons. Several reported gains are point estimates, e.g., higher combined win-rate against “six of eight opponents.”
  2. The experiment setup is ambiguous. The main text cites Llama 2 as the base model while Appendix L states it is Gemma-2 2B-IT. Both of which are outdated and do not provide sufficient signal to evaluate the approach's scalability. Similar contradiction is stated for the pairwise judge as the main text says it is an (unspecified) LLM judge while the appendix says it is a pairwise finetuned judge.
Questions for authors
  1. Theorem 1 rely on the assumption that λ‾R<β\overline{\lambda}_R < \beta. Can you estimate λ‾R\overline{\lambda}_R for the setting used in Section 4? How does it align with theoretical results?
  2. Could the authors also elaborate on why Appendix B states that the paper considers the monotone regime with λ‾R<β2\overline{\lambda}_R < \frac{\beta}{2}? Is it only a typo or it does not refer to Theorem 1?
  3. Could the authors provide evaluations on other datasets and judges? Currently the experimental section is the weakest point of the paper and it provides not enough evidence that the method generalizes and scales.
  4. The related works and experimental comparison baselines are missing approaches connected to Stackelberg Learning from Human Feedback (SLHF) [1,2] which provide a different risk tolerance than NLHF. Could the authors elaborate on how their method differ and include at least one of them in Section 4?

[1]: Self-Improving Robust Preference Optimization, Choi et al., ICLR (2025) [2]: Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Game, Pasztor et al., ICLR (2026)

7 comments
AuthorsOpenReview ↗

Opening & Q1, Q2 Response

Thank you for the thoughtful review. We address the specific questions first, and then we address the weaknesses.

Q1 Estimating Risk Distortion

The quantity λˉR\bar{\lambda}_R in Theorem 2 is a global worst-case curvature quantity for the finite preference-game operator. In the setting of the theorem, it can be verified directly. In the revision, we added Appendix F.6 (colored blue{\color{darkblue}\text{blue}}), which derives explicit sufficient regimes for entropic risk and CVaR in terms of properties of the finite preference matrix, together with numerical examples comparing the resulting certificates, numerically evaluated finite-game curvature, and observed convergence behavior. These bounds are corollaries of results that were already present in Appendix K. You can also view the examples here: anon-website

We do not estimate λˉR\bar{\lambda}_R for the LLM experiments in Section 4, because doing so in a way that verifies the theorem would require certifying the corresponding global property of the neural self-play operator over the full sequence-level policy space, with an effectively enormous response space. A Jacobian estimate along the observed training trajectory, or on a sampled set of responses, would characterize only restricted directions or sampled subgames and would not establish the global condition λˉR<β\bar{\lambda}_R<\beta. Moreover, the relationship between such local diagnostics and the global convergence property required by the theorem is itself nontrivial and is not established.

This distinction is not specific to risk-sensitive preference learning. Existing risk-neutral NLHF methods make the same separation between their convergence theory and neural implementations. For example, Nash-MD establishes its convergence guarantees for the tabular game and separately introduces deep-learning implementations; EGPO likewise explicitly distinguishes its tabular setting with rigorous guarantees from its neural experiments. Neither work computes a neural self-play Jacobian/Hessian spectrum for its LLM runs or verifies that the neural implementation satisfies an analogous global monotonicity certificate. Thus, introducing risk has not created a new empirical verification gap: certifying global monotonicity after composition with a large nonlinear neural parameterization is a broader theory-to-practice problem for preference-game methods.

Indeed, even substantially weaker local neural-curvature diagnostics constitute a separate empirical problem. The GAN literature provides a useful precedent: dedicated empirical studies developed local diagnostics based on game Jacobians, player Hessians, and stability spectra, using matrix-free derivative products and iterative eigensolvers. Such diagnostics can illuminate local training dynamics, but they still do not certify a global condition of the form required here. Developing analogous diagnostics for LLM self-play would therefore be an interesting separate direction, rather than an empirical calculation needed for the present theorem.

What can be verified is the condition in the setting in which the theorem is stated. Appendix F.6 gives explicit, verifiable finite-game regimes for entropic risk and CVaR, uniformly over opponent policies, and numerical examples comparing the analytical certificates with numerical curvature and convergence behavior. We also explicitly verify the condition for the submitted finite Bradley--Terry experiment. These results show that the assumption is non-vacuous and directly checkable in the finite-game setting of the theory.

Accordingly, the theoretical and empirical results have complementary roles. Appendix F.6 gives explicit regimes in which the theorem is guaranteed to apply and illustrates how the sufficient condition varies with risk and regularization. Section 4 evaluates the proposed risk-sensitive algorithms at LLM scale and demonstrates stable optimization and improved robustness empirically, but we do not interpret this empirical stability as a verification of the theorem's global sufficient condition. It is the neural-scale evaluation of the proposed algorithms, just as neural experiments in prior NLHF work are not certificates of their tabular convergence assumptions.

Q2 Typo

Yes β/2\beta/2 appearing in the future work is a typo. Thanks for catching this; we have fixed it in the revision.

AuthorsOpenReview ↗

Q3 Further Evals and Empirical Scope

Q3 Further Evaluations and Empirical Scope

Broader evaluation across additional datasets, base models, and evaluators would certainly be valuable. However, the experimental setting in this paper was chosen deliberately to enable controlled, apples-to-apples comparison with the closest state-of-the-art NLHF baselines, rather than to constitute a broad empirical benchmark. In particular, we use the same data/base model setting as the prior methods against which we compare---Nash-MD, EGPO with IPO---so that differences can be attributed to the introduction of risk sensitivity and bias correction rather than to changes in the underlying training environment.

Within that matched setting, the experiments are designed to isolate the main algorithmic questions raised by the paper. We compare risk-sensitive and risk-neutral objectives at matched opponent-sampling budgets; study multiple forms and levels of risk; compare corrected and uncorrected finite-KK risk estimation; compare OMD and extragradient dynamics; evaluate across held-out safety strata and diverse opponents; and test the two-timescale correction directly through measured bias, self-play risk, and downstream performance. We also evaluate preference and safety with distinct evaluation models, rather than relying on the training objective alone. Thus the empirical section is intended to test the mechanisms introduced by the theory and algorithms, not simply to report one end-to-end number.

We do not claim that these experiments establish universal empirical generalization across model families or datasets. Our principal contributions are algorithmic and theoretical: the RSPG formulation, its equilibrium and convergence analysis, the finite-sample bias characterization, and the novel two-timescale bias correction. The LLM experiments provide supporting evidence that these mechanisms remain meaningful and effective at a practically relevant neural scale.

A substantially broader empirical study would require retraining the full set of matched baselines, risk-sensitive variants, risk sweeps, and bias-corrected variants across multiple datasets and base-model families. This is particularly costly here because each training update requires sampling multiple opponent responses, and extragradient approximately doubles the policy-update cost. Our available compute is limited to L40S-class GPUs; even the present study required restricting model size and generation length to keep the matched experiments feasible. Scaling to substantially larger models would require reducing batch size, opponent sample count, or generation length—precisely the quantities whose effects we study—or access to substantially greater compute.

For this reason, we prioritized a controlled setting that matches the most relevant prior work and allows us to perform the mechanism-level comparisons required by the paper. We view multi-dataset and multi-model scaling as an important empirical extension, but carrying it out carefully would constitute a substantially broader experimental study rather than a missing component of the theoretical and algorithmic contribution presented here.

AuthorsOpenReview ↗

Q4 Relationship to SLHF and SRPO

Q4 Relationship to SLHF and SRPO

We agree that SLHF and SRPO are relevant related work, and we have added discussion of these approaches to the related-work section in the revision (Appendix A in blue). However, we do not agree with the characterization that they provide a different risk tolerance than NLHF in the sense studied in this paper. Their robustness arises from changing the preference-optimization game or information structure, rather than from replacing expected preference payoffs with a risk functional.

SLHF changes the underlying game and solution concept. NLHF is a simultaneous-move game: each player chooses a policy without observing the opponent's realized response. SLHF instead introduces a sequential Leader--Follower game in which the Leader first produces a response and the Follower observes that realized response before generating a conditional refinement. Its objective nevertheless retains the expected pairwise preference probability with KL regularization. Thus SLHF's robustness is induced by the Follower's informational advantage and the Leader's optimization against a response-conditioned best response. This is a different Stackelberg solution concept, not a risk-sensitive extension of the NLHF payoff.

Our construction changes a different object. RSPG retains the NLHF self-play formulation but replaces the expected opponent-dependent payoff by a convex risk functional. Setting that risk functional to expectation recovers NLHF exactly. The risk parameter therefore controls sensitivity to the distribution of preference outcomes themselves---for example, lower tails or variability---rather than changing which player moves first or what information one player observes. The framework also permits risk to act over the opponent distribution, prompt distribution, within-prompt heterogeneity such as annotators or labels, and the response-pair sampler. This distinction is important in our experiments, where we measure improvements in the tails of prompt and severity distributions even against fixed opponents.

SRPO is related for a similar but distinct reason. SRPO introduces a conditional self-improvement policy that maps an initial completion yy to an improved completion y′y', and jointly optimizes this policy with a generative policy through a min--max objective. The pairwise preference term in that objective is again an expectation of the preference probability, with KL regularization; it is not replaced by a convex risk functional. SRPO's notion of robustness is instead that the resulting generative solution is independent of the behavior policy used to generate the preference data, together with the ability to improve responses through conditional self-revision. SRPO is also formulated as an offline preference-optimization method, whereas the Section 4 experiments study online self-play.

For these reasons, SLHF and SRPO are useful conceptual comparisons but are not matched experimental baselines for the contribution tested in Section 4. Our risk-neutral NLHF baselines---Nash-MD, OMD/online IPO, and EGPO---keep the game, opponent construction, and solution concept fixed while replacing our nonlinear risk aggregation by expectation. Consequently, the RSPG-versus-NLHF comparison directly isolates the effect of introducing risk sensitivity. Likewise, our corrected-versus-uncorrected experiments isolate the effect of correcting the finite-sample bias created by estimating that risk.

Adding SLHF or SRPO would simultaneously change the underlying game formulation, conditional response structure, solution concept, and, for SRPO, the online/offline training setup. Such a comparison could be useful in a broader empirical benchmark of different preference-optimization paradigms, but it would not isolate the algorithmic contribution studied here. We therefore include SLHF and SRPO in the related-work discussion while retaining matched NLHF methods as the appropriate Section 4 baselines.

Reviewer_PEEiOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends Nash Learning from Human Feedback by replacing each response's expected payoff against an opponent policy with a risk-adjusted value. The resulting preference game targets a risk-adjusted quantal response equilibrium through an IPO objective. The authors show that a shared translation-equivariant risk functional preserves an operator identity that lifts symmetric self-play fixed points to Nash equilibria. They analyze convergence under monotonicity assumptions, introduce two-timescale tracking of finite-batch risk-estimation bias, and study equilibrium sensitivity and statistical estimation. For a fixed finite response set, sufficient pairwise sampling coverage, and population regularity, the offline analysis obtains a KL estimation rate of order 1/n. Experiments include a synthetic preference game and Gemma-2 2B-IT LoRA fine-tuning on PKU-SafeRLHF, reporting improvements in several safety and lower-tail win-rate comparisons under automated evaluation.

Critical strengths
  • S1. The formulation connects risk-sensitive preference objectives to tractable equilibrium and statistical questions. Equations (3)–(7) incorporate opponent uncertainty into the learning target. Lemma 1 and Theorem 1 show why symmetric self-play fixed points still yield Nash equilibria even though nonlinear risk breaks constant-sum utilities. Theorem 16 obtains a 1/n offline KL rate with explicit coverage and conditioning dependence by estimating preference probabilities before applying risk. Appendix D further clarifies which placements of risk change the equilibrium and which modify the fitting dynamics. These results give the framework value beyond a particular optimizer or benchmark comparison and provide the main basis for my positive recommendation.
  • S2. The analysis of finite-batch bias is supported by useful diagnostics and promising downstream results. Appendix I.1 constructs bias corrections from the same samples as the operator estimate and separates residual bias, tracking lag, and variance. For the operator-averaging variant, Corollary 10 gives an O(T−1+m−4)O(T^{-1}+m^{-4}) squared-distance bound for smooth risks under its assumptions. At K=4, jackknife correction reduces absolute bias relative to the finite reference pool from 0.109 to 0.047; its variance is 1.86 times that of the uncorrected estimator (Appendix L.8.1, Table 1). Figure 2 reports cross-stratum averages of 88.5% for the mean combined win-rate and 60.9% for its lower-tail CVaR0.25\mathrm{CVaR}_{0.25} with TT-OMD delta, compared with 85.0% and 51.1% for uncorrected entropic OMD. These are encouraging point estimates, complemented by the training curves in Appendix L.8.2. The estimator study establishes the relevance of the bias problem; the attribution of the downstream gains requires the clarification in W2.
Critical weaknesses
  • W1. The condition in the main convergence statement does not match the geometry used in its proof. Section 2.2 defines the payoff Jacobian with respect to policy probabilities, JRπ=∂PR/∂πJ_R^\pi=\partial P_R/\partial\pi. Appendix F.4 instead proves Theorem 6 using the logit Jacobian JRθ=JRπ[diag⁡(π)−ππ⊤]J_R^\theta=J_R^\pi[\operatorname{diag}(\pi)-\pi\pi^\top], and Appendix H assumes strong monotonicity in logits. Appendix F.5 explicitly notes that a policy-space curvature bound alone does not bound the logit-space quantity.

    For example, consider the risk-neutral case R=ER=\mathbb{E} with

    $ P=[[0.5, 1, 0], [0, 0.5, 1], [1, 0, 0.5]], \pi = (0.8,0.1,0.1), \beta = 0.01. $

    The policy-space symmetric tangent curvature is zero, so the policy-space condition holds. At this policy, however, the symmetric logit Jacobian of FRF_R has an eigenvalue of approximately −0.05062-0.05062 on 1⊥\mathbf{1}^\perp. This is a counterexample to the implication between the two monotonicity conditions. Appendix F.4 already supplies a separate sufficient condition, so a revision can retain that result while correcting the main statement. The authors should distinguish the two moduli and state which condition supports each convergence or sensitivity claim. The self-play identity and the offline result under its own assumptions remain separate contributions.

  • W2. The LLM implementation needs an explicit connection between its scalar tracker and the theoretical bias target. Algorithm 1 tracks the vector bias bm(θ)b_m(\theta) and applies the correction inside the IPO preconditioner (Appendix I.3, Eq. (157)). Appendix L.8.2 describes the practical tracker as a “single running scalar” averaged over batches and prompts (PDF p. 128, lines 6873–6876). If this scalar is applied as the same additive correction to both action values, it cancels from the difference in Eq. (7); the corresponding preconditioner satisfies Σ(ρ)1=0\Sigma(\rho)\mathbf{1}=0. The manuscript does not make clear whether the implementation has this form or applies the scalar differently.

    Figures 33–35 provide useful evidence about the scalar risk-estimation diagnostic. Relating that diagnostic to the bias of the actual pairwise target or policy update is necessary to support the proposed explanation of the downstream gains. An explicit update formula or concise pseudocode, together with a diagnostic for that target on the existing response pools, would address this uncertainty. The reported comparisons remain empirical evidence for the implemented training procedures; the current description leaves their correspondence to the analyzed correction unclear.

Questions for authors
  • Q1 (W1): Which curvature quantity and metric are required by Theorem 2 and Propositions 2–3? Please distinguish the policy-space and logit-space moduli and reconcile the main statements with Appendices F.4–F.5, H, J, and K. This would determine the scope of the guarantees used in the assessment.
  • Q2 (W2): What is the exact corrected IPO residual or update in the LLM experiments, and where does the scalar tracker enter it? Please identify the bias quantity being estimated and relate the reported diagnostics to the bias of that actual target. A check using the existing response pools would help establish whether the observed gains support the proposed correction mechanism.
  • Q3: Appendix L.8.2 selects step 4250 using the best corrected model's combined win-rate on the Random stratum (PDF p. 125). Are the selection prompts disjoint from the reported Random evaluation prompts? If they overlap, please provide a key paired comparison on unused prompts at a predetermined endpoint or under a validation-only selection rule. This affects interpretation of the gains because the prompt-bootstrap intervals do not account for checkpoint selection or the single-seed training variability acknowledged in the appendix.
9 comments
AuthorsOpenReview ↗

Opening & Q1 (W1) response

Opening

We thank the reviewer for recognizing our key contributions, and providing thoughtful questions. Below we address each of your questions (and aligned weaknesses).

Q1 (W1) Result Geometry

Our analysis establishes sufficient conditions separately in policy space and in parameter space; it does not infer monotonicity in one geometry from monotonicity in the other.

Specifically, Theorem 5 in Appendix F.3 proves strong monotonicity in the policy-space pairing under the policy-space curvature condition. The same subsection then states last-iterate linear convergence of deterministic extragradient applied to the policy-space variational inequality, under the corresponding regularity and step-size conditions. Thus, the policy-space monotonicity and convergence result has its own justification; it does not rely on the parameter-space argument in F.4.

Appendix F.4 separately defines the logit Jacobian JRθ(π)=JRπ(π)[diag⁡(π)−ππ⊤]J_R^\theta(\pi)=J_R^\pi(\pi)\bigl[\operatorname{diag}(\pi)-\pi\pi^\top\bigr] and proves the parameter-space result in Theorem 6 under its own condition λˉRθ<β\bar\lambda_R^\theta<\beta. The distinction is explicit in F.4 and at the beginning of F.5, which states that a policy-space curvature bound alone does not provide the parameter-space bound. Thus, an example separating the two monotonicity conditions does not contradict either result under its respective assumptions.

Concretely, the downstream results use these two moduli separately. Theorem 2 is the policy-space statement, with policy-space modulus μRπ\mu_R^\pi, and is supported by Theorem 5 and the policy-space VI convergence result in Appendix F.3. Proposition 2 and Appendices H--I, which analyze stochastic and two-timescale updates in logit coordinates, require the parameter-space modulus μRθ>0\mu_R^\theta>0, for which λˉRθ<β\bar\lambda_R^\theta<\beta from Theorem 6 is a sufficient condition. Proposition 3 and Appendix J likewise use the parameter-space modulus through the logit-space stability result of Theorem 13. In contrast, Theorem 4 and Appendix K use their own policy-space population monotonicity condition, stated directly in policy coordinates, with modulus μRπ\mu_R^\pi. We will make these dependencies explicit so that the two curvature quantities and their associated guarantees are not conflated.

We acknowledge that reusing JRJ_R and λˉR\bar\lambda_R across the two geometries in places in the appendix (before we explicitly distinguish them in F.5), together with the main-text cross-reference, obscures this distinction. Honestly the space constraints in the main contributed to notation overload; with the extra camera ready page we will have room to make this distinction very clear. We will distinguish JRπ,λˉRπJ_R^\pi,\bar\lambda_R^\pi from JRθ,λˉRθJ_R^\theta,\bar\lambda_R^\theta throughout, explicitly label Theorem 2 as a policy-space statement, and direct its supporting reference to Theorem 5 and the convergence consequence in F.3. We will describe Theorem 6 (Appendix F.4) as the separate parameter-space formulation, whose condition is used in the subsequent logit-space analysis. Both results retain their respective assumptions and guarantees; no implication between the two curvature conditions is asserted or needed.

AuthorsOpenReview ↗

Q2: Connection between payoff-space diagnostic and bias correction (Part I)

We thank the reviewer for pointing out an ambiguity in the presentation in Appendix L.8 (will be L.9 in the revision). The key distinction is between the payoff-space diagnostic reported in Figure 40/Table 6 and the vector-valued correction actually used in the LLM update. Below we explain in two comments.

Figure 40 and Table 6 intentionally operate in payoff space. They provide a tractable surrogate for the finite-KK bias--variance regime before mapping that error into the high-dimensional LoRA parameter space. The scalar bˉt\bar b_t used in that diagnostic is therefore a payoff-space analogue/summary of the tracker; it is not the quantity subtracted from the LLM gradient. The actual training-time tracker ξt\xi_t is vector-valued, with the same structure as the trainable LoRA parameters.

We make that connection explicit below.

For prompt xix_i, completion a∈{yi,yi′}a\in\lbrace y_i,y_i'\rbrace, and KK sampled opponents, let R^ia\widehat R_{ia} denote the finite-KK estimate of the risk-adjusted payoff (PRμ)ia(\mathcal P_R\mu)_{ia}. Following the bias convention of Appendix I, define bia:=(PRμ)ia−E[R^ia]b_{ia}:=(\mathcal P_R\mu)_{ia}-\mathbb E[\widehat R_{ia}], so that bb is the corresponding component of the operator bias bm=E[F^m]−FRb_m=\mathbb E[\widehat F_m]-F_R. Because FRF_R contains −PR-\mathcal P_R, this has the opposite sign from the usual payoff-estimation bias.

For entropic risk, we have q^ia=1K∑k=1Ke−τPiak\widehat q_{ia}=\frac1K\sum_{k=1}^K e^{-\tau P_{iak}} and R^ia=−1τlog⁡q^ia\widehat R_{ia}=-\frac1\tau\log \widehat q_{ia}, and the delta-method estimate of this operator-bias component is

b^iaΔ=−Var^k[e−τPiak]2τK q^ia 2.\widehat b_{ia}^{\Delta}=-\frac{\widehat{\text{Var}}_k[e^{-\tau P_{iak}}]}{2\tau K\,\widehat q_{ia}^{\,2}}.

The jackknife produces the analogous leave-one-out estimate. Importantly, these estimates are computed separately for each completion; the implementation does not apply one common scalar correction to both responses.

Define the IPO log-ratio margin

\log\frac{\pi_\phi(y_i\mid x_i)} {\pi_{\rm ref}(y_i\mid x_i)}- \log\frac{\pi_\phi(y_i'\mid x_i)} {\pi_{\rm ref}(y_i'\mid x_i)},$$ where $\phi$ denotes the trainable LoRA parameters. The finite-$K$ IPO residual is \$r_i(\phi)= s_i(\phi)-(\widehat R_{iy_i}-\widehat R_{iy_i'})/\beta\$. If the payoff bias were corrected instantaneously, the residual would be $$r_i^{\rm deb}(\phi)=s_i(\phi)-\frac{ [\widehat R_{iy_i}+\widehat b_{iy_i}]- [\widehat R_{iy_i'}+\widehat b_{iy_i'}] }{\beta}.$$ Writing $\Delta\widehat b_i=\widehat b_{iy_i}-\widehat b_{iy_i'}$, we have $r_i^{\rm deb}=r_i-\frac{\Delta\widehat b_i}{\beta}$. Since the risk targets are detached when differentiating the policy loss, the corresponding estimate of the bias in the LoRA gradient is $$\widehat g_t^{\,\mathrm{bias}}= \frac{2}{\beta B} \sum_{i=1}^{B} \Delta\widehat b_i\, \nabla_\phi s_i(\phi_t).$$ Thus each prompt contributes a **scalar pairwise payoff-bias difference** $\Delta\widehat b_i$, multiplied by that prompt's LoRA-gradient direction $\nabla_\phi s_i$. Averaging these products gives a vector with the same dimensions as the trainable LoRA parameters. The two-timescale implementation tracks this vector: \$\xi_t= (1-\gamma)\xi_{t-1}+\gamma \widehat g_t^{\,\mathrm{bias}}\$, and applies $g_t^{\rm applied}=g_t^{\rm IPO}-\xi_{t-1}$. Equivalently, $\widehat g_t^{\,\mathrm{bias}}$ is obtained by automatic differentiation of $$\frac{2}{\beta B} \sum_i \operatorname{sg}(\Delta\widehat b_i)\, s_i(\phi).$$ Thus, the LLM implementation tracks the **parameter-space pushforward of the response-specific payoff-bias correction**. In tabular logit coordinates, $\nabla_\theta s_i=e_{y_i}-e_{y_i'}$, so the same construction reduces to $\widehat g^{\,\mathrm{bias}}=\frac{2}{\beta}\Sigma(\rho)\widehat b_m$. Under the LoRA parameterization, automatic differentiation applies the corresponding chain-rule map into the trainable parameter space. This is the connection to the preconditioned correction analyzed in Algorithm 1. This resolves the reviewer's cancellation concern. If the same scalar correction were added to every action value, it would indeed vanish because $\Sigma(\rho)\mathbf 1=0$. **That is not what our implementation does.** The two completions receive separately estimated payoff-bias corrections, and the IPO update depends on their difference $\widehat b_{iy_i}-\widehat b_{iy_i'}$. Any common component correctly cancels, because a common shift cannot affect a pairwise IPO target; response-specific finite-$K$ bias remains and produces a nonzero gradient correction.
AuthorsOpenReview ↗

Q2: Connection between payoff-space diagnostic and bias correction (Part II)

Next we comment on the relation to figure 40 and table 6 (of the revision, Fig 35 and table 2 of the original)

Relation to Figure 40 and Table 6

Figure 40 and Table 6 deliberately stop one step earlier in this pipeline. They measure finite-KK bias and variance in the scalar payoff estimate, rather than attempting to estimate and visualize a very high-dimensional LoRA bias vector:

\longrightarrow \text{per-completion payoff bias}\longrightarrow \Delta\widehat b_i\,\nabla_\phi s_i\longrightarrow \widehat g_t^{\,\mathrm{bias}}\longrightarrow \xi_t.$$ For OMD, the payoff-space diagnostic uses the finite-$K$ plug-in risk estimate $\widehat\rho_K$. For the two-timescale methods, the diagnostic reports the residual payoff-space bias after accounting for the tracker. These scalar quantities are used only to diagnose the finite-$K$ bias--variance regime; they should not be confused with the vector-valued LoRA correction used in training. The payoff-space calculation isolates the finite-$K$ estimation error **upstream of** the parameter-space weighting. Each prompt's pairwise payoff-bias difference is subsequently mapped into parameter space by multiplication with $\nabla_\phi s_i$. Because the gradient directions vary across prompts and common-mode bias cancels between paired completions, Figure 40/Table 6 should be interpreted as a low-dimensional surrogate for the corresponding bias--variance regime of the policy update, rather than as a direct measurement of the LoRA-gradient bias. The existing Appendix L experiments already provide complementary empirical checks of this mechanism; the ambiguity is in how we described their relationship to the implemented update, rather than in the absence of a diagnostic. The large response-pool experiments directly measure the finite-$K$ payoff-estimation bias and its reduction by the delta and jackknife corrections; Figure 38 compares that measured bias with the amount recovered by the tracker across training checkpoints; Figure 40/Table 6 characterize the resulting payoff-space bias--variance regime; and Figure 41 checks the predicted downstream effect through the implicit-reward margin, KL drift, and self-play entropic risk. We therefore do not believe an additional experiment is necessary to establish the correspondence between the implemented correction and the analyzed update. The update formula above gives the exact mapping from the response-specific payoff-bias estimates to the vector-valued LoRA correction, while the existing diagnostics measure the finite-$K$ bias mechanism upstream and its consequences downstream. Hopefully the distinction/relationship is now clear. We will revise Appendix L to make this distinction explicit, in particular clarifying that $\bar b_t$ is a **scalar payoff-space diagnostic/summary** and not the vector-valued training-time tracker $\xi_t$. For completeness, pseudocode for the implemented computation is: ```python # phi: trainable LoRA parameters # xi: vector bias tracker with the same structure as phi for i in batch: for a in (y[i], y_prime[i]): # K judge payoffs against sampled opponents p = judge_scores(x[i], a, opponents[i]) # finite-K entropic payoff estimate w = exp(-tau * p) q = mean(w) R_hat[i, a] = -(1.0 / tau) * log(q) # estimate of operator bias: # b = R_true - E[R_hat] b_hat[i, a] = ( -var(w, ddof=1) / (2 * tau * K * q**2) ) # pairwise payoff-bias difference delta_b[i] = stopgrad( b_hat[i, y[i]] - b_hat[i, y_prime[i]] ) # IPO log-ratio margin s[i] = ( log(pi_phi(y[i] | x[i]) / pi_ref(y[i] | x[i])) - log( pi_phi(y_prime[i] | x[i]) / pi_ref(y_prime[i] | x[i]) ) ) residual[i] = ( s[i] - stopgrad( R_hat[i, y[i]] - R_hat[i, y_prime[i]] ) / beta ) # ordinary finite-K IPO gradient g_ipo = grad_phi(mean(residual**2)) # vector-valued estimate of its finite-K bias bias_surrogate = (2.0 / beta) * mean(delta_b * s) g_bias_hat = grad_phi(bias_surrogate) # slow policy update uses the current vector tracker g_applied = g_ipo - xi optimizer_step(phi, g_applied) # fast-timescale vector bias tracker xi = (1 - gamma) * xi + gamma * g_bias_hat ```
Reviewer_qS4wOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper considers the extension of Nash Learning from Human Feedback (NLHF) framework, where risk-sensitive preference game (RSPG) is introduced, replacing the expectation operator in NLHF with a class of “risk adjusted payoff functional”, in order to fill the gap of preference uncertainty. It provides structural results for symmetric self-play, convergence guarantees under monotonicity conditions, two-timescale bias correction, and offline statistical guarantees. Experiment results are provided to investigate whether risk-sensitive training improves robustness without decrement on the performance.

Critical strengths

The paper provides an elegant generalization of the NLHF framework and provides solid results on self-play algorithm with Nash equilibrium, convergence guarantees as well as bias correction in low-sample settings. The risk adjusted payoff functional covers a board scope of settings, which might inspire subsequent research.

Critical weaknesses
  1. Limited methodological novelty. The framework offers an interesting risk-sensitive extension of NLHF, but its main technical ingredients largely build on established risk measures, monotone operator theory, and two-timescale stochastic approximation. The contribution appears to lie primarily in their integration and problem-specific analysis rather than fundamentally new methodology.

  2. Limited empirical support for the strongest claims. While two-timescale bias correction shows encouraging improvements, risk adjustment alone does not consistently outperform risk-neutral baselines (e.g., OMD-Ent vs. OMD in Figure 2). Thus, the claimed empirical Pareto-dominance appears stronger than what the results show.

Questions for authors
  1. Applicability of Theorem 2. For which common risk functionals and parameter regimes does the condition (\bar{\lambda}_R<\beta) hold? In particular, can the authors verify it for the entropic risk settings used in the experiments and clarify the treatment of non-smooth risks such as CVaR? (I currently lean toward Weak Reject (2). I would be open to reconsidering my rating toward Weak Accept (3) if the authors could provide convincing, nontrivial and broadly-used examples of risk functionals for which Theorem 2 applies, and clarify how its assumptions can be verified in practically relevant settings.)

  2. Experimental validation. Could the authors provide additional matched ablations to disentangle the effects of risk adjustment and two-timescale bias correction, and clarify how consistently the improvements hold across evaluation settings?

10 comments
AuthorsOpenReview ↗

Opening & Q1 Theorem 2 Bounds (part I)

Opening

We thank the reviewer for recognizing our key contributions, and providing thoughtful questions. Below we address each of the questions, and then we address the weaknesses.

Q1 Applicability of Theorem 2

Yes. We provide explicit, nontrivial sufficient regimes for two widely used risk functionals, entropic risk and CVaR, and show how the corresponding conditions can be verified directly from a finite preference game. We have added a new section in the appendix (Appendix F.6, in {\color{darkblue}{\text{blue}}) and supporting numerical examples whose plots and description can also be found linked here for your convenience. In particular, for entropic risk we obtain closed-form ranges in both the regularization strength and risk parameter, and for CVaR we obtain analogous ranges without requiring differentiability or smoothing. These results are developed in the new Appendix F.6, building on the entropic and CVaR analysis already used in Appendix K.

Given your stated willingness, we hope that you will reconsider your score in light of these risk examples and corresponding theoretical and empirical characterizations of convergence.

Brief summary of theoretical bounds:

Let rr upper-bound the maximum payoff-row span of the preference matrix. Building on Appendix K.7, the strengthened entropic analysis gives the following dimension-independent regimes:

Entropic riskSufficient regime
Fixed risk parameter τ>0\tau>0β>eτr−1−τr2τ\beta>\frac{e^{\tau r}-1-\tau r}{2\tau}
Fixed regularization β>0\beta>00<τ<1rlog⁡(1+4βr)0<\tau<\frac1r\log\left(1+\frac{4\beta}{r}\right)

Thus, for any fixed entropic-risk level there is an explicit sufficient regularization range, and conversely, for any fixed β>0\beta>0 there is an explicit nontrivial interval of entropic-risk parameters for which the regularized operator remains strongly monotone. These bounds hold for every finite number of responses and do not require transitive preferences.

Likewise, extending the finite-difference analysis of CVaR in Appendix K.8 yields:

Lower-tail CVaRSufficient regime
Fixed tail mass q∈(0,1]q\in(0,1]β>r2q\beta>\frac{r}{2q}
Fixed regularization β>r/2\beta>r/2r2β<q≤1\frac{r}{2\beta}<q\le1

Hence CVaR also admits explicit regularization and risk-parameter regimes. Importantly, this result does not require differentiability of CVaR: the proof uses a finite-difference bound that remains valid when the active tail changes, so no smoothing or fixed-tail assumption is needed.

These conditions are directly verifiable in the finite-game setting of our theory. Given a finite preference matrix, one computes or upper-bounds its maximum payoff-row span rAr_A and substitutes it into the corresponding inequality. The resulting certificates are uniform over opponent policies and do not require searching over equilibria or numerically maximizing curvature over the policy simplex.

Finite preference games are also practically relevant: a fixed collection of LLM-generated candidate responses evaluated by a fixed pairwise preference scorer defines exactly such a game. The revised appendix therefore makes the admissible parameter regimes and verification procedure explicit for the entropic and CVaR examples already developed in Appendix K.

The numerical studies in Appendix F.6 (figures provided in the anonymous repository) complement the analytic certificates by comparing the closed-form bounds, the numerically computed distortion, and the actual stability of the learning dynamics. Across transitive and cyclic Bradley--Terry games with both entropic risk and CVaR, the sufficient conditions correctly identify a globally well-behaved regime but are conservative: convergence can persist substantially below the certified threshold, while further decreasing regularization can produce loss of stability, cycling, and multiple attracting equilibria. The experiments also recover the expected advantage of extragradient over mirror descent in rotational regimes and, for nonsmooth CVaR, show the predicted piecewise changes in stability as the active tail changes. Thus the numerical results illustrate both the validity and the conservatism of the sufficient conditions, as well as the qualitatively different behavior that can arise once those conditions are violated.

The full details, including the supporting corollaries and numerical analyses of the sufficient conditions, empirical convergence, regions of attraction, and bifurcations, are given in Appendix F.6.

AuthorsOpenReview ↗

Q1 Theorem 2 Bounds: Empirical Verification (Part II)

Empirical Verification

Theorem 2 is a convergence result for a finite preference game, and its assumptions are directly verifiable in that setting. For a finite response set, one computes or upper-bounds the payoff-row span rAr_A and substitutes it into the entropic or CVaR conditions above to get certificates that are uniform over opponent policies and require neither locating the equilibrium nor numerically maximizing curvature over the simplex. Such games need not be synthetic: a fixed collection of LLM-generated candidate responses evaluated by a fixed pairwise preference scorer defines such a finite preference matrix, and the theorem applies to updates of the probabilities assigned to those candidates. The guarantee concerns this finite candidate-selection game, not unrestricted generation or subsequent neural-network fine-tuning.

The distinction between finite-game guarantees and neural implementations is not specific to risk sensitivity. Directly verifying the monotonicity condition for a full neural LLM is not currently tractable, as is the case with any competing algorithmic framework such as those for risk-neutral settings. At LLM scale, certifying the corresponding global property of the induced neural optimization mapping is a broader open theory-to-practice problem. This issue is not specific to the introduction of risk. Existing KL-regularized NLHF methods such as Nash-MD and EGPO establish convergence via monotonicity properties of an idealized policy-space/tabular game operator, but their neural LLM implementations likewise dont verify global monotonicity of the resulting parameter-space mapping. Composing the policy-space operator with a nonlinear neural parameterization does not in general preserve such a guarantee.

We therefore do not claim to empirically certify the global monotonicity condition for the full LLM parameterization. Local or finite-sample curvature probes would measure restricted directions or sampled subgames and wouldn't establish the global condition required by the finite-game theorem. Developing practically verifiable conditions connecting finite-game monotonicity to large neural parameterizations remains an open theory-to-practice problem for preference-game methods more generally.

The obstacle for neural training is theoretical before it is computational. KL regularization supplies favorable geometry in the finite policy game, but this does not automatically survive nonlinear neural parameterization. Increasing its coefficient does not add a uniform positive multiple of the identity to the neural self-play Jacobian and thus doesn't by itself certify even local strong monotonicity. A curvature-based local convergence analysis would first need to specify the neural learning dynamics, an appropriate local equilibrium or stability notion, and sufficient conditions for convergence in a neighborhood. These issues arise for risk-neutral NLHF as well as for risk-sensitive learning.

Even estimating the resulting local quantity requires additional methodology. The response-space and neural-weight dimensions create different obstacles due to the scale of the problem. A useful diagnostic has to account for prompt and response sampling, the opponent's dependence on the policy, and the relation between the estimated quantity and the proposed local convergence conditions. We do not assume that a computationally feasible, statistically reliable estimator satisfying these requirements is available for our LLM implementation.

The GAN and deep-learning literature provide a precedent for this type of local spectral investigation. Fiez et al (2020) formulate local equilibrium and stability conditions and inspect selected eigenvalues of game Jacobians, player Hessians, and Stackelberg-related operators in GAN experiments, using matrix-free derivative products, iterative eigensolvers, and regularized inverse-Hessian-vector products. More broadly, Bérard et al. (2020), Mescheder et al (2017) and Ghorbani et al. (2019), e.g., study neural optimization landscapes through dedicated Hessian and spectral analyses. These works illustrate that constructing and validating meaningful local curvature diagnostics is itself a substantial methodological undertaking; such diagnostics don't automatically certify global convergence. Our finite-game examples answer the applicability question in the setting of Thm 2. Developing an analogous local-stability theory and empirical diagnostic methodology for LLM self-play is a separate research direction.

Bérard et al (2020). A Closer Look at the Optimization Landscapes of Generative Adversarial Networks. ICLR. Fiez et al. (2020). Implicit Learning Dynamics in Stackelberg Games: Equilibria Characterization, Convergence Analysis, and Empirical Study. ICML. Mescheder et al (2017), The Numerics of GANs, Neurips 2017 Ghorbani et al (2019). An Investigation into Neural Net Optimization via Hessian Eigenvalue Density. ICML.

AuthorsOpenReview ↗

Q2 Matched ablations and consistency across settings

Q2 Matched ablations and consistency across settings

Disentangling risk adjustment and two-timescale bias correction. Our experiments already include both relevant comparisons: risk-neutral versus risk-adjusted training without correction, and risk-adjusted training with versus without two-timescale correction. Section 4 and Appendix L include the former comparison; the latter is additionally examined through the controlled study in Appendix L.9.2 (formerly L8.2). At each K∈{2,4,6,8}K\in\lbrace 2,4,6,8\rbrace, the uncorrected, delta-corrected, and jackknife-corrected policies share the same initialization, entropic risk parameter τ=10\tau=10, regularization coefficient, and training hyperparameters, differing only in how the risk estimate enters the gradient. Thus, the fixed-KK comparisons evaluate the incremental effect of correction without changing the risk objective or opponent-sample budget.

Crucially, these experiments also quantify how much of the intended risk-adjusted update is recovered. Varying KK changes the finite-sample bias and variance of the risk estimate, while comparing delta and jackknife correction at a fixed KK examines different amounts of bias removal and their associated variance costs. Appendix L.9.1 (formerly L.8.1) measures these quantities directly. For example, Table 5 (formerly Table 1) reports that at K=4K=4, delta correction removes 21% of the plug-in estimator’s bias, whereas jackknife correction removes 57%, with estimator-variance inflation factors of 1.07 and 1.86, respectively. The table reports the corresponding quantities across sampling budgets, making explicit how both the original bias and the fraction recovered depend on KK and the correction method.

Appendix L.9.2 (aka L.8.2 in previous version) then measures the correction actually realized during training, rather than relying solely on these estimator-level diagnostics. At K=4K=4, the jackknife tracker removes 56–76% of the measured bias at checkpoints 1500, 3000, and 4500, compared with 23–26% for the delta tracker. Figures 34–35 additionally examine the bias–variance tradeoff and the variance contribution after tracking. These measurements connect the training comparisons to the proposed mechanism: the methods differ in how accurately they recover the same underlying risk-adjusted update, and the experiments quantify both the remaining bias and the variance cost.

Together, these comparisons address the requested disentanglement: the risk-neutral comparison evaluates introducing risk adjustment; the fixed-KK comparisons evaluate correcting its estimation bias; and the KK sweep and delta-versus-jackknife comparisons characterize how the effects depend on the amount of bias present, the fraction removed, and the associated variance. We will make this experimental rationale explicit and relabel the uncorrected “OMD” curves in Appendix L.9.2 as “OMD-Ent (no TT)” to avoid suggesting that they are risk-neutral.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.