- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper targets the problem that standard preference-learning techniques are optimising for point-wise estimates of the population's preference distribution neglecting the tails.
The main contributions of the paper are
- The authors propose Risk-Sensitive Preference Games (RSPG) as a modelling framework to address this shortcoming of the previous works. Instead of optimising for the expected preference signal, this framework focuses convex risk measures.
- The authors address the two main limitations of optimising risk measures instead of expected payoff, namely, non-convexity and biased sample estimation. They propose solutions for both and efficient learning algorithms that converge to the equilibrium in RSPG.
- Their empirical results show pareto-dominance over previous baselines.
Critical strengths
- Clean, unifying formalization. The operator-level constant-sum observation lets the single-policy self-play machinery of NLHF carry over unchanged. gives an explicit, interpretable condition on how much risk distortion the KL regularization can absorb. This matters because it makes risk sensitivity a principled extension of NLHF ( recovers it exactly) rather than an ad-hoc reweighting.
- Bias identified as the binding constraint, and corrected for free. The paper identifies estimator bias, rather than variance, as what limits training at the small rollout budgets used for LLMs, and corrects it with no additional samples. This is backed by direct measurement showing that bias dominates up to K ≈ 150, and jackknife tracking removes 56–76% of measured bias at K = 4.
Critical weaknesses
- The headline claims (“Pareto-dominance”, “robustness without a performance tax”) rest on 100–136 prompts per stratum, with confidence intervals bootstrapped over prompts only. The controlled study uses a single training seed, and I could not find seed variation for the main comparisons. Several reported gains are point estimates, e.g., higher combined win-rate against “six of eight opponents.”
- The experiment setup is ambiguous. The main text cites Llama 2 as the base model while Appendix L states it is Gemma-2 2B-IT. Both of which are outdated and do not provide sufficient signal to evaluate the approach's scalability. Similar contradiction is stated for the pairwise judge as the main text says it is an (unspecified) LLM judge while the appendix says it is a pairwise finetuned judge.
Questions for authors
- Theorem 1 rely on the assumption that . Can you estimate for the setting used in Section 4? How does it align with theoretical results?
- Could the authors also elaborate on why Appendix B states that the paper considers the monotone regime with ? Is it only a typo or it does not refer to Theorem 1?
- Could the authors provide evaluations on other datasets and judges? Currently the experimental section is the weakest point of the paper and it provides not enough evidence that the method generalizes and scales.
- The related works and experimental comparison baselines are missing approaches connected to Stackelberg Learning from Human Feedback (SLHF) [1,2] which provide a different risk tolerance than NLHF. Could the authors elaborate on how their method differ and include at least one of them in Section 4?
[1]: Self-Improving Robust Preference Optimization, Choi et al., ICLR (2025) [2]: Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Game, Pasztor et al., ICLR (2026)
7 comments
Q3 Further Evals and Empirical Scope
Q3 Further Evaluations and Empirical Scope
Broader evaluation across additional datasets, base models, and evaluators would certainly be valuable. However, the experimental setting in this paper was chosen deliberately to enable controlled, apples-to-apples comparison with the closest state-of-the-art NLHF baselines, rather than to constitute a broad empirical benchmark. In particular, we use the same data/base model setting as the prior methods against which we compare---Nash-MD, EGPO with IPO---so that differences can be attributed to the introduction of risk sensitivity and bias correction rather than to changes in the underlying training environment.
Within that matched setting, the experiments are designed to isolate the main algorithmic questions raised by the paper. We compare risk-sensitive and risk-neutral objectives at matched opponent-sampling budgets; study multiple forms and levels of risk; compare corrected and uncorrected finite- risk estimation; compare OMD and extragradient dynamics; evaluate across held-out safety strata and diverse opponents; and test the two-timescale correction directly through measured bias, self-play risk, and downstream performance. We also evaluate preference and safety with distinct evaluation models, rather than relying on the training objective alone. Thus the empirical section is intended to test the mechanisms introduced by the theory and algorithms, not simply to report one end-to-end number.
We do not claim that these experiments establish universal empirical generalization across model families or datasets. Our principal contributions are algorithmic and theoretical: the RSPG formulation, its equilibrium and convergence analysis, the finite-sample bias characterization, and the novel two-timescale bias correction. The LLM experiments provide supporting evidence that these mechanisms remain meaningful and effective at a practically relevant neural scale.
A substantially broader empirical study would require retraining the full set of matched baselines, risk-sensitive variants, risk sweeps, and bias-corrected variants across multiple datasets and base-model families. This is particularly costly here because each training update requires sampling multiple opponent responses, and extragradient approximately doubles the policy-update cost. Our available compute is limited to L40S-class GPUs; even the present study required restricting model size and generation length to keep the matched experiments feasible. Scaling to substantially larger models would require reducing batch size, opponent sample count, or generation length—precisely the quantities whose effects we study—or access to substantially greater compute.
For this reason, we prioritized a controlled setting that matches the most relevant prior work and allows us to perform the mechanism-level comparisons required by the paper. We view multi-dataset and multi-model scaling as an important empirical extension, but carrying it out carefully would constitute a substantially broader experimental study rather than a missing component of the theoretical and algorithmic contribution presented here.
Q4 Relationship to SLHF and SRPO
Q4 Relationship to SLHF and SRPO
We agree that SLHF and SRPO are relevant related work, and we have added discussion of these approaches to the related-work section in the revision (Appendix A in blue). However, we do not agree with the characterization that they provide a different risk tolerance than NLHF in the sense studied in this paper. Their robustness arises from changing the preference-optimization game or information structure, rather than from replacing expected preference payoffs with a risk functional.
SLHF changes the underlying game and solution concept. NLHF is a simultaneous-move game: each player chooses a policy without observing the opponent's realized response. SLHF instead introduces a sequential Leader--Follower game in which the Leader first produces a response and the Follower observes that realized response before generating a conditional refinement. Its objective nevertheless retains the expected pairwise preference probability with KL regularization. Thus SLHF's robustness is induced by the Follower's informational advantage and the Leader's optimization against a response-conditioned best response. This is a different Stackelberg solution concept, not a risk-sensitive extension of the NLHF payoff.
Our construction changes a different object. RSPG retains the NLHF self-play formulation but replaces the expected opponent-dependent payoff by a convex risk functional. Setting that risk functional to expectation recovers NLHF exactly. The risk parameter therefore controls sensitivity to the distribution of preference outcomes themselves---for example, lower tails or variability---rather than changing which player moves first or what information one player observes. The framework also permits risk to act over the opponent distribution, prompt distribution, within-prompt heterogeneity such as annotators or labels, and the response-pair sampler. This distinction is important in our experiments, where we measure improvements in the tails of prompt and severity distributions even against fixed opponents.
SRPO is related for a similar but distinct reason. SRPO introduces a conditional self-improvement policy that maps an initial completion to an improved completion , and jointly optimizes this policy with a generative policy through a min--max objective. The pairwise preference term in that objective is again an expectation of the preference probability, with KL regularization; it is not replaced by a convex risk functional. SRPO's notion of robustness is instead that the resulting generative solution is independent of the behavior policy used to generate the preference data, together with the ability to improve responses through conditional self-revision. SRPO is also formulated as an offline preference-optimization method, whereas the Section 4 experiments study online self-play.
For these reasons, SLHF and SRPO are useful conceptual comparisons but are not matched experimental baselines for the contribution tested in Section 4. Our risk-neutral NLHF baselines---Nash-MD, OMD/online IPO, and EGPO---keep the game, opponent construction, and solution concept fixed while replacing our nonlinear risk aggregation by expectation. Consequently, the RSPG-versus-NLHF comparison directly isolates the effect of introducing risk sensitivity. Likewise, our corrected-versus-uncorrected experiments isolate the effect of correcting the finite-sample bias created by estimating that risk.
Adding SLHF or SRPO would simultaneously change the underlying game formulation, conditional response structure, solution concept, and, for SRPO, the online/offline training setup. Such a comparison could be useful in a broader empirical benchmark of different preference-optimization paradigms, but it would not isolate the algorithmic contribution studied here. We therefore include SLHF and SRPO in the related-work discussion while retaining matched NLHF methods as the appropriate Section 4 baselines.
Opening & Q1, Q2 Response
Thank you for the thoughtful review. We address the specific questions first, and then we address the weaknesses.
Q1 Estimating Risk Distortion
The quantity λˉR in Theorem 2 is a global worst-case curvature quantity for the finite preference-game operator. In the setting of the theorem, it can be verified directly. In the revision, we added Appendix F.6 (colored blue), which derives explicit sufficient regimes for entropic risk and CVaR in terms of properties of the finite preference matrix, together with numerical examples comparing the resulting certificates, numerically evaluated finite-game curvature, and observed convergence behavior. These bounds are corollaries of results that were already present in Appendix K. You can also view the examples here: anon-website
We do not estimate λˉR for the LLM experiments in Section 4, because doing so in a way that verifies the theorem would require certifying the corresponding global property of the neural self-play operator over the full sequence-level policy space, with an effectively enormous response space. A Jacobian estimate along the observed training trajectory, or on a sampled set of responses, would characterize only restricted directions or sampled subgames and would not establish the global condition λˉR<β. Moreover, the relationship between such local diagnostics and the global convergence property required by the theorem is itself nontrivial and is not established.
This distinction is not specific to risk-sensitive preference learning. Existing risk-neutral NLHF methods make the same separation between their convergence theory and neural implementations. For example, Nash-MD establishes its convergence guarantees for the tabular game and separately introduces deep-learning implementations; EGPO likewise explicitly distinguishes its tabular setting with rigorous guarantees from its neural experiments. Neither work computes a neural self-play Jacobian/Hessian spectrum for its LLM runs or verifies that the neural implementation satisfies an analogous global monotonicity certificate. Thus, introducing risk has not created a new empirical verification gap: certifying global monotonicity after composition with a large nonlinear neural parameterization is a broader theory-to-practice problem for preference-game methods.
Indeed, even substantially weaker local neural-curvature diagnostics constitute a separate empirical problem. The GAN literature provides a useful precedent: dedicated empirical studies developed local diagnostics based on game Jacobians, player Hessians, and stability spectra, using matrix-free derivative products and iterative eigensolvers. Such diagnostics can illuminate local training dynamics, but they still do not certify a global condition of the form required here. Developing analogous diagnostics for LLM self-play would therefore be an interesting separate direction, rather than an empirical calculation needed for the present theorem.
What can be verified is the condition in the setting in which the theorem is stated. Appendix F.6 gives explicit, verifiable finite-game regimes for entropic risk and CVaR, uniformly over opponent policies, and numerical examples comparing the analytical certificates with numerical curvature and convergence behavior. We also explicitly verify the condition for the submitted finite Bradley--Terry experiment. These results show that the assumption is non-vacuous and directly checkable in the finite-game setting of the theory.
Accordingly, the theoretical and empirical results have complementary roles. Appendix F.6 gives explicit regimes in which the theorem is guaranteed to apply and illustrates how the sufficient condition varies with risk and regularization. Section 4 evaluates the proposed risk-sensitive algorithms at LLM scale and demonstrates stable optimization and improved robustness empirically, but we do not interpret this empirical stability as a verification of the theorem's global sufficient condition. It is the neural-scale evaluation of the proposed algorithms, just as neural experiments in prior NLHF work are not certificates of their tabular convergence assumptions.
Q2 Typo
Yes β/2 appearing in the future work is a typo. Thanks for catching this; we have fixed it in the revision.