Risk-Sensitive Preference Games: Learning from Strategic Interaction & Uncertainty

A growing line of work reframes preference-based fine-tuning of large language models game-theoretically: Nash Learning from Human Feedback (NLHF) recasts the problem as a zero-sum game over policies. However, optimization is over expected pairwise payoffs, thereby conflating policies with similar win rates but different tail behavior. As such, these methods are agnostic to where in the data distribution they succeed or fail: strong average performance can mask systematic failure across prompts, annotators, or safety-critical strata. Risk-sensitive preference games, introduced here, address this gap: players optimize convex risk measures of their preference loss, exploiting structure in preference uncertainty. We introduce risk-sensitive preference games, in which players optimize convex risk measures of their preference loss, exploiting structure in preference uncertainty. While risk transforms generally break zero-sum structure at the certainty-equivalent level, we show that translation invariance ensures constant-sum structure is preserved at the level of risk-adjusted payoff operators. This is sufficient to retain monotonicity, yielding fast convergence of sample-efficient self-play methods. The harder challenge is robustness. We establish algorithmic stability and offline sample complexity bounds that scale with risk, requiring simultaneous control of structural bias from nonlinear risk transformations, statistical bias in risk estimation, and concentration tailored to the risk-sensitive setting. To address statistical bias, we introduce a hierarchical game formulation and a two-timescale extragradient algorithm with bias correction that converges to the Stackelberg equilibrium and is especially effective in low-sample regimes. Empirically, risk-adjusted policies are robust across data strata, stable across risk choices, and match or exceed risk-neutral performance thereby achieving robustness without a performance tax.