Win Rate Is Not Security: Reducing LLM Exploitability

As large language models (LLMs) evolve, an increasing body of research evaluates them on strategic games such as poker to study their behavior in environments with hidden information. We argue that there is a major gap in current evaluations that centered on win rate, where LLMs with high win rates can fail against a well-constructed adversary. In a controlled Kuhn poker audit, half of the eight audited LLM systems earn above-equilibrium payoffs against a fixed opponent pool but fall below equilibrium against opponents that maximize their loss. Reducing this shortfall from equilibrium, known as exploitability, remains scarce in research on LLM game-playing agents. We propose an exploitability bound in terms of the training opponent, the requested policy updates, and the mismatch between the requested and learned policies. We further show how distillation error bounds the increase in exploitability relative to the target policy. We propose a new training method that uses exploitability to constrain and guide training. Across 18 poker runs, 94.4% have exploitability below the absolute game value at the selected checkpoint, at the final checkpoint, and throughout the final 20 rounds. A capability-preserving extension also maintains performance close to the unmodified model on general capability benchmarks.