acceptodds
Under review as a conference paper at ICLR 2027

Win Rate Is Not Security: Reducing LLM Exploitability

Abstract

As large language models (LLMs) evolve, an increasing body of research evaluates them on strategic games such as poker to study their behavior in environments with hidden information. We argue that there is a major gap in current evaluations that centered on win rate, where LLMs with high win rates can fail against a well-constructed adversary. In a controlled Kuhn poker audit, half of the eight audited LLM systems earn above-equilibrium payoffs against a fixed opponent pool but fall below equilibrium against opponents that maximize their loss. Reducing this shortfall from equilibrium, known as exploitability, remains scarce in research on LLM game-playing agents. We propose an exploitability bound in terms of the training opponent, the requested policy updates, and the mismatch between the requested and learned policies. We further show how distillation error bounds the increase in exploitability relative to the target policy. We propose a new training method that uses exploitability to constrain and guide training. Across 18 poker runs, 94.4% have exploitability below the absolute game value at the selected checkpoint, at the final checkpoint, and throughout the final 20 rounds. A capability-preserving extension also maintains performance close to the unmodified model on general capability benchmarks.

Primary area:
reinforcement learning
Keywords:
Large Language Models, Exploitability, Imperfect-Information Games, Policy Distillation, Counterfactual Regret Minimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Yh1TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that LLM game agents are evaluated mainly by win rate against a population of opponents, and that this does not measure worst-case security. A high payoff against a pool says nothing about how much an opponent who adapts to the agent's weaknesses can win. The proposed measure is exploitability, normalized by the absolute game value |v*|.

Critical strengths
  1. The paper moves the evaluation of LLM game agents from win rate against a population to exact worst-case exploitability. Win rate against a pool measures how much an agent profits from the opponents it happens to meet, while exploitability measures how much an opponent who adapts to the agent can take from it. The same concern applies to any leaderboard ranked by pairwise results against a sampled population, so I expect the audit protocol to be reused independently of the training method.

  2. The paper separates the requested policy from the expressed policy and reports its experiments completely. Because an LLM shares parameters across information sets, an update that targets the action probabilities at one information set can change the probabilities that the model plays at others. The paper makes this gap an explicit quantity and measures it throughout training. Every run stays in the reported denominators, and the paper discloses its negative results. This makes the strength of each claim easy to judge.

Critical weaknesses
  1. The method and the evidence apply only to games small enough to solve exactly, which is much narrower than the paper's thesis about win-rate evaluation, and the most direct baseline is missing.

The experiments use games with 6 to 270 information sets per seat, and each training step extracts the full policy and computes an exact best response to it. At this size the exact Nash equilibrium is cheap to compute and already serves as the fallback target, so the method in effect stores a known near-equilibrium table in one LoRA adapter per game, seat, and seed. The paper also gives no route, even for evaluation alone, to games where exact best responses are intractable, and those are the settings where its broader thesis matters.

  1. The bounds are correct, yet they neither produce the reported results nor select the training objective.

Theorem 1 and Corollary 1 describe iterative training, while the final method replaces iterative RM+ training with a fixed target because Figure 3 shows that iterative training is unstable. In the fixed-target stage, Theorem 1 only supplies an upper bound B_T on the target's exploitability, which the linear program already computes exactly, and the main text never reports the measured size of the three terms.

Proposition 1 shows only that target-reach weights give the tightest bound of this form, which matters little when the bound is vacuous, and I could not find an ablation that compares the two weightings. The 94.4% stable sub-value rate therefore rests on Proposition 2, where an exact audit with rollback keeps every accepted policy below the ceiling. That guarantee is sound but nearly tautological, and it is available only because exact evaluation is cheap in these games.

  1. How the figures are presented can be improved.
Questions for authors
  1. The thesis concerns win-rate evaluation in general, so the paper would be much stronger if its evaluation reached a game where exact best responses are intractable. An approximate best response trained with RL, for example, gives a lower bound on exploitability in a larger game, and one such experiment would support the broader claim more than further ablations on small games.
1 comment

Many thanks for your review!

  1. Neural realization and continued security. The certified policy is computed by the LLM's forward passes over complete legal action strings. Shared parameters couple decisions at different information sets, so realizing a known equilibrium and retaining it through updates are training problems. In our 100-round ungated Qwen3-4B controls at learning rate 10−410^{-4}, all 12 reloaded finals are below ∣v∣|v|, but only 5/12 pass the final-20-round criterion. These settings cover Kuhn seat 1, Leduc both seats and river seat 1, with seeds 2026 to 2028. The capability-constrained Kuhn adapter reaches 0.259∣v∣0.259|v| while retaining MMLU, HellaSwag and GSM8K scores within 0.002 of the same-stack base model. This connects strategic protection with a general model's retained capabilities.
  2. Exact evaluation and enumerable games. The game supplies a controlled teacher and an exact adversarial test of neural realization. Among eight audited systems, four beat the fixed pool while losing to their best response. We also evaluated five open models at both seats under shared prompts and identical complete-action log-probability extraction: all ten remain above ∣v∣|v|, at 4.54.5 to 8.1∣v∣8.1|v|. Thus the central finding persists under a common interface. A learned best response supplies a lower bound on exploitability. Sequence-form optimization supplies the upper-bound guarantees needed to certify protection against every opponent in these enumerable games.
  3. Fixed-target theory. Theorem 1 analyzes moving-target search through regret, opponent fidelity and neural realization. Its BTB_T summarizes that search stage. For fixed-target distillation, we prove a separate theorem taking the exact teacher exploitability E(q)\mathcal E(q) as input. For teacher qq and student pp, ∣E(p)−E(q)∣≤∑IωI TV(qI,pI)|\mathcal E(p)-\mathcal E(q)|\le\sum_I\omega_I\,\mathrm{TV}(q_I,p_I), where E\mathcal E is exploitability, TV is total variation and ωI=rq(I)πc(I)ΔI\omega_I=r_q(I)\pi_c(I)\Delta_I. Here rqr_q is reach contributed by teacher actions, πc\pi_c sums chance reach over histories in II, and ΔI\Delta_I is the terminal-payoff range reachable from II. The constant is tight, attained in a one-decision game. These weights are computable before training and identify strategically costly fitting errors. For a teacher at 0.6∣v∣0.6|v|, the sharpened KL form gives sufficient excess reach-weighted KL-loss budgets of 1.9×10−41.9\times10^{-4} on Kuhn and 1.0×10−51.0\times10^{-5} on Leduc.
  4. Measured certificates and their limits. Our independent audit evaluates the reach-weighted distillation bound on 84 poker and negotiation trajectories and 7,003 checkpoints, with every observed exploitability covered. The 73 poker finals comprise 21 Kuhn, 23 Leduc and 29 river trajectories. The sharpened KL-based closed form certifies all 16 sub-value Kuhn finals, versus 13 with the original Theorem 2. This gives Proposition 1's reach-weighted fitting objective a concrete certificate consequence. We also quantify the limit of bounds that add a threat-independent realization error to teacher exploitability: their unavoidable certificate floor is at least ∣v∣|v| for 8/17 sub-value Leduc finals and 10/23 river finals. These limits concern that analytic bound class. Exact LP evaluation certifies the actual policies. The 17/18 stable-result rate includes all 18 poker refinement attempts and measures the complete gated procedure. It requires selected and final checkpoints and every final-20-round policy below ∣v∣|v|.
  5. Acceptance, progress and numerical error. The acceptance rule states a safety invariant with rollback. We add a liveness theorem in realization-plan space: convex interpolation of a current plan and target qq preserves a ceiling τ\tau when both are below it. For target weight 0<λ≤10<\lambda\le1, let ηk\eta_k be the worst-opponent loss increase from realizing the requested mixed plan as a neural policy. If ηk≤λ[τ−E(q)]\eta_k\le\lambda[\tau-\mathcal E(q)] at each step, we obtain a geometric bound relative to teacher exploitability, up to an error floor.

We also prove that an audit with acceptance ceiling τc\tau_c and underestimation at most ε\varepsilon for every candidate gives E≤τc+ε\mathcal E\le\tau_c+\varepsilon for every accepted policy. With the 0.99∣v∣0.99|v| ceiling, uniform underestimation below 0.01∣v∣0.01|v| preserves sub-value certification. Our rational verifier checks feasibility and payoff witnesses using exact Fraction arithmetic, independently of floating-point solver tolerances.

Reviewer_L8XyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that win rate against sampled opponents says nothing about worst-case loss. An audit of eight LLM systems on Kuhn poker finds every policy exploitable above 1x|v*|, and four of them beat a fixed pool while losing to their best response. Theorem 1 bounds average exploitability by regret, best-response gap and a "neural realization" error. Theorem 2 bounds the student's exploitability by the target's plus Delta*sqrt(KL). The method runs RM+ search, takes the least-exploitable convex mixture of the checkpoints (falling back to Nash), and distills it with exact-best-response acceptance checks. On Qwen3-4B, 17 of 18 runs in Kuhn, Leduc and a river endgame end below the game value.

Critical strengths

Exact exploitability is the right yardstick for strategic robustness, and it is underused in LLM game-playing work. The audit (Table 1) is a useful corrective.

The reporting is transparent: per-seed tables, retained policy tables, and an explicit extraction contract.

Critical weaknesses

The certified object is a lookup table, not LLM behavior. The policy is a softmax over action-string log-probabilities under one fixed prompt per information set; Kuhn has six per seat. The guarantee does not transfer to sampled or reasoned play, or to other prompts, as the authors concede. Every step requires enumerating the game tree, so the method only applies to games CFR solves outright. What is gained by embedding a solved table in 4B parameters?

The audit also compares unlike interfaces. Frontier models are sampled at temperature 1 with reasoning on and about 40% of GPT-5.5 outputs dropped, while the trained model is read off exact log-probabilities. SPIRAL is scored on canonical Kuhn after training on a modified variant.

The theory is standard, and the results come mostly from the search stage. Theorem 1 is regret-to-exploitability plus a triangle inequality. Theorem 2 is convexity plus a Pinsker-type bound. Proposition 2 is tautological. The Theorem 2 bound falls below the game value only on Kuhn, so on Leduc and river the sub-value results come from the empirical acceptance gate, not the theory. In Kuhn, five of six initializers are already well below threshold before distillation (Table 11). The success rate of the search stage itself is not reported. The natural baseline is missing: plain reach-weighted distillation of exact Nash without the gate. "Nash from base" fails 0/9 only because the gate rejects updates and stops the run.

The 1x|v*| threshold is a convention, not a security notion. Three seeds, one model, one river board. The capability check is a single LoRA adapter.

Questions for authors

Does the certificate survive a different prompt template, or sampled play with reasoning?

What does ungated distillation of exact Nash achieve under the same compute?

What fraction of RM+ search runs yield a sub-value checkpoint or hull?

How does Table 1 change if all systems are evaluated under the same log-probability extraction?

1 comment

Thank you for the careful review. We address your questions below.

  1. What is gained by embedding a solved policy in an LLM?

We respectfully disagree with the “lookup table” characterization. We evaluate actions by rating all valid moves and applying a softmax. In a finite game, every policy is a map from information sets to action distributions; our implementation fits this definition. A true lookup table would fail across game scenarios. We compare against this degenerate case (“card-blind” floor in the Appendix) and quantify the gap as the realization gap.

  1. What does ungated Nash distillation achieve?

We refer the reviewer to our detailed response to Reviewer REBp, where we provide the requested experiments, baselines, and comparisons for ungated Nash distillation.

  1. What happens under another prompt or sampled play?

We acknowledge these limitations in the revised text. Every certificate covers only one specified policy; a different prompt or decoding method yields a different policy requiring a new audit. We add a theoretical result: if a new prompt induces an action-string logit perturbation at information set II with oscillation oIo_I, exploitability rises by at most ∑Irp(I)πc(I)ΔItanh⁡(oI/4)\sum_I r_p(I)\pi_c(I)\Delta_I\tanh(o_I/4). We also restrict our study to solvable games, a stated limitation, because improvements are quantifiable there.

  1. What changes under the same extraction interface?

No conclusions change. We use the exact same log-probability extraction, with identical prompts, for all open models. All ten cases were above ∣v∣|v| (4.54.5--8.1∣v∣8.1|v|). Closed models can only be sampled (parse rates: GPT-5.5: 60.5%/64.5%). We add a confidence interval for sampled-policy exploitability treating every unparsed output as worst-case. Regarding SPIRAL: its Kuhn variant has zero exploitability by construction; on standard canonical Kuhn the model has 5.000∣v∣5.000|v| and 7.000∣v∣7.000|v|, agreeing with our point that equilibria differ under different payoff specifications.

  1. Additional theoretical results and the “tautological” concern

We do not consider the theory trivial. We add:

(i) ∣E(p)−E(q)∣≤∑Irq(I)πc(I)ΔI TV(qI,pI)|\mathbb{E}(p)-\mathbb{E}(q)|\le\sum_I r_q(I)\pi_c(I)\Delta_I\,\mathrm{TV}(q_I,p_I)

(ii) Sharper Theorem~2: E(p)≤E(q)+ΔπˉcΦq(p)/2\mathbb{E}(p)\le\mathbb{E}(q)+\Delta\sqrt{\bar{\pi}_c\Phi_q(p)/2}, explaining all Kuhn experiments.

(iii) Impossibility: any certificate of the form “target exploitability plus a threat-independent realization term” is at least E(q)+d(q→p)\mathbb{E}(q)+d(q\to p), showing the exact LP gate is necessary.

(iv) Saturation: exploitability reaches 2.495∣v∣2.495|v| while loss decreases monotonically between two exact equilibria.

Proposition 2 is sound by construction, which is its purpose: the guarantee does not depend on perfect optimization. We add liveness (reach-weighted interpolation toward the target is always admissible) and audit error (ε\varepsilon moves the guarantee to τc+ε\tau_c+\varepsilon, explaining the 0.99∣v∣0.99|v| ceiling).

  1. How often does search succeed, and does this extend beyond Qwen?

Starting below ∣v∣|v| does not mean staying below. Our example in 5(iv) starts at an exact equilibrium and leaves the sub-value regime. Of 8 initializers (out of 18) starting above ∣v∣|v|, 7 end stable and sub-value. In 7 of 8 new settings (REBp reply), the target was the exact equilibrium, not the archive, and every run still ends sub-value. Thus results are not purely dependent on the search starting point. The 1×∣v∣1\times|v| threshold is a convention and does not affect the conclusion that exploitability is drastically reduced. Search-stage success: the mixture policy is sub-value in 5/6 cells (vs. 4/6 for the best single archived policy) and in all 13 archives (best single policy: 11). We will report the fraction of sub-value checkpoints along each RM+ trajectory, showing success extends beyond Qwen.

Reviewer_REBpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how much worst-case strategic protection survives distillation into a pretrained LLM policy. In finite games, it defines the trained policy by normalizing the model's scores for complete legal action strings under a fixed prompt interface. The analysis separates requested from expressed policies and bounds added exploitability from fixed target distillation, while the method searches for policies, chooses an archive mixture or equilibrium teacher and audits updates and reloaded checkpoints. It reports 17 of 18 stable Qwen3-4B poker refinements from retained search initializers, with exploitability below the absolute game value at selected and final checkpoints and throughout the final 20 rounds.

Critical strengths
  1. A principled transfer objective. The sound conditional bound in Theorem 2 separates teacher quality from neural realization error and controls added exploitability against every opponent. This gives fitting error a strategic meaning that payoff against a fixed pool alone lacks.

  2. Improvement with useful controls. Table 11 reports seven of eight initially above threshold policies becoming stably sub-value. Matched feedback and target controls show intermediate regressions and the importance of target quality. Reloaded checkpoint audits and restoration of rejected updates support the reported stability check.

Critical weaknesses
  1. Modest comparative advantage. Combined routing gives 11/12 stable outcomes versus 10/12 for Nash-only under shared initialization and acceptance checks (Table 4). The direct base comparisons all stop without final policies and change both target and initializer, as the authors acknowledge. A completed comparison would clarify what search and archive construction add in final quality or total work.

  2. Limited evidence of broader benefits. Training uses one 4B model in small enumerated games with a single river board and capability preservation tested on one Kuhn adapter. Complete games enable exact adversarial evaluation, but these experiments leave benefits in other models and larger interactions open.

Questions for authors
  1. Could you complete an additional equilibrium distillation comparison with the same base model and final audit, allowing optimization before threshold entry to finish? Please compare final exploitability, entry reliability and total work.
2 comments

Thank you for the review. We clarify the following points.

  1. Routing and checking. Routing gives 11/12 stable outcomes, compared with 10/12 for Nash-only and 7/12 for mixture-only, under shared initialization and acceptance checks. In a separate target-only ablation, the archive mixture produces lower final exploitability than the best single archived policy in 10 of 11 trained pairs across 13 archives. Checking ensures security as optimization continues, as the new comparisons below demonstrate.

  2. Broader experiments. We added Llama-3.1-8B-Instruct on Kuhn. All five completed settings end below ∣v∣|v|, with three stable over the final 20 rounds. On two additional river boards, all three completed board--seat pairs are stable and sub-value, compared with one of three for direct Nash200 at matched nominal total optimizer-step budgets (Nash200 denotes ungated distillation of an exact Nash target for 200 rounds). These results extend successful construction to a second model and new board instances. In 7 of these 8 completed settings, the teacher is exact Nash because the archive mixture is above ∣v∣|v|. Successful sub-value construction therefore extends beyond cases with a secure archive teacher.

  3. Completed equilibrium-distillation comparison. We completed the requested ungated comparison from the same base model, allowing optimization before threshold entry to finish. 7/12 runs cross back above ∣v∣|v| during the final 20 rounds after previously entering the sub-value regime, reaching as high as 3.5∣v∣3.5|v|. Reaching a secure checkpoint does not ensure that further training preserves security. The additional-board comparison demonstrates the stability advantage. In the two dry-board seats, our worst final-window exploitability is 0.780∣v∣0.780|v| and 0.0625∣v∣0.0625|v|, versus 3.865∣v∣3.865|v| and 1.489∣v∣1.489|v| for direct Nash200. Our procedure has no threshold violation in either window. In seat 0, direct Nash is stable at round 100, with window maximum 0.243∣v∣0.243|v|, but loses stability by round 200. Our matched-step procedure remains sub-value throughout its final window.

  4. Why decreasing loss is insufficient. On Kuhn, we distilled between two exact equilibrium policies with bluff rates 1/31/3 and 00. Although both endpoints are equilibria and the loss decreases monotonically, exploitability reaches 2.495∣v∣2.495|v| and remains above ∣v∣|v| for a sustained interval. Intermediate policies combine the target at some information sets with the initializer at others, producing value bets without the corresponding bluffs. This explains why decreasing distillation loss cannot replace the audit step.

Reviewer_REBpOpenReview ↗

Thank you

I remain supportive of acceptance with a stronger empirical basis after this response. Including these results in the revision would strengthen the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.