acceptodds
Under review as a conference paper at ICLR 2027

Assessing dynamic predictability in LLM strategic behavior

Abstract

Unpredictability is central to strategic behavior in competitive settings. Yet, evaluations of large language models (LLMs) in mixed-strategy games typically measure only the overall frequencies of actions, not whether those actions can be predicted from the history of play. Drawing on methods from behavioral game theory and decision neuroscience, we introduce the Protean Adversarial Choice Assay (PACA), a repeated matching-pennies paradigm in which an adaptive opponent detects and exploits statistical regularities in a model's action history. We evaluate seven LLMs from Anthropic, OpenAI, and Google under two framings of the same game: a minimal prompt that conceals the strategic context and elicits a choice framed as an outcome prediction, and a strategically informed prompt that explains the game and prescribes the equilibrium strategy of unpredictable 50/50 play. Informed prompting moves behavior farther from equilibrium, and win rates fall by 10.9 percentage points on average (45.6% down to 34.7%), with six of seven models showing a decline. Under this framing, the worst-performing models over-alternate most strongly despite maintaining near-balanced overall action frequencies. These findings show that assessment of equilibrium play should account for history-dependence, and that explicitly prescribing the equilibrium strategy can be counterproductive to its execution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, strategic reasoning, mixed-strategy equilibrium, sequential dependence, game theory

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_G3t6OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PACA, a repeated matching pennies assay in which an adaptive opponent (Proteus) detects and exploits statistical regularities in an LLM's action history. Seven models from Anthropic, OpenAI, and Google are evaluated under two framings of the same game: a minimal prompt that conceals the strategic context, and an informed prompt that explains the game and prescribes the equilibrium strategy of unpredictable 50/50 play. The headline finding is that informed prompting moves behavior farther from equilibrium and lowers win rates from 45.6% to 34.7% on average across seven models, with the worst models over alternating most strongly while keeping near balanced marginal frequencies. The main experiment runs at temperature 0, with 98 sessions of 200 rounds each, and models accessed through OpenRouter.

Critical strengths

The experimental design has real virtues. The closed loop adaptive opponent is a sensible device, the stay and shift decomposition is informative, and the supplemental analyses, temperature sweeps, label swaps, and lag structure, are thorough. The core finding is genuinely counterintuitive and worth reporting: explicitly prescribing equilibrium play makes execution worse. The methodological recommendation, that evaluations of equilibrium play should test history dependence rather than marginal frequencies alone, is correct and overdue.

Critical weaknesses

The headline result is heavily confounded by the decoding setup. The main experiment is run at temperature 0. Under greedy decoding an autoregressive model cannot produce a mixed strategy by construction; the best it can do is a deterministic policy that looks random, and period 2 alternation is precisely what that looks like. Telling a deterministic system to be unpredictable and observing that it responds with an exploitable cycle is close to tautological. The temperature robustness checks exist, but they sit in an appendix while the abstract carries the temperature 0 numbers. To a first approximation the paper measures what greedy decoding does to a request for randomness, and that phenomenon has been documented extensively since 2023, along with the broader finding that LLMs drift away from mixed strategy equilibria.

That leaves a modest novelty core, one methodological device plus one framing effect, measured in a single 2 by 2 game. Matching pennies is the simplest possible zero sum game, and it is a very narrow base from which to generalize about LLM strategic behavior. Reproducibility is also weak by construction: all models are accessed through OpenRouter with versions drifting across the access window, and the paper itself tables model provenance dates because the exact weights will never be recoverable; the 98 sessions of 200 rounds are a modest sample for the lag 1 to 4 history dependence claims. Finally, on venue fit, this is a behavioral science study of API models with no method, no model, and no theory beyond textbook mixed strategy equilibrium. It is competently executed, and parts of it would interest a cognitive science or behavioral economics audience, but as an ICLR contribution it is analysis in search of a machine learning result.

Questions for authors
  1. Does the informed framing effect on win rate survive as the headline effect when the main experiment is run at temperatures where a mixed strategy is even representable, with session counts matched to the temperature 0 study?
  2. Can you demonstrate the effect in at least one game with a richer action space, and does Proteus beat a fixed iid 50/50 baseline as an opponent in that setting?
  3. What is the marginal contribution of the adaptive opponent over a simple alternation detecting strategy, given that the dominant failure mode reported is over alternation?
3 comments
AuthorsOpenReview ↗

Response to Reviewer G3t6 (1/3)

We appreciate the reviewer’s thoughtful acknowledgement of the strengths and opportunities in our work.

W1 & Q1: Thank you for this careful consideration of the decoding setup. We have now run an independent replication with requested temperatures 0 and 1, with seven 200-round sessions per model, prompt, and temperature (Appendix H). Win rates dropped by 13.6 percentage points at requested temperature 0 (p = .012) and by 9.9 at requested temperature 1 (95% CI [−20.1, +0.3]; t(6) = −2.36, p = .055; exact Wilcoxon p = .047), with six out of seven models declining at temperature 1. The effect was thus somewhat attenuated but not eliminated. Win rates, marginal bias, and runs z were closely correlated across temperatures (r = .91–.98; Figure H1). Over-alternation in particular persisted under sampling: in the models where the setting took effect, informed-prompt runs z went from 10.8 to 10.0 for Sonnet and from 8.5 to 7.0 for Haiku. A manipulation check also showed that the serving endpoints for Opus, GPT-5.4, and GPT-5.4-mini did not apply the temperature parameter, so for these three models the comparison reflects repeated sampling at provider defaults, and their main-study results were not obtained by greedy decoding either. Restricted to the four models in which the setting took effect, the correlations were .92–.99.

We also note that deterministic decoding does not necessarily imply predictable behavior across rounds in a stochastic interactive environment. Proteus's choices incorporate random draws, and its completed choices enter the model’s subsequent prompts. Even a model using greedy decoding could therefore condition its actions on this external source of entropy, producing less predictable behavior without stochastic token sampling. Consistent with this possibility, under minimal prompting all seven models exhibited significant dependence on Proteus's recent choices, whereas this dependence largely disappeared under informed prompting. We do not take this association as direct evidence that models extracted randomness from the opponent, but it demonstrates why the closed-loop setting cannot be reduced to a deterministic model simply being asked to generate random choices.

W2:

On matching pennies: The reason that we chose matching pennies is because it is the simplest game with only mixed strategy equilibria, and hence the archetypal game that sharply illustrates the distinction between marginal bias and sequential dependence. These properties are exactly why matching pennies has remained such an influential setting for game theory, harkening back to von Neumann and Morgenstern and yet still studied in contemporary research across several fields including economics, biology, and computer science. We agree that there may be strategic phenomena in richer games that matching pennies does not capture, such as non-uniform equilibrium mixtures or highly asymmetric roles, and we now state this explicitly in our limitations. However, our contributions (see below) do not depend simply on generalizing particular behaviors from matching pennies to other games. Our methodological contribution, the standard that evaluations of equilibrium play must test unpredictability given the history and not just action frequencies, and PACA, which implements it, applies to any competitive game whose equilibrium involves mixing, since that requirement is part of all of them. Our empirical finding, that prescribing the equilibrium can make play worse, is a demonstration that this can happen, and the simplest game in which the requirement appears is a natural place to show it. We also report a new experiment that extends beyond it (see Q2).

On version drift: all main-study sessions were run on a single day, and each model returned a single served identifier across both conditions (Tables A1–A2), so no version change was recorded within any comparison. On reproducibility: the counterbalanced replication, run 11 days later, again recorded a single identifier per model and reproduced the main effect (Appendix E). The more recent replication to assess temperature effects also reproduced the finding. As with any evaluation of closed models, the weights themselves cannot be recovered, but we will release full round-level logs, code, and opponent seeds, and PACA can be rerun on any current or future model.

AuthorsOpenReview ↗

Response to Reviewer G3t6 (2/3)

(cont'd)

On sample size: the history dependence we report is not a subtle effect that requires a large sample. It was detected in nearly every individual 200-round session. Under the minimal prompt, the opponent’s recent choices significantly predicted the model’s choices in all 49 sessions across the seven models. Under the informed prompt, the models’ own recent choices significantly predicted their next choice in all 21 Anthropic sessions and 11 out of 14 Gemini sessions (Table D3). In the Anthropic models, the effect is strong enough that a single session alone rules out chance by a wide margin (mean runs z of 7–10 per session). These patterns recurred in the counterbalanced replication and supplemental checks. For comparison, the classic experiments that established these analyses detected serial dependence in sequences of 105 rounds per pair (O’Neill, 1987; Brown & Rosenthal, 1990); our sessions are nearly twice as long.

On venue fit: the paper’s machine-learning contribution lies in both an evaluation method for foundation models and an empirical finding about them, which fall within ICLR’s scope (our primary area is foundation and frontier models). Behavioral and game-theoretic evaluations of LLMs have appeared at ICLR (Bigelow et al., 2024; Huang et al., 2025; Liu et al., 2025) and other ML venues (Jia et al., NeurIPS 2025; Wang et al., ICML 2026).

The method, PACA, evaluates the strategies models actually execute in adaptive interaction. By measuring how predictable their choices are given the history of play, against an opponent that adapts to whatever regularities they produce, it reveals failures that conventional evaluations of equilibrium play, which focus on aggregate action frequencies, overlook. More broadly, it addresses a growing challenge in evaluating agentic systems: determining whether models reliably carry out appropriate strategies when interacting with other adaptive agents. The finding is that explicitly giving an LLM the equilibrium strategy can make the behavior it executes substantially worse than minimal instructions do. This is a counterintuitive result about how language models translate instructions into behavior, beyond the established observation that they struggle to generate random sequences. It bears directly on how LLM agents are prompted, and on monitoring of chain-of-thought: models’ reasoning often stated the right strategy and recognized their departures from it, yet was an unreliable guide to the policy they executed. We do not claim new theory. The standard we apply, unpredictability given the history of play, is textbook; our point is that current LLM evaluations of equilibrium play do not test it.

Q2: Thank you for prompting us to test this. We extended PACA to a four-action version of matching pennies, similar to the hide-and-seek games studied in experimental game theory (Crawford & Iriberri, 2007), where independent uniform play wins 25% of rounds (Section 4.4.3; Appendix J). Against Proteus, a simulated independent uniform player had a win rate of .2495 (95% CI [.2477, .2513]). Informed prompting again reduced win rates, from 23.6% to 14.2% (mean difference −9.4 percentage points, 95% CI [−15.0, −3.8]; t(6) = −4.11, p = .006), with a decline in all seven models. The informed prompt here prescribed independent uniform choice but omitted the fair-coin analogy and the warning about alternation and repetition, so the decline did not depend on those components. The accompanying behavior differed from the binary game: informed prompting increased switching but also produced pronounced imbalances in overall choice frequencies, exceeding the independent-uniform reference range in 27 out of 28 informed sessions. In the most extreme case, Gemini 3 Flash Preview chose STAR on only 0.1% of informed rounds and won 2.0% of rounds. The disadvantage is thus not limited to two-action alternation, though the form of exploitable behavior changed with the action space.

AuthorsOpenReview ↗

Response to Reviewer G3t6 (3/3)

Q3: Thank you for inquiring about this. We emphasized over-alternation because it is the simplest pattern that cannot be detected from marginal frequencies, and because the models that showed it most strongly had the lowest win rates. However, models differed in which regularity they showed, and we have revised the results and added Figure D2 to make this clearer. Figure D2 classifies every round by the prediction Proteus acted on. Under the informed prompt, Proteus predicted a switch on most rounds only for Sonnet and Haiku (89% and 84%). For GPT-5.4, GPT-5.4-mini, and Gemini 3.1 Flash Lite, it instead acted on a side bias on 83–86% of rounds, winning 60–68% of them, and almost never predicted a switch. Under the minimal prompt, Proteus predicted repetition more often than switching for every model (e.g., 76% of rounds for Opus, 72% for Gemini 3 Flash Preview). Outside Sonnet and Haiku, then, most of what Proteus acted on consisted of predictions an alternation detector never makes, and the four-action game showed yet another profile, dominated by marginal bias (Q2). This variation is why PACA uses a general adaptive opponent: an evaluation must detect whatever regularity a model produces without knowing in advance which it will be.

Bigelow, E., Lubana, E. S., Dick, R., Tanaka, H., & Ullman, T. (2024). In-context learning dynamics with random binary sequences. ICLR 2024. Huang, J.-T., Li, E. J., Lam, M. H., Liang, T., Wang, W., Yuan, Y., Jiao, W., Wang, X., Tu, Z., & Lyu, M. R. (2025). Competing large language models in multi-agent gaming environments. ICLR 2025. Jia, J., Yuan, Z., Pan, J., McNamara, P. E., & Chen, D. (2025). LLM strategic reasoning: Agentic study through behavioral game theory. NeurIPS 2025. Liu, R., Geng, J., Peterson, J. C., Sucholutsky, I., & Griffiths, T. L. (2025). Large language models assume people are more rational than we really are. ICLR 2025.

Reviewer_7YWiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper examines the strategic behavior of LLMs, with a focus on analyzing their ability to act randomly. The authors introduce an evaluation scheme based on matching pennies (PACA), where an LLM must play against an adaptive opponent who attempts to exploit statistical patterns in the LLM’s gameplay, such that the LLM must behave uniform randomly in order to win. 7 LLMs from Anthropic, OpenAI, and Google are evaluated. The paper finds that LLMs fail to act fully randomly under both “informed” and “minimal” gameplay prompt instructions, as they are prone to over-alternation bias. However, this aspect of their behavior is clear only when sequential action choice dependence is analyzed, and is hidden by the marginal action choice statistic. The validation occurs only on matching pennies.

Critical strengths

The overall approach of using techniques/ideas from behavioral game theory and neuroscience to analyze LLM behavior is both interesting and timely.

Critical weaknesses

In my view, the paper has two critical weaknesses.

First, the paper fails to establish the broader importance of its findings, which is in part due to the limited evaluation on matching pennies only. What real-world scenarios/properties do we expect matching pennies to encapsulate? While the authors relate this study to the phenomenon where LLMs act “unfaithfully” towards their words, I find the relationship weak. If that is the setting they intend to study, why not study that problem setting directly, or construct an abstract matrix game that directly captures the relevant property of that setting? The findings

Second, the paper’s findings and proposed contributions do not present sufficient novelty for the standards of the conference. Regarding the finding that marginal statistics fail to capture important aspects of LLM behavior -- this has already been observed by [1]. Regarding the proposed evaluation methodology: I am not sure how the proposed evaluation methodology allows for substantial novel insights over existing evaluation methods. The main insight -- that LLMs cannot randomize effectively -- has already been established (using more general methods) by a body of literature studying whether LLMs can faithfully induce target distributions, as discussed in Section 2.2. I’m unsure how the game-based evaluation methodology can support future new insights. As such, I cannot consider it a contribution of the paper.

[1] Wang et al. Discovering Differences in Strategic Behavior Between Humans and LLMs. ICML 2026.

Questions for authors

N/A

1 comment
AuthorsOpenReview ↗

Response to Reviewer 7YWi

We thank the reviewer for raising opportunities to improve our work.

W1: Thank you for noting limitations on the scope of our conclusions. The reason that we chose matching pennies is because it is the simplest game with only mixed strategy equilibria, and hence the archetypal game that sharply illustrates the distinction between marginal bias and sequential dependence. These properties are exactly why matching pennies has remained such an influential setting for game theory, harkening back to von Neumann and Morgenstern and yet still studied in contemporary research across several fields including economics, biology, and computer science. Numerous studies have mapped the game on to real-world field settings like penalty kicks and tennis serves; similar principles could apply to LLMs bidding in auctions or conducting audits, for instance. Our results also bear on a natural practice in such deployments, giving the model the game-theoretic solution, which here made play more predictable.

For present purposes, the simplicity of the game is an advantage. Matching pennies isolates the need to remain unpredictable, a requirement embedded in every competitive game whose equilibrium involves mixing. The gap we identify can arise in any such game, and we see no reason that embedding the requirement in a more complex game would make it easier to meet. The extension to the four-action version of the game finds the same framing disadvantage in all seven models (Section 4.4.3). We now mention this in our discussion.

We agree that our study is not a test of unfaithful reasoning in general, and we do not intend it as one. The trace analysis is secondary. Models frequently stated that they should randomize and identified their own alternation as exploitable, yet continued to alternate. This indicates that the failure lies in executing the strategy rather than understanding it.

W2: Thank you for pointing us to Wang et al. (2026), which we now cite. While we agree this is an important, sophisticated, and relevant study as it involves sequential dependencies in strategic play, these projects are geared toward different goals. Their focus is on probing and comparing the depths of LLM and human strategic behavior and its underlying structural factors. To this end, they use opponent bots with relatively simple strategies that each track a single type of pattern; all 15 of their adaptive and non-adaptive bots are simpler than Proteus, which searches several histories and targets whichever regularity the model produces. In contrast, our focus is on the assessment of equilibrium play. Most evaluations of LLM equilibrium play rely on overall action frequencies or quantities derived from them without considering sequential dependencies (Silva, 2024; Xu et al., 2024; Guo et al., 2025; Vidler & Walsh, 2025; Lekeas & Stamatopoulos, 2026). We show how this omission can significantly affect evaluations, and argue for a more comprehensive standard, in line with the history of behavioral game theory.

In terms of results, Wang et al. find that LLMs can be capable of deeper strategic behavior than humans. We do not dispute that LLMs are capable of this. It is also consistent with our minimal condition, in which every model’s choices depended significantly on the opponent’s recent choices (a dependence which largely disappeared in the informed condition). We find that they can also exhibit surprising patterns of vulnerability even in quite simple settings, which is not incompatible. Thus, both projects contribute to our broader understanding of LLM strategic behavior from different angles. We appreciate the opportunity to clarify this in our revision.

We also would not characterize our findings as saying that LLMs cannot randomize effectively, let alone consider this our main insight. Indeed, in our minimal condition they performed relatively well. Rather, we find counterintuitive differences depending on the nature of the prompt.

Finally, while strategic behavior might invoke processes like randomization, it extends beyond this, which is why researchers in psychology and economics have separately studied random sequence generation and mixed strategy equilibrium play.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We are grateful for the reviewers’ thoughtful comments, which have helped us to meaningfully improve the paper, and for their acknowledgement that our "core finding is genuinely counterintuitive and worth reporting" and our central methodological recommendation is "correct and overdue".

We have responded to each reviewer comment below in ways that include clearer positioning and more in depth analysis. Most notably, we conducted four new experiments:

  1. A replication comparing requested temperatures 0 and 1 (Section 4.4.1; Appendix H).
  2. A new "opponent-only" condition, in which only the exploitative opponent's existence is mentioned but without prescribing a strategy (Section 4.4.2; Appendix I).
  3. A variant of matching pennies with four actions (Section 4.4.3; Appendix J).
  4. A replication changing the model's role from matcher to mismatcher (Section 4.1; Appendix G). These experiments broaden the scope of our findings and deepen our insight into the phenomenon. We also added Figure D2, which shows which regularities the opponent exploited in each model.
AuthorsOpenReview ↗

Response to Reviewer ADg6

We appreciate the reviewer’s thoughtful evaluation of our work including its strengths, and their suggestions for improvement.

W1 & Q1: Thank you for this suggestion, which helps separate the effect of strategic information from that of prescribing a strategy. We ran a new "opponent-only" condition that adds a single sentence to the minimal prompt revealing the adversary, without naming the game, warning about patterns, or prescribing a strategy, and that keeps the minimal condition's outcome-prediction objective (Section 4.4.2; Appendix I). We compared it with seed-matched minimal and informed sessions from our temperature replication at requested temperature 1.

Revealing the opponent changed how models played without changing how often they won. Alternation rose in all seven models (mean +20.8 percentage points, p < .001), and the lag-1 coefficient predicting the model's current choice from its previous one went from positive to negative (mean of the per-model medians +0.43 to −1.56), while win rates were essentially unchanged (−0.6 percentage points, 95% CI [−4.5, +3.3]). This appears to be because the models' predictability changed in form rather than extent: Proteus shifted from predicting repetition on 35% of minimal-prompt rounds to predicting switches on 43% of opponent-only rounds, winning about 59% and 61% of those rounds, respectively.

The Brown–Rosenthal regressions show the same shift. The models' own past choices predicted their current choice in more sessions (18/28 to 25/28), while the opponent's past choices did so in fewer (28/28 to 17/28). Under the informed prompt, dependence on the opponent nearly disappeared (2/28 sessions), and win rates fell by 9.1 percentage points on the same seeds. One interpretation, in terms of the distinction drawn by von Neumann and Morgenstern (1944) between predicting the opponent and avoiding being predicted, is that revealing the opponent moved models from the former toward the latter, and the informed prompt moved them further.

The performance decline thus required the full informed prompt, and our ablations point to its prescription of the equilibrium strategy. This supports the reviewer's reading: the main result reflects a failure to implement the prescribed strategy under this framing, not evidence that revealing the strategic setting is itself always harmful.

W2 & Q2: We acknowledge limitations on the scope of our conclusions, as the reviewer notes. We have nonetheless expanded the scope in several ways.

First, we ran a new replication with requested temperatures 0 and 1 (Section 4.4.1; Appendix H), addressing the reviewer's point that the earlier temperature checks did not compare the two conditions. The informed-prompt disadvantage was −13.6 percentage points at temperature 0 and −9.9 at temperature 1, with six out of seven models declining at the latter setting, and the models' failure patterns were closely correlated across settings. For three models, the serving endpoints did not apply the temperature parameter, which we now report.

Second, we tested variations on the game itself. In a four-action version of matching pennies, informed prompting reduced win rates in all seven models (Section 4.4.3; Appendix J), though with more marginal bias than in the binary game. When the model played the mismatching rather than the matching role, the framing disadvantage appeared in both roles (Appendix G).

Third, to clarify the range of regularities Proteus exploits, we added Figure D2, which classifies its predictions as side bias, predicted switching, or predicted repetition, and reports the associated win rates. The results reveal substantial heterogeneity across models: Proteus primarily exploited over-alternation in Sonnet and Haiku, but side biases in GPT-5.4, GPT-5.4-mini, and Gemini 3.1 Flash Lite, and under minimal prompting it more often predicted repetition. We agree that Proteus is not the only possible adaptive opponent. It provides an interpretable means of detecting and exploiting multiple forms of predictability without assuming in advance which regularities a model will exhibit. These results do not establish that the findings generalize to opponents with other learning rules, which we now note in our limitations.

W2 & Q3: Thank you for this point about interpreting the regression. We agree that it should be read descriptively rather than as evidence of causal mediation. We have revised Section 4.2 accordingly. The main point the regression illustrates does not depend on it: the two most exploited models, Sonnet and Haiku, kept near-balanced action frequencies, and Proteus acted mostly on predicted switches against them (Figure D2), so their losses arose from sequential dependence that frequency-based evaluation would not detect.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.