- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces PACA, a repeated matching pennies assay in which an adaptive opponent (Proteus) detects and exploits statistical regularities in an LLM's action history. Seven models from Anthropic, OpenAI, and Google are evaluated under two framings of the same game: a minimal prompt that conceals the strategic context, and an informed prompt that explains the game and prescribes the equilibrium strategy of unpredictable 50/50 play. The headline finding is that informed prompting moves behavior farther from equilibrium and lowers win rates from 45.6% to 34.7% on average across seven models, with the worst models over alternating most strongly while keeping near balanced marginal frequencies. The main experiment runs at temperature 0, with 98 sessions of 200 rounds each, and models accessed through OpenRouter.
Critical strengths
The experimental design has real virtues. The closed loop adaptive opponent is a sensible device, the stay and shift decomposition is informative, and the supplemental analyses, temperature sweeps, label swaps, and lag structure, are thorough. The core finding is genuinely counterintuitive and worth reporting: explicitly prescribing equilibrium play makes execution worse. The methodological recommendation, that evaluations of equilibrium play should test history dependence rather than marginal frequencies alone, is correct and overdue.
Critical weaknesses
The headline result is heavily confounded by the decoding setup. The main experiment is run at temperature 0. Under greedy decoding an autoregressive model cannot produce a mixed strategy by construction; the best it can do is a deterministic policy that looks random, and period 2 alternation is precisely what that looks like. Telling a deterministic system to be unpredictable and observing that it responds with an exploitable cycle is close to tautological. The temperature robustness checks exist, but they sit in an appendix while the abstract carries the temperature 0 numbers. To a first approximation the paper measures what greedy decoding does to a request for randomness, and that phenomenon has been documented extensively since 2023, along with the broader finding that LLMs drift away from mixed strategy equilibria.
That leaves a modest novelty core, one methodological device plus one framing effect, measured in a single 2 by 2 game. Matching pennies is the simplest possible zero sum game, and it is a very narrow base from which to generalize about LLM strategic behavior. Reproducibility is also weak by construction: all models are accessed through OpenRouter with versions drifting across the access window, and the paper itself tables model provenance dates because the exact weights will never be recoverable; the 98 sessions of 200 rounds are a modest sample for the lag 1 to 4 history dependence claims. Finally, on venue fit, this is a behavioral science study of API models with no method, no model, and no theory beyond textbook mixed strategy equilibrium. It is competently executed, and parts of it would interest a cognitive science or behavioral economics audience, but as an ICLR contribution it is analysis in search of a machine learning result.
Questions for authors
- Does the informed framing effect on win rate survive as the headline effect when the main experiment is run at temperatures where a mixed strategy is even representable, with session counts matched to the temperature 0 study?
- Can you demonstrate the effect in at least one game with a richer action space, and does Proteus beat a fixed iid 50/50 baseline as an opponent in that setting?
- What is the marginal contribution of the adaptive opponent over a simple alternation detecting strategy, given that the dominant failure mode reported is over alternation?
3 comments
Response to Reviewer G3t6 (2/3)
(cont'd)
On sample size: the history dependence we report is not a subtle effect that requires a large sample. It was detected in nearly every individual 200-round session. Under the minimal prompt, the opponent’s recent choices significantly predicted the model’s choices in all 49 sessions across the seven models. Under the informed prompt, the models’ own recent choices significantly predicted their next choice in all 21 Anthropic sessions and 11 out of 14 Gemini sessions (Table D3). In the Anthropic models, the effect is strong enough that a single session alone rules out chance by a wide margin (mean runs z of 7–10 per session). These patterns recurred in the counterbalanced replication and supplemental checks. For comparison, the classic experiments that established these analyses detected serial dependence in sequences of 105 rounds per pair (O’Neill, 1987; Brown & Rosenthal, 1990); our sessions are nearly twice as long.
On venue fit: the paper’s machine-learning contribution lies in both an evaluation method for foundation models and an empirical finding about them, which fall within ICLR’s scope (our primary area is foundation and frontier models). Behavioral and game-theoretic evaluations of LLMs have appeared at ICLR (Bigelow et al., 2024; Huang et al., 2025; Liu et al., 2025) and other ML venues (Jia et al., NeurIPS 2025; Wang et al., ICML 2026).
The method, PACA, evaluates the strategies models actually execute in adaptive interaction. By measuring how predictable their choices are given the history of play, against an opponent that adapts to whatever regularities they produce, it reveals failures that conventional evaluations of equilibrium play, which focus on aggregate action frequencies, overlook. More broadly, it addresses a growing challenge in evaluating agentic systems: determining whether models reliably carry out appropriate strategies when interacting with other adaptive agents. The finding is that explicitly giving an LLM the equilibrium strategy can make the behavior it executes substantially worse than minimal instructions do. This is a counterintuitive result about how language models translate instructions into behavior, beyond the established observation that they struggle to generate random sequences. It bears directly on how LLM agents are prompted, and on monitoring of chain-of-thought: models’ reasoning often stated the right strategy and recognized their departures from it, yet was an unreliable guide to the policy they executed. We do not claim new theory. The standard we apply, unpredictability given the history of play, is textbook; our point is that current LLM evaluations of equilibrium play do not test it.
Q2: Thank you for prompting us to test this. We extended PACA to a four-action version of matching pennies, similar to the hide-and-seek games studied in experimental game theory (Crawford & Iriberri, 2007), where independent uniform play wins 25% of rounds (Section 4.4.3; Appendix J). Against Proteus, a simulated independent uniform player had a win rate of .2495 (95% CI [.2477, .2513]). Informed prompting again reduced win rates, from 23.6% to 14.2% (mean difference −9.4 percentage points, 95% CI [−15.0, −3.8]; t(6) = −4.11, p = .006), with a decline in all seven models. The informed prompt here prescribed independent uniform choice but omitted the fair-coin analogy and the warning about alternation and repetition, so the decline did not depend on those components. The accompanying behavior differed from the binary game: informed prompting increased switching but also produced pronounced imbalances in overall choice frequencies, exceeding the independent-uniform reference range in 27 out of 28 informed sessions. In the most extreme case, Gemini 3 Flash Preview chose STAR on only 0.1% of informed rounds and won 2.0% of rounds. The disadvantage is thus not limited to two-action alternation, though the form of exploitable behavior changed with the action space.
Response to Reviewer G3t6 (3/3)
Q3: Thank you for inquiring about this. We emphasized over-alternation because it is the simplest pattern that cannot be detected from marginal frequencies, and because the models that showed it most strongly had the lowest win rates. However, models differed in which regularity they showed, and we have revised the results and added Figure D2 to make this clearer. Figure D2 classifies every round by the prediction Proteus acted on. Under the informed prompt, Proteus predicted a switch on most rounds only for Sonnet and Haiku (89% and 84%). For GPT-5.4, GPT-5.4-mini, and Gemini 3.1 Flash Lite, it instead acted on a side bias on 83–86% of rounds, winning 60–68% of them, and almost never predicted a switch. Under the minimal prompt, Proteus predicted repetition more often than switching for every model (e.g., 76% of rounds for Opus, 72% for Gemini 3 Flash Preview). Outside Sonnet and Haiku, then, most of what Proteus acted on consisted of predictions an alternation detector never makes, and the four-action game showed yet another profile, dominated by marginal bias (Q2). This variation is why PACA uses a general adaptive opponent: an evaluation must detect whatever regularity a model produces without knowing in advance which it will be.
Bigelow, E., Lubana, E. S., Dick, R., Tanaka, H., & Ullman, T. (2024). In-context learning dynamics with random binary sequences. ICLR 2024. Huang, J.-T., Li, E. J., Lam, M. H., Liang, T., Wang, W., Yuan, Y., Jiao, W., Wang, X., Tu, Z., & Lyu, M. R. (2025). Competing large language models in multi-agent gaming environments. ICLR 2025. Jia, J., Yuan, Z., Pan, J., McNamara, P. E., & Chen, D. (2025). LLM strategic reasoning: Agentic study through behavioral game theory. NeurIPS 2025. Liu, R., Geng, J., Peterson, J. C., Sucholutsky, I., & Griffiths, T. L. (2025). Large language models assume people are more rational than we really are. ICLR 2025.
Response to Reviewer G3t6 (1/3)
We appreciate the reviewer’s thoughtful acknowledgement of the strengths and opportunities in our work.
W1 & Q1: Thank you for this careful consideration of the decoding setup. We have now run an independent replication with requested temperatures 0 and 1, with seven 200-round sessions per model, prompt, and temperature (Appendix H). Win rates dropped by 13.6 percentage points at requested temperature 0 (p = .012) and by 9.9 at requested temperature 1 (95% CI [−20.1, +0.3]; t(6) = −2.36, p = .055; exact Wilcoxon p = .047), with six out of seven models declining at temperature 1. The effect was thus somewhat attenuated but not eliminated. Win rates, marginal bias, and runs z were closely correlated across temperatures (r = .91–.98; Figure H1). Over-alternation in particular persisted under sampling: in the models where the setting took effect, informed-prompt runs z went from 10.8 to 10.0 for Sonnet and from 8.5 to 7.0 for Haiku. A manipulation check also showed that the serving endpoints for Opus, GPT-5.4, and GPT-5.4-mini did not apply the temperature parameter, so for these three models the comparison reflects repeated sampling at provider defaults, and their main-study results were not obtained by greedy decoding either. Restricted to the four models in which the setting took effect, the correlations were .92–.99.
We also note that deterministic decoding does not necessarily imply predictable behavior across rounds in a stochastic interactive environment. Proteus's choices incorporate random draws, and its completed choices enter the model’s subsequent prompts. Even a model using greedy decoding could therefore condition its actions on this external source of entropy, producing less predictable behavior without stochastic token sampling. Consistent with this possibility, under minimal prompting all seven models exhibited significant dependence on Proteus's recent choices, whereas this dependence largely disappeared under informed prompting. We do not take this association as direct evidence that models extracted randomness from the opponent, but it demonstrates why the closed-loop setting cannot be reduced to a deterministic model simply being asked to generate random choices.
W2:
On matching pennies: The reason that we chose matching pennies is because it is the simplest game with only mixed strategy equilibria, and hence the archetypal game that sharply illustrates the distinction between marginal bias and sequential dependence. These properties are exactly why matching pennies has remained such an influential setting for game theory, harkening back to von Neumann and Morgenstern and yet still studied in contemporary research across several fields including economics, biology, and computer science. We agree that there may be strategic phenomena in richer games that matching pennies does not capture, such as non-uniform equilibrium mixtures or highly asymmetric roles, and we now state this explicitly in our limitations. However, our contributions (see below) do not depend simply on generalizing particular behaviors from matching pennies to other games. Our methodological contribution, the standard that evaluations of equilibrium play must test unpredictability given the history and not just action frequencies, and PACA, which implements it, applies to any competitive game whose equilibrium involves mixing, since that requirement is part of all of them. Our empirical finding, that prescribing the equilibrium can make play worse, is a demonstration that this can happen, and the simplest game in which the requirement appears is a natural place to show it. We also report a new experiment that extends beyond it (see Q2).
On version drift: all main-study sessions were run on a single day, and each model returned a single served identifier across both conditions (Tables A1–A2), so no version change was recorded within any comparison. On reproducibility: the counterbalanced replication, run 11 days later, again recorded a single identifier per model and reproduced the main effect (Appendix E). The more recent replication to assess temperature effects also reproduced the finding. As with any evaluation of closed models, the weights themselves cannot be recovered, but we will release full round-level logs, code, and opponent seeds, and PACA can be rerun on any current or future model.