acceptodds
Under review as a conference paper at ICLR 2027

Fool Me Once: Scaling Interaction Reveals How LLMs Adapt to and Exploit One Another

12 views

Abstract

Large Language Models (LLMs) act strategically toward the models they interact with, adapting to an opponent's behavior and exploiting what it cannot see. We scale test-time interaction up to 100 turns in matrix games grounded in real-world harms, where LLMs play against fixed strategies and against each other. Before a history of interactions exists, models cooperate in 81.6% of first turns. After a single observed interaction, their cooperation rises to 92.3% against an always-cooperating opponent but falls to 45.8% against an always-defecting one. Furthermore, withholding a player's decision history from its opponent encourages exploitation in games that reward defection and increases cooperation in games where mutual cooperation creates the best outcome for both players. Over 100 turns of Chicken, withholding this history raises outcomes where the player defects while its opponent cooperates from 4.5% to 69.0% of turns, and telling the opponent that the history is hidden does not undo this. A model's first action therefore cannot reveal how it will respond to other models as interactions accumulate. Long-horizon behavior is already difficult to evaluate, and interacting models add strategic complexity that single-model evaluations miss. Thus, pre-deployment safety evaluations must test models against other models, including adversarial and competitive ones, over long interaction horizons.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multi-agent safety, LLM evaluation, repeated games, game theory, information asymmetry, test-time interaction, long-horizon evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dvQvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies LLM behavior in repeated, harm-framed Prisoner’s Dilemma, Chicken, and Stag Hunt games. Scripted-opponent experiments measure adaptation, while same-model self-play tests how asymmetric access to action histories changes cooperation and exploitation over up to 100 turns.

Critical strengths

The Chicken result is substantial: exploitative outcomes increase from 4.5% to 69.0% over turns 2–100. The effect appears across all four long-horizon models and persists when the disadvantaged player is informed about the missing history.

Comparing three incentive structures is useful. The per-model results and scenario-level confidence intervals give concrete evidence of behavior that first-turn evaluations miss.

Critical weaknesses

The history intervention both hides Player 1’s actions from Player 2 and explicitly tells Player 1 that those actions cannot be observed. Their separate contributions are unresolved, limiting the explanation of why exploitation increases.

The higher-stakes interpretation is weakly supported. Severity labels are LLM-assigned, and level 9 contains only nine scenarios (Appendix R). Narrative and domain differences remain confounded with severity.

Questions for authors

Does exploitation still increase when Player 1 is not explicitly told that its history is hidden?

Does the severity association remain within risk domains, with scenario-level uncertainty, or should Section 4.4 present it as a descriptive association?

2 comments

TL;DR

  • What happens when the visibility notice is removed from asymmetric history experiments? In Chicken games, (D, C) outcomes account for 0.5% of turns without the notice. In PD games, (D, C) outcomes made up 6.8% of turns without the notice.
  • Changes to the severity claim. We removed the broader claim about cooperation as scenario stakes rise.
  • Comments on experimental design. We clarified the wording in the abstract and methods to make it explicit that the question we wanted to answer with the asymmetric history experiment is how models behave when they know that they have privileged information.

W1/Q1: Does exploitation still increase when Player 1 is not explicitly told that Player 2 cannot see its (Player 1’s) past decisions?

A: We ran two additional 10-turn asymmetric self-play experiments where we (1) ablated the visibility notice from Player 1’s prompt and (2) used the fixed strategy prompt, which does not mention visibility at all, in the asymmetric setting (see §4.2, Appendix Q, and Table 34). The visibility notice first appeared on turn 2 in our original asymmetric history experiment.

1. From turns 2-10 in Chicken games, (D, C) outcomes account for 0.5% of turns without the notice and 19.8% with it. In PD games, (D, C) outcomes made up 6.8% of turns without the notice and 22.8% with it.

2. From turns 2-10 in Chicken games, (D, C) outcomes account for 1.09% of turns with the fixed-strategy prompt. In PD games, (D, C) outcomes made up 7.15% of turns with the fixed-strategy prompt.

We were specifically interested in how models behave when they are aware that they have privileged information that is unique to them. The visibility notice was a key part of our experiment and we don’t believe we obscured this. However, we do acknowledge that, in some places, our presentation of findings could be interpreted as making a more general point outside the scope of our work. Thus, we have made the scope of our conclusions more explicit. Without the visibility notice we do not expect Player 1 to discover that it has privileged information given Player 2’s behavior in both PD and Chicken Games. For reference in PD, Player 2 cooperates in 87% of turns without the disclosure and 86.6% with it, and in Chicken, it cooperates in 99.6% of turns without the disclosure and 99.4% with it.

W2/Q2: How should we interpret the results by severity?

A: We revised §4.4 and the conclusion to remove the more general claim about cooperation rates in higher-stakes scenarios given the fact that the point is not central to our paper and that there are too few level-9 scenarios for us to make any conclusive claims.

We hope these changes address your key concerns about clarity and soundness. We would appreciate it if you would reconsider your overall score.

We wanted to follow up and see whether the experiments without the visibility notice and the revisions to the severity claim address your concerns. Thank you again for your time.

Reviewer_jxT9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

LLMs play repeated Prisoner's Dilemma, Chicken and Stag Hunt framed by GT-HarmBench harm scenarios, for up to 100 turns. They face scripted opponents and themselves, under symmetric or asymmetric history. Under asymmetric history, Player 1 sees both histories and is told that Player 2 cannot see its moves. Models condition on the opponent after one turn, and asymmetric history raises (D, C) outcomes (Chicken: 4.5% to 69.0% over 100 turns). Telling Player 2 about the asymmetry does not help. The authors conclude that safety evaluations must include long-horizon multi-agent play.

Critical strengths

A timely question with a clean, readable design: scripted anchors, cluster-bootstrap intervals, and reported refusals.

The disclosure arm is interesting. Informed Player 2s cooperate more in the Prisoner's Dilemma, not less.

Critical weaknesses

The exploitation result is confounded with an explicit cue. Player 1 is told "The other player cannot see your actions." Uninformed Player 2 then acts as a near-unconditional cooperator (97.7% in Chicken). Yet in Experiment 1 the same models cooperate most against a literal Always Cooperate opponent when no such sentence is given. The same strategic situation yields cooperation without the sentence and defection with it. So the effect is a response to a statement that retaliation is impossible, not exploitation discovered through interaction. Needed controls: (a) asymmetric information without the statement; (b) symmetric information with the false statement; (c) the statement against scripted Always Cooperate.

Moreover, the prompt casts the model as the user's advisor and ranks (D, C) as the user's best outcome. Choosing it when retaliation is impossible is a best response, and it is not shown to be a safety failure.

The novelty is limited and some conclusions outrun the design. Without monitoring, reputation cannot sustain cooperation. Every reported direction, including the Stag Hunt reversal, is what standard repeated-game reasoning predicts. Conditional cooperation in LLMs is already documented (Akata et al.; Fontana et al.). The 100-turn runs mainly show that effects established by turn 25 persist. The CoT-monitoring claim is unsupported: the judges see OpenAI reasoning summaries, and Anthropic models had thinking off. "Higher stakes reduce cooperation" rests on nine level-9 scenarios with a different game mix.

Self-play only. One episode per cell. Mixed temperatures. A same-family LLM judge. Utilitarian accuracy ignores third-party harm.

Questions for authors

Can you run controls (a)-(c) above?

Why do models cooperate at about 92% against scripted Always Cooperate but defect against an LLM that cooperates at about 98% under asymmetry, if not because of the sentence?

Do the effects persist when the prompt does not rank outcomes for the user?

Please stratify the severity analysis by game.

10 comments

TL;DR

  • Control A. In an asymmetric history setting, we removed the sentence telling Player 1 that Player 2 can’t see its past actions. In Chicken games, exploitation was 0.5% without this notice compared to 19.8% with it (Appendix Q, Table 34).
  • Control C. When giving Player 1 the asymmetric history visibility notice against an always cooperative opponent, it cooperated 8.1 pp less than without the notice. Player 1 cooperated 11.6 pp less often against an LLM opponent than against an always cooperative opponent (both settings feature the visibility notice), despite the LLM opponent cooperating on 99.4% of turns (Appendix P, Table 33).
  • Repeated Play. Defection is strictly dominant in one-shot PD. In our asymmetric experiments, Player 1 cannot elicit reciprocity via cooperation because Player 2 cannot see its (Player 1’s) past actions. If Player 1 was maximizing its payoff, we would expect defection on every turn. Despite this, Player 1 cooperated on 56.3% of turns 2-10 in asymmetric self-play (Table 9). In PD, Player 2 cooperated 13.4 pp more often after we told it about its information disadvantage (§4.3).
  • Severity. We removed the broader claim about cooperation in higher stakes scenarios (§4.4 and Conclusion) and instead report the differences between history conditions separately by game and severity level (Appendix T, Table 40).

W1/Q1: Visibility notice and controls (a)-(c).

A: The asymmetric history experiments in our paper explicitly evaluate what models do when they know they have privileged information. As described in our methods, we directly tell Player 1 that Player 2 cannot see its past actions.

  • Control (a). We ran control (a) where we omitted the visibility notice from the asymmetric self-play prompt (Appendix Q). In Chicken, exploitative outcomes accounted for 19.8% of turns with the disclosure and 0.5% without it (Table 34). In the same Chicken scenarios, Player 2 cooperated on 99.6% of turns in both symmetric self-play and control (a). Whether models can learn that they have a competitive advantage (in our case privileged information) on their own as interactions accumulate is an interesting question, but separate from what we chose to evaluate. Given Player 2’s near perfect cooperation in both symmetric self-play and control (a), we think it would be nearly impossible for Player 1 to discern that it has an information advantage based solely on Player 2’s behaviour, and at no point expected it to do so.
  • Control (c). We ran control (c) where we give Player 1 the same visibility notice against a fixed, always cooperative opponent that is received in asymmetric self-play experiments (Appendix P). Against an always cooperative opponent, Player 1 cooperated 8.1 pp less often with the visibility notice in its prompt than without it.
  • Control (b). We chose not to run control (b) given that it tests what Player 1 does when falsely told its actions are hidden, which we believe introduces additional confounds related to Player 2’s behavior and does not answer the concerns you raised.
  • Abstract. In response to your comments, we revised the abstract to define asymmetric history and clarify that the 69.0% exploitation rate in Chicken is specifically from runs where Player 1’s past actions are hidden from Player 2 and Player 1 was aware that its actions were hidden.

W1/Q2: Why does Player 1 behave differently against Always Cooperate and an almost always cooperative LLM.

A: Player 1 cooperates 11.6 pp (95% CI [8.3, 15.0]) less often in asymmetric self-play than against always-cooperate with the visibility notice. In asymmetric self-play, Player 2 cooperated on 99.4% of turns (Appendix P). Thus we find a significant gap in cooperation against an LLM opponent vs a fixed-strategy, over the same set of scenarios, that cannot be explained by the presence of the visibility notice alone. Additionally, we firmly believe that these experiments cannot be categorized as “the same strategic situation” because of the difference in opponent policies, even when their observed cooperation rates are similar.

Reviewer_GhEiOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors presents a series of results obtained using LLM agents based on the GT-HarmBench benchmark. The authors present a series of results (such as the availability of symmetric vs asymmetric history). The paper reads like a presentation of additional experiments of games from GT-HarmBench (which already presented a series of very similar results of agents playing games in "harmful" situations) under different conditions. The mapping to reality of the setup is difficult to understand (in a game you know the move of your adversary).

Critical strengths
  • The general area of understanding the behaviour of LLMs in game settings is indeed interesting.
Critical weaknesses
  • Unfortunately, there have been a large number of papers in this space. Some of the papers are also cited by the authors. The main issue is that most of these papers are essentially variations of existing works. They are well-executed, but they are not particularly insightful or novel. This paper is in this category: its novelty is extremely limited. There are several papers about interacting agents (in different scenarios and situations). In fact, essentially, the authors take an existing benchmark (GT-HarmBench) and run some repeated games under a variety of mildly different configurations. The idea of testing LLM agents in "harmful" situations is presented in the original GT-HarmBench itself. This is the type of work that I consider ok for a coursework for a University module, but not sufficient for publication at ICLR. In general, as a community I think we should refrain from publishing variations of existing work just to have an additional paper at ICLR in my opinion. It is sufficient to check Table 1. There are variations with respect to the existing papers, but there is a need for a new ICLR paper presenting these results? This might be ok potentially for a workshop, even if, also in workshops, I would prefer to see interesting new results and not extensions/applications of existing works.

  • The choice of having models playing against only fixed strategies appear very limiting. The interesting aspect is probably to check what happens when you have two LLMs playing against each other. Moreover, the choice of having asymmetric history is also not completely easy to understand: if I understood correctly one agent does not have access to the history. Which practical situation are you mapping? If you play a game, by definition you know the move of your adversary.

  • I have substantial concerns about the interpretation of the statistical results presented by the authors: the results presented in Figures 5 and 6 have overlapping confidence intervals. It is not possible to derive clear observations from these results (they are essentially statistically indistinguishable). The results are essentially non-conclusive (hence my "Soundness" score).

Questions for authors

I don't have specific questions for the authors that could materially affect my assessment. My concerns are about the actual topic/design of the experiments and the evaluation of the results as discussed in weaknesses.

3 comments

TL;DR

  • Experimental design. In experiments 2-4 LLMs play against one another for 10 or 100 turns (§3, §4.2, §4.3).
  • What situation are we modeling? In games with imperfect monitoring, players can have different or unequal access to information. Financial markets are one example. Kriegspiel is another. We have included a non-exhaustive list of games and practical scenarios that, like our asymmetric history experiments, feature imperfect monitoring.
  • Statistical Significance. Figure 5 now shows how much exploitation differs between symmetric and asymmetric history experiments (§4.3) and figure 6 now shows how much Player 1’s cooperation differs between history conditions at each severity level and in each risk domain (§4.4). We find the statistical significance of these results more easy to assess.

First, unless we are misinterpreting your comment, you state that a key limitation of our paper is that models only play against fixed-strategies and that an interesting result would involve two models playing one another. We do exactly this in experiments 2-4 (the “self-play” experiments) where two instances of the same LLM play each other for 10 or 100 turns (§3, §4.2, and §4.3). We use our early experiments involving play against programmatically fixed opponents to establish a basic understanding and evaluation of LLM play in iterated games, which we believe is foundational for our later LLM v. LLM games.

Additionally, we find asymmetric history access to be especially relevant and practical. To begin, we disagree with the notion that if you play a game that by definition you know the move of your adversary. Of course this is true for games with perfect monitoring but there are also well-established classes of games with imperfect information or imperfect monitoring. Kriegspiel is one example of imperfect monitoring. Poker is an example of imperfect information. Fog of war is another practical example.

We find asymmetric information access to be especially applicable to markets like first-price sealed-bid auctions and Erlei and Meub’s work on LLM-Agent Interactions on Markets with Information Asymmetries, which we cite directly in our manuscript. For your convenience, other examples include: (1) Green and Porter’s model of market competition where firms operate on observed market price with no direct observability over their competitors’ production decisions; (2) Atakan and Ekmekci’s simultaneous-move, repeated game, where one player can perfectly observe its opponent’s actions, while the other receives an imperfect signal; (3) Goeschl and Jarke’s 12-round trust game where they compare results when trustors can see their counterpart’s actions versus when those actions are hidden or observable only through costly monitoring. Unequal information is also central to Bayesian games.

An especially relevant example is (4) Redwood’s Ctrl-Z paper, which can be understood as a game between an attacker and a defender. The attacker can propose shell commands and attempts to run malicious code. The defender’s goal is to get the untrusted agent to accomplish its task while preventing attacks. The defender resamples commands without telling the untrusted agent and can use the suspicion scores of discarded commands when deciding whether to audit. The defender can also return to earlier steps and sample commands before deciding if it wants to audit them.

In response to your concerns about statistical significance related to Figures 5 and 6, we have updated figure 5 to highlight the difference in exploitation rates between symmetric history and each of the two asymmetric history conditions and figure 6 to highlight the difference in Player 1 cooperation between symmetric and asymmetric history experiments at each severity level and in each risk domain. We previously reported related comparisons in the appendix.

We hope this clarifies our work.

(1) Green, E. J., and R. H. Porter. 1984. “Noncooperative Collusion Under Imperfect Price Information.” Econometrica 52 (1). 10.2307/1911462.

(2) Atakan, Alp E., and Mehmet Ekmekci. “Reputation in the Long-Run with Imperfect Monitoring.” Journal of Economic Theory 157 (2015): 553–605. 10.1016/j.jet.2015.01.012.

(3) Goeschl, Timo, and Johannes Jarke. “Trust, but Verify? Monitoring, Inspection Costs, and Opportunism under Limited Observability.” Journal of Economic Behavior & Organization 142 (2017): 320–30. 10.1016/j.jebo.2017.07.028.

(4) Bhatt, Aryan, Cody Rushing, Adam Kaufman, et al. “Ctrl-Z: Controlling AI Agents via Resampling.” arXiv preprint, April 14, 2025. arXiv:2504.10374

Thanks for your reply. I will briefly respond to your comments:

  • Self-play assumes that the two agents have the same model (not two different LLMs). In any case, this was a specific sub-experiment (the majority of the experiments are about fixed strategies).

  • I am not sure about the long comment about asymmetric history, which is not relevant for the contents of this paper.

  • The games you cite in your rebuttal are not those studied in your paper. That reply is very confusing: the authors are making claims that are not presented in the paper.

  • When you have overlapping confidence intervals you cannot make claims about differences in performance.

For these reasons, I will keep my score.

Thank you for taking time to review some of our comments. It seems there is some additional confusion related to the contents of our paper that we are more than happy to address.

1. Self-play

We agree that self-play means that both players are the same model. We call our experiments self-play experiments because, in each LLM v. LLM game, both players are the same model. We state this in section §3 Methodology, lines 215-216.

2. Fixed strategies

In any case, this was a specific sub-experiment (the majority of the experiments are about fixed strategies).

We outline 4 experiments. Experiment (1) is the only experiment that involves fixed strategies. Experiments (2), (3), and (4) all involve self-play with LLMs (§3, lines 160–184). In our supplementary materials, you can find turn-level data from our experiments. Our results are derived from 68,040 turns against fixed strategies and 254,280 self-play turns.

3. Asymmetric history

I am not sure about the long comment about asymmetric history, which is not relevant for the contents of this paper.

Asymmetric history is a core part of our paper. Experiments (2), (3), and (4) all involve self-play runs where Player 1 can see both players’ past actions and Player 2 can see only its own (§3, lines 216–223, §4.2, lines 278–286, and §4.3, lines 330–335 and 395–405). We discussed asymmetric history in our earlier reply to answer your question about what practical setup our experiments model and to give examples of how players can have limited access to each other’s actions in games.

Thank you again for engaging with our paper. We hope these details and references are helpful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.