acceptodds
Under review as a conference paper at ICLR 2027

Enhancing Decision-Making with Large Language Models through Multi-Agent Fictitious Play

Abstract

Large language model (LLM)-based multi-agent systems (MAS) have demonstrated great potential in solving tasks with execution complexity, by distributing subtasks across cooperative agents. However, this divide-and-conquer paradigm falls short in mixed-motive environments such as negotiation and competitive markets, where we control only one agent and the others pursue their own payoffs. In these environments, the quality of our strategy depends on how the others respond, so the strategies are mutually dependent and leave no independently solvable subtasks to divide. Reasoning over this dependence within a single LLM call unfolds recursive mutual anticipation, which quickly exceeds the model's capability. Inspired by the classic learning dynamic of Fictitious Play, we propose Multi-Agent Fictitious Play (MAFP), a novel MAS paradigm that converts this recursive anticipation into iterative strategy updates. To obtain a policy for a distinguished agent, MAFP instantiates a simulated agent for all agents. At each iteration, every agent generates a new decision that seeks to maximize its expected utility against other agents' past decisions. After several rounds, MAFP synthesizes a policy from the distinguished agent's history for deployment. Throughout the process, each agent only needs first-order reasoning, as it best responds to a fixed pool of observed decisions. We evaluate MAFP on decision-making tasks that require deciding a strategy for competitive scenarios prior to acting. Across scenarios spanning competitive games and negotiation, MAFP outperforms other multi-round baselines, and retains higher payoff when others adapt against it, validating it as an effective MAS framework for decision-making.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Decision-Making, Large Language Models, Multi-Agent Systems

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YAxSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose Multi-Agent Fictitious Play (MAFP), a multi-agent framework for mixed-motive scenarios inspired by Fictitious Play.

MAFP instantiates a simulated agent for each player by querying a frozen LLM to generate executable policies. At each round, a new strategy is added to the agent’s strategy pool by querying a frozen LLM to generate an approximate best response against the uniform mixture over each opponent’s pool. After K such rounds, the agent’s strategy pool is merged into a final strategy for deployment.

The paper evaluates MAFP on 13 two-player environments (including zero-sum and general-sum games) against 4 baselines (CoT, self-reflection, debate, ToM). For each method and environment, the authors report the tournament score and the Stackelberg value, with MAFP showing the best performance on average across both metrics. The authors additionally test the generalisation of MAFP to multi-player environments (Avalon, Doudizhu, Texas Hold’em) and a long-horizon economic simulation (CoffeeBench) against the same baselines.

Critical strengths
  • The evaluation of MAFP is comprehensive, including environments with varying incentive structures, number of players, and a long-horizon economic simulation. All methods are evaluated in a round-robin schedule to obtain the tournament score, and additionally, the authors report the Stackelberg value to assess robustness against adaptive opponents.
  • The MAFP formalisation is clear and grounded in game-theoretic concepts.
Critical weaknesses
  • [1] and [2] are not cited despite their relevance to the problem setting. While the methodology differs from MAFP, both papers use population self-play where the policies are executable code generated by an LLM.
  • MAFP uses best-of-n for the Best Response (BR) component, which is orthogonal to Fictitious Play, and could be applied to any of the baselines that already condition on opponent policies. None of the baselines incorporate simulation-based feedback during strategy generation, which could inflate the improvement of MAFP with respect to them. This is partly supported by the ablation provided in Tables 1 and 2, where removing best-of-n from the strategy selection process leads to a performance drop on average.

[1] Yoram Bachrach, Edan Toledo, Karen Hambardzumyan, Despoina Magka, Martin Josifoski, Minqi Jiang, Jakob Foerster, Roberta Raileanu, Tatiana Shavrina, Nicola Cancedda, Avraham Ruderman, Katie Millican, Andrei Lupu, and Rishi Hazra. 2025. Combining code generating large language models and self-play to iteratively refine strategies in games. In Proceedings of the 34th International Joint Conference on Artificial Intelligence (IJCAI '25)

[2] Daniel Hennes, Zun Li, John Schultz, and Marc Lanctot. 2026. Code-Space Response Oracles: Generating Interpretable Multi-Agent Policies with Large Language Models. In Proceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS '26)(Extended Abstract). Full version: arXiv:2603.10098.

Questions for authors
  • Could the authors differentiate their proposed framework and contribution from [1] and [2]?
  • Since all experiments have already been repeated for 8 seeds, could the authors include uncertainty measures in all tables and figures?
  • Could the authors provide additional baselines that make use of simulation-based feedback? For example, for the CoT baseline, generating multiple executable policies, and selecting the policy that obtains the highest payoff in simulated matches against the others.

[1] Yoram Bachrach, Edan Toledo, Karen Hambardzumyan, Despoina Magka, Martin Josifoski, Minqi Jiang, Jakob Foerster, Roberta Raileanu, Tatiana Shavrina, Nicola Cancedda, Avraham Ruderman, Katie Millican, Andrei Lupu, and Rishi Hazra. 2025. Combining code generating large language models and self-play to iteratively refine strategies in games. In Proceedings of the 34th International Joint Conference on Artificial Intelligence (IJCAI '25)

[2] Daniel Hennes, Zun Li, John Schultz, and Marc Lanctot. 2026. Code-Space Response Oracles: Generating Interpretable Multi-Agent Policies with Large Language Models. In Proceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS '26)(Extended Abstract). Full version: arXiv:2603.10098.

4 comments
AuthorsOpenReview ↗

Response to Reviewer YAxS

W1 & Q1: Missing citations of [1] and [2], and the difference of MAFP from them.

A: We thank the reviewer for pointing out these related works. LLM-PSRO [1] and CSRO [2] build on PSRO, a framework for computing approximate equilibria, and replace its best-response oracle with an LLM that writes policies as code for interpretability. We agree that MAFP appears similar to [1] and [2], since all three use an LLM as the best responder. However, MAFP differs from them in two aspects:

Different problem. [1] and [2] inherit the problem of PSRO, namely game solving. The target is a Nash equilibrium, which is the well-defined solution of a two-player zero-sum game, and both works are evaluated on such games. In contrast, MAFP may be applied to general-sum environments and is not designed to find an equilibrium per se. Our goal, as stated in Section 3, is to design an effective strategy for our own agent in the setting where we control only this agent, while the other agents pursue their own goals and may adapt to our strategy.

Different method. Although all three use an LLM as the best responder, what the LLM responds to is different. At every iteration, [1] and [2] explicitly construct a payoff matrix over the current population, which requires playing its policies against each other, and solve this matrix game for an equilibrium. The LLM then best responds to this equilibrium mixture. MAFP does not construct a payoff matrix or solve a matrix game. Following fictitious play (FP), the LLM of each agent directly best responds to the empirical history of the other agents, namely the uniform mixture over all of their past strategies (Eq. 5).

Our contributions. (1) We abstract the learning dynamic of classical FP into a principle, namely replacing the complex mutual anticipation with iterative updates. This learning dynamic can exist outside of game solving. (2) We design on this basis an LLM-based multi-agent framework for strategy improvement in mixed-motive environments. (3) We validate the effectiveness of this framework on two-player zero-sum games, general-sum games, multi-player games, and an economic simulation, which demonstrates the value of this learning dynamic for LLM-based multi-agent systems. Such an application can inspire research on mixed-motive multi-agent systems, beyond the fully cooperative LLM-based agents that are widely adopted.

We will cite and discuss [1] and [2] in Section 2.


W2 & Q2 & Q3: Best-of-nn selection as a confound, baselines with simulation-based feedback, and uncertainty measures.

A: We thank the reviewer for the suggestions. We apply best-of-nn selection to every strategy generation and update step of all methods, including the aggregation step of MAFP, and adjust the number of iterations to achieve a matched-budget comparison. We then rerun both the tournament and the adaptive-opponent evaluation on the latest strategies of all methods, and report the uncertainty of the metrics:

MethodTSSV
CoT-BoN0.187 ± 0.0180.030 ± 0.030
SR-BoN0.181 ± 0.018−0.024 ± 0.029
Debate-BoN0.216 ± 0.0200.027 ± 0.029
ToM-BoN0.111 ± 0.026−0.063 ± 0.024
MAFP-BoN0.317 ± 0.0220.098 ± 0.027

The results show that MAFP ranks first on both metrics under matched budgets. We further run a statistical test between MAFP and the second-best baseline on each metric. On TS, MAFP leads Debate-BoN by +0.101 ± 0.026 (pp = 0.003). On SV, MAFP leads CoT-BoN by +0.068 ± 0.039 (pp = 0.061). This indicates that noise alone would produce a lead at least this large with a probability of 0.3% on TS and 6.1% on SV. These results further demonstrate the effectiveness of MAFP. In the revision, we will replace the current comparison with this matched-budget comparison suggested by the reviewer, with uncertainty estimates.


References

[1] Bachrach, Yoram, et al. "Combining Code Generating Large Language Models and Self-Play to Iteratively Refine Strategies in Games." IJCAI Demo Track (2025).

[2] Hennes, Daniel, et al. "Code-Space Response Oracles: Generating Interpretable Multi-Agent Policies with Large Language Models." arXiv preprint arXiv:2603.10098 (2026).

W1, Q1. Thank you to the authors for their comments. As acknowledged in my review, the methodology is different. However, with respect to the problem setting, Fictitious Play is a special case of PSRO, and it was formulated as an equilibrium finding method. The paper’s angle on the problem may differ, but CSRO also evaluates the solutions against external opponent populations to assess their robustness.

W2, Q2 & Q3. Thank you to the authors for the provided ablation with best-of-n being applied to all methods. Firstly, it is not entirely clear from the rebuttal what CoT-BoN and SR-BoN actually implement, since they do not have a predefined pool of opponents. Could the authors clarify this? Secondly, my question was whether MAFP’s reported advantage would survive if the other baselines received the same simulation feedback that MAFP already had. What I expected was a comparison of the baselines with BoN against MAFP as presented in the paper.

Since the tournament pools differ, we cannot compare the reported TS in Table 1 for the version of MAFP presented in the paper (only best-of-on for the BR component) with the BoN baselines. However, Table 2 reports an SV metric of 0.034 for MAFP, which falls within the interval reported for both CoT-BoN (0.030) & Debate-BoN (0.027), assuming the SV evaluation protocol remained unchanged. If so, MAFP’s advantage may partially stem from the simulation feedback rather than the method itself, which was my original concern.

AuthorsOpenReview ↗

Follow-up Response to Reviewer YAxS (1/2)

Dear Reviewer YAxS,

Thank you for the quick response and the specific questions. We clarify them below.

W1, Q1

Thank you for pointing out the specific concern. Here we clarify the difference between MAFP and CSRO.

  • Regarding Fictitious Play (FP) being a special case of PSRO. It is indeed true that the original PSRO paper (not LLM-PSRO nor CSRO) is presented as a framework that includes a broad range of algorithms that span both (i) double oracle and (ii) classical FP. However, when used in the context of neural policies, the double oracle (i.e., strategic generation) variant is almost always what is referred to. This is true for CSRO as well.

  • Consequently, CSRO requires the explicit solving of restricted games (with actions being the added code strategies per CSRO iteration), which in turn requires one to run simulations/sampling to obtain the head-to-head performance between all pairs of code strategies. In contrast, MAFP does not require these potentially costly pairwise simulations; the LLM is tasked, in context, to find an approximate best-repsonse to the average of the opponent's past code policies.

  • The distinction between CSRO and MAFP extends past methodological details when moving beyond 2-player-zero-sum (2p0s) games. CSRO is focused on finding Nash equilibrium (NE) in 2p0s games. However, in general-sum games, NE are not necessarily "optimal" in the same minimax/maximin manner as with 2p0s games and equilibrium quality/selection is a huge challenge. Hence CSRO as a "NE-solver" may not even lead to a desirable outcome. Indeed, the goal of MAFP in non-2p0s games (Section 3) is not necessarily to find NE, but to apply the simple FP dynamic and to show that practically, it yields strong performance on different metrics (TS and SV, the latter of which corressponds essentially to exploitability for 2p0s games).

  • Regarding "CSRO also evaluates the solutions against external opponent populations to assess their robustness." We argue that our evaluation is actually more thorough than CSRO, both in fidelity and breadth.

    Fidelity when evaluating on 2p0s games. CSRO was only evaluated in repeated Rock-Paper-Scissors and repeated Leduc poker against a catalog of predefined opponent strategies. For repeated RPS, the opponents are 43 hand-coded heuristic bots, including randbot, rockbot, copybot, and iocainebot. In repeated Leduc poker, the only 3 opponents are a Nash strategy computed by CFR+ on the single-hand game, AlwaysCall, and AlwaysFold. The majority of these opponents are very simple (i.e., they are unlikely to be near Nash nor best-response to any strategy), such as rockbot, AlwaysCall, and AlwaysFold, with the Nash strategy being an exception. Performing well against these predefined strategies does not imply a NE of the game (good performance against a NE does not imply low exploitability). In contrast, evaluating MAFP under SV means that we play against an (approximate) best-response; in other words, we are actually trying to measure exploitability --- the gold standard for 2p0s games.

    Breadth. As mentioned, CSRO only focused on 2p0s games. In contrast, our evaluation for MAFP includes important general-sum environments like Negotiation and Buy-Sell. This is where "external opponent populations" come in useful. One of our key emprical contributions is that our method performs well against a variety of "reasonable" opponent (e.g., those obtained from competing methods like TOM), rather than extremely simple predefined strategies used in CSRO, but also against a best-responding opponent (note that for general-sum environments, an opponent best responding may not be "bad" for us).

We hope you can recognize the contribution of MAFP. We will add this discussion to the revision.

AuthorsOpenReview ↗

Follow-up Response to Reviewer YAxS (2/2)

W2 & Q2 & Q3

Thank you for the specific questions. We have supplemented the experiment as suggested.

Implementation: Regarding the experimental design in our previous response. Since not all methods involve opponents, e.g. CoT, SR, and Debate, we let the LLM generate an opponent strategy to apply the BoN selection, and select the candidates against it. We adjust the number of iterations of each method so that all methods use the same number of LLM calls.

Since an additional opponent is introduced as a reference for the selection, we considered that, for a fair comparison, the BoN selection should also be applied to the aggregation step of MAFP, which does not involve opponents.

Supplemented experiments: Following your suggestion, we remove the selection from the aggregation step of MAFP, so that it is the version presented in the paper, and re-adjust the number of iterations of the other methods to achieve a matched budget. The results are as follows:

MethodTSSV
CoT-BoN0.196 ± 0.0210.015 ± 0.030
SR-BoN0.191 ± 0.021−0.036 ± 0.030
Debate-BoN0.226 ± 0.0230.019 ± 0.033
ToM-BoN0.115 ± 0.033−0.077 ± 0.023
MAFP0.262 ± 0.0170.034 ± 0.027

The new results show that MAFP still outperforms the other baselines on both TS and SV under matched budgets. The other methods all improve with the BoN selection against opponent strategies, but the design of MAFP, in which the strategies of multiple agents co-evolve, produces strategies of higher quality.

We will use these results in the revision.

We hope that our additional results and clarifications address your concerns. We would greatly appreciate it if you could reconsider your assessment, and we are happy to discuss further if you have any other concerns.

Sincerely,

The Authors


We attach the full results here.

TS

MethodTicTacNimConn4PigBrkThruKuhnPkLiarDiceIPDBlindAucNegotBuySellUltimatResExchAvg.
CoT-BoN0.037 ± 0.0130.125 ± 0.0490.089 ± 0.1730.021 ± 0.0460.280 ± 0.179−0.024 ± 0.0260.173 ± 0.0380.452 ± 0.0160.251 ± 0.008−0.269 ± 0.150−0.067 ± 0.1780.497 ± 0.0500.988 ± 0.0120.196 ± 0.021
SR-BoN0.042 ± 0.0160.086 ± 0.0350.299 ± 0.1280.086 ± 0.0360.119 ± 0.108−0.043 ± 0.0300.012 ± 0.0910.468 ± 0.0140.249 ± 0.010−0.068 ± 0.157−0.163 ± 0.0930.446 ± 0.1020.952 ± 0.0360.191 ± 0.021
Debate-BoN0.034 ± 0.0200.113 ± 0.0410.095 ± 0.1320.119 ± 0.0670.179 ± 0.1850.012 ± 0.0170.274 ± 0.0520.454 ± 0.0170.246 ± 0.009−0.132 ± 0.2880.046 ± 0.1740.524 ± 0.0370.976 ± 0.0240.226 ± 0.023
ToM-BoN0.045 ± 0.0130.006 ± 0.087−0.524 ± 0.111−0.057 ± 0.102−0.369 ± 0.1410.058 ± 0.051−0.235 ± 0.1140.548 ± 0.0310.255 ± 0.0080.312 ± 0.1730.096 ± 0.1330.380 ± 0.1000.976 ± 0.0240.115 ± 0.033
MAFP0.046 ± 0.0150.009 ± 0.0920.289 ± 0.053−0.039 ± 0.0550.098 ± 0.0870.065 ± 0.0520.086 ± 0.0980.470 ± 0.0190.255 ± 0.0110.549 ± 0.0590.367 ± 0.0940.241 ± 0.1530.976 ± 0.0240.262 ± 0.017

SV

MethodTicTacNimConn4PigBrkThruKuhnPkLiarDiceIPDBlindAucNegotBuySellUltimatResExchAvg.
CoT-BoN0.000 ± 0.0000.000 ± 0.000−0.009 ± 0.171−0.054 ± 0.066−0.232 ± 0.234−0.281 ± 0.051−0.196 ± 0.1740.410 ± 0.0470.195 ± 0.032−0.669 ± 0.156−0.196 ± 0.2110.230 ± 0.1471.000 ± 0.0000.015 ± 0.030
SR-BoN0.000 ± 0.0000.000 ± 0.0000.000 ± 0.109−0.179 ± 0.066−0.554 ± 0.139−0.308 ± 0.042−0.321 ± 0.1440.361 ± 0.0480.176 ± 0.021−0.479 ± 0.202−0.346 ± 0.0780.252 ± 0.1450.929 ± 0.071−0.036 ± 0.030
Debate-BoN−0.009 ± 0.0090.000 ± 0.000−0.321 ± 0.187−0.107 ± 0.123−0.232 ± 0.196−0.196 ± 0.068−0.214 ± 0.1630.338 ± 0.0610.177 ± 0.026−0.320 ± 0.262−0.186 ± 0.1620.317 ± 0.0571.000 ± 0.0000.019 ± 0.033
ToM-BoN0.000 ± 0.000−0.125 ± 0.125−0.714 ± 0.2650.036 ± 0.108−0.786 ± 0.089−0.205 ± 0.069−0.696 ± 0.1090.200 ± 0.0000.177 ± 0.0350.020 ± 0.202−0.022 ± 0.2040.120 ± 0.1451.000 ± 0.000−0.077 ± 0.023
MAFP0.000 ± 0.000−0.125 ± 0.1250.086 ± 0.172−0.188 ± 0.116−0.531 ± 0.168−0.195 ± 0.056−0.391 ± 0.1040.157 ± 0.0260.185 ± 0.0200.167 ± 0.1730.144 ± 0.0930.130 ± 0.1511.000 ± 0.0000.034 ± 0.027

The results show that, on TS, MAFP attains the highest score in 5 of the 13 scenarios, more than any other method. On SV, MAFP attains the highest score on Conn4, KuhnPk, Negot, and BuySell, and ties for the highest on TicTac and ResExch.

Reviewer_5VSCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MAFP to prepare a policy for one agent before it interacts with other self-interested agents. Using a frozen LLM, the method generates policy programs for all players in simulation, then repeatedly produces candidates against the opponents’ past policies and selects the best candidate by simulated payoff. The target agent’s policy history is finally combined into a program for deployment. Experiments cover 13 two-player scenarios, three multiplayer games, and CoffeeBench; the two-player evaluation includes tournament payoffs and performance against an approximate best-response opponent.

Critical strengths

Representing complete strategies as executable programs allows decision rules to be run repeatedly and scored directly, providing a clear implementation for comparing strategy-design methods. Evaluation spans several game classes and an economic simulation, while the history-pool and best-of-n ablations provide initial evidence for the roles of these components (Tables 1–3).

Critical weaknesses

a.Closely related LLM-PSRO and CSRO methods are neither discussed nor compared, leaving the substantive contribution beyond existing LLM-generated policy-population methods unclear. b. MAFP generates three candidates per round and selects them through simulation, whereas baselines match only iteration counts; without uncertainty estimates, the benefits of the method and additional computation remain insufficiently separated (§5.2, Appendix B.3). c. The final aggregation produces a new program whose equivalence to the historical policy mixture is not established; Figure 4 reports payoffs against the current pool, leaving the effect on exploitability unclear (Eq. 7).

Questions for authors
  1. Does the advantage over the existing baselines hold under matched policy-generation and simulation budgets? How does MAFP compare with LLM-PSRO and CSRO under the same budgets?
  2. How close is the six-round responder to an exact best response in small games such as Kuhn Poker? Are the rankings and payoff gaps across the eight seeds stable when the response budget is increased (Table 2)?
  3. How does the aggregation in Eq. (7) compare with uniformly sampling a historical policy at the start of each game, in terms of payoff and performance against adaptive opponents?
  4. How often do the best-response and aggregation steps fall below the agent’s own pool average in individual games and seeds, and by how much (Figure 4, Appendix A)?
  5. In CoffeeBench, does “cumulative revenue” mean gross sales or net income? How are any transactions among the method’s three controlled agents counted in this metric (Figure 5, §5.4)?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 5VSC (1/3)

W1 & Q1(b): Relation to LLM-PSRO and CSRO, and the comparison with them.

A: We thank the reviewer for pointing out these related works. LLM-PSRO [1] and CSRO [2] build on PSRO, a framework for computing approximate equilibria, and replace its best-response oracle with an LLM that writes policies as code for interpretability. We agree that MAFP appears similar to [1] and [2], since all three use an LLM as the best responder. However, MAFP differs from them in two aspects:

Different problem. [1] and [2] inherit the problem of PSRO, namely game solving. The target is a Nash equilibrium, which is the well-defined solution of a two-player zero-sum game, and both works are evaluated on such games. In contrast, MAFP may be applied to general-sum environments and is not designed to find an equilibrium per se. Our goal, as stated in Section 3, is to design an effective strategy for our own agent in the setting where we control only this agent, while the other agents pursue their own goals and may adapt to our strategy.

Different method. Although all three use an LLM as the best responder, what the LLM responds to is different. At every iteration, [1] and [2] explicitly construct a payoff matrix over the current population, which requires playing its policies against each other, and solve this matrix game for an equilibrium. The LLM then best responds to this equilibrium mixture. MAFP does not construct a payoff matrix or solve a matrix game. Following fictitious play (FP), the LLM of each agent directly best responds to the empirical history of the other agents, namely the uniform mixture over all of their past strategies (Eq. 5).

Our contributions. (1) We abstract the learning dynamic of classical FP into a principle, namely replacing the complex mutual anticipation with iterative updates. This learning dynamic can exist outside of game solving. (2) We design on this basis an LLM-based multi-agent framework for strategy improvement in mixed-motive environments. (3) We validate the effectiveness of this framework on two-player zero-sum games, general-sum games, multi-player games, and an economic simulation, which demonstrates the value of this learning dynamic for LLM-based multi-agent systems. Such an application can inspire research on mixed-motive multi-agent systems, beyond the fully cooperative LLM-based agents that are widely adopted.

We will cite and discuss [1] and [2] in Section 2.

Regarding the comparison under the same budgets, [1] and [2] follow PSRO and address a different problem from ours, as discussed above. We therefore do not treat them as competing baselines.


W2 & Q1(a): The advantage under matched budgets, and missing uncertainty estimates.

A: We thank the reviewer for suggesting improvements to the evaluation.

We apply best-of-nn selection to every strategy generation and update step of all methods, including the aggregation step of MAFP, and adjust the number of iterations to achieve a matched-budget comparison. We then rerun both the tournament and the adaptive-opponent evaluation on the latest strategies of all methods, and report the uncertainty of the metrics:

MethodTSSV
CoT-BoN0.187 ± 0.0180.030 ± 0.030
SR-BoN0.181 ± 0.018−0.024 ± 0.029
Debate-BoN0.216 ± 0.0200.027 ± 0.029
ToM-BoN0.111 ± 0.026−0.063 ± 0.024
MAFP-BoN0.317 ± 0.0220.098 ± 0.027

The results show that MAFP ranks first on both metrics under matched budgets. We further run a statistical test between MAFP and the second-best baseline on each metric. On TS, MAFP leads Debate-BoN by +0.101 ± 0.026 (pp = 0.003). On SV, MAFP leads CoT-BoN by +0.068 ± 0.039 (pp = 0.061). This indicates that noise alone would produce a lead at least this large with a probability of 0.3% on TS and 6.1% on SV. These results further demonstrate the effectiveness of MAFP. In the revision, we will replace the current comparison with this matched-budget comparison suggested by the reviewer, with uncertainty estimates.


References

[1] Bachrach, Yoram, et al. "Combining Code Generating Large Language Models and Self-Play to Iteratively Refine Strategies in Games." IJCAI Demo Track (2025).

[2] Hennes, Daniel, et al. "Code-Space Response Oracles: Generating Interpretable Multi-Agent Policies with Large Language Models." arXiv preprint arXiv:2603.10098 (2026).

AuthorsOpenReview ↗

Response to Reviewer 5VSC (2/3)

W3 & Q3: Comparison of the aggregation in Eq. 7 with uniformly sampling one strategy from the pool, in payoff and against adaptive opponents.

A: We agree that the strategy produced by AggM\mathrm{Agg}_M is not equivalent to the pool mixture. AggM\mathrm{Agg}_M consolidates the strategies in the pool into a single strategy, and our analysis only requires this strategy to perform no worse than the pool mixture (Assumption 2 in App. A). Fig. 4 shows that this holds on average against the pools of the simulated opponents. We further compare the aggregated strategy with the mixture itself, namely uniformly sampling one strategy from the agent's own pool for each match, on TS and SV under the settings of Tables 1 and 2:

Final strategyTSSV
Pool mixture (random sampling)0.269 ± 0.0150.036 ± 0.021
Aggregated strategy AggM\mathrm{Agg}_M (Ours)0.335 ± 0.0160.034 ± 0.027

The results show that the aggregated strategy attains a clearly higher TS than the mixture, namely higher than the average over the strategies in the pool, which indicates that it combines their strengths instead of averaging them. Meanwhile, it attains almost the same SV as the mixture. Note that the mixture randomizes over the four strategies in the pool, so an adaptive opponent cannot tailor its response to any single one of them, and the aggregated strategy reaches the same level with a single strategy. In the revision, we will state in Section 4 that AggM\mathrm{Agg}_M consolidates the pool into a single strategy instead of reproducing the mixture, and add this comparison.


Q2: Closeness of the six-round responder to an exact best response in small games, and stability under a larger response budget.

A:

Larger response budget

We extend the adaptive opponent from 6 to 12 rounds for the five main methods on all 13 scenarios and 8 seeds. The table reports the SV after each round, where round 0 is the initial program of the opponent:

Round0123456789101112
CoT0.2290.019−0.059−0.085−0.111−0.125−0.123−0.135−0.139−0.140−0.144−0.145−0.145
SR0.1830.053−0.019−0.057−0.079−0.104−0.105−0.116−0.116−0.119−0.128−0.128−0.125
Debate0.2600.0570.016−0.024−0.046−0.057−0.062−0.065−0.080−0.089−0.091−0.094−0.099
ToM0.232−0.013−0.076−0.095−0.103−0.115−0.136−0.145−0.161−0.168−0.173−0.174−0.170
MAFP0.3550.1910.1230.0900.0710.0380.0340.0280.0200.0160.0000.000−0.002

For a more intuitive comparison, we also report the gap between the SV at each round and that at round 12:

Round0123456789101112
CoT0.3740.1650.0870.0610.0350.0200.0220.0100.0060.0050.0010.0000.000
SR0.3080.1780.1060.0680.0460.0210.0200.0080.0090.006−0.003−0.0030.000
Debate0.3590.1560.1140.0740.0530.0420.0370.0340.0190.0100.0080.0050.000
ToM0.4020.1570.0950.0760.0670.0550.0350.0260.0090.003−0.002−0.0040.000
MAFP0.3570.1930.1240.0920.0720.0390.0360.0300.0220.0180.0010.0010.000

As shown in the tables, extending the adaptation from 6 to 12 rounds lowers the SV of all methods by a similar amount (0.02 to 0.04). The ranking is unchanged, and the lead of MAFP over the best baseline stays the same (0.096 at round 6 and 0.097 at round 12).

We further replace the model of the adaptive opponent with a stronger one, GPT-6.1-sol:

MethodQwen3.7-Flash (Table 2) SVRankGPT-6.1-sol SVRank
CoT−0.1234−0.1984
SR−0.1053−0.1763
Debate−0.0622−0.1702
ToM−0.1365−0.2565
MAFP0.0341−0.0831

The stronger opponent lowers the SV of every method by 0.07 to 0.12. The ranking of the five methods is unchanged, and MAFP keeps a lead of 0.087 over the best baseline (0.096 in Table 2).

Exact best response on small games

We select three simpler games, Tic-Tac-Toe, Nim, and Kuhn Poker, whose game trees are small enough to compute the exact best response. For each shipped program, we extract its behavioral strategy and compute its expected payoff against the exact best response, which gives the exact SV. The results are as follows, where LLM-based is the SV estimated with the 6-round adaptive opponent (Table 2) on the same games, and Δ is Exact BR minus LLM-based:

MethodLLM-basedExact BRΔ
CoT−0.152−0.139+0.013
SR−0.143−0.154−0.011
Debate−0.115−0.104+0.011
ToM−0.141−0.175−0.035
MAFP−0.107−0.099+0.008

From the results, the average gap between the LLM-based SV and the exact SV is 11% of the exact SV. We note that the exact SV is slightly higher than the LLM-based SV on some methods because the chance events in Kuhn Poker and the sampling error of the 16-match estimate can push the LLM-based SV below its expected value.

AuthorsOpenReview ↗

Response to Reviewer 5VSC (3/3)

Q4: How often and by how much the best-response and aggregation steps fall below the agent's own pool average.

A: As suggested, we count how often each step of Fig. 4 falls below the average of the agent's own pool:

StepMarginBelow average
Best response, iteration 1+0.221 ± 0.02015/104
Best response, iteration 2+0.202 ± 0.02014/104
Best response, iteration 3+0.180 ± 0.0177/104
Aggregation+0.059 ± 0.02216/104

The results show that the best response falls below the pool average in 11.5% of the samples (36/312) and the aggregation in 15.4% (16/104). Among these samples, the best response and the aggregation fall below the pool average by 0.165 and 0.195 on average.


Q5: Whether cumulative revenue in CoffeeBench is gross sales or net income, and how transactions within a team are counted.

A: The cumulative revenue in Fig. 5 is gross sales. To reflect the overall performance of each method, we sum the gross sales of the three roles controlled by the same method.

Due to time and space limits, we did not present more results in the main text. We have now conducted a more in-depth analysis and report additional results to further examine the advantage of MAFP in CoffeeBench.

The cumulative net income at each checkpoint is as follows:

MethodDay 15Day 30Day 45Day 60Day 75Day 90
CoT867 ± 581,133 ± 1431,917 ± 3512,558 ± 6492,731 ± 8912,248 ± 1,044
SR254 ± 54548 ± 187960 ± 2761,811 ± 3782,338 ± 4051,533 ± 413
Debate−219 ± 46−899 ± 110−1,412 ± 115−1,896 ± 109−2,512 ± 121−3,607 ± 171
ToM−888 ± 44−2,339 ± 55−3,613 ± 90−4,511 ± 164−6,056 ± 158−7,394 ± 204
MAFP1,734 ± 1813,483 ± 4945,871 ± 9617,776 ± 1,0848,860 ± 1,2268,768 ± 1,341

MAFP has the highest net income at every checkpoint, and its margin over CoT grows from 0.9k USD at day 15 to 6.5k USD at day 90. Debate and ToM lose money at every checkpoint.

Meanwhile, we report the gross sales and the net income of each role over the 90 days:

MethodFarmerRoasterRetailerTotal
Gross sales
CoT2,528 ± 2226,073 ± 75415,356 ± 1,30123,957 ± 2,120
SR0 ± 04,809 ± 65117,053 ± 36821,861 ± 728
Debate1,824 ± 49403 ± 310,045 ± 29712,272 ± 329
ToM245 ± 202,999 ± 344450 ± 03,695 ± 334
MAFP6,557 ± 1,0088,291 ± 73723,506 ± 1,47338,354 ± 2,998
Net income
CoT−1,536 ± 76−532 ± 2794,316 ± 7742,248 ± 1,044
SR−2,320 ± 0−1,252 ± 2805,105 ± 2531,533 ± 413
Debate−2,234 ± 8−2,473 ± 31,100 ± 169−3,607 ± 171
ToM−2,122 ± 21−973 ± 214−4,300 ± 0−7,394 ± 204
MAFP995 ± 61462 ± 3717,711 ± 5778,768 ± 1,341

MAFP has the highest gross sales and the highest net income in every role. We will supplement these analyses in the revision.

Reviewer_9ygHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Multi-Agent Fictitious Play (MAFP) to address recursive strategic reasoning in mixed-motive multi-agent environments through iterative strategy updates. The method simulates agents that generate strategies based on their opponents’ previous strategies and combines them into a final policy. The experiments show promising results; however, several issues remain unclear.

Critical strengths
  • The paper addresses an interesting problem in multi-agent systems where agents have different objectives and their decisions depend on each other.
  • The method considers the opponents’ historical strategies rather than only their latest strategies. The ablation results show that this improves performance.
  • The results show improvements over the baseline methods in several games and negotiation tasks.
Critical weaknesses
  • The method is based on the existing Fictitious Play approach, and it is unclear what is new beyond using LLMs for policy generation.
  • In Sec. 4, the authors use AggM to combine several strategies into one final policy. However, it is unclear whether this policy preserves their original behavior.
  • In Section 5.2, the ablation study shows that removing the history pool reduces the average tournament score from 0.330 to 0.160 and the Stackelberg value from 0.034 to -0.099. The authors explain that relying on the opponent’s latest strategy can lead to overfitting. However, even without a history pool, the method still uses the simulated opponent's current strategy.
  • The proposed method relies on the history of simulated opponents’ strategies. However, in real-world settings, an agent may encounter new opponents with no available history or unknown behaviors. It is unclear how well the generated policy would perform in such cases.
  • In Section 4, the authors use a fixed number of iterations to generate and improve strategies. The main experiments use K=3, while K=7 is considered for multiplayer games. Although the results improve with additional iterations, it is unclear how the authors determine the required number of iterations and whether these are sufficient for convergence across different policies.
  • In Sec. 5.1, the Stackelberg values are estimated using an adaptive opponent that updates its strategy over six rounds. However, the authors do not clearly justify why six rounds are sufficient; some opponents may require more rounds to identify weaknesses.
Questions for authors

1-How well would MAFP perform against new opponents with no available history or unknown behaviors, given that the ablation method still uses the simulated opponent's current strategy? 2- How do the authors verify that the generated final policy preserves the behavior of the original mixture of strategies? 3- How do the authors know whether six rounds are enough to identify weaknesses in the generated policies? Would increasing the number of adaptation rounds or using stronger opponents affect the reported results?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 9ygH (1/2)

W1: Novelty beyond using LLMs for policy generation in fictitious play.

A: Classical fictitious play (FP) is a learning dynamic for finding equilibria in games with explicitly given payoffs. Our work studies a different problem, namely how to design the strategy of an LLM-based agent that faces other agents pursuing their own goals (Section 3).

Our contribution is to abstract this learning dynamic into a principle, namely replacing the complex mutual anticipation with iterative updates, and to design on this basis an LLM-based multi-agent framework for strategy improvement in mixed-motive environments.

We validate the effectiveness of this framework on two-player zero-sum games, general-sum games, multi-player games, and an economic simulation, which goes beyond the scope of classical FP for finding equilibria in zero-sum games.


W2 & Q2: Whether the aggregated strategy preserves the behavior of the pool mixture.

A: We would like to clarify that the aggregated strategy is not an exact representation of the pool mixture. AggM\mathrm{Agg}_M consolidates the strategies in the pool into a single strategy, and our analysis only requires this strategy to perform no worse than the pool mixture (Assumption 2 in App. A). Fig. 4 shows that this holds on average against the pools of the simulated opponents. We further compare the aggregated strategy with the mixture itself, namely uniformly sampling one strategy from the agent's own pool for each match, on TS and SV under the settings of Tables 1 and 2:

Final strategyTSSV
Pool mixture (random sampling)0.269 ± 0.0150.036 ± 0.021
Aggregated strategy AggM\mathrm{Agg}_M (Ours)0.335 ± 0.0160.034 ± 0.027

The results show that the aggregated strategy attains a clearly higher TS than the mixture, namely higher than the average over the strategies in the pool, which indicates that it combines their strengths instead of averaging them. Meanwhile, it attains almost the same SV as the mixture. Note that the mixture randomizes over the four strategies in the pool, so an adaptive opponent cannot tailor its response to any single one of them, and the aggregated strategy reaches the same level with a single strategy. In the revision, we will state in Section 4 that AggM\mathrm{Agg}_M consolidates the pool into a single strategy instead of reproducing the mixture, and add this comparison.


W3: The history-pool ablation and its explanation.

A: We would like to clarify that MAFP and its w/o pool variant differ in whether the best response at each iteration is computed against the simulated opponent's latest strategy or against all the strategies it has produced so far. Since the agent maximizes its expected payoff over all the strategies in the pool, it takes every strategy the opponent has used into account, and thus does not overfit to any single one of them. We are happy to explain further if anything remains unclear.


W4 & Q1: Performance against new opponents with no available history.

A: We would like to clarify that MAFP does not use any history of the real opponents. Instead, it is a multi-agent framework in which LLM-based agents improve their strategies through simulated matches before the actual deployment. Our evaluation is conducted exactly in the setting of opponents with no available history. The SR, Debate, and ToM strategies in TS are generated independently and never appear in the MAFP process. The adaptive opponent in the SV evaluation is also generated independently and never appears in the MAFP process.

AuthorsOpenReview ↗

Response to Reviewer 9ygH (2/2)

W5: How the number of iterations KK is chosen and whether it suffices for convergence.

A: We chose K=3K=3 as a trade-off between API cost and performance. As shown in Fig. 3, MAFP keeps improving across iterations and shows the potential for further gains, while the other methods do not improve consistently. To address this concern, we extend the iterations to K=11K=11 on both the two-player and the multi-player scenarios. Following Fig. 3, we run an internal tournament among the per-iteration strategies of each method to examine how its performance changes across iterations. Note that the scores are comparable only within each row, and comparisons across rows are not meaningful. The results on the two-player scenarios are as follows:

MethodK=0K=0K=3K=3K=7K=7K=11K=11Δ11−0\Delta_{11-0}
SR0.2110.1760.1710.195−0.017
Debate0.1540.0910.1600.067−0.087
ToM0.2580.2290.1770.161−0.097
MAFP0.0660.2010.2110.220+0.153

The results show that MAFP is already close to saturation at K=3K=3 on the two-player scenarios (0.201, against 0.220 at K=11K=11).

The results on the multi-player scenarios are as follows:

MethodK=0K=0K=3K=3K=7K=7K=11K=11Δ11−0\Delta_{11-0}
SR0.2150.1520.1750.173−0.041
Debate0.2020.2060.1590.103−0.098
ToM0.2420.1630.0900.180−0.061
MAFP0.0980.1460.2070.211+0.113

The results show that MAFP is close to saturation at K=7K=7 on the multi-player scenarios (0.207, against 0.211 at K=11K=11).

These two results show that our choices of K=3K=3 for the two-player scenarios and K=7K=7 for the multi-player scenarios are reasonable. Meanwhile, MAFP is the only method that improves steadily across iterations, while the multi-round baselines fluctuate or decline.


W6 & Q3: Whether six adaptation rounds are enough, and the effect of more rounds or stronger opponents.

A: We thank the reviewer for raising this concern. We chose six rounds as an affordable budget. We address this concern from two aspects, namely more adaptation rounds and a stronger opponent model.

More adaptation rounds

We extend the adaptive opponent from 6 to 12 rounds for the five main methods on all 13 scenarios and 8 seeds. The table reports the SV after each round, where round 0 is the initial program of the opponent:

Round0123456789101112
CoT0.2290.019−0.059−0.085−0.111−0.125−0.123−0.135−0.139−0.140−0.144−0.145−0.145
SR0.1830.053−0.019−0.057−0.079−0.104−0.105−0.116−0.116−0.119−0.128−0.128−0.125
Debate0.2600.0570.016−0.024−0.046−0.057−0.062−0.065−0.080−0.089−0.091−0.094−0.099
ToM0.232−0.013−0.076−0.095−0.103−0.115−0.136−0.145−0.161−0.168−0.173−0.174−0.170
MAFP0.3550.1910.1230.0900.0710.0380.0340.0280.0200.0160.0000.000−0.002

For a more intuitive comparison, we also report the gap between the SV at each round and that at round 12:

Round0123456789101112
CoT0.3740.1650.0870.0610.0350.0200.0220.0100.0060.0050.0010.0000.000
SR0.3080.1780.1060.0680.0460.0210.0200.0080.0090.006−0.003−0.0030.000
Debate0.3590.1560.1140.0740.0530.0420.0370.0340.0190.0100.0080.0050.000
ToM0.4020.1570.0950.0760.0670.0550.0350.0260.0090.003−0.002−0.0040.000
MAFP0.3570.1930.1240.0920.0720.0390.0360.0300.0220.0180.0010.0010.000

As shown in the tables, extending the adaptation from 6 to 12 rounds lowers the SV of all methods by a similar amount (0.02 to 0.04). The ranking is unchanged, and the lead of MAFP over the best baseline stays the same (0.096 at round 6 and 0.097 at round 12).

Stronger opponents

We further replace the model of the adaptive opponent with a stronger one, GPT-6.1-sol:

MethodQwen3.7-Flash (Table 2) SVRankGPT-6.1-sol SVRank
CoT−0.1234−0.1984
SR−0.1053−0.1763
Debate−0.0622−0.1702
ToM−0.1365−0.2565
MAFP0.0341−0.0831

The stronger opponent lowers the SV of every method by 0.07 to 0.12. The ranking of the five methods is unchanged, and MAFP keeps a lead of 0.087 over the best baseline (0.096 in Table 2).

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer 9ygH,

As the discussion period closes in less than one day, we would be grateful for your feedback on whether our responses have addressed your concerns. We would be glad to clarify any remaining concerns and would appreciate it if you could consider re-evaluating our work in light of these responses.

Best regards,

The Authors

AuthorsOpenReview ↗

Follow-up Before the Discussion Closes

Dear Reviewer 9ygH,

We truly appreciate the time you have devoted to reviewing our paper. As the discussion period is drawing to a close, we wanted to gently check whether our responses have addressed your concerns. We would appreciate it if you could consider re-evaluating our work in light of these results.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.