- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose Multi-Agent Fictitious Play (MAFP), a multi-agent framework for mixed-motive scenarios inspired by Fictitious Play.
MAFP instantiates a simulated agent for each player by querying a frozen LLM to generate executable policies. At each round, a new strategy is added to the agent’s strategy pool by querying a frozen LLM to generate an approximate best response against the uniform mixture over each opponent’s pool. After K such rounds, the agent’s strategy pool is merged into a final strategy for deployment.
The paper evaluates MAFP on 13 two-player environments (including zero-sum and general-sum games) against 4 baselines (CoT, self-reflection, debate, ToM). For each method and environment, the authors report the tournament score and the Stackelberg value, with MAFP showing the best performance on average across both metrics. The authors additionally test the generalisation of MAFP to multi-player environments (Avalon, Doudizhu, Texas Hold’em) and a long-horizon economic simulation (CoffeeBench) against the same baselines.
Critical strengths
- The evaluation of MAFP is comprehensive, including environments with varying incentive structures, number of players, and a long-horizon economic simulation. All methods are evaluated in a round-robin schedule to obtain the tournament score, and additionally, the authors report the Stackelberg value to assess robustness against adaptive opponents.
- The MAFP formalisation is clear and grounded in game-theoretic concepts.
Critical weaknesses
- [1] and [2] are not cited despite their relevance to the problem setting. While the methodology differs from MAFP, both papers use population self-play where the policies are executable code generated by an LLM.
- MAFP uses best-of-n for the Best Response (BR) component, which is orthogonal to Fictitious Play, and could be applied to any of the baselines that already condition on opponent policies. None of the baselines incorporate simulation-based feedback during strategy generation, which could inflate the improvement of MAFP with respect to them. This is partly supported by the ablation provided in Tables 1 and 2, where removing best-of-n from the strategy selection process leads to a performance drop on average.
[1] Yoram Bachrach, Edan Toledo, Karen Hambardzumyan, Despoina Magka, Martin Josifoski, Minqi Jiang, Jakob Foerster, Roberta Raileanu, Tatiana Shavrina, Nicola Cancedda, Avraham Ruderman, Katie Millican, Andrei Lupu, and Rishi Hazra. 2025. Combining code generating large language models and self-play to iteratively refine strategies in games. In Proceedings of the 34th International Joint Conference on Artificial Intelligence (IJCAI '25)
[2] Daniel Hennes, Zun Li, John Schultz, and Marc Lanctot. 2026. Code-Space Response Oracles: Generating Interpretable Multi-Agent Policies with Large Language Models. In Proceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS '26)(Extended Abstract). Full version: arXiv:2603.10098.
Questions for authors
- Could the authors differentiate their proposed framework and contribution from [1] and [2]?
- Since all experiments have already been repeated for 8 seeds, could the authors include uncertainty measures in all tables and figures?
- Could the authors provide additional baselines that make use of simulation-based feedback? For example, for the CoT baseline, generating multiple executable policies, and selecting the policy that obtains the highest payoff in simulated matches against the others.
[1] Yoram Bachrach, Edan Toledo, Karen Hambardzumyan, Despoina Magka, Martin Josifoski, Minqi Jiang, Jakob Foerster, Roberta Raileanu, Tatiana Shavrina, Nicola Cancedda, Avraham Ruderman, Katie Millican, Andrei Lupu, and Rishi Hazra. 2025. Combining code generating large language models and self-play to iteratively refine strategies in games. In Proceedings of the 34th International Joint Conference on Artificial Intelligence (IJCAI '25)
[2] Daniel Hennes, Zun Li, John Schultz, and Marc Lanctot. 2026. Code-Space Response Oracles: Generating Interpretable Multi-Agent Policies with Large Language Models. In Proceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS '26)(Extended Abstract). Full version: arXiv:2603.10098.
4 comments
W1, Q1. Thank you to the authors for their comments. As acknowledged in my review, the methodology is different. However, with respect to the problem setting, Fictitious Play is a special case of PSRO, and it was formulated as an equilibrium finding method. The paper’s angle on the problem may differ, but CSRO also evaluates the solutions against external opponent populations to assess their robustness.
W2, Q2 & Q3. Thank you to the authors for the provided ablation with best-of-n being applied to all methods. Firstly, it is not entirely clear from the rebuttal what CoT-BoN and SR-BoN actually implement, since they do not have a predefined pool of opponents. Could the authors clarify this? Secondly, my question was whether MAFP’s reported advantage would survive if the other baselines received the same simulation feedback that MAFP already had. What I expected was a comparison of the baselines with BoN against MAFP as presented in the paper.
Since the tournament pools differ, we cannot compare the reported TS in Table 1 for the version of MAFP presented in the paper (only best-of-on for the BR component) with the BoN baselines. However, Table 2 reports an SV metric of 0.034 for MAFP, which falls within the interval reported for both CoT-BoN (0.030) & Debate-BoN (0.027), assuming the SV evaluation protocol remained unchanged. If so, MAFP’s advantage may partially stem from the simulation feedback rather than the method itself, which was my original concern.
Follow-up Response to Reviewer YAxS (1/2)
Dear Reviewer YAxS,
Thank you for the quick response and the specific questions. We clarify them below.
W1, Q1
Thank you for pointing out the specific concern. Here we clarify the difference between MAFP and CSRO.
-
Regarding Fictitious Play (FP) being a special case of PSRO. It is indeed true that the original PSRO paper (not LLM-PSRO nor CSRO) is presented as a framework that includes a broad range of algorithms that span both (i) double oracle and (ii) classical FP. However, when used in the context of neural policies, the double oracle (i.e., strategic generation) variant is almost always what is referred to. This is true for CSRO as well.
-
Consequently, CSRO requires the explicit solving of restricted games (with actions being the added code strategies per CSRO iteration), which in turn requires one to run simulations/sampling to obtain the head-to-head performance between all pairs of code strategies. In contrast, MAFP does not require these potentially costly pairwise simulations; the LLM is tasked, in context, to find an approximate best-repsonse to the average of the opponent's past code policies.
-
The distinction between CSRO and MAFP extends past methodological details when moving beyond 2-player-zero-sum (2p0s) games. CSRO is focused on finding Nash equilibrium (NE) in 2p0s games. However, in general-sum games, NE are not necessarily "optimal" in the same minimax/maximin manner as with 2p0s games and equilibrium quality/selection is a huge challenge. Hence CSRO as a "NE-solver" may not even lead to a desirable outcome. Indeed, the goal of MAFP in non-2p0s games (Section 3) is not necessarily to find NE, but to apply the simple FP dynamic and to show that practically, it yields strong performance on different metrics (TS and SV, the latter of which corressponds essentially to exploitability for 2p0s games).
-
Regarding "CSRO also evaluates the solutions against external opponent populations to assess their robustness." We argue that our evaluation is actually more thorough than CSRO, both in fidelity and breadth.
Fidelity when evaluating on 2p0s games. CSRO was only evaluated in repeated Rock-Paper-Scissors and repeated Leduc poker against a catalog of predefined opponent strategies. For repeated RPS, the opponents are 43 hand-coded heuristic bots, including randbot, rockbot, copybot, and iocainebot. In repeated Leduc poker, the only 3 opponents are a Nash strategy computed by CFR+ on the single-hand game, AlwaysCall, and AlwaysFold. The majority of these opponents are very simple (i.e., they are unlikely to be near Nash nor best-response to any strategy), such as rockbot, AlwaysCall, and AlwaysFold, with the Nash strategy being an exception. Performing well against these predefined strategies does not imply a NE of the game (good performance against a NE does not imply low exploitability). In contrast, evaluating MAFP under SV means that we play against an (approximate) best-response; in other words, we are actually trying to measure exploitability --- the gold standard for 2p0s games.
Breadth. As mentioned, CSRO only focused on 2p0s games. In contrast, our evaluation for MAFP includes important general-sum environments like Negotiation and Buy-Sell. This is where "external opponent populations" come in useful. One of our key emprical contributions is that our method performs well against a variety of "reasonable" opponent (e.g., those obtained from competing methods like TOM), rather than extremely simple predefined strategies used in CSRO, but also against a best-responding opponent (note that for general-sum environments, an opponent best responding may not be "bad" for us).
We hope you can recognize the contribution of MAFP. We will add this discussion to the revision.
Follow-up Response to Reviewer YAxS (2/2)
W2 & Q2 & Q3
Thank you for the specific questions. We have supplemented the experiment as suggested.
Implementation: Regarding the experimental design in our previous response. Since not all methods involve opponents, e.g. CoT, SR, and Debate, we let the LLM generate an opponent strategy to apply the BoN selection, and select the candidates against it. We adjust the number of iterations of each method so that all methods use the same number of LLM calls.
Since an additional opponent is introduced as a reference for the selection, we considered that, for a fair comparison, the BoN selection should also be applied to the aggregation step of MAFP, which does not involve opponents.
Supplemented experiments: Following your suggestion, we remove the selection from the aggregation step of MAFP, so that it is the version presented in the paper, and re-adjust the number of iterations of the other methods to achieve a matched budget. The results are as follows:
| Method | TS | SV |
|---|---|---|
| CoT-BoN | 0.196 ± 0.021 | 0.015 ± 0.030 |
| SR-BoN | 0.191 ± 0.021 | −0.036 ± 0.030 |
| Debate-BoN | 0.226 ± 0.023 | 0.019 ± 0.033 |
| ToM-BoN | 0.115 ± 0.033 | −0.077 ± 0.023 |
| MAFP | 0.262 ± 0.017 | 0.034 ± 0.027 |
The new results show that MAFP still outperforms the other baselines on both TS and SV under matched budgets. The other methods all improve with the BoN selection against opponent strategies, but the design of MAFP, in which the strategies of multiple agents co-evolve, produces strategies of higher quality.
We will use these results in the revision.
We hope that our additional results and clarifications address your concerns. We would greatly appreciate it if you could reconsider your assessment, and we are happy to discuss further if you have any other concerns.
Sincerely,
The Authors
We attach the full results here.
TS
| Method | TicTac | Nim | Conn4 | Pig | BrkThru | KuhnPk | LiarDice | IPD | BlindAuc | Negot | BuySell | Ultimat | ResExch | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CoT-BoN | 0.037 ± 0.013 | 0.125 ± 0.049 | 0.089 ± 0.173 | 0.021 ± 0.046 | 0.280 ± 0.179 | −0.024 ± 0.026 | 0.173 ± 0.038 | 0.452 ± 0.016 | 0.251 ± 0.008 | −0.269 ± 0.150 | −0.067 ± 0.178 | 0.497 ± 0.050 | 0.988 ± 0.012 | 0.196 ± 0.021 |
| SR-BoN | 0.042 ± 0.016 | 0.086 ± 0.035 | 0.299 ± 0.128 | 0.086 ± 0.036 | 0.119 ± 0.108 | −0.043 ± 0.030 | 0.012 ± 0.091 | 0.468 ± 0.014 | 0.249 ± 0.010 | −0.068 ± 0.157 | −0.163 ± 0.093 | 0.446 ± 0.102 | 0.952 ± 0.036 | 0.191 ± 0.021 |
| Debate-BoN | 0.034 ± 0.020 | 0.113 ± 0.041 | 0.095 ± 0.132 | 0.119 ± 0.067 | 0.179 ± 0.185 | 0.012 ± 0.017 | 0.274 ± 0.052 | 0.454 ± 0.017 | 0.246 ± 0.009 | −0.132 ± 0.288 | 0.046 ± 0.174 | 0.524 ± 0.037 | 0.976 ± 0.024 | 0.226 ± 0.023 |
| ToM-BoN | 0.045 ± 0.013 | 0.006 ± 0.087 | −0.524 ± 0.111 | −0.057 ± 0.102 | −0.369 ± 0.141 | 0.058 ± 0.051 | −0.235 ± 0.114 | 0.548 ± 0.031 | 0.255 ± 0.008 | 0.312 ± 0.173 | 0.096 ± 0.133 | 0.380 ± 0.100 | 0.976 ± 0.024 | 0.115 ± 0.033 |
| MAFP | 0.046 ± 0.015 | 0.009 ± 0.092 | 0.289 ± 0.053 | −0.039 ± 0.055 | 0.098 ± 0.087 | 0.065 ± 0.052 | 0.086 ± 0.098 | 0.470 ± 0.019 | 0.255 ± 0.011 | 0.549 ± 0.059 | 0.367 ± 0.094 | 0.241 ± 0.153 | 0.976 ± 0.024 | 0.262 ± 0.017 |
SV
| Method | TicTac | Nim | Conn4 | Pig | BrkThru | KuhnPk | LiarDice | IPD | BlindAuc | Negot | BuySell | Ultimat | ResExch | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CoT-BoN | 0.000 ± 0.000 | 0.000 ± 0.000 | −0.009 ± 0.171 | −0.054 ± 0.066 | −0.232 ± 0.234 | −0.281 ± 0.051 | −0.196 ± 0.174 | 0.410 ± 0.047 | 0.195 ± 0.032 | −0.669 ± 0.156 | −0.196 ± 0.211 | 0.230 ± 0.147 | 1.000 ± 0.000 | 0.015 ± 0.030 |
| SR-BoN | 0.000 ± 0.000 | 0.000 ± 0.000 | 0.000 ± 0.109 | −0.179 ± 0.066 | −0.554 ± 0.139 | −0.308 ± 0.042 | −0.321 ± 0.144 | 0.361 ± 0.048 | 0.176 ± 0.021 | −0.479 ± 0.202 | −0.346 ± 0.078 | 0.252 ± 0.145 | 0.929 ± 0.071 | −0.036 ± 0.030 |
| Debate-BoN | −0.009 ± 0.009 | 0.000 ± 0.000 | −0.321 ± 0.187 | −0.107 ± 0.123 | −0.232 ± 0.196 | −0.196 ± 0.068 | −0.214 ± 0.163 | 0.338 ± 0.061 | 0.177 ± 0.026 | −0.320 ± 0.262 | −0.186 ± 0.162 | 0.317 ± 0.057 | 1.000 ± 0.000 | 0.019 ± 0.033 |
| ToM-BoN | 0.000 ± 0.000 | −0.125 ± 0.125 | −0.714 ± 0.265 | 0.036 ± 0.108 | −0.786 ± 0.089 | −0.205 ± 0.069 | −0.696 ± 0.109 | 0.200 ± 0.000 | 0.177 ± 0.035 | 0.020 ± 0.202 | −0.022 ± 0.204 | 0.120 ± 0.145 | 1.000 ± 0.000 | −0.077 ± 0.023 |
| MAFP | 0.000 ± 0.000 | −0.125 ± 0.125 | 0.086 ± 0.172 | −0.188 ± 0.116 | −0.531 ± 0.168 | −0.195 ± 0.056 | −0.391 ± 0.104 | 0.157 ± 0.026 | 0.185 ± 0.020 | 0.167 ± 0.173 | 0.144 ± 0.093 | 0.130 ± 0.151 | 1.000 ± 0.000 | 0.034 ± 0.027 |
The results show that, on TS, MAFP attains the highest score in 5 of the 13 scenarios, more than any other method. On SV, MAFP attains the highest score on Conn4, KuhnPk, Negot, and BuySell, and ties for the highest on TicTac and ResExch.
Response to Reviewer YAxS
W1 & Q1: Missing citations of [1] and [2], and the difference of MAFP from them.
A: We thank the reviewer for pointing out these related works. LLM-PSRO [1] and CSRO [2] build on PSRO, a framework for computing approximate equilibria, and replace its best-response oracle with an LLM that writes policies as code for interpretability. We agree that MAFP appears similar to [1] and [2], since all three use an LLM as the best responder. However, MAFP differs from them in two aspects:
Different problem. [1] and [2] inherit the problem of PSRO, namely game solving. The target is a Nash equilibrium, which is the well-defined solution of a two-player zero-sum game, and both works are evaluated on such games. In contrast, MAFP may be applied to general-sum environments and is not designed to find an equilibrium per se. Our goal, as stated in Section 3, is to design an effective strategy for our own agent in the setting where we control only this agent, while the other agents pursue their own goals and may adapt to our strategy.
Different method. Although all three use an LLM as the best responder, what the LLM responds to is different. At every iteration, [1] and [2] explicitly construct a payoff matrix over the current population, which requires playing its policies against each other, and solve this matrix game for an equilibrium. The LLM then best responds to this equilibrium mixture. MAFP does not construct a payoff matrix or solve a matrix game. Following fictitious play (FP), the LLM of each agent directly best responds to the empirical history of the other agents, namely the uniform mixture over all of their past strategies (Eq. 5).
Our contributions. (1) We abstract the learning dynamic of classical FP into a principle, namely replacing the complex mutual anticipation with iterative updates. This learning dynamic can exist outside of game solving. (2) We design on this basis an LLM-based multi-agent framework for strategy improvement in mixed-motive environments. (3) We validate the effectiveness of this framework on two-player zero-sum games, general-sum games, multi-player games, and an economic simulation, which demonstrates the value of this learning dynamic for LLM-based multi-agent systems. Such an application can inspire research on mixed-motive multi-agent systems, beyond the fully cooperative LLM-based agents that are widely adopted.
We will cite and discuss [1] and [2] in Section 2.
W2 & Q2 & Q3: Best-of-n selection as a confound, baselines with simulation-based feedback, and uncertainty measures.
A: We thank the reviewer for the suggestions. We apply best-of-n selection to every strategy generation and update step of all methods, including the aggregation step of MAFP, and adjust the number of iterations to achieve a matched-budget comparison. We then rerun both the tournament and the adaptive-opponent evaluation on the latest strategies of all methods, and report the uncertainty of the metrics:
The results show that MAFP ranks first on both metrics under matched budgets. We further run a statistical test between MAFP and the second-best baseline on each metric. On TS, MAFP leads Debate-BoN by +0.101 ± 0.026 (p = 0.003). On SV, MAFP leads CoT-BoN by +0.068 ± 0.039 (p = 0.061). This indicates that noise alone would produce a lead at least this large with a probability of 0.3% on TS and 6.1% on SV. These results further demonstrate the effectiveness of MAFP. In the revision, we will replace the current comparison with this matched-budget comparison suggested by the reviewer, with uncertainty estimates.
References
[1] Bachrach, Yoram, et al. "Combining Code Generating Large Language Models and Self-Play to Iteratively Refine Strategies in Games." IJCAI Demo Track (2025).
[2] Hennes, Daniel, et al. "Code-Space Response Oracles: Generating Interpretable Multi-Agent Policies with Large Language Models." arXiv preprint arXiv:2603.10098 (2026).