acceptodds
Under review as a conference paper at ICLR 2027

Coding Agents are Zero-shot Coordinators

Abstract

We focus on zero-shot coordination (ZSC), where independently developed agents must cooperate with unseen partners at test time (e.g., autonomous driving). Self-play (SP), a paradigm in which agents learn by interacting with copies of themselves, leads them to adopt specialised conventions that fail to generalise to new partners. Existing ZSC methods rely on restrictive assumptions and perform poorly on benchmarks such as Overcooked V2 and Yokai, achieving poor inter-seed cross-play (IXP), which measures mutual compatibility across independent runs of the same algorithm. Recent demonstrations of large-scale LLM coordination, together with rapid advancements in their coding capabilities, motivate a different approach: using general-purpose LLMs to directly generate ZSC policies as code. We find that coding agents, LLMs equipped with an off-the-shelf coding harness and runtime environment for iterative policy refinement, achieve state-of-the-art IXP on both Overcooked V2 and Yokai. Their policies also remain compatible across model families and novel Overcooked V2 configurations. Beyond ZSC, we consider a setting in which agents may pre-agree on conventions before deployment but are prohibited from joint training, as in self-play. Subsequently, we find that coding agents generate policies that reliably follow the H-group set of conventions used by human players in Hanabi. Taken together, our results show that coding agents significantly improve over existing ZSC methods, offering a promising route toward robust, adaptable, and human-compatible coordination.

Primary area:
reinforcement learning
Keywords:
zero-shot coordination, LLM agents, cooperative multi-agent learning, code-as-policy

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_jhT2OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents an empirical study of whether off-the-shelf LLM coding agents can generate executable policies for zero-shot coordination with independently developed partners. Given task descriptions, policy interfaces, and rollout tools, each coding agent iteratively writes, evaluates, and revises policy code in an isolated session with a budget of up to two hours, without access to the actual cross-play partners. The LLM weights remain fixed, and the resulting decentralized policies act on local observations and private memory without deployment-time LLM calls. Evaluation covers two-player Overcooked V2, Yokai, and Hanabi through self-play, inter-seed cross-play, and cross-model pairings, with additional tests on two new Overcooked layouts and external Hanabi bots. The reported results show substantial inter-seed cross-play gains over the selected RL baselines on Overcooked V2 and Yokai. In Hanabi, supplying H-group conventions improves compatibility with a convention-following bot, while self-play and inter-seed cross-play remain below strong RL baselines. The paper also compares policy-generation objectives and model capabilities to examine how these factors affect the performance and compatibility of generated policies.

Critical strengths
  1. The generated policies are executable and inspectable, with no deployment-time LLM calls. The reported inter-seed cross-play gains on Overcooked V2 and Yokai are substantial relative to the evaluated baselines.

  2. Independent runs, cross-model pairings, and new layouts provide evidence beyond self-play. The Hanabi tests with human-proxy and convention-following bots also extend evaluation beyond the generated-policy population.

Critical weaknesses
  1. The paper offers limited methodological novelty or new insight into coordination. It uses existing coding agents with rollout-guided code refinement, and Co-pi-tree already develops executable cooperative policies through interaction with a source partner [1]. The independent-generation setting is a relevant distinction, but the analysis provides little explanation of why the resulting policies are compatible or under what partner conditions that compatibility breaks down.

  2. The comparisons rely largely on early ZSC baselines and do not adequately cover subsequent population-based methods. For example, Overcooked V2 includes Self-Play, State-Augmented, Other-Play, and FCP, but omits MEP [2], HSP [3], COLE [4], and GAMMA [5]. Without comparisons to these methods in the same environment and under a common evaluation protocol, the reported gains do not establish an advantage over a sufficiently representative set of ZSC approaches.

  3. Partner generalization is insufficiently evaluated. The main Overcooked V2 and Yokai results use partners produced through the coding-agent generation setup, leaving robustness to independently designed conventions largely untested. Evaluation against a common, fixed, diverse, held-out partner pool, as in ZSC-Eval [6] and ICRL4AHT [7], is missing. Inter-seed and cross-model compatibility provide limited evidence for generalization across a broader partner distribution. The Hanabi human-proxy and convention-following bot tests provide useful external-partner evidence, but the claims about human compatibility lack direct human-AI evaluation.

  4. The sources of coordination gains are not isolated. There is no comparable-budget, same-model control separating rollout feedback from additional generation and selection, and the SP-versus-XP experiment changes both the objective and selection rule. The mutual co-adaptation interpretation is also stronger than the code evidence: in Appendix G.4, the cook infers signalling conventions while the signaller follows a fixed mapping. Without targeted ablations, shared priors, fixed conventions, fallback behavior, and online adaptation remain competing explanations.

[1] Zhang, B., et al. Distilling LLM Reasoning into an Interpretable Policy Tree for Human-AI Collaboration. arXiv:2606.08596, 2026. Paper.

[2] Zhao, R., et al. Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination. AAAI, 2023. Paper.

[3] Yu, C., et al. Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased. ICLR, 2023. Paper.

[4] Li, Y., et al. Cooperative Open-ended Learning Framework for Zero-Shot Coordination. ICML, 2023. Paper.

[5] Liang, Y., et al. Learning to Cooperate with Humans using Generative Agents. NeurIPS, 2024. Paper.

[6] Wang, X., et al. ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordination. NeurIPS Datasets and Benchmarks, 2024. Paper.

[7] Jing, Y., et al. Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamwork. ICML, 2026. Paper.

Questions for authors
  1. How do MEP, HSP, COLE, and GAMMA perform under the same environment and evaluation protocol?

  2. How do the methods compare on a common, diverse, held-out partner pool, and in direct human-AI experiments?

  3. How much do rollout feedback and online convention inference contribute when generation budgets, selection rules, and ordinary task memory are controlled?

2 comments

We thank the reviewer for their positive comments on our results. We appreciate their feedback and address their concerns below.

No new insight into coordination

We kindly refer the reviewer to our response to reviewer fPME regarding the main contributions of the paper. In light of these contributions, we strongly disagree that the paper provides no new insight.

On difference from Co-π-tree

We differ from Co-π-tree in several substantive ways:

  1. Independent policy generation. Co-π-tree constructs and refines policies through interaction with an externally supplied source partner. This is essentially best-response optimisation against a fixed partner, which can be formulated as a single-agent problem by treating the partner as part of the environment. This development setting is fundamentally different from the ZSC setting considered in our work, where policies must become mutually compatible despite being developed independently using self-play feedback.

  2. General-purpose coding agents. Co-π-tree uses a specialised scaffold, validated solely on Overcooked. We demonstrate that off-the-shelf coding agents can generate effective executable policies across three established coordination benchmarks.

  3. More demanding coordination challenges. Existing methods already achieve strong IXP in the original Overcooked environment, which poses limited coordination challenges (as highlighted in the Overcooked V2 paper). We evaluate on Overcooked V2 and Yōkai, which introduce challenges including partial observability, asymmetric information, and test-time convention formation. Importantly, these benchmarks remain unaddressed by the existing ZSC methods.

  4. Complementary evidence from Hanabi. We demonstrate competitive cross-play with human-proxy bots and strong compatibility with Wilflame when supplied with convention descriptions (Section 5.2). Our hybrid method further achieves the highest mean cross-play returns with both partner types while retaining competitive SP and IXP (Section 5.5, Table 3).

Hence, our findings extend substantially beyond Co-π-tree.

On in-depth analysis

We agree that an in-depth analysis of agent behaviour is important. We kindly refer the reviewer to our response to reviewer fPME on our significant additions to our qualitative analysis. We also refer the reviewer to our response to reviewer tPkP under “On characterising failure conditions.”

Comparison with MEP and Gamma

We kindly refer the reviewer to our response to reviewer wVgq on our experiments with GAMMA and MEP. We find that both these methods perform very poorly on Overcooked V2.

On ablations

Appendix E compares single-shot generation, the coding agent’s first attempt, and iterative refinement with rollout feedback. Iterative refinement improves mean IXP over the first attempt in all 30 model–environment comparisons and over single-shot generation in 25 of 30. Access to the coding harness alone does not consistently improve coordination, highlighting the contribution of iterative feedback.

Appendix I provides learning curves showing improvements in SP and IXP over successive attempts. Section 5.4 further compares models under SP and XP objectives: stronger models achieve higher SP under the SP objective, alongside a larger SP–IXP gap, and higher IXP under the XP objective.

Generation budgets are already controlled through a common two-hour wall-clock limit per run. We are unclear what is meant by controlling “ordinary task memory.”

On mutual co-adaptation

We kindly refer the reviewer to our response to reviewer tPkP on mutual co-adaptation, where we provide additional examples illustrating richer forms of test-time adaptation beyond the belief-mapping example initially presented.

On comparison with a common diverse held-out partner pool

We kindly refer the reviewer to our response to reviewer tPkP on the pitfalls of evaluating against a common diverse held-out partner pool and why we believe it is not indicative of ZSC performance in the real world.

On direct human-AI experiments

Such studies are costly, which motivates AH2AC2 (Dizdarević et al., 2025): a benchmark specifically designed to provide inexpensive, reproducible human-proxy evaluation partners trained on human gameplay. We use these proxies alongside the convention-following Wilflame bot. As reported in Section 5.2, generated code policies achieve XP returns comparable to established baselines with the human proxies and substantially higher returns with Wilflame. Our hybrid policies further improve these results, achieving the highest mean XP returns with both partner types (Section 5.5, Table 3).

We agree that direct human evaluation is important, but it lies beyond this paper’s scope. Our results establish compatibility with these bots and provide encouraging evidence for human–AI coordination. We will make the scope of this evidence clearer in the paper.

I’m remaining my rating. My main concern remains that the Overcooked V2 and Yokai results don’t show whether the cross-play gains hold when all methods face the same diverse set of unseen partners.

Reviewer_tPkPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates general-purpose coding agents as a new approach to zero-shot coordination. Using frontier LLMs equipped with coding harnesses, the authors iteratively generate executable cooperative policies through code development and environment rollouts. The paper reports strong inter-seed and inter-model cross-play performance on Overcooked V2 and Yokai, and further explores test-time convention formation and compatibility with human-proxy and convention-following agents in Hanabi.

Critical strengths

1. A novel and interesting perspective on zero-shot coordination.

The paper explores ZSC from a relatively new perspective by treating general-purpose coding agents as cooperative policy generators, rather than relying on specialized MARL algorithms. I find this research direction interesting and potentially valuable, as it opens up an alternative approach to studying cooperative multi-agent systems.

2. Promising empirical findings.

The paper reports impressive IXP performance on Overcooked V2 and Yokai. The cross-model evaluations, experiments on newly designed Overcooked V2 layouts, and Hanabi results provide several interesting observations about the potential of coding agents for cooperative policy generation.

Critical weaknesses

1. My main concern is the gap between the paper's central claim, "Coding Agents Are Zero-Shot Coordinators," and the scientific understanding provided by its experiments.

The paper presents several successful applications of coding agents to cooperative games, but these results are largely organized as performance improvements and individual demonstrations. What remains unclear is whether the coding-agent workflow can systematically identify and construct appropriate coordination mechanisms across different cooperative problems, and under what conditions it succeeds or fails.

For an empirical paper, I do not expect a novel coordination algorithm or a complete theoretical explanation of the underlying LLM capabilities. However, I would expect a more systematic characterization of the observed phenomenon. Different cooperative environments may require fundamentally different coordination mechanisms, such as role specialization, communication protocols, partner modeling, or shared conventions. Can the coding-agent workflow discover mechanisms appropriate to these different requirements? What properties of an environment or coordination problem make this possible?

The current results demonstrate that coding agents can generate effective cooperative policies in several settings, but offer limited insight into when and why meaningful coordination emerges. Consequently, it remains unclear what generalizable scientific understanding this work provides beyond demonstrating the policy-generation capabilities of frontier LLMs.

2. The origin and generality of the reported convention-formation capability remain unclear.

The paper emphasizes test-time convention formation, but it is unclear whether the observed conventions genuinely emerge through online interaction or are largely determined by the underlying LLM's prior knowledge and offline policy generation. For example, Appendix G.4 presents a predefined signaling scheme combined with belief updating, which does not necessarily demonstrate the emergence of a new convention through mutual adaptation.

Although the novel Overcooked V2 layouts partially address concerns about benchmark familiarity, they do not rule out the possibility that LLMs already possess sufficient knowledge of Overcooked to construct intuitive signaling strategies. Overcooked provides relatively intuitive opportunities for signaling through positions, movements, and orientations. A capable LLM with substantial prior knowledge of the game may readily construct such a signaling scheme A more systematic analysis would help clarify whether coding agents can discover appropriate conventions in less familiar coordination problems, rather than merely applying existing coordination heuristics.

3. Insufficient evaluation of coordination with diverse unseen partners.

A central challenge in zero-shot coordination is the ability to cooperate with diverse, previously unseen partners. However, the paper does not sufficiently investigate this aspect. The main Overcooked V2 and Yokai experiments primarily evaluate cross-play among coding-agent-generated policies. Even when generated by different LLMs, these policies may share similar priors and coordination preferences. Strong performance within this population therefore does not necessarily demonstrate robust coordination with a broader range of unseen partners.

Although the Hanabi experiments with human-proxy agents provide useful evidence, the paper lacks a systematic analysis of how coding-agent-generated policies perform with heterogeneous partners, particularly those developed through different learning paradigms or following different conventions. This limits the extent to which the reported results support the paper's broader ZSC claim.

4. Minor concern: Clarifying the Scientific Contribution of the Coding-Agent Perspective.

The paper demonstrates promising coordination capabilities of general-purpose coding agents, but the distinctive scientific contribution of this perspective could be more clearly articulated. For example, prior work such as Leveraging Dual Process Theory in Language Agent Framework for Real-time Simultaneous Human-AI Collaboration (DPT-Agent, ACL 2025) has already shown that LLM-based code generation can support cooperation through explicitly designed mechanisms, including partner modeling, belief updating, and test-time adaptation.

A potentially more interesting contribution of the present work is whether general-purpose coding agents can autonomously discover and synthesize appropriate coordination mechanisms, without explicitly specifying these mechanisms in advance. Discussing this distinction would help clarify what new insight the paper provides beyond demonstrating successful cooperative policy generation.

Questions for authors
  • Autonomous discovery of coordination mechanisms. Beyond achieving strong cross-play performance, what evidence suggests that coding agents can autonomously discover coordination mechanisms appropriate to different cooperative tasks? Could the authors provide a more systematic analysis of the mechanisms found across environments, and discuss under what conditions this process succeeds or fails?

  • Origin of test-time conventions. How can we distinguish conventions genuinely established through test-time interaction from those determined during offline policy generation or inherited from the LLM's prior knowledge? In particular, what evidence supports mutual convention formation, rather than predefined signaling strategies combined with one-sided belief updating?

  • Coordination with diverse unseen partners. To what extent do the generated policies coordinate with partners outside the coding-agent-generated population, such as independently trained RL agents or agents following different conventions? Could the authors clarify how the current IXP and inter-model XP results support generalization to more diverse unseen partners?

2 comments

We are glad to hear that the reviewer found our perspective on zero-shot coordination interesting and novel. We also thank them for their constructive feedback, which has helped us improve our paper.

On general scientific insight

We kindly refer the reviewer to our response to reviewer fPME on the main contributions of the paper and why we feel they are valuable to the ZSC community.

On characterisation of the observed phenomenon

We agree that an in-depth analysis of agent behaviour is important. We kindly refer the reviewer to our response to reviewer fPME on our significant additions to our qualitative analysis.

On characterising failure conditions

Coding agents saturate the evaluated Overcooked V2 and Yōkai benchmarks, including our novel Overcooked V2 layouts. Consequently, these benchmarks offer limited scope for identifying where current models fail, motivating more challenging ZSC benchmarks. Hanabi remains the exception: purely code-based policies underperform established methods in SP and IXP, although the hybrid approach in Section 5.5 (Table 3) substantially closes this gap by combining recurrent neural policies with conventions encoded by the generated policies.

On extracting general coordination principles

Our results demonstrate that coding agents can solve challenging ZSC tasks. We agree that whether they can articulate the underlying principles and translate them into a general learning algorithm, potentially one that learns tabula rasa, is an important question, but it is outside the scope of this work. Establishing that effective solution methods exist is itself a valuable contribution, even before extracting general principles from them. We see similar results in other fields, including the reported AI-generated Navier–Stokes proof (OpenAI, 2026b) and LLM agents’ success in NetHack (Kenny, 2026) and Craftax (Creus Castanyer, 2026). These illustrate the value of reporting solutions to longstanding challenges: we can verify at a high level how the agents solve these tasks, while still lacking a deeper understanding of when, why, and where they fail. We believe these broader questions are important but leave them to future work; they do not diminish the value of establishing and reporting the capabilities demonstrated here.

Recent developments on Craftax further illustrate this progression: subsequent AI-led research (Kondratyuk et al., 2026) developed an RL training recipe that enabled neural policies trained from scratch to beat the benchmark. Developing analogous learning methods for ZSC is a promising direction for future work; our contribution establishes the capabilities that motivate it.

On prior knowledge

We kindly refer the reviewer to our response to reviewer fPME on our experiments that probe the prior knowledge of the model on the evaluated benchmarks.

On test-time conventions

Generating a policy offline does not preclude test-time adaptation. The generated code can maintain memory across timesteps and condition actions on the agent’s observations of its partner and their interaction history within the episode.

The Overcooked V2 test-time convention environments require agents to establish conventions linking their joint behaviour to the dish they should prepare. The simple belief-update policy shown in the appendix achieves strong performance, but it represents only one form of adaptation. Additional examples in Appendix H demonstrate richer mutual adaptation: both agents respond to their partner’s behaviour, signal intended conventions, detect deviations from an established convention, and revise their behaviour to establish a new one. Thus, the generated policies are not restricted to selecting among fixed mappings; they can establish and renegotiate conventions during interaction.

On comparison with a common diverse held-out partner pool

We agree that IXP has its limitations and alone does not establish general coordination with unseen partners, as discussed in Section 6.

Some prior work constructs evaluation populations through researcher-defined choices, including training seeds, architectures, and checkpoints (Strouse et al., 2021), or behavioural reward functions (Wang et al., 2024). However, diversity and held-out status alone do not establish that these populations represent real-world coordination partners. The ZSC literature (Treutlein et al., 2021) also distinguishes robustness to a population whose members may adopt arbitrary conventions from mutual coordination between agents developed to avoid such conventions, which is the focus of this work.

Accordingly, we use IXP, a primary and well-established evaluation metric in the ZSC literature (Hu et al., 2020; Hu et al., 2021; Forkel et al., 2026), as the main evaluation metric. In this formulation, both partners are ZSC-aware: they seek conventions that independently developed partners can infer. Cross-play with partners relying on arbitrary conventions therefore measures adaptation to those conventions, rather than directly measuring mutual ZSC capability.

We acknowledge that the ZSC community can benefit from better evaluation procedures; that said, it is outside the scope of this work. Subsequently, we believe that constructed partner pools as done in prior work (Strouse et al., 2021; Wang et al., 2024) do not resolve this problem, since they do not establish that the generated population represents real-world ZSC partners (also acknowledged in Wang et al., 2024).

References

Roger Creus Castanyer. I let GPT-6 Astra run its own research. It beat Craftax and reached the bottom of NetHack. Blog post, September 2026.

Kenny. An LLM Beat NetHack. Vaguely Aligned, September 21, 2026.

Dan Kondratyuk, Ming-Chang Chiu, Junpeng Lao, and Joshua V. Dillon. A 15M-parameter RL agent trained from scratch vs GPT-6 Astra: who would win? Rekursiv, October 7, 2026.

Johannes Treutlein, Michael Dennis, Caspar Oesterheld, and Jakob Foerster. A New Formalism, Method and Open Issues for Zero-Shot Coordination. arXiv:2106.06613, 2021.

Xihuai Wang, Shao Zhang, Wenhao Zhang, Wentao Dong, Jingxiao Chen, Ying Wen, and Weinan Zhang. ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordination. Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2024.

Reviewer_wVgqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether existing LLM-based coding agents can generate policies for zero-shot coordination through iterative code generation and simulator feedback. The resulting policies are evaluated on Overcooked V2, Yokai, and Hanabi.

Critical strengths
  • The paper is very well written. The problem is well defined, and the approach and evaluation are presented clearly and concisely.

  • The idea of using LLM coding harnesses to generate policies for zero-shot coordination is interesting and reasonable.

Critical weaknesses

The results are meaningful, but the paper primarily explores an application of existing LLM-based coding agents. In its current form, I feel the work may be better suited to an NLP-focused venue such as EMNLP. The followings are some concerns I have.

  • Scalability to more complex coordination is not sufficiently established: The generated policies rely on explicit decision rules and programmed belief updates. This approach may work well when states, objects, locations, and coordination situations can be described explicitly and captured by relatively compact programs. However, it is unclear how well it scales as partner strategies and possible coordination conventions become more diverse, or when coordination requires more continuous and context-dependent adjustments. The Hanabi results make this concern concrete: the generated policies underperform strong existing ZSC methods in both self-play and inter-seed cross-play. The paper does not sufficiently analyze this gap or explain whether it reflects limitations in generating policies for more complex coordination. Its applicability to richer environments, including continuous state and action spaces, therefore remains uncertain.

  • The characterization of existing methods is too broad: Important related approaches such as GAMMA [1] and TALENTS [2] are missing from the discussion. GAMMA models diverse partner strategies, while TALENTS explicitly infers partner strategies and adapts during interaction. These approaches are directly relevant to the paper’s discussion of adaptation and its characterization of existing methods as relying on fixed strategies. They have also demonstrated successful coordination in Overcooked variants, although their performance on the specific Overcooked V2 and Yokai settings here is not established. The reported failures of selected baselines do not justify a broader conclusion that existing coordination approaches generally struggle in these environments. I am not asking the authors to evaluate every method, but the related work and claims should better reflect what has and has not been established.

  • The comparison of assumptions and requirements is not sufficiently balanced: The paper emphasizes restrictive assumptions and computational costs of existing methods, while its own approach relies on detailed game descriptions, simulator access, coding tools, and substantial pretrained model knowledge. Some model-free RL methods can learn from observations, actions, and rewards without an explicit description of the game rules. These approaches use different information and resources, so it is not clear that the coding-agent approach is generally less restrictive or more broadly applicable. Moreover, the paper does not establish that the poor Overcooked V2 and Yokai performance of the evaluated methods is caused by the restrictive assumptions highlighted in the introduction.

Questions for authors
  1. What explains the weaker Hanabi performance, and what does this imply about the approach’s ability to handle more complex belief reasoning and a broader range of coordination strategies?
2 comments

We thank the reviewer for their positive comments on the paper’s clarity and the presented coding agent based approach to zero-shot coordination. We address their concerns below.

I feel the work may be better suited to an NLP-focused venue such as EMNLP.

We kindly refer the reviewer to our response to reviewer fPME on why we believe our paper is a good fit for the ICLR main conference under the reinforcement learning track.

Related approaches such as GAMMA (Liang et al., 2024) and TALENTS (Li et al., 2025) are missing from the discussion.

We evaluate two population-based methods, MEP (Zhao et al., 2023) and GAMMA (Liang et al., 2024), on all six Overcooked-v2 environments. For MEP, we train a best-response (BR) policy directly against the population using prioritised sampling (β=3). For GAMMA, we train a VAE on trajectories collected from the same populations, then train a BR policy against partners sampled from the VAE. Neither method achieves consistently strong self-play (SP) or inter-population cross-play (IXP) performance in our experiments. As discussed in Section 2 (Related Work), these methods also incur substantial computational costs: in our setup, training five population members and one best response requires approximately six times the environment-step budget of training a single policy, excluding VAE training.

Moreover, many of the reviewer’s suggested methods are evaluated only on the original Overcooked benchmark. As discussed in the Overcooked V2 paper, this benchmark is fully observable and presents a limited range of coordination challenges that several ZSC methods already address successfully.

EnvironmentMEP SPMEP IXPGAMMA SPGAMMA IXP
Demo Cook Simple0.52 ± 1.940.69 ± 1.394.39 ± 9.383.87 ± 3.09
Demo Cook Wide10.00 ± 24.4910.55 ± 10.4913.21 ± 18.384.35 ± 5.32
Grounded Coordination Simple1.84 ± 8.863.83 ± 2.479.39 ± 17.416.84 ± 5.68
Grounded Coordination Ring82.11 ± 51.0682.87 ± 19.6022.91 ± 32.6717.36 ± 15.56
Test-Time Simple6.54 ± 8.963.88 ± 1.793.58 ± 8.657.42 ± 6.78
Test-Time Wide11.84 ± 26.619.66 ± 7.3124.23 ± 43.339.72 ± 11.94

Mean ± sample standard deviation across six BR seeds, evaluated with greedy actions over 1,024 games per pairing. SP measures BR self-play; each seed’s IXP score averages cross-play with the other five BRs.

the paper does not establish that the poor Overcooked V2 and Yokai performance of the evaluated methods is caused by the restrictive assumptions highlighted in the introduction.

The baselines are drawn from the respective benchmark papers, which already establish their limitations. Examples include:

  1. Self-play: Section 7 of the Overcooked V2 paper explains that self-play agents develop arbitrary communication conventions, such as using ingredients placed on intermediate counters to communicate intentions (Gessler et al., 2025, §7).

  2. Other-Play: Other-Play requires prior knowledge of the environment’s symmetries. Cui et al. (2021) explicitly acknowledge that “discovering symmetries for a given environment is a computationally hard problem.” The Yōkai paper identifies incomplete symmetry coverage as a limitation: accounting for only a subset of the relevant symmetries can leave incompatible conventions intact (Ruhdorfer et al., 2026, §7).

  3. IPPO with high entropy: The high-entropy IPPO paper itself demonstrates that high entropy regularisation alone cannot recover the optimal symmetry-equivariant policy in some Dec-POMDPs. The entropy needed to prevent symmetry breaking can also prevent effective coordination, particularly when successful behaviour requires nearly deterministic actions (Forkel et al., 2026, §3, limitation discussion). The Yōkai benchmark similarly reports that increasing entropy can destabilise learning before optimal cross-play performance is reached, particularly in multiplayer settings (Ruhdorfer et al., 2026, §7).

  4. Off-Belief Learning: The Yōkai paper identifies two difficulties for OBL: ambiguous hints complicate belief modelling, and longer fictitious rollouts increase computational and memory requirements (Ruhdorfer et al., 2026, §§5.2 and 7). Computational cost also limits evaluations in Hanabi: Forkel et al. (2026) restrict their four- and five-player experiments to IPPO because of computational constraints, omitting OBL results for those settings.

comparison of assumptions and requirements is not sufficiently balanced

We clarify the resources they receive below:

  1. Simulator access: The tabula-rasa baselines also interact extensively with simulators, training on billions of sampled transitions. We do not provide coding agents with the simulator’s source code; they receive the same interaction-based access as the tabula-rasa methods.

  2. Game descriptions: We provide descriptions of the game’s rules, objective, observations, and legal actions, much like the information a human player needs before playing. Such descriptions are straightforward to provide and specify the task without supplying a solution.

  3. Learning without explicit rules: While model-free RL can learn without these descriptions, specifying the rules is straightforward for a large class of problems; discovering effective strategies remains the main challenge. Examples include chess, Go, Hanabi, and Overcooked. Our results show that, given task descriptions, coding agents outperform the evaluated tabula-rasa baselines in IXP across all tested Overcooked V2 and Yōkai configurations.

  4. Practical accessibility and cost: LLMs and coding harnesses are now widely available and easily accessible. Our total expenditure for the experiments reported in the paper was approximately USD 1,000. For comparison, training a single seed of level-6 OBL in two-player Hanabi requires more than 240 hours on an L40S GPU (Forkel et al., 2026, §4, footnote 2). At a standard L40S rental rate of USD 1.09 per hour, this corresponds to more than USD 1000 for the four evaluated seeds. Hence, even for the current benchmarks, coding agents are comparable to, if not cheaper than, tabula-rasa ZSC methods in terms of cost.

On weaker Hanabi performance

We kindly refer the reviewer to our response to reviewer fPME on the weak performance in Hanabi and the scalability of the approach.

References

Brandon Cui, Hengyuan Hu, Luis Pineda, and Jakob N. Foerster. K-level Reasoning for Zero-Shot Coordination in Hanabi. Advances in Neural Information Processing Systems, 34:8215–8228, 2021.

Yancheng Liang, Daphne Chen, Abhishek Gupta, Simon S. Du, and Natasha Jaques. Learning to Cooperate with Humans using Generative Agents. Advances in Neural Information Processing Systems, 2024.

Rui Zhao et al. Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination. Proceedings of the AAAI Conference on Artificial Intelligence, 2023.

Reviewer_fPMEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies the use of coding agents to address zero-shot coordination challenges in multi-agent environments. Instead of traditional neural network training, the approach uses LLMs to generate cooperative policies as executable code. The models use an off-the-shelf coding harness and runtime environment to perform iterative policy refinement through environment rollouts. The authors evaluate these coding agents on Overcooked V2, Yokai, and Hanabi. The paper reports state-of-the-art inter-seed cross-play on both Overcooked V2 and Yokai.

Critical strengths
  • 1. The paper is clear and well-written.

  • 2. Using LLMs as coding agents to tackle the ZSC task is an interesting idea in the direction of standard multi-agent reinforcement learning paradigms.

  • 3. The results show the interesting possibilities of coding agent LLMs acting as ZSC coordinators, demonstrating good performance in complex environments like Overcooked V2 and Yokai.

Critical weaknesses
  • 1. The approach does not involve any actual training. The authors are simply using existing LLM models, such as GPT and Claude, in a closed-loop evaluation setup. Because nothing is actually being trained, this setup makes the paper more suitable for a datasets and benchmarks track rather than a main conference RL track.

  • 2. Framing "independent runs" as "independent seeds" (Lines 253-254) is a very convenient change in terminology. In standard MARL, different seeds use the exact same inputs and training setup, and policies diverge naturally due to underlying stochasticity (such as network initialization or exploration). In contrast, this paper forces divergence by explicitly feeding the LLM distinct paraphrases of the task specification. This does not measure the natural behavioral variance of a learning algorithm; it merely measures the LLM's sensitivity to prompt variations.

  • 3. It is difficult to determine if the agents are genuinely demonstrating zero-shot coordination or simply exploiting shared knowledge already baked into their pre-training data. The models likely already have prior knowledge of these environments and human conventions from their training corpora, which gives them an unfair advantage over RL baselines that learn from scratch.

  • 4. The agents are instructed to optimize for cross-play, but the only feedback they receive during the iterative refinement process comes from self-play rollouts. Optimizing for unseen partners using only self-play feedback naturally biases the generated code toward self-play conventions, which contradicts the entire premise of zero-shot coordination.

  • 5. The evaluation relies entirely on closed-source proprietary models (OpenAI and Anthropic). Including evaluations against open-source LLMs such as Qwen, Gemma, and LLaMA is missing but necessary to demonstrate whether this approach generalizes beyond specific gated APIs

  • 6. The paper lacks an in-depth analysis of actual agent behavior. A deeper qualitative and quantitative investigation into how the generated policies behave and adapt differently during self-play versus cross-play is needed to truly understand the mechanics of the coordination being produced.

  • 7. The lack of good performance in the Hanabi environment requires more attention. The authors admit that coding agents underperform existing ZSC baselines in Hanabi for both self-play and inter-seed cross-play. However, the paper does not sufficiently analyze the root causes of this failure or explain why the proposed iterative coding method struggles specifically in this domain.

Questions for authors
  • 1. Given that the method relies entirely on prompt-driven code generation without parameter updates, and "independent seeds" are simulated by changing the input text, how do you justify classifying this as a multi-agent reinforcement learning algorithm suitable for the RL track?

  • 2. How can we formally distinguish whether the coding agents are executing genuine zero-shot coordination versus simply retrieving and exploiting shared strategic priors and human conventions already baked into their pre-training data?

  • 3. How do you justify expecting the model to generate robust cross-play policies when the only evaluative feedback it receives during the iterative refinement process comes from self-play rollouts?

  • 4. Do you plan to evaluate this framework on capable open-source models like LLaMA, Qwen, or Gemma to prove that this approach generalizes beyond gated, heavily subsidized proprietary APIs?

  • 5. Can you provide a deeper behavioral analysis of why the coding agents fail to match existing ZSC baselines in Hanabi? What specific limitations in the code-generation process cause it to struggle in this domain compared to Overcooked V2 and Yokai?

3 comments

We thank the reviewer for their positive comments on the paper’s writing and our research findings. We address their concerns below.

This setup makes the paper more suitable for a datasets and benchmarks track rather than a main conference RL track.

Our work introduces neither a dataset nor a benchmark; it showcases that coding agents can generate executable policies that solve existing zero-shot coordination benchmarks. The absence of parameter updates does not make this a benchmarks and datasets contribution. We refer the reviewer to an established line of research using LLMs to generate and refine solutions across diverse problems, including AlphaEvolve (Novikov et al., 2025), GEPA (Agrawal et al., 2025), and FunSearch (Romera-Paredes et al., 2024).

We reiterate our main contributions of the work:

  1. Current-generation LLMs can address coordination challenges that remain unresolved by established ZSC methods.

    • We believe this finding directly advances the field’s understanding of which approaches can achieve robust zero-shot coordination and is valuable to the ZSC community.
    • Before this work, established tabula-rasa methods had not solved these challenges, while prior work had found LLM-generated policies to perform poorly even on simple zero-shot coordination tasks (Hayler et al., 2026). Our findings substantially revise this picture of what is achievable.
  2. Iterative code refinement using only self-play feedback can still produce policies that achieve strong IXP performance across independently generated agents.

  3. Coding agents can generate policies with different types of coordination strategies, such as role specialisation, grounded communication, test-time convention formation, and convention following, depending on the task. This is evidenced by their strong IXP and cross-play scores with human-proxy and Wilflame bots.

  4. The Hybrid results (Section 5.5, Table 3) suggest that code-based and neural policies have complementary strengths, and combining them can improve performance.

Our research question, benchmarks, and baselines are centred on multi-agent learning and coordination. Prior ZSC work has been published in the reinforcement learning track (Hu et al., 2020; Hu et al., 2021; Forkel et al., 2026), and hence we believe this is the most appropriate track for our work.

"independent runs" as "independent seeds"; forces divergence

The characterisation of prompt paraphrasing as a “convenient change in terminology” is incorrect and misrepresents its purpose. As stated in lines 255–257, each run receives a meaning-preserving paraphrase, reflecting our assumption that independently coding agents need not receive identically worded prompts. Paraphrasing also removes identical wording as a potential shared coordination cue, making the evaluation more stringent rather than undermining its validity. We will make this additional rationale explicit in the paper.

Crucially, stochasticity comes from LLM generation, not prompt variation. Even with identical prompts, stochastic generation leads to distinct policy-development trajectories and final policies. As in conventional ZSC, independent runs can therefore produce different policies given the same inputs.

We address this concern by running Astra with identical prompts across all generations. Every run produces a distinct policy, while self-play (SP) and inter-seed cross-play (IXP) performance are maintained. These results contradict the suggestion that policy diversity and coordination performance merely reflect sensitivity to prompt paraphrases.

Astra with identical prompts

OCV2 layoutSPIXP
Demo Cook Simple203 ± 11181 ± 14
Demo Cook Wide209 ± 9197 ± 9
Grounded Coordination Simple136 ± 3136 ± 1
Grounded Coordination Ring159 ± 15146 ± 6
Test-time Simple157 ± 7156 ± 3
Test-time Wide206 ± 16187 ± 18

Mean ± SD across six independent policies per layout; 1,024 episodes per pairing. SP uses each policy’s self-play score; each policy-level IXP score averages its five other partners in both roles.

in-depth analysis of actual agent behaviour

We agree that an in-depth analysis of agent behaviour is important. The generated policies contain hundreds of lines of code each (Appendix I), making it challenging to fully characterise their coordination behaviour through code inspection alone. Nevertheless, we have significantly expanded our qualitative analysis in Appendix H, presenting code snippets with explanations across all three benchmarks. These include grounded communication, signalling intended conventions, modelling partners, detecting departures from an established convention, and adapting behaviour in response. These examples provide deeper insight into the policies’ coordination behaviour and, together with our quantitative results, provide evidence that the models possess the zero-shot coordination capabilities tested by these benchmarks.

on prior knowledge of the models

We systematically investigate the models’ prior knowledge of the evaluation environments using a series of targeted questions (see Appendix G for the questions and grading rubrics). These questions elicit readily verifiable answers, which we assess using an LLM judge and an explicit rubric that awards partial credit for incomplete or partly correct responses.

Table 9 in Appendix G reveals uneven knowledge of Overcooked V2, with the largest gaps in coordination strategies (Q4) and layout reconstruction (Q5). Although the Claude models demonstrate familiarity with the benchmark, the OpenAI models exhibit little benchmark-specific knowledge: neither identifies any of the six tasks, and both score poorly on coordination strategies and layout reconstruction. Nevertheless, GPT-6 Astra outperforms all evaluated ZSC baselines in IXP across the six layouts. This provides evidence that the models possess genuine ZSC capabilities, rather than simply retrieving and exploiting strategic priors and human conventions from their pre-training data.

Furthermore, while the published Overcooked V2 descriptions discuss solution concepts, they do not provide the corresponding executable policies or trained neural-network weights. To further investigate generalisation beyond previously described strategies, we evaluate the models on two novel layouts, Block Grid and Pipeline Cook. As discussed in Section 5.3, these layouts are more complex than the original benchmark layouts and do not readily admit effective ZSC strategies. Nevertheless, the generated policies achieve strong performance and remain compatible across independent generations and model types. Strong performance on these layouts provides additional evidence that coding agents can construct effective coordination policies on novel layouts.

How do you justify expecting the model to generate robust cross-play policies?

We would like to clarify that this is the main claim of the paper, which is validated by experimental results. We will make this clearer in the paper.

The paper reads as follows:

  1. Hypothesis: Current-generation language models, given simulator access and a ZSC objective, can generate policies that coordinate effectively with independently developed partners, even when iterative refinement relies exclusively on self-play feedback.

  2. Empirical validation: We test this hypothesis through extensive experiments on three established ZSC benchmarks. The results provide strong supporting evidence, with coding agents outperforming established ZSC baselines in IXP across all evaluated Overcooked V2 and Yokai configurations.

On open-source models

We conducted additional experiments with Kimi K3.1 Pro and DeepSeek V4.1 Flash, two leading open-source models, at a total cost of approximately USD 80. As shown in Table 1, both models outperform all existing ZSC baselines in IXP across all six Overcooked V2 layouts. These results demonstrate that our findings generalise beyond proprietary models and gated APIs.

Optimising for unseen partners using only self-play feedback naturally biases the generated code toward self-play conventions, which contradicts the entire premise of zero-shot coordination.

We agree that self-play can produce arbitrary conventions that fail to generalise to unseen partners. This does not contradict the premise of zero-shot coordination; it is precisely the problem that motivates ZSC. The challenge is to develop policies that coordinate with independently developed partners despite having access only to self-play feedback during development. Prior ZSC work (Hu et al., 2020; Hu et al., 2021; Forkel et al., 2026) adopts this same setting, training policies through self-play and evaluating their ability to coordinate with independently trained partners at test time.

This setting reflects problems in the real world; for example, Tesla and Waymo could each train their vehicles extensively with self-play, without access to the other company’s vehicles during training. Yet their vehicles must still coordinate when they encounter one another on public roads. The separation between training partners and test-time partners is the defining challenge of ZSC, not a contradiction.

Weak performance in Hanabi

As discussed in Sections 5.1 and 6, generated code policies achieve lower SP and IXP returns in Hanabi than established ZSC methods. We hypothesise that this gap reflects the difficulty of implementing reliable, history-dependent belief updates in explicit code, which may limit the policies’ ability to exploit complex conventions.

The results suggest complementary strengths: recurrent neural policies, such as LSTMs, can learn representations of interaction histories and achieve strong SP and IXP performance, while LLMs excel at convention selection as demonstrated by strong IXP scores in Overcooked V2 and Yokai. Additionally, they can translate textual descriptions of conventions into executable policies, as demonstrated by their strong cross-play performance in the H-group setting. This ability to incorporate conventions from textual instructions is absent from the tabula-rasa RL baselines.

To combine these strengths, we evaluate a hybrid method that trains a recurrent neural policy using IPPO+HE, augmented with a cross-entropy minimisation objective from a fixed generated code policy. This encourages the neural policy to follow the code policy’s conventions while continuing to improve through reinforcement learning. Section 5.5 describes the method, and Table 3 reports the results.

The hybrid policies achieve SP and IXP returns comparable to the established baselines while obtaining the highest mean XP returns with both the human-proxy and Wilflame bots. Their XP returns with Wilflame reach 21.97 (Astra) and 21.69 (Fable), exceeding Wilflame’s own SP return of 21.05. These results show that coding agents’ lower Hanabi scores do not indicate a fundamental ceiling on performance: a simple hybrid approach combines deep RL’s strong task performance with coding agents’ ability to incorporate human conventions, yielding both strong SP and IXP and superior coordination with external partners.

Although hybrid methods are promising, current coding limitations need not persist. LLM-based systems already generate programs containing millions of lines of code (Anthropic, 2026), while Code as Policies and CaP-X demonstrate executable robot policies operating in continuous environments (Liang et al., 2023; Fu et al., 2026). Given this progress, future generations of models may close the remaining Hanabi performance gap using purely code-based policies.

References

Lakshya A. Agrawal et al. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. arXiv:2507.19457, 2025.

Anthropic. Formalizing Fermat’s Last Theorem. Research, September 4, 2026.

Bernardino Romera-Paredes et al. Mathematical discoveries from program search with large language models. Nature, 625:468–475, 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.