acceptodds
Under review as a conference paper at ICLR 2027

PopGen: Generating Semantically Diverse Populations with LLMs for Zero-shot coordination

Abstract

In multi-agent reinforcement learning (MARL), exposing a cooperator to partners with diverse strategies during training is key to better collaboration at test time. However, existing methods typically rely on trajectory-level proxies to measure diversity, which may label two distinct trajectories of the same strategy as diverse. Our key insight is instead to leverage LLMs' ability to describe behaviorally distinct agents in environments, similar to how humans can recognize high-level strategies. To this end, we introduce , a framework that synthesizes semantically diverse agents from scratch using LLMs. first generates a diverse set of textual descriptions of agent behaviors, then translates each behavior into a code policy that can be executed within the environment. We show that training a cooperator against a population achieves robust zero-shot cooperation performance across four collaborative environments, and that the resulting policies adapt to diverse partners. Additionally, we measure how diversity scales with larger populations and provide insights on how population diversity impacts cooperation and how it can conflict with the quality of the agents.

Primary area:
reinforcement learning
Keywords:
Multi-agent Reinforcement Learning, Zero-shot coordination

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ogVZOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes PopGen, a framework for generating semantically diverse teammate populations for zero-shot coordination. It first generates diverse behavior descriptions using LLM-based oversampling, then translates them into executable policy code through iterative refinement with VLM feedback. The generated teammates are used to train a cooperator via population play. Experiments across four environments demonstrate improved coordination with unseen teammates, while ablations investigate semantic diversity, behavior adherence, and policy generation methods.

Critical strengths
  1. Practical and efficient teammate generation. Directly generating executable policies avoids the expensive RL training required by reward-function-based approaches such as SemDiv. The experiments demonstrate improvements in generation efficiency and behavioral adherence, making this a practically valuable alternative.

  2. Promising empirical results. PopGen achieves strong zero-shot coordination performance across four environments. The analyses of population diversity, robustness to unseen teammates, and the diversity-quality trade-off provide useful empirical insights.

Critical weaknesses
  1. Limited methodological novelty over SemDiv and insufficient positioning. The central idea of using LLMs to generate semantically diverse teammate behaviors and grounding them into executable agents was already explored by SemDiv (Li et al., 2025). PopGen's main differences appear to be replacing sequential behavior generation with oversampling and replacing reward-function-based policy training with direct policy code generation. While useful, these modifications seem incremental, and the paper should more clearly distinguish its contributions from SemDiv. The related work should also discuss Macop (Learning to Coordinate with Anyone, Yuan et al., 2023), which explores adaptive generation of diverse teammates based on coordination difficulty. This is particularly relevant to the paper's proposed future direction of online adaptive behavior generation.

  2. Insufficient evaluation of the claimed improvements over SemDiv and semantic diversity. The comparisons with SemDiv are fragmented: Figure 5 compares text generation strategies, representing SemDiv only through multi-turn prompting, while Figure 6 compares policy code against reward-function-based training. Neither evaluates the complete SemDiv teammate generation pipeline under a unified protocol. Although the original methods have different test-time assumptions, a controlled end-to-end comparison of their generated populations would strengthen the claims. Furthermore, the semantic diversity metric relies on distances between text embeddings, which may not reflect meaningful differences between the resulting policies. The authors should validate this relationship and clarify the embedding inputs and normalization procedure (the embedding model itself is specified in Section 5.1).

Questions for authors

See Critical Weaknesses

2 comments
AuthorsOpenReview ↗

Reply to reviewer ogVZ

We thank the reviewer for the thoughtful review and for recognizing PopGen as a practical and efficient alternative to RL-based teammate generation, with strong empirical results across four environments. We address each concern below.

Novelty and positioning relative to SemDiv (W1). We agree that SemDiv first proposed generating teammates from behavior descriptions, and PopGen builds on this idea. Our contribution is to identify and address the two bottlenecks that limit it in practice.

  1. Diversity of the generated behaviors. Generating diverse text with an LLM is not trivial, as diversity often comes at the cost of quality. SemDiv generates behaviors sequentially and does not analyze their diversity. PopGen reaches 30% higher diversity than sequential generation for a 6% drop in quality (Section 5.3, Appendix). Oversampling and merging also scales with a stronger generator, improving both quality and diversity, while sequential generation loses diversity.
  2. Grounding behaviors into agents. Reward functions do not guarantee that the trained policy exhibits the intended behavior; SemDiv relies on rule-based filters, environment-specific thresholds, and manual trajectory description to discard failed policies. PopGen instead writes policy code and refines it with an advisor until rollouts match the description. This raises adherence from 0.43 to 0.66 on average and cuts generation time from about 48 minutes to 8 seconds per policy (Section 5.4, Table 9).

Together, these changes make it practical to generate large, complex populations, which a reward-function pipeline cannot reach at a similar budget. We will state these differences from SemDiv explicitly in the related work.

We thank the reviewer for pointing us to Macop, which generates teammates that are hard for the current cooperator to coordinate with and continually trains the cooperator against them. This is closely related to the online adaptive generation we propose as future work, where PopGen could generate behaviors targeting the cooperator's weaknesses. We will discuss Macop in the related work and conclusion.

End-to-end comparison with SemDiv (W2). We separated the two stages on purpose, so each comparison changes one component while the other is held fixed. Figure 5 compares behavior generation methods on the same environments, and Figure 6 compares policy code and reward functions from the same set of behaviors. We did not run the full SemDiv pipeline for two reasons. First, SemDiv assumes a different test-time setting: at execution, an LLM uses a natural-language description of the teammate's behavior to select the cooperator's best policy head, whereas we assume the cooperator has no information about its partner at test time. Second, SemDiv filters trained policies with manually set, per-environment thresholds (e.g., on task reward and on reward with existing partners), which are hard to reproduce faithfully in new environments. We therefore compared the two components that transfer to our setting, behavior generation and agent grounding. PopGen generates more diverse behaviors than SemDiv, and the policy-code population also yields a stronger cooperator (Figure 6). We will clarify this reasoning in the paper.

Validity of the semantic diversity metric (W3). We agree that distances between text embeddings need not reflect differences between the resulting policies. To test this, for every pair of PopGen policies in each environment we computed both the semantic diversity between their behaviors (Equation 2) and the Jensen–Shannon divergence (JSD) between their trajectory distributions, introduced in TrajeDi. As most PopGen policies are deterministic, we add Gaussian noise (σ=0.25\sigma = 0.25) to continuous actions. The table reports the Spearman correlation ρ\rho between the two over all pairs, with Mantel permutation pp-values. A discount factor γ\gamma sets the comparison horizon, from one-step actions (γ=0\gamma = 0) to whole trajectories (γ=1\gamma = 1).

γ\gammaPMRLBFPPOvercooked
0+0.26 (p<0.001)+0.28 (p=0.061)+0.35 (p=0.005)+0.14 (p<0.001)
0.5+0.24 (p=0.001)+0.35 (p=0.022)+0.30 (p=0.020)+0.15 (p<0.001)
1+0.19 (p<0.001)+0.30 (p=0.054)+0.37 (p=0.004)+0.15 (p<0.001)

The correlation is positive in all 12 settings and significant (p<0.05p < 0.05) in 10, so policies whose descriptions are semantically farther apart also act more differently in the environment. We will add this analysis to the appendix.

Thank you for suggesting we clarify the embedding inputs and normalization. The embedding model takes as input the textual description of each behavior, generated by the LLM in Stage 1. We normalize embeddings to unit length, and Equation 2 takes the Euclidean distance between unit vectors, 2(1−ek⊤eℓ)\sqrt{2(1 - e_k^\top e_\ell)}, ranging from 0 (identical) to 2. We will clarify this in Section 4.1.

Thank you for the detailed response. The new analysis of semantic diversity and the clarification of the embedding procedure address part of my concern. The efficiency and adherence gains are also valuable. The comparison with SemDiv remains at the component level, and I still view the methodological contribution as incremental. I therefore maintain my rating of 3.

Reviewer_rYTCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the zero-shot collaboration (ZSC) problem in multi-agent reinforcement learning: collaborators need to collaborate with partners they have never met before during testing. The core premise is that the effectiveness of population play depends on how well the training population covers the policy space, and existing diversity surrogate metrics, calculated from trajectories, actions, or hidden states, may misjudge two different trajectories under the same policy as "diverse.” The author's key insight is that, when humans describe strategies at a high level, an LLM can use that interface to capture semantic diversity.

POP GEN is a two-stage framework. Phase 1: generate a diverse set of text behavior descriptions using oversampling. Phase 2: translate each description into executable Python policy code using an iterative refinement loop: the generator proposes the policy, the VLM adherence evaluator scores the rollout, and the LLM advisor provides structured feedback for use in the next round. The obtained population was directly used for population play (POP PLAY); collaborators were trained with PPO, and MEP-style rank-based preferred partner sampling was adopted. The experiment covered 4 environments (PMR, LBF, Predator Prey, Overcooked) against 4 population-play baselines (Self-Play, FCP, MEP, CoMeDi).

Critical strengths

The core insights are indeed novel and highly motivated. The criticism of the track-level diversity proxy metric is very persuasive and is based on a specific failure pattern. Using an LLM as a semantic interface, rather than a policy, reward generator, or planner, is a clean and underexplored perspective.

The two-stage decomposition is modular and reasonable. Decoupling "explore diversity in text space" from "ground behaviors in executable code" avoids the quality-diversity tension that plagues single-stage generation. The oversample-and-merge technique (VS-OS) is a simple and fully developed example of this principle.

Critical weaknesses

The held-out partners are scripted, not human, and this is the ZSC paper's core limitation. The paper's Limitations section acknowledges this, but it should be more prominent. ZSC (Hu et al., 2020) is defined as collaborating with humans. The paper cites Carroll et al. (2019) and Liang et al. (2024) as evidence of "migration trained on humanoid populations,” but this paper does not demonstrate such migration. The script partners for each environment are reasonable proxies, but the authors designed them, raising the question of whether they were influenced by the POP GEN semantic vocabulary at the time of design.

The held-out partners may be overly aligned with the semantic space of POP GEN. The paper states that the held-out behavior "never provided to the LLMs during our experiments," which is good. But the authors also designed the prompt and POP GEN's held-out partners. The authors describe the held-out partners using the same natural-language vocabulary as POP GEN (such as "Fill only", "Fetch & Serve", "Stag Only"). If the held-out partners happen to be described by the same semantic vocabulary generated by POP GEN, this may elevate the result relative to the partner distribution from different semantic priors.

The statistical significance is weak, and the Overcooked results do not support it. In 5 runs, the error stick overlapped in PMR and Overcooked, which was not enough to support "POP PLAY attains the highest reward in all four environments."

Questions for authors

The held-out partner design. Was the held-out partner designed before or after the POP GEN output was seen? Who designed it? Is it constrained to use the same semantic vocabulary that POP GEN can use? If so, how does this affect the interpretation of the result?

Overcooked variance. The Overcooked eval reward for POP PLAY is 31.2 ± 44.6, std > mean. Please report the run-by-run figures and the pairing test with CoMeDi. Does the claim that "highest in all four environments" hold true in Overcooked?

Behavioral diversity. The paper measures semantic diversity in the text space, but does not measure behavioral diversity in the environment. Could you report behavioral diversity measures for the POP GEN population vs. the baseline (e.g., state-access divergence, cross-play reward variance) and show that semantic diversity predicts behavioral diversity better than track-level proxy metrics?

LLM evaluation verification. The paper uses Gemini-3-Flash as the generator, adherence classifier, and quality judge. Could you report the adherence and mass scores from independent judges (e.g., GPT-5.6 or Claude), along with the consistency rate?

Zhao et al. — Maximum Entropy Population-Based Training, missed one author.

2 comments
AuthorsOpenReview ↗

Reply to reviewer rYTC (part 1/2)

We thank the reviewer for the detailed review and the insightful questions. We appreciate the reviewer recognizing the novelty of the method, its design choices, and its effectiveness. Below, we address each question.

Held-out partner design (Q1). We curated held-out partners to test coordination with diverse behaviors. Following SemDiv, we identify key aspects of the environments to derive the behaviors (e.g. wing attacker, defender in football), and implement them as scripted partners (i.e. a python script). We use an LLM to do the latter, over multiple attempts. For a fair evaluation, the behaviors were designed prior to seeing any PopGen output, and we manually inspected PopGen outputs to avoid clear overlap with the held-out set.

To address the concern about scripted held-out partners, we also evaluated PopPlay on three Overcooked layouts from ZSC-Eval (Counter Circuit, Coordination Ring, and Forced Coordination), against its published baselines. ZSC-Eval releases checkpoints of neural-network partners with diverse behaviors and skill levels, which we use as held-out evaluation partners. Following ZSC-Eval, we report BR-Prox, the interquartile mean of the cooperator's return with each partner divided by the return of that partner's best response. Its authors show that rankings of this evaluation protocol closely match those from human evaluations. PopPlay outperforms the strongest published baselines on all three layouts, improving BR-Prox by 10.4% on average (17.5% on Counter Circuit, 10.7% on Forced Coordination, and 2.9% on Coordination Ring). Its generalization therefore extends beyond our scripted partners to learned neural-network partners, although a human study remains the definitive test (Limitations). We will add these results to the final paper.

MethodCounter CircuitCoordination RingForced Coordination
PopPlay (ours)0.818 ± 0.0340.785 ± 0.0350.764 ± 0.049
Self-Play0.105 ± 0.0470.273 ± 0.0430.565 ± 0.108
FCP (Strouse et al., 2021)0.590 ± 0.0610.706 ± 0.0210.520 ± 0.073
TrajeDi (Lupu et al., 2021)0.686 ± 0.0330.714 ± 0.0240.546 ± 0.030
MEP (Zhao et al., 2023)0.696 ± 0.0090.763 ± 0.0060.524 ± 0.070
HSP (Yu et al., 2023)0.568 ± 0.0330.650 ± 0.0170.667 ± 0.042
COLE, 25 partners (Li et al., 2023)0.409 ± 0.0610.463 ± 0.0920.507 ± 0.050
E3T (Yan et al., 2023)0.531 ± 0.0320.591 ± 0.0260.690 ± 0.038

Overcooked variance (Q2). We thank the reviewer for raising this. We first clarify that 31.2 ± 44.6 is PopPlay's worst-case reward in Overcooked (Figure 3c), i.e., the reward with the held-out partner on which the cooperator scores lowest in each run. PopPlay's average reward over the four held-out partners is 120.9 ± 59.3 (Figure 2, Table 4), and Figure 2 shows each run's average reward as a dot. We realize that the placement of Figure 3c made the two numbers easy to confuse, and we will label them more clearly.

The run-by-run average rewards in Overcooked are:

MethodRun 1Run 2Run 3Run 4Run 5Mean ± std
PopPlay (ours)208.9116.441.9114.7122.6120.9 ± 59.3
CoMeDi44.096.673.387.9129.986.3 ± 31.5
Self-Play132.54.9134.339.36.063.4 ± 65.4
FCP18.06.042.6130.06.040.5 ± 52.2
MEP33.65.998.619.910.933.8 ± 37.7

Most of PopPlay's variance comes from runs 1 and 3. The other three runs lie between 114 and 123. Overcooked has larger run-to-run variance than the other environments for all methods.

PopPlay indeed has more sources of variance. Each run generates a new set of behaviors and new policy code, so the training population itself differs from run to run. As noted in our Limitations, this makes PopPlay's performance more dependent on the generated population. Nevertheless, its std in Overcooked (59.3) is comparable to Self-Play's (65.4) and FCP's (52.2), and PopPlay attains the highest mean reward.

AuthorsOpenReview ↗

Reply to reviewer rYTC (part 2/2)

Trajectory diversity in PopGen (Q3). We thank the reviewer for this suggestion. We analyzed the behavioral diversity of each Overcooked population with the generalized Jensen-Shannon divergence (JSD) between the trajectory distributions of a population's policies, as introduced in TrajeDi. A discount factor γ\gamma sets the horizon of comparison, we report γ∈{0,0.5,1}\gamma \in \{0, 0.5, 1\} (i.e. from single action, to full trajectory). We report results in the table below.

The results show that trajectory diversity does not necessarily track cooperator performance in Overcooked. MEP has the highest JSD0\mathrm{JSD}_0 and JSD0.5\mathrm{JSD}_{0.5}, yet its cooperator is the weakest (Figure 2). This is expected, since MEP is trained with a population-entropy objective that directly rewards partners for acting differently. The strongest baseline, CoMeDi, ranks third in JSD0\mathrm{JSD}_0 and second in JSD0.5\mathrm{JSD}_{0.5}. PopGen ranks second in JSD0\mathrm{JSD}_0 and lowest in JSD0.5\mathrm{JSD}_{0.5}, while its cooperator performs best. At γ=1\gamma = 1, every baseline reaches the ceiling of ln⁡4≈1.386\ln 4 \approx 1.386, since over long episodes almost every pair of policies produces disjoint trajectories, so JSD1\mathrm{JSD}_1 cannot tell the populations apart.

These results suggest that trajectory-level measures can be optimized without covering more strategies, and thus may not reflect the diversity that matters for coordinating with semantically different partners. This supports measuring diversity at the semantic level. We will add these results to the paper.

PopulationJSD0JSD0.5JSD1
PopGen (ours)0.696 ±\pm 0.1580.836 ±\pm 0.1831.323 ±\pm 0.111
Self-Play0.413 ±\pm 0.0730.913 ±\pm 0.0751.386 ±\pm 0.000
FCP0.392 ±\pm 0.0540.877 ±\pm 0.0691.386 ±\pm 0.000
MEP0.741 ±\pm 0.0941.220 ±\pm 0.0851.386 ±\pm 0.000
CoMeDi0.642 ±\pm 0.0200.998 ±\pm 0.0331.386 ±\pm 0.000

LLM evaluation verification (Q4). We thank the reviewer for this suggestion. To assess the adherence judge's performance, we re-judged a random subset of 200 rollouts with the original Gemini-3-Flash judge and two other judges, GPT-6-Luna and Gemini-3.8-Flash. All the judges used the same prompt, only the LLM differs. For each judge, we compute the precision, recall and F1 for the performance, as well as self-consistency to capture the variance (over 5 trials). The rollout golden-truth are obtained via a combination of strong LLM (GPT-6-sol) and human annotation.

Gemini-3-Flash – the judge employed in the paper – has high precision (92%), so rollouts it marks as adherent are almost always adherent. Its main error is low recall (45%), as it misses about half of the adherent rollouts. Most of these (46/53) contain long waiting phases that the 30 sampled states cannot distinguish from being stuck. The adherence rates reported in the paper are therefore conservative (45% vs. 69% in the golden annotations). The results show that using more recent models can improve the performance (Gemini-3.8-Flash keeps precision high (85%) while raising recall from 45% to 79%). We will add these results to the appendix.

JudgeAdherence ratePrecision vs. GTRecall vs. GTF1 vs. GTSelf-consistency
Gemini-3-Flash (paper's judge)37.791.644.960.390.6
GPT-6-Luna35.289.546.461.190.0
Gemini-3.8-Flash65.684.979.382.090.9
Reviewer_moVUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes POPGEN, a framework for zero-shot coordination that uses LLMs to generate semantically diverse agent populations. The methodology operates in two primary stages: first, an LLM generates a diverse set of natural-language behavior descriptions, carefully balancing semantic diversity with the overall quality and realism of the behaviors. Second, a code generator translates these high-level textual descriptions into executable agent policies. These policies are rolled out in the environment, and a VLM acts as a judge to evaluate how well the agent's actions adhere to the intended semantic behavior, to provide feedback for iterative refinement. Ultimately, a cooperator agent trained against this synthetic population demonstrates robust zero-shot coordination when paired with unseen, held-out partners across multiple collaborative environments. Finally, a cooperator agent trained against this synthetic population demonstrates good zero-shot coordination when paired with unseen, held-out partners across multiple collaborative environments.

Critical strengths
  • 1. The paper is well-written, logically structured, and easy to follow, making the multi-stage framework highly accessible.

  • 2. The POPGEN pipeline is thoughtfully designed. The approach to synthesizing a diverse set of behavior policies from high-level semantics, coupled with LLM based judging and the iterative loop of VLM-based refinement. provides a robust solution to generating meaningful diversity for zero-shot coordination.

  • 3. The work provides valuable empirical insights into population scaling for cross-play training. By analyzing how semantic diversity grows with population size and directly impacts downstream cross play performance, the paper helps the multi-agent reinforcement learning community address the difficult question of how many diverse agents are required for efficient training and optimal performance.

Critical weaknesses
  • 1. While generating executable "Policy Code" from semantics is simple and elegant for grid-world environments (like Overcooked or Level-Based Foraging) where behaviors are easily coded via heuristics, this approach may not be practical for highly complex games like Hanabi, GRF, or SMAC. Writing a purely scripted, competent policy for environments with massive strategy spaces and complex dynamics is notoriously difficult for LLMs without relying on an RL backbone.

  • 2. The ablation proves that Python "Policy Code" achieves higher adherence than RL reward functions, hard-coding policies as scripts inherently limits the capability ceiling of the generated agents. Neural network policies can learn highly nuanced, reactive control in noisy state spaces, whereas LLM-written Python heuristics are typically rigid and rule-based. This forces a trade-off where the training population might be semantically diverse but mechanically brittle or sub-optimal in skill level.

  • 3. Supporting the scalability concern, the paper's own data shows that the framework struggles to enforce behavior in complex games. While the Policy Code approach achieves high adherence in simple discrete environments, Table 11 shows that for Football, the best-attempt adherence on the test set is only 34.7%. This indicates the refinement loop struggles to successfully patch code for complex dynamics.

  • 4. Zero-Shot Coordination is primarily motivated by the need to train agents that can collaborate effectively with novel human partners at test time. However, the paper evaluates POPPLAY exclusively against synthetic, held-out scripted partners. Without a human study, it remains an open question whether the semantic diversity generated by LLMs accurately reflects the types of diversity needed for human-AI teaming.

  • 5. The zero-shot coordination evaluation relies on a very small, hand-crafted set of just four scripted partners per environment (Table 2). These held-out partners represent extreme, one-dimensional strategies (e.g., in Overcooked: "Fill only" or "Wait & Serve"). Successfully coordinating with four simplistic, extreme heuristic scripts does not necessarily prove true zero-shot robustness against a complex, naturally diverse test distribution.

  • 6. The paper emphasizes the low inference latency of the final Policy Code, but glosses over the massive computational cost of the generation phase. The iterative refinement loop (Algorithm 1) requires multiple environment rollouts evaluated frame-by-frame by a VLM judge for every candidate policy. Scaling this pipeline to larger populations or longer-horizon tasks would incur prohibitive API costs and latency.

  • 7. The entire pipeline hinges on the environment_description fed to the LLM in Stage 1. The paper does not provide an ablation on how sensitive the generated semantic diversity is to the phrasing, completeness, or granularity of this initial prompt. If an environment is too complex to describe perfectly in a text prompt, the Stage 1 behavior generation will likely fail to cover the true strategic space.

Questions for authors
  • 1. Given the significant drop in best-attempt test adherence for Football (34.7%), how can the purely scripted "Policy Code" approach scale to complex environments (e.g., Hanabi, SMAC) where heuristic scripting is generally intractable?

  • 2. Does the inherent rigidity of hard-coded Python scripts result in mechanically brittle or sub-optimal training partners compared to RL-trained baselines, and does this limit the cooperator's capability ceiling?

  • 3. The current evaluation relies on just four extreme, one-dimensional heuristic scripts per environment. Can you provide preliminary results evaluating POPPLAY cooperators against more naturalistic or human-proxy partners (e.g., human behavior cloning)?

  • 4. What is the total computational overhead (API cost and wall-clock time) required for the LLM based diverse policy generation, code generation, judge LLM and VLM-based iterative refinement loop, and does this become a bottleneck for scaling?

  • 5. How sensitive is the Stage 1 behavior generation to the phrasing, completeness, and granularity of the environment_description prompt?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer moVU (Part 1/2)

We thank the reviewer for the detailed review and the insightful questions. We appreciate that the reviewer recognized the clarity of the method's presentation, its practical significance, and the empirical analysis regarding scaling of the method.

Below, we address each question.

Policy code in complex environments (Q1). Policy Code scales to complex environments with discrete actions, and its limitation lies in continuous action spaces. Table 11 reports adherence on two environments harder than our main four: SMACv2, with discrete actions, and Football, with continuous control. In discrete environments, Policy Code achieves the highest test adherence against both alternatives. It reaches 51.5% on SMACv2, vs. 45.8% for LLM-A and 33.0% for Reward Function. It reaches 61.0% on LBF (vs. 25.4% and 12.5%) and 63.3% on Overcooked (vs. 45.3% and 40.7%). On SMACv2, refinement raises adherence by +40.0 points on dev and +25.5 on test, the largest gains across environments.

In continuous environments, Policy Code keeps the best average test adherence (68.9% vs. 62.3% for Reward Function). However, its margin over the best alternative falls from +19.8 to +6.6 points. The gap between dev and test also doubles (24.8 vs. 10.4 points), e.g., from 99.0% to 66.0% on Football. A likely cause is that continuous control requires hard-coded thresholds that overfit to the dev initializations. For example, Football Policy Code designs many hand-picked constants for their strategy logics, such as distance gates for when to attack and when to defend, which may not generalize to stronger opponents with faster movements.

Limitation of scripted partners (Q2). A strong cooperator requires partners that reliably exhibit their intended behaviors, which we measure as adherence (Section 4.2). In our experiments, scripted Policy Code partners achieve the highest adherence: 63.8% on test, vs. 45.5% for Reward Function and 42.2% for LLM-A (Table 11). With the behavior set held fixed, cooperators trained against Policy Code partners outperform those trained against Reward Function partners in all four environments (Figure 6).

As we discussed in Q1, Policy Code partners can be brittle because of hard-coded thresholds. However, they show intended behavior even when they fail, just miscalibrated. A Football attacker still shows ball pushing behavior even if their decision logic is slightly off. Reward Function partners often fail by never producing the behavior in unseen configurations, such as diverging to the corner of the field or oscillations. Because each Policy Code partner keeps their behavior consistent, the cooperator gets a cleaner training signal for adapting to each partner.

Additionally, Policy Code partners are cheaper to generate and rollout (Table 13). Under a fixed budget, the cooperator thus could see more partners during training.

PopPlay with human-like partners (Q3). We added an evaluation of PopPlay on three Overcooked layouts from ZSC-Eval (Counter circuit, Coordination ring, Forced coordination), against its published baselines. ZSC-Eval releases checkpoints of neural-network partners with diverse behaviors and skill levels, which we use as held-out evaluation partners. Following ZSC-Eval, we report BR-Prox, the interquartile mean of the cooperator’s return with each partner divided by the return of that partner’s best response. Its authors show that rankings of this evaluation protocol closely match those from human evaluations. PopPlay outperforms the strongest published baselines in all three environments, improving BR-Prox by 10.4% on average (17.5% on Counter circuit, 10.7% on Forced Coordination, and 2.9% on Coordination ring). Its generalization therefore extends beyond our scripted partners to learned neural-network partners, although a human study remains the definitive test (Limitations). We will update the final paper with these results.

MethodCounter CircuitCoordination RingForced Coord.
PopGen (Ours)0.818 ± 0.0340.785 ± 0.0350.764 ± 0.049
Self-play0.105 ± 0.0470.273 ± 0.0430.565 ± 0.108
FCP (Strouse et al, 2021)0.590 ± 0.0610.706 ± 0.0210.520 ± 0.073
TrajeDi (Lupu et al., 2021)0.686 ± 0.0330.714 ± 0.0240.546 ± 0.030
MEP (Zhao et al., 2023)0.696 ± 0.0090.763 ± 0.0060.524 ± 0.070
HSP (Yu et al., 20230.568 ± 0.0330.650 ± 0.0170.667 ± 0.042
COLE (25 partners) (Li et al., 2023)0.409 ± 0.0610.463 ± 0.0920.507 ± 0.050
E3T (Yan et al., 2023)0.531 ± 0.0320.591 ± 0.0260.690 ± 0.038
AuthorsOpenReview ↗

Reply to Reviewer moVU (Part 2/2)

Computational overhead (Q4). Stage 1 is a one-time cost of about 254 s per behavior set (Figure 9b). Stage 2 makes up to 5 attempts per behavior. Each attempt uses one generator call, M adherence calls to the VLM (both Gemini-3-Flash), and one advisor call (Gemini-3-Pro). Policy Code and Reward Function share this loop and produce code of similar length, so their API costs are comparable. At OpenRouter list prices, we estimate Stage 2 costs at around $0.60 per behavior and $1.00 per aligned agent (Overcooked run, 177 behaviors, up to 3 attempts, 2.0 on average). Around 60% of this comes from the advisor's reasoning tokens and ~35% from the VLM adherence calls.

In wall-clock time, Policy Code is the fastest method in Table 13, with generation about 355× faster than Reward Function. The baselines must also train their partner populations with MAPPO, a step that PopGen replaces with code generation. Once generated, Policy Code partners add negligible compute on top of environment steps (Section 4.3), which leaves refinement as the main bottleneck. Its cost is tunable: more attempts raise the best adherence (e.g., in Table 14, one attempt reaches 56.7% test adherence and five reach 66.3%).

Sensitivity of Stage 1 to the input prompt (Q5). We generate sets of 30 behaviors in Overcooked, using different variants of the original description as input to PopGen's Stage 1. We use Gemini-3.8-Flash as the generator and GPT-6-Luna as the quality judge, over 3 seeds. As reported in the table below, Stage 1 is robust to how the description is written but sensitive to what it contains. Paraphrasing, reordering the prose, or halving the description by removing implementation details keeps diversity (±0.03) and quality (±0.3/15) at the original level. The content of the generated sets also stays roughly the same: Chamfer similarity (the mean similarity of each behavior to its closest counterpart in the other set) is 0.80–0.81, vs. 0.82 between seeds of the original.

Without the coordination rules, diversity inflates (0.95 vs. 0.80), but quality falls by 3.4/15, because the added variety comes from behaviors that violate the specification. A coarse 142-word summary preserves diversity but lowers quality by 1.4/15 and drifts from the original content (Chamfer similarity 0.73). For practitioners, stating the task constraints clearly matters more than tuning the wording. We will add this analysis to the final version of the paper.

Description (words)MutationDiversity ↑Quality /15 ↑Chamfer sim. vs original
original (1671)— (seed noise floor)0.80 ± 0.0112.50.82
paraphrase (1796)phrasing0.82 ± 0.0212.30.80
prose_reordered (1359)phrasing / format0.80 ± 0.0312.30.80
no_impl_details (837)completeness0.81 ± 0.0112.20.81
no_coordination_rules (1227)completeness0.95 ± 0.029.10.74
coarse_summary (142)granularity0.79 ± 0.0411.10.73

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.