acceptodds
Under review as a conference paper at ICLR 2027

Trait-Consistent Population Modeling for Human–AI Zero-Shot Cooperation

Abstract

Human–AI and multi-agent zero-shot cooperation (ZSC) requires agents that generalize to unseen human and novel agent behaviors. Population-based training is a central paradigm for ZSC, with the key challenge of designing training populations that capture the structured diversity of human behavior. To cope with this challenge, we introduce **Trait-Embedded Agent Modeling (TEAM)**, a framework for inducing human-aligned behaviors in the training population. Drawing on studies of human gameplay, particularly the widely adopted Bartle's taxonomy, our proposed population exhibits traits inspired by general behavioral patterns observed in social games, without requiring task-specific reward modifications. We instantiate this idea through three lightweight modifications to the learning objectives, yielding a population of socializers, achievers, and explorers. To support our approach of pattern-focused diversity, we formalize ZSC as training on a set of Markov decision processes (MDPs), with a newly sampled MDP at test time. This connects ZSC to existing generalization theory and quantitatively motivates focused population coverage. Our evaluation shows that TEAM outperforms strong baselines on AI-AI and real-human tests on challenging ZSC benchmarks, including Overcooked and StarCraft. We further provide an empirical space-coverage study of the trained population, showing TEAM induces semantically meaningful and interpretable state-action space coverage.

Primary area:
reinforcement learning
Keywords:
Human–AI, Multi-agent RL, Zero-shot Coordination

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_s44bOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TEAM, a population-training method for human–AI zero-shot cooperation. TEAM organizes training partners into three human-inspired traits: socializers, achievers, and explorers. These traits are implemented through empowerment, reward reallocation, and entropy regularization, respectively. A recurrent ego agent is then trained against the frozen population. Experiments cover Coin Game, SMAC, Overcooked v1/v2, and a 32-participant Overcooked study.

Critical strengths
  • The paper addresses a meaningful practical issue: a training population should represent plausible human behavior rather than maximize diversity without direction. Using interpretable behavioral priors makes the population easier to understand and control.

  • The paper is generally well written and easy to follow. The two-stage training procedure, trait implementations, and evaluation questions are presented in a clear structure.

Critical weaknesses
  • The problem and the proposed traits have limited novelty. Training against heterogeneous partners, modeling human preference differences, and adapting to unseen partners have been studied by FCP (Strouse et al., 2021), MEP (Zhao et al., 2023), HSP (Yu et al., 2023), CoMeDi (Sarkar et al., 2023), and GAMMA (Liang et al., 2024), while Melting Pot evaluates social generalization across diverse partners and mixed incentives (Leibo et al., 2021; Agapiou et al., 2022). Socializer, achiever, and explorer are also taken directly from Bartle’s taxonomy. The main novelty is therefore using a fixed human-inspired taxonomy to organize a ZSC training population, rather than introducing a new cooperation problem or behavioral abstraction.

  • The experiments do not isolate the value of the taxonomy. TEAM simultaneously introduces three additional objectives, multiple coefficient levels, mixed-trait agents, and several checkpoints. The single-trait ablation only shows that combining the three populations is better than using one population alone. It does not show that the Bartle structure is better than an equally sized population built from the same objectives and coefficients without trait labels.

  • The comparison is incomplete for the paper’s novelty claim. The main experiments include FCP, MEP, HSP, E3T, and CEC, but omit convention-based and data-driven style populations such as CoMeDi (Sarkar et al., 2023) and GAMMA (Liang et al., 2024). The method is also not evaluated on a broader mixed-motive social-generalization benchmark such as Melting Pot (Leibo et al., 2021; Agapiou et al., 2022). Coin Game provides only a limited test of cooperation–competition trade-offs, while the remaining tasks largely assume a shared objective.

References

  • Strouse, D. J., et al. (2021). Collaborating with Humans without Human Data. NeurIPS 2021.

  • Zhao, R., et al. (2023). Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination. AAAI 2023.

  • Yu, C., et al. (2023). Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased. ICLR 2023.

  • Sarkar, B., Shih, A., & Sadigh, D. (2023). Diverse Conventions for Human-AI Collaboration. NeurIPS 2023.

  • Liang, Y., et al. (2024). Learning to Cooperate with Humans using Generative Agents. NeurIPS 2024.

  • Leibo, J. Z., et al. (2021). Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot. ICML 2021.

  • Agapiou, J. P., et al. (2022). Melting Pot 2.0. arXiv:2211.13746.

Questions for authors
  • How does TEAM differ conceptually from HSP’s hidden-utility populations and GAMMA’s latent partner styles beyond using three named traits?

  • Why are CoMeDi and GAMMA not included as empirical baselines? Would the conclusions hold on a mixed-motive social-generalization benchmark such as Melting Pot?

  • Can the authors validate the three traits using an independent human-behavior clustering or human-coded annotation, rather than profiles generated by TEAM itself? Are the human-study differences statistically significant?

5 comments
AuthorsOpenReview ↗

Reponse 1/3 - Weakness 1 (Novelty)

Dear Reviewer s44b,

We thank you for your detailed feedback and the time dedicated to reviewing our work. We considered each point you raised very carefully and added new experiments, ablations, and clarification of novelty over prior work. Our response is given below.


Summary of our response:

(1) W1: Clarified novelty vs. HSP and GAMMA. (2) W2: Ablation and trait justification replacing our traits with no-label objectives (3) W3: New experiments on Melting Pot (Harvest, Stug), where TEAM outperforms all baselines. (4) Q3: New Appendix G: detailed human analysis.


Weaknesses:

  1. Novelty (We answer here both W1 and Q1). The paper does not suggest that the novelty is derived from introducing heterogeneous partners or adaptation. Our main novelty is a new method for capturing general personality traits. This is an unexplored dimension of human behavior in ZSC, which enables a single human model across tasks. In contrast, prior methods such as HSP build a separate human model for each task from utilities over events.

Why do the per-task utilities of HSP differ from our traits approach? HSP draws on behavioral economics and models humans via a personal utility, while TEAM models traits. In personality psychology, traits are general dispositions that shape human behaviors across many situations [2], whereas in economics, a utility represents a person’s preferences over a given set of options [3]. The two constructs can differ: for instance, economic preferences and personality traits have been found to be only weakly correlated and not interchangeable [4]. In our settings, the key difference is what each method’s human model is defined over. In HSP, the options are hand-designed task events, so a separate utility model must be designed for each task. For example, in Overcooked, HSP constructs bias around placing a tomato in the pot; this bias does not exist in SMAC, where a new utility model had to be rebuilt. In contrast, TEAM’s traits are defined over styles of engagement (e.g., with a partner) rather than over specific tasks: a socializer maximizes its partner’s influence on future states, similarly, in every environment, so the same human modeling applies across tasks. We believe that this task-independent approach is a key step toward making human-aligned ZSC practical across diverse tasks.

GAMMA human-aligned partner generation requires collecting actual human trajectories. This represents a different approach from ours: rather than modeling human behavior through learning objectives, GAMMA learns to reproduce diversity from collected human trajectories. Note that the human data is optional here as GAMMA can produce trajectories from non-human data as well, but this aspect of GAMMA, which uses encoder-based generation, differs substantially from our work.

Other baselines. Methods such as MEP, FCP, E3T, and CoMeDi are not designed to model human behavior, so we refer to the related work section for the discussion of their differences from our approach.

Regarding novelty of the Bartle taxonomy, Viewing the use of existing human trait taxonomies as a limitation of novelty somewhat overlooks the central idea behind our work. The paper's innovation lies in drawing on human-behaviour literature and deriving our traits from established, observed taxonomies that capture real human traits. This is essential to ensuring that our traits are meaningful and human-grounded. Conversely, introducing our own novel traits or taxonomy would lack human grounding and weaken the paper's central motivation.

We thank you for the valuable feedback. We will incorporate the corresponding clarifications into the revised paper within the next day or two, before the end of the discussion period. The changes will be marked in red.

AuthorsOpenReview ↗

Reponse 2/3 - Weaknesses 2-3a (Trait Abalation and Environments)

  1. Ablation and trait justification. We assume the reviewer means one of three:

(i) No traits at all. We use the same population size, checkpoints, architecture, and training budget but without any trait modification. We run this ablation with and without the self-play round in phase 2.

(ii) Replacing objective with non-trait terms. We first replace only the explorer with lower temperatures, which removes the “trait labels” of the explorer but leaves the entire mechanism and other traits intact. Similarly, we replace the MI term of the socializer with I(ait; st+k∣st)I(a^{t}_{i};\, s^{t+k} \mid s^{t}), in which the socializer optimizes the empowerment term on its own action and future state rather than on the other agent, but uses the same αi\alpha_i coefficients. This removes only the socializer label but leaves all other traits intact.

(ii) Each of the K agents is a jointly random draw of (α_soc, α_ach, α_exp) over the same ranges, with the same K, checkpoints and budget. I.e., and sampled agent gets values of (0.6, 1, 0.01) for the coefficients. We include it under “joint sampled”.

Comparison is in the following Table:

coord ringcountercrampedforcedassym advantage
No traits and self-play148.9$\pm$ 7.281.08$\pm$9.5183.9 $\pm$6.639.8$\pm$5.4146.2$\pm$31.55
No traits and no self-play143.3$\pm$8.574.09$\pm$ 9.56183.2$\pm$ 10.337.8$\pm$ 6.4129.9 ±\pm 33.2
No explorer label189.2$\pm$ 7.4110.4$\pm$ 6.9190.3$\pm$4.648.3$\pm$5.7263.6$\pm$20.8
No socializer label185.2$\pm$ 7.1112.3$\pm$ 5.1186.3$\pm$5.149.7$\pm$6.2269.9$\pm$22.9
Jointly sampled180.8$\pm$ 6.3103.3$\pm$ 8.2177.3$\pm$4.444.1$\pm$5.9241.2$\pm$24.3
TEAM (original)197.9$\pm$9.9126.1$\pm$ 7.8195.1$\pm$7.155.1 ±\pm 6.9283.2$\pm$ 24.1

If you intended us to run an additional comparison, please let us know, and we will be happy to run it.

3a. Additional environments. To address your concerns, we added experiments on Melting Pot’s Harvest and Stug Hunt, evaluating AI-AI (ego agents from all methods against one another). Note that to our knowledge, population-based ZSC research, as cited in our paper (as well as all the works cited in your response, e.g., HSP, MEP), is designed and tested primarily on 2-player settings. Since Melting Pot is mainly designed for collaboration in larger groups, many of its settings are not suitable. For our experiments, we use Harvest and Stug in 2-player settings to remain consistent with the original baseline design. We selected these environments because they support meaningful 2-player collaboration.

TEAMFCPMEPHSPE3T
Harvest17.9 $\pm$2.510.1 ±\pm 4.47.6$\pm$5.911.9 ±\pm 2.39.1 ±\pm 2.6
Stug hunt in the matrix223.94 ±\pm 4.91210.3 ±\pm 4.3211.9 ±\pm 4.0215.1 ±\pm 3.07211.3$\pm$ 4.52

Harvest (Commons Harvest) is a common-pool-resource dilemma: agents are rewarded for eating apples, yet apples regrow only if enough neighbors' trees remain uneaten, testing restraint of agents in eating apples versus over-harvesting. In Stag Hunt, each player collects stag or hare resources and then zaps the other to interact, and the payoff rewards both choosing stag (high but risky) over hare (low but safe). Experimental specification. We use 12 seeds. The table shows standard errors. Training and compute budget fixed among all methods. Since all the cited diversity techniques are designed for 2-player settings, we use Harvest and Stug with only two players. The results presented include both cross-seed and cross-play. We use the main achiever mechanism based on reward allocation for TEAM.

HSP biases: Harvest: greedy harvester, sustainable harvester. Stug: Prefers stag resources, prefers hare resources, avoids interacting, Zaps as soon as possible.

Due to the limited rebuttal time, we were able to obtain the additional empirical results but have not yet completed their full analysis. We will provide a detailed analysis in the revised paper.

AuthorsOpenReview ↗

Reponse 3/3 - Weakness 3b (Baselines) and Questions

3b.Additional baselines. CoMeDi and GAMMA were not included originally since they are not implemented on the JAXMARL framework, on which our experiments are conducted. Since GAMMA is an improvement over CoMeDi with better reported results (they report in their paper a GAMMA+CoMeDi column), we focus on the GAMMA comparison. The table shows their results on the 5 Overcooked-v1 layouts:

coord ringcountercrampedforcedassym advantage
GAMMA+CoMeDi142.5$\pm$8.781.4$\pm$9.5160.08$\pm$7.944.7 ±\pm 7.1198.1$\pm$ 33.7
GAMMA+MEP157.2$\pm$7.6104$\pm$ 12.6162.3$\pm$7.548.3 ±\pm 6.7245.1$\pm$ 22.4
TEAM197.9$\pm$9.9126.1$\pm$ 7.8195.1$\pm$7.155.1 ±\pm 6.9283.2$\pm$ 24.1

For GAMMA to run with TEAM and our baselines, we had to convert the JAX state into a suitable input representation for GAMMA. The JAX official implementation states [1]: “The implementation aims to be as close as possible to the original Overcooked-AI environment, including dynamics, collision logic, and action and observation spaces.” So no major differences should exist between JAX and the original Overcooked. Yet, since this raises some mismatch between different baselines, we leave it as a side comparison (which will be included in the revision).

Questions:

  1. TEAM versus GAMA and HSP. Addressed in weakness 1.

  2. Additional baselines and Meltingpot. Both parts of the question were addressed in weakness 3.

  3. Trait validation and statistical significance. The six heatmaps' behavioural categories (plate/onion/soup pickups, etc.) that we also used to produce Fig 5b are standard Overcooked event counts, and match the categories used to identify partner behaviours in the ZSC-Eval Overcooked benchmark [5]. We added the citation to the paper already for clarity and to ground the work in existing literature. Regarding further human identification, we uploaded a revised version of the paper. We refer you to Appendix G. We conducted a detailed study in which we visualize human trajectories without relying on these categories and show higher similarity to the TEAM population than to the baseline methods. Our main findings are that (i) TEAM and its three traits produce, more frequently than baselines, trajectories that were similarly observed in human players, as given in Sec. G.1G.1 Tables 8 and 9, and (ii) when comparing action selection and per-tile visitation over the entire layout, the TEAM population exhibits higher overall similarity to human trajectories than baselines. This is given in Sec. G.2G.2 Table 11. A summary of the results will be incorporated into the main paper in a subsequent revision (ICLR allows one additional page if a paper is accepted). The revised paper also includes significance tests on the human study (see Fig. 5 and explanation in Appendix C.3).

We hope the clarifications and additional results address the reviewer's concerns. We look forward to your reply and would appreciate the opportunity for further discussion should any questions remain.


[1] https://jaxmarl.foersterlab.com/environments/overcooked/

[2] Fleeson, W., & Jayawickreme, E. (2015). Whole trait theory. Journal of Research in Personality, 56, 82–92.

[3] Mas-Colell, A., Whinston, M. D., & Green, J. R. (1995). Microeconomic Theory. Oxford University Press.

[4] Becker, A., Deckers, T., Dohmen, T., Falk, A., & Kosse, F. (2012). The relationship between economic preferences and psychological personality measures. Annual Review of Economics, 4, 453–478.

[5] Wang, Xihuai, et al. "Zsc-eval: An evaluation toolkit and benchmark for multi-agent zero-shot coordination." Advances in Neural Information Processing Systems 37 (2024): 47344-47377.

Reviewer_cQydOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduced a psychology/cognitive science informed algorithm for inducing human-like behavior in zero-shot multi-agent coordination. They present results showing this method leads to identifiable structure in the behavior of output agents and that it leads to an agent with better zero-shot AI-AI and Human-AI collabroation. Their algorithm works in 2 phases where you first train "trait-based" partners which use training objectives inspired by psychology/cognitive science literature (but not the math of models in those fields as far as I can tell) and then trains an agent to collaborate with these trait-based partners. They show that this 2-stage process leads to better zero-shot collaboration.

Critical strengths

This paper has a really nice story about how one could use psychology/cognitive science to inform human-AI collaboration systems. The setup is natural enough and the results are to be comprehensive, covering many of the pieces one would need for this kind of result: (1) They show that their methods lead to AI agents with identifiable structure in their behavior at a statistical level of analysis (2) They show their method leads to better zero-shot AI-AI perfomance and human-AI performance (3) They have some generalization-bounds that I didn't look at clearly but could be nice. I'm more an expert on the intersection of cognitive science & AI, less a theory-person so I'll let other reviewers critique this. But this is a nice addition.

Critical weaknesses

I described it above but I'll summarize it here. I think this story is not yet complete, and unfortunately, for this kind of paper I think the story needs be complete for the results to be meaningful.

What the story needs:

  1. Method produces more human-like behavior (claim this but not enough evidence yet)
  2. We identify qualitative (at level of visualizing trajectories of behavior) human-like behavioral traits that lead to better AI-AI or Human-AI performance (missing)
  3. We show that this is not cherry-picked and we show this behavior shows up statistically (this is closest to what they have, but missing ontological grounding to make meaningful, i.e. 2)
  4. We show that this leads to better performance. Here an argument for why (2) or (3) plausibly lead to better performance is fine. But they actually don't even give this argument.

I think for this kind of story 2+3 are needed. Its possible to generate the results with further analysis on existing trajectories. I welcome the authors to do this.

Questions for authors

Q: why is gthe CEC performance so low? in their paper, don't they show that their method outperforms FCP? Did you change how CEC is trained? Q: why is table 2 missing CEC? Q: why do you call the analysis L1? It seems like you're not actually computing L1 you're looking at action-likelihood? why is action-lielkhood a good number?

3 comments
AuthorsOpenReview ↗

Reponse 1/2 - Weaknesses

Dear Reviewer cQyd

We thank you for your insightful comments, which we found extremely helpful. Thanks to your suggestions, we believe we improved the presented story about aligning ZSC with human trait taxonomies. We welcome you to read our reply and the revised paper, as detailed below.

Weaknesses:

  1. We uploaded a revised paper with additional analyses in Appendix G. A summary of the results will be incorporated into the main paper as well in a subsequent revision (ICLR allows one additional page in the final revision). We first summarize the new analyses and then explain how they address your concerns.

Summery. The new analysis includes (i) (Sec. G.1G.1) visualizations of the trajectories that were observed most frequently in human-player data. For each layout, we visualized the most observed trajectories as measures between “interact” actions (two interact actions, or four; we check both shorter trajectories and longer ones separately). The illustration shows, for each tile, the percentage of the most frequent action for that tile as measured during the trajectory. We check how frequently similar trajectories were observed in TEAM and each trait, and against baseline population methods. TEAM achieves higher overall proximity. We also produce similar visualizations for TEAM trajectories. (ii) (Sec. G.2G.2) We show visitation heatmaps averaged over all possible trajectories, and show visitation frequency in each tile compared to TEAM and baselines. (iii) (Sec. G.2G.2) We check action distribution in each tile, compared to all methods, from all possible trajectories. In both (ii) and (iii), TEAM achieves the best overall human proximity. Sec. G.3G.3 also ablates alternative traits and checks if they produce trajectories similar to the human compared to the original TEAM population.

Method produces more human-like behavior. This is addressed by showing, for each human-visualized trajectory, whether it is similarly observed in TEAM and baselines. Human visualization is in Figs. 9-11, with comparison to TEAM, and numerical comparison is given in the tables in Section G.1G.1 and G.2G.2 (Tables 8-12)

We identify qualitative (at the level of visualizing trajectories of behavior) human-like behavioral traits that lead to better AI-AI or Human-AI performance. We provide both visualization and analysis.

We show that this is not cherry-picked, and we show that this behavior shows up statistically. We provide a numerical measure of the frequency of each observed trajectory so the reader can tell how frequent each trajectory is. In Figs 9-11, each trajectory is labeled with its observation frequency. Importantly, we also quantify the differences in trajectory frequencies between humans and each method.

We show that this leads to better performance. Here an argument for why (2) or (3) plausibly lead to better performance is fine. But they actually don't even give this argument. This is where the bound gives a justification. The proximity we demonstrate in (2)+(3) gives a better test-time performance guarantee due to human proximity, according to the bound in Theorem 1 (Eq. 6). We discuss it in further depth in Sec. G.4G.4 and show how the quantity we use in the per-tile comparison to evaluate proximity between humans and TEAM relates to the distance used in the bound.

AuthorsOpenReview ↗

Reponse 2/2 - Questions

Questions:

1-3. CEC performance. We used CEC's officially released code (Jha et al., 2025) with its original hyperparameters and training configuration unchanged, as these were presumably optimized for Overcooked v1. We believe the performance discrepancy may stem from two factors. First, the CEC paper evaluates cross-play only against IPPO, E3T, and FCP, whereas our evaluation also includes MEP, HSP, and TEAM but without IPPO, due to the simplicity of vanilla IPPO. Second, the CEC paper reports results over only six seeds, while we use 20 (different) seeds per method, which may contribute to differences in the reported performance. CEC versus FCP. Our FCP results on the Jaxmarl framework reproduce the ones reported in the CEC paper; here, again, we used (Jha et al., 2025) released code unchanged.

  1. Why is Table 2 missing CEC? We focus on population-based methods because Table 2 measures the human proximity of the Phase 1 population after training. CEC follows a different paradigm: it trains against evolving copies of itself across a large number of online-randomized environments (potentially tens of thousands based on its reported number of steps, although the exact number is unspecified). Thus, a meaningful comparison would require evaluating CEC’s final ego against humans, but this would compare a single CEC ego with the full populations of other methods. Accounting for CEC’s previous ego iterations as potential partners is also impractical given the scale of its training. We therefore excluded CEC to avoid this mismatch and ensure a fair comparison. If the reviewer considers this comparison essential, we would be happy to include it. We will clarify this distinction in the revised paper.

5-6. Analysis metric instead of L1L_1. the L1L_1 title we wrote in the table title was inaccurate; we apologize for the oversight, and we corrected the table in the revised paper. To clarify, when calculating the distance between a human and an AI policy, we take the human action with probability 1 and subtract the probability the AI agent assigns to this action; this measure is a proxy for L1L_1. Regarding your question of whether it is a good measure, the proxy gives L1 ≤ 2 × proxy, which provides mathematical justification for using it. The full mathematical derivation of the inequality is in the revised paper we uploaded in Appendix D3. We thank you again for the insightful comments. If further clarification or revision is needed, please let us know during the discussion period, and we will be happy to address it.


We hope the clarifications and additional results address the reviewer's concerns. We would appreciate the opportunity for further discussion should any questions remain. We are looking forward to your reply!

Dear Reviewer cQyd,

We would like to thank you again. You gave us valuable suggestions, which we made our best effort to address adequately.

The discussion period is nearing its end. Could you please read our reply and revised paper so that there is sufficient time for us to respond accordingly during the discussion period?

We also edited the main paper to include some of the new results. You can see a new table under Section 7.1 and a reference to the appendix for the full analysis. Changes are marked in red. If you have any comments on this, please let us know.

Thank you again for your time and consideration.

Reviewer_DEymOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes TEAM, a framework for zero-shot cooperation with OOD human and AI partners. Th essense is to structure training populations around human-inspired behavioral traits: socializers, achievers, explorers. Diverse trait strengths and training checkpoints generates population with varied behaviors and proficient levels. This population is then frozen and is thus used to train a recurrent ego policy, which tries to adapt to different partners. The paper also models partner variation as distribution of induced MDPs. Experiments on various ZSC benchmarks including Overcooked, SMAC, Coin Game, report improvements in cooperation performance over several baselines. Behavioral analyses and ablations further suggest that the proposed objectives produce differentiated and complementary partner behaviors. I think the main contribution is a fairly standard method to training-population design that reduces reliance on task-specific event rewards.

Critical strengths
  1. I think the paper is quite well-motivated. It attempt to addresses a central challenge in zero-shot cooperation, which is generating training partners that behavioral differences are relevant to the deployment partner.

  2. The empirical eval is quite broad. The experiments cover Overcooked V1, Overcooked V2, SMAC, and Coin Game, with various eval protocols using 20 training seeds. This breadth in expeirment provides useful evidence on generalization to OOD AI partners under different coordination scenarios.

  3. The paper examines concrete interaction behaviors such as ingredient handling and counter placement, to analyze differences among the different generated partners. Ablations that held population constant show the combined population achieves higher average performance than populations containing only one trait. This supports behaviorial variations.

Critical weaknesses
  1. I do believe the paper's motivation is strong, but I'm quite worried about its generalizability. All experiments are under game environments, and the human study only covers Overcooked. It's not yet enough to convince me the proposed traits would help in practical collaboration tasks like communication or other real world complex interactions.

  2. I think the added value beyong prior work needs to be clearer. The method somewhat combines existing ideas, including two-phase population training and entropy-based ops. The paper should also clearly explain what this combination adds beyond earlier approaches e.g. (paper title: LEARNING ZERO-SHOT COOPERATION WITH HUMANS, ASSUMING HUMANS ARE BIASED), which already model human preferences in populations.

  3. The results of this paper motivate training with partners similar to test time partners, but doesnt show why the mentioned traits are the right choice.

Questions for authors
  1. Appendix D.3 defines the distance as ( 1 - the probability that the agent assigns to the observed human action). Could you clarify how this score relates to the L1 distance between policies used in the theory, especially under the condition that the human policy is stochastic?

  2. Your results motivates training with partners close to those expected at test time. How does this support the specific choice of socializers, achievers, and explorers? Does the analyses provide any guidance for choosing these traits over others?

6 comments
AuthorsOpenReview ↗

Reponse 1/3 - Weakness 1 (Evaluation and Generalizability)

Dear Reviewer DEym,

Thank you for the considerable effort you put into reviewing our work. We have taken your feedback into careful consideration. Below, we address each of the points raised, including new experiments and clarification of novelty.


Summary of our response:

(1) W1: New experiments on MPE World Comm and Melting Pot (Harvest, Stug), where TEAM outperforms all baselines. (2) W2: Clarified novelty vs. HSP. (3) W3: New Appendix G: replacing our traits with alternatives reduces similarity to humans. (4) Q1: we show the proxy upper-bounds the L1 distance (‖p−q‖₁ ≤ 2·proxy).


Weaknesses:

1a. Overcooked human-AI evaluation and game-based AI-AI experiments. To our knowledge, recent methods in the ZSC and human-AI literature that we compare against and/or discuss in the paper conduct human-AI experiments exclusively on Overcooked, with additional AI-AI evaluation only on game-based environments (e.g., the CEC, E3T, HSP, MEP, FCP, and GAMMA methods, which are landmark works within the field, published at top-tier conferences). We therefore follow the established evaluation convention in this area of ZSC, and ask the reviewer to take this into account rather than consider the Overcooked-focused human evaluation or game-based environments a specific limitation of TEAM. We will add this as a future work direction that should be addressed. Also, note that our evaluation is more extensive than typical ZSC papers, which usually conduct AI-AI experiments only on Overcooked v1 and one or two additional game-based benchmarks. For instance, (i) HSP tests exclusively on three classical Overcooked v1 layouts and two more complicated layouts with several ingredients/orders. (ii) MEP tests on five classical Overcooked v1 layouts and one simple matrix game, and (iii) GAMMA tests exclusively on six Overcooked layouts (five classical and one with multi-ingredient). Our evaluation includes five classical Overcooked v1 layouts, two Overcooked v2 layouts (with multi-ingredient and implicit communication), 2 SMAC maps, and the coin game.

Regarding communication-based environments, Overcooked v2, which we already have in the paper, includes implicit communication: the agents have asymmetric information (a recipe indicator visible only to some of them), and they need to convey the recipe to one another implicitly through actions. This is explained in depth in the Overcooked v2 paper (Gessler et al., 2025). To address your remaining concerns, we provide additional experiments on another communication-based environment.

1b. Additional experiments. We conduct AI-AI experiments on (i) world communication [MPE], in which a leader with global access to all locations communicates data to assist other agents in catching prey. (ii) The melting-pot benchmark [1], specifically on the Stug and Stug environments, which pose social dilemmas beyond the ones evaluated in traditional ZSC:

TEAMFCPMEPHSPE3T
World communication37.8 ±\pm 3.419.57$\pm$ 4.522.3 ±\pm 6.130.09 ±\pm 2.821.3$\pm$ 5.1
Harvest17.9 $\pm$2.510.1 ±\pm 4.47.6$\pm$5.911.9 ±\pm 2.39.1 ±\pm 2.6
Stug hunt in the matrix223.94 ±\pm 4.91210.3 ±\pm 4.3211.9 ±\pm 4.0215.1 ±\pm 3.07211.3$\pm$ 4.52

Harvest (Commons Harvest) is a common-pool-resource dilemma: agents are rewarded for eating apples, yet apples regrow only if enough neighbors' trees remain uneaten, testing restraint of agents in eating apples versus over-harvesting. In Stag Hunt, each player collects stag or hare resources and then zaps the other to interact, and the payoff rewards both choosing stag (high but risky) over hare (low but safe).

Experimental specification. We use 12 seeds. The table shows standard errors. Training and compute budget are fixed among all methods. Since all the cited diversity techniques are designed for 2-player settings, we use Harvest and Stug with only two players. The results presented include both cross-seed and cross-play. We use the main achiever mechanism based on reward allocation for TEAM.

HSP biases: Harvest: greedy harvester, sustainable harvester. Stug: Prefers stag resources, prefers hare resources, avoids interacting, Zaps as soon as possible. World Comm: forest hider, food-greedy, adversary-avoider, leader-ignorer.

To summarize, Our evaluation covers various aspects of collaboration and interaction: human tests (Overcooked), partial observability (Overcooked 2), collaboration against an adversarial team (SMAC), Mixed motives and social dilemmas among agents (Coin Game, Meltingpot), and communication (World Communication).

AuthorsOpenReview ↗

Reponse 2/3 - Weakness 2-3 (Novelty and Trait Justification)

  1. Novelty. Our main novelty is a new method for capturing general personality traits. This is an unexplored dimension of human behavior in ZSC, which enables a single human model across tasks. In contrast, prior methods such as HSP build a separate human model for each task from utilities over events.

Why do the per-task utilities of HSP differ from our traits approach? HSP draws on behavioral economics and models humans via a personal utility, while TEAM models traits. In personality psychology, traits are general dispositions that shape human behaviors across many situations [2], whereas in economics, a utility represents a person’s preferences over a given set of options [3]. The two constructs can differ: for instance, economic preferences and personality traits have been found to be only weakly correlated and not interchangeable [4]. In our settings, the key difference is what each method’s human model is defined over. In HSP, the options are hand-designed task events, so a separate utility model must be designed for each task. For example, in Overcooked, HSP constructs bias around placing a tomato in the pot; this bias does not exist in SMAC, where a new utility model had to be rebuilt. In contrast, TEAM’s traits are defined over styles of engagement (e.g., with a partner) rather than over specific tasks: a socializer maximizes its partner’s influence on future states, similarly, in every environment, so the same human modeling applies across tasks. We believe that this task-independent approach is a key step toward making human-aligned ZSC practical across diverse tasks.

Algorithmic contributions. As described in the method section, we introduce several techniques for inducing trait diversity. These are not the paper's central novelties, and they fall under Contribution 2 (the TEAM framework). For clarity, we write here the difference from prior work. For achievers, inducing diverse division of labor by tuning partner rewards for ZSC is novel; prior reward-allocation methods focused on sustainable cooperation and used a different allocation scheme [5]. Likewise, using empowerment as a diversity term alongside reward, rather than solely as a training aid in reward-free settings, is novel. We should also note that the paper does not claim novelty over the two-phase scheme; it is phrased in the introduction as a known framework we build on.

We thank you for the valuable feedback. will incorporate the corresponding details into the revised paper within the next day or two, before the end of the discussion period. The changes will be marked in red.

  1. Why use these three traits over others?

Evidence from behavioural literature that the three traits are prominent. The trait selection is grounded in Sec. 4’s review of human behavior in social games. We selected our traits as they were identified as prominent in social scenarios by human-psychology studies. For instance, the comprehensive review of Hamari & Tuunanen (2014), where all three traits appear in their Table 1 across different games. Also, in studies on games such as World of Warcraft (Zackariasson et al., 2010) and Ultima Online (Hamari & Tuunanen, 2014).

Additional empirical evidence. In response to your request, we added Appendix G, which provides a detailed analysis of trajectories collected from human players and shows that: (i) human similarity decreases when the traits are replaced with non-trait modifications such as random reward mixing, and alternative game-intensity traits (motivating our three traits over traits/modifications). This is given in Sec. G.3,G.3, Table 13. (ii) Our traits yield higher similarity to human trajectories than baseline population-based methods when similarity is measured over entire trajectories. This is shown in Sec. G.1G.1 under Table 9, and Sec. G.2G.2 under Table 11. (iii) Better ZSC performance of our traits, where the alternative traits achieve 109.5-149.6, compared to 171.48 for TEAM (full results are in Sec. G.3,G.3, under Table 13). A summary of the results will be incorporated into the main paper in a subsequent revision.

AuthorsOpenReview ↗

Reponse 3/3 - Questions

Questions

  1. We uploaded a revised version with a new Table 2 name, “distance from human policy”, fixing the “L1 title” in Table 2. The current title is inaccurate. Since real humans do not produce a policy or a distribution of actions, the Human-collected trajectories contain only tuples of state and a chosen action. For this reason, to calculate the distance between a human and an AI policy, we take the human action with probability 1, and subtract the probability the AI agent assigns to this action. We clarify in Sec. 7.1 that for L1 a stochastic policy is required, and the current measure is a proxy. Regarding your question about the relationship, Let p=πH(⋅∣s)p = \pi_H(\cdot \mid s), q=πθ(⋅∣s)q = \pi_\theta(\cdot \mid s), and proxy=ap(a)(1-q(a)). so L1 ≤ 2 × proxy, i.e., the proxy gives an upper bound to the L1 distance. Full mathematical derivation is in the revised paper we uploaded in Appendix C.3.

  2. Addressed in weakness 3

We hope the clarifications and additional results address the reviewer's concerns. We would appreciate the opportunity for further discussion should any questions remain. We are looking forward to your reply!


[1] Leibo, Joel Z., et al. "Scalable evaluation of multi-agent reinforcement learning with melting pot." International conference on machine learning. PMLR, 2021.

[2] Fleeson, W., & Jayawickreme, E. (2015). Whole trait theory. Journal of Research in Personality, 56, 82–92.

[3] Mas-Colell, A., Whinston, M. D., & Green, J. R. (1995). Microeconomic Theory. Oxford University Press.

[4] Becker, A., Deckers, T., Dohmen, T., Falk, A., & Kosse, F. (2012). The relationship between economic preferences and psychological personality measures. Annual Review of Economics, 4, 453–478.

[5] McKee, Kevin R., et al. "Social diversity and social preferences in mixed-motive reinforcement learning." arXiv preprint arXiv:2002.02325 (2020).

Reviewer_PoHbOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TEAM, a new way to generate training populations that can enhance zero-shot cooperation, so that an AI agent can work with unfamiliar human and AI partners in cooperative settings. Instead of seeking arbitrary diversity, TEAM builds partners around three human-inspired behavioral traits adapted from Bartle’s player taxonomy. Socializers help their teammates through an empowerment bonus. Achievers take on more of the workload through reward reallocation. Explorers pursue unconventional strategies through high policy entropy. Each trait has a coefficient that controls its strength. A recurrent ego agent is then trained against this population.

The authors also frame zero-shot cooperation as generalization over a set of partner-induced MDPs. Using domain-randomization theory, they argue that ego performance depends on how many training partners are close to the human partner, not just on broad coverage.

Critical strengths
  • 1. The three trait objectives (an empowerment bonus, reward reallocation and an entropy coefficient) are lightweight and need far less domain knowledge than hand-designed event rewards. The paper applies them across four quite different environments, including SMAC and Coin Game This is a practical, reusable contribution: an easy way to inject interpretable, controllable variation into ZSC populations. It is the main reason I lean toward acceptance.

  • 2. Treating each frozen partner as inducing its own MDP connects ZSC to established domain-randomization generalization theory (Chen et al., 2022). This gives a clean, algorithm-agnostic way to reason about population design.

  • 3. TEAM achieves the best mean score across Overcooked V1/V2, SMAC, Coin Game and a 32-participant human study. The per-trait ablation (Table 5) shows that combining traits beats any single trait, supporting the value of structured diversity. Individual margins are often small, but the gains are consistent across settings.

Critical weaknesses
  • 1. Although the behavioral analyses reveal different play styles, it remains unclear how well the three traits represent the diversity of human cooperation. The human study is also limited to 32 participants playing Overcooked, leaving generalization to other tasks and populations uncertain.

  • 2. The evaluation is limited to controlled game benchmarks, leaving performance in more complex environments with larger teams, longer planning horizons, and changing human behavior unclear. This is particularly relevant because the theoretical bound assumes an optimal ego policy, which may be harder to approximate in such settings.

Questions for authors
  • 1. How sensitive are the human-proximity results in Table 2 to the choice of ε = 0.54? Reporting proximity fractions across several thresholds would help establish whether TEAM’s advantage over the baseline populations is robust.

  • 2. E3T and CEC have the lowest average scores in Table 7. Were their method-specific hyperparameters tuned for this setup, and do their results reproduce the original findings under comparable settings? How were TEAM’s α grids and mixed-trait combinations selected, and how sensitive is performance to these choices?

  • 3. What challenges would arise when extending TEAM to larger teams, longer horizons, or partners whose behavior changes within an episode? In particular, how would these settings affect empowerment estimation and the applicability of the theoretical bound, which assumes stationary partners and an optimal ego policy?

2 comments
AuthorsOpenReview ↗

Reponse 1/2 - Weaknesses

Dear Reviewer PoHb,

We truly appreciate the time and effort you dedicated to this review, and we thank you for the positive assessment of our work. Your helpful suggestions are addressed below.

Critical Weaknesses:

  1. Additional human-based analysis. To address this weakness, we added a new section (Appendix G in the revised paper) analyzing trajectories collected from human data. We visualize the most frequently visited human trajectories and measure how frequently similar trajectories were generated by TEAM and the baselines, with TEAM achieving the highest similarity in its trajectories (Sec. G.1G.1 Table 8). We also visualize per-tile visitation and actions on humans and compare them to those produced by TEAM and baselines (Sec. G.2G.2 Table 11). TEAM achieves the highest overall similarity here as well. Due to trajectory-level and per-tile measurements of both humans and TEAM, this analysis better captures the diversity related to human play behind what we already have in Table 2 and Fig. 5(b). A summary of these findings will be incorporated into the main paper in a subsequent revision (ICLR allows one additional page in the camera-ready version).

Overcooked-only human-AI validation. To our knowledge, recent ZSC methods that we compare against and/or discuss in the paper conduct human-AI experiments exclusively on Overcooked, e.g., CEC, E3T, HSP, MEP, FCP, GAMMA (all published at the top tier conferences). This is somewhat of a field convention, which we agree should be addressed. We will acknowledge this in the conclusion as a direction for future work.

  1. Environments with larger teams, and controlled game benchmarks. The ZSC research we cite and address in the paper, including all baselines, focuses primarily on two-player settings. Evaluating larger teams would therefore deviate from the design/settings considered in all these works. We agree that together with the broader reliance on controlled game benchmarks in the ZSC literature, this warrants further discussion. We will add a dedicated discussion addressing these issues and provide practical guidance on scaling TEAM to settings with more than two agents. For example, in a setting with K agents, one can sample a subset of agents and replace only the subgroup (instead of all agents) with background trait-based population members, reducing computational requirements.

Relation to bound. Obtaining optimality indeed becomes more challenging with larger teams and longer horizons. Note, however, that the bound requires the ego to be optimal only with respect to the fixed (frozen) Phase 1 population. Because the partners are frozen, the ego solves a stationary problem rather than adapting to a co-evolving large number of partners. We emphasize that we use the bound to guide population design (proximity to human and test partners) rather than to predict absolute performance. We will add further discussion of how more complex settings affect the conditions underlying the bound, such as ego optimality.

Environments with longer horizons. To account for tasks with longer horizons, we conduct additional AI-AI experiments on the melting pot benchmark [1] with Harvest, and stug hunt in the matrix, with a longer horizon of 1K steps per episode.

TEAMFCPMEPHSPE3T
Harvest17.9 $\pm$2.510.1 ±\pm 4.47.6$\pm$5.911.9 ±\pm 2.39.1 ±\pm 2.6
stug hunt in the matrix223.94 ±\pm 4.91210.3 ±\pm 4.3211.9 ±\pm: 4.0215.1 ±\pm 3.07211.3$\pm$: 4.52

Harvest (Commons Harvest) is a common-pool-resource dilemma: agents are rewarded for eating apples, yet apples regrow only if enough neighbors' trees remain uneaten, testing the restraint of agents' eating apples versus over-harvesting. In Stag Hunt, each player collects stag or hare resources and then zaps the other to interact, and the payoff rewards both choosing stag (high but risky) over hare (low but safe).

Experimental specification. We use 12 seeds. The table shows standard errors. Training and compute budget fixed among all methods. Since all the cited diversity techniques are designed for 2-player settings, we use Harvest and Stug with only two players. The results presented include both cross-seed and cross-play. We use the main achiever mechanism based on reward allocation for TEAM.

HSP biases: Harvest: greedy harvester, sustainable harvester. Stug: Prefers stag resources, prefers hare resources, avoids interacting, Zaps as soon as possible.

AuthorsOpenReview ↗

Reponse 2/2 - Questions

Questions

  1. The revised paper now includes an extended Table (Table 4, page 16) with different ϵ\epsilon selections, showing the robustness of TEAM to different thresholds. TEAM leads on all thresholds.

  2. Part A (CEC and E3T). We used the E3T and CEC JAX implementations released by (Jha et al., 2025). For E3T, we reproduced results very similar to those reported in (Jha et al., 2025), no modifications were made. For CEC, we used the original hyperparameters without modification, as these were presumably optimized for Overcooked v1. We experienced performance discrepancies that we believe may stem from two factors. First, the CEC paper evaluates cross-play only against IPPO, E3T, and FCP, whereas our evaluation also includes MEP, HSP, and TEAM but excludes IPPO due to the simplicity of vanilla IPPO. Second, the CEC paper reports results over six seeds, whereas we use 20 different seeds per method, which may contribute to differences in performance. To verify that we could reproduce the reported results, we additionally evaluated CEC against only FCP, E3T, and IPPO. In this setting, CEC achieved an average reward of 125, which is much closer to the results reported in their paper. We used the fine-tuned CEC.

Part B (αi\alpha_i values). In our main experiments, we selected αi\alpha_i values to be consecutive values of 0.1 to ensure sufficient diversity between consecutive population members. As long as this minimum gap is maintained, TEAM remains robust to changes in the specific i values. We did a controlled experiment on three Overcooked v1 maps and show that shifting all αi\alpha_i weights by 0.03 and 0.06 has no significant effect on performance.

coord ringcountercrampedforcedassym advantage
(0.03 shift)196.9 ±\pm 7.8127.09$\pm$ 8.1197.8$\pm$6.855.55 ±\pm 6.1288.4$\pm$ 20.2
(0.06 shift)198.9 ±\pm 8.9126.18$\pm$ 7.3194.9$\pm$6.658.49 ±\pm 6.4280.3$\pm$ 20.11
  1. Part A. Challenges that arise with: (i) Larger team. For population training (phase 1), achiever is straightforward to extend to more agents and requires only different reward allocation for different members, same as explorer. For socializers, empowerment term modifications are needed, i.e., combining the state from all agents and feeding it into the empowerment term (explained below in iii). Phase 2 is unchanged, as we freeze the population from phase 1 and train the ego to produce a policy that can collaborate well with the population composition that was obtained in phase 1. (ii) Longer horizon. The new tests on Harvest and Stug are with longer, 1k horizons, in which TEAM outperformed baselines, showing transferability without modifications to longer horizons. (iii) effect of (i) and (ii) on empowerment. For very long horizons, practitioners can adjust the sampling range of future states used in the empowerment term in order to sample states further into the future. For settings with more than two players, where the socializer needs to empower multiple partners, practitioners can combine all partner states into a global state representing the group and use it in the empowerment term, allowing empowerment to capture all partners.

Part B. Changing partners and effect on the bound. Longer horizons increase the bound learning term O(...)O(...) which grows sublinearly, so small mismatches with the human partner become more costly. Larger teams enlarge the partner class, making partners harder to identify, and make it harder for training partners to lie close to the human team. Both effects strengthen the case for proximity over coverage, which motivates TEAM. Non-stationary partners violate the bound’s stationarity assumption. A natural extension models such partners as switching between stationary behaviors: the bound then applies within each segment, at the cost of re-identifying the partner after each switch. We will add this discussion to the paper.


Your feedback is valuable to us; should any questions or concerns remain, we would be happy to engage in further discussion.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.