- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TEAM, a population-training method for human–AI zero-shot cooperation. TEAM organizes training partners into three human-inspired traits: socializers, achievers, and explorers. These traits are implemented through empowerment, reward reallocation, and entropy regularization, respectively. A recurrent ego agent is then trained against the frozen population. Experiments cover Coin Game, SMAC, Overcooked v1/v2, and a 32-participant Overcooked study.
Critical strengths
-
The paper addresses a meaningful practical issue: a training population should represent plausible human behavior rather than maximize diversity without direction. Using interpretable behavioral priors makes the population easier to understand and control.
-
The paper is generally well written and easy to follow. The two-stage training procedure, trait implementations, and evaluation questions are presented in a clear structure.
Critical weaknesses
-
The problem and the proposed traits have limited novelty. Training against heterogeneous partners, modeling human preference differences, and adapting to unseen partners have been studied by FCP (Strouse et al., 2021), MEP (Zhao et al., 2023), HSP (Yu et al., 2023), CoMeDi (Sarkar et al., 2023), and GAMMA (Liang et al., 2024), while Melting Pot evaluates social generalization across diverse partners and mixed incentives (Leibo et al., 2021; Agapiou et al., 2022). Socializer, achiever, and explorer are also taken directly from Bartle’s taxonomy. The main novelty is therefore using a fixed human-inspired taxonomy to organize a ZSC training population, rather than introducing a new cooperation problem or behavioral abstraction.
-
The experiments do not isolate the value of the taxonomy. TEAM simultaneously introduces three additional objectives, multiple coefficient levels, mixed-trait agents, and several checkpoints. The single-trait ablation only shows that combining the three populations is better than using one population alone. It does not show that the Bartle structure is better than an equally sized population built from the same objectives and coefficients without trait labels.
-
The comparison is incomplete for the paper’s novelty claim. The main experiments include FCP, MEP, HSP, E3T, and CEC, but omit convention-based and data-driven style populations such as CoMeDi (Sarkar et al., 2023) and GAMMA (Liang et al., 2024). The method is also not evaluated on a broader mixed-motive social-generalization benchmark such as Melting Pot (Leibo et al., 2021; Agapiou et al., 2022). Coin Game provides only a limited test of cooperation–competition trade-offs, while the remaining tasks largely assume a shared objective.
References
-
Strouse, D. J., et al. (2021). Collaborating with Humans without Human Data. NeurIPS 2021.
-
Zhao, R., et al. (2023). Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination. AAAI 2023.
-
Yu, C., et al. (2023). Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased. ICLR 2023.
-
Sarkar, B., Shih, A., & Sadigh, D. (2023). Diverse Conventions for Human-AI Collaboration. NeurIPS 2023.
-
Liang, Y., et al. (2024). Learning to Cooperate with Humans using Generative Agents. NeurIPS 2024.
-
Leibo, J. Z., et al. (2021). Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot. ICML 2021.
-
Agapiou, J. P., et al. (2022). Melting Pot 2.0. arXiv:2211.13746.
Questions for authors
-
How does TEAM differ conceptually from HSP’s hidden-utility populations and GAMMA’s latent partner styles beyond using three named traits?
-
Why are CoMeDi and GAMMA not included as empirical baselines? Would the conclusions hold on a mixed-motive social-generalization benchmark such as Melting Pot?
-
Can the authors validate the three traits using an independent human-behavior clustering or human-coded annotation, rather than profiles generated by TEAM itself? Are the human-study differences statistically significant?
5 comments
Reponse 2/3 - Weaknesses 2-3a (Trait Abalation and Environments)
- Ablation and trait justification. We assume the reviewer means one of three:
(i) No traits at all. We use the same population size, checkpoints, architecture, and training budget but without any trait modification. We run this ablation with and without the self-play round in phase 2.
(ii) Replacing objective with non-trait terms. We first replace only the explorer with lower temperatures, which removes the “trait labels” of the explorer but leaves the entire mechanism and other traits intact. Similarly, we replace the MI term of the socializer with , in which the socializer optimizes the empowerment term on its own action and future state rather than on the other agent, but uses the same coefficients. This removes only the socializer label but leaves all other traits intact.
(ii) Each of the K agents is a jointly random draw of (α_soc, α_ach, α_exp) over the same ranges, with the same K, checkpoints and budget. I.e., and sampled agent gets values of (0.6, 1, 0.01) for the coefficients. We include it under “joint sampled”.
Comparison is in the following Table:
| coord ring | counter | cramped | forced | assym advantage | |
|---|---|---|---|---|---|
| No traits and self-play | 148.9$\pm$ 7.2 | 81.08$\pm$9.5 | 183.9 $\pm$6.6 | 39.8$\pm$5.4 | 146.2$\pm$31.55 |
| No traits and no self-play | 143.3$\pm$8.5 | 74.09$\pm$ 9.56 | 183.2$\pm$ 10.3 | 37.8$\pm$ 6.4 | 129.9 33.2 |
| No explorer label | 189.2$\pm$ 7.4 | 110.4$\pm$ 6.9 | 190.3$\pm$4.6 | 48.3$\pm$5.7 | 263.6$\pm$20.8 |
| No socializer label | 185.2$\pm$ 7.1 | 112.3$\pm$ 5.1 | 186.3$\pm$5.1 | 49.7$\pm$6.2 | 269.9$\pm$22.9 |
| Jointly sampled | 180.8$\pm$ 6.3 | 103.3$\pm$ 8.2 | 177.3$\pm$4.4 | 44.1$\pm$5.9 | 241.2$\pm$24.3 |
| TEAM (original) | 197.9$\pm$9.9 | 126.1$\pm$ 7.8 | 195.1$\pm$7.1 | 55.1 6.9 | 283.2$\pm$ 24.1 |
If you intended us to run an additional comparison, please let us know, and we will be happy to run it.
3a. Additional environments. To address your concerns, we added experiments on Melting Pot’s Harvest and Stug Hunt, evaluating AI-AI (ego agents from all methods against one another). Note that to our knowledge, population-based ZSC research, as cited in our paper (as well as all the works cited in your response, e.g., HSP, MEP), is designed and tested primarily on 2-player settings. Since Melting Pot is mainly designed for collaboration in larger groups, many of its settings are not suitable. For our experiments, we use Harvest and Stug in 2-player settings to remain consistent with the original baseline design. We selected these environments because they support meaningful 2-player collaboration.
| TEAM | FCP | MEP | HSP | E3T | |
|---|---|---|---|---|---|
| Harvest | 17.9 $\pm$2.5 | 10.1 4.4 | 7.6$\pm$5.9 | 11.9 2.3 | 9.1 2.6 |
| Stug hunt in the matrix | 223.94 4.91 | 210.3 4.3 | 211.9 4.0 | 215.1 3.07 | 211.3$\pm$ 4.52 |
Harvest (Commons Harvest) is a common-pool-resource dilemma: agents are rewarded for eating apples, yet apples regrow only if enough neighbors' trees remain uneaten, testing restraint of agents in eating apples versus over-harvesting. In Stag Hunt, each player collects stag or hare resources and then zaps the other to interact, and the payoff rewards both choosing stag (high but risky) over hare (low but safe). Experimental specification. We use 12 seeds. The table shows standard errors. Training and compute budget fixed among all methods. Since all the cited diversity techniques are designed for 2-player settings, we use Harvest and Stug with only two players. The results presented include both cross-seed and cross-play. We use the main achiever mechanism based on reward allocation for TEAM.
HSP biases: Harvest: greedy harvester, sustainable harvester. Stug: Prefers stag resources, prefers hare resources, avoids interacting, Zaps as soon as possible.
Due to the limited rebuttal time, we were able to obtain the additional empirical results but have not yet completed their full analysis. We will provide a detailed analysis in the revised paper.
Reponse 3/3 - Weakness 3b (Baselines) and Questions
3b.Additional baselines. CoMeDi and GAMMA were not included originally since they are not implemented on the JAXMARL framework, on which our experiments are conducted. Since GAMMA is an improvement over CoMeDi with better reported results (they report in their paper a GAMMA+CoMeDi column), we focus on the GAMMA comparison. The table shows their results on the 5 Overcooked-v1 layouts:
| coord ring | counter | cramped | forced | assym advantage | |
|---|---|---|---|---|---|
| GAMMA+CoMeDi | 142.5$\pm$8.7 | 81.4$\pm$9.5 | 160.08$\pm$7.9 | 44.7 7.1 | 198.1$\pm$ 33.7 |
| GAMMA+MEP | 157.2$\pm$7.6 | 104$\pm$ 12.6 | 162.3$\pm$7.5 | 48.3 6.7 | 245.1$\pm$ 22.4 |
| TEAM | 197.9$\pm$9.9 | 126.1$\pm$ 7.8 | 195.1$\pm$7.1 | 55.1 6.9 | 283.2$\pm$ 24.1 |
For GAMMA to run with TEAM and our baselines, we had to convert the JAX state into a suitable input representation for GAMMA. The JAX official implementation states [1]: “The implementation aims to be as close as possible to the original Overcooked-AI environment, including dynamics, collision logic, and action and observation spaces.” So no major differences should exist between JAX and the original Overcooked. Yet, since this raises some mismatch between different baselines, we leave it as a side comparison (which will be included in the revision).
Questions:
-
TEAM versus GAMA and HSP. Addressed in weakness 1.
-
Additional baselines and Meltingpot. Both parts of the question were addressed in weakness 3.
-
Trait validation and statistical significance. The six heatmaps' behavioural categories (plate/onion/soup pickups, etc.) that we also used to produce Fig 5b are standard Overcooked event counts, and match the categories used to identify partner behaviours in the ZSC-Eval Overcooked benchmark [5]. We added the citation to the paper already for clarity and to ground the work in existing literature. Regarding further human identification, we uploaded a revised version of the paper. We refer you to Appendix G. We conducted a detailed study in which we visualize human trajectories without relying on these categories and show higher similarity to the TEAM population than to the baseline methods. Our main findings are that (i) TEAM and its three traits produce, more frequently than baselines, trajectories that were similarly observed in human players, as given in Sec. Tables 8 and 9, and (ii) when comparing action selection and per-tile visitation over the entire layout, the TEAM population exhibits higher overall similarity to human trajectories than baselines. This is given in Sec. Table 11. A summary of the results will be incorporated into the main paper in a subsequent revision (ICLR allows one additional page if a paper is accepted). The revised paper also includes significance tests on the human study (see Fig. 5 and explanation in Appendix C.3).
We hope the clarifications and additional results address the reviewer's concerns. We look forward to your reply and would appreciate the opportunity for further discussion should any questions remain.
[1] https://jaxmarl.foersterlab.com/environments/overcooked/
[2] Fleeson, W., & Jayawickreme, E. (2015). Whole trait theory. Journal of Research in Personality, 56, 82–92.
[3] Mas-Colell, A., Whinston, M. D., & Green, J. R. (1995). Microeconomic Theory. Oxford University Press.
[4] Becker, A., Deckers, T., Dohmen, T., Falk, A., & Kosse, F. (2012). The relationship between economic preferences and psychological personality measures. Annual Review of Economics, 4, 453–478.
[5] Wang, Xihuai, et al. "Zsc-eval: An evaluation toolkit and benchmark for multi-agent zero-shot coordination." Advances in Neural Information Processing Systems 37 (2024): 47344-47377.
Reponse 1/3 - Weakness 1 (Novelty)
Dear Reviewer s44b,
We thank you for your detailed feedback and the time dedicated to reviewing our work. We considered each point you raised very carefully and added new experiments, ablations, and clarification of novelty over prior work. Our response is given below.
Summary of our response:
(1) W1: Clarified novelty vs. HSP and GAMMA. (2) W2: Ablation and trait justification replacing our traits with no-label objectives (3) W3: New experiments on Melting Pot (Harvest, Stug), where TEAM outperforms all baselines. (4) Q3: New Appendix G: detailed human analysis.
Weaknesses:
Why do the per-task utilities of HSP differ from our traits approach? HSP draws on behavioral economics and models humans via a personal utility, while TEAM models traits. In personality psychology, traits are general dispositions that shape human behaviors across many situations [2], whereas in economics, a utility represents a person’s preferences over a given set of options [3]. The two constructs can differ: for instance, economic preferences and personality traits have been found to be only weakly correlated and not interchangeable [4]. In our settings, the key difference is what each method’s human model is defined over. In HSP, the options are hand-designed task events, so a separate utility model must be designed for each task. For example, in Overcooked, HSP constructs bias around placing a tomato in the pot; this bias does not exist in SMAC, where a new utility model had to be rebuilt. In contrast, TEAM’s traits are defined over styles of engagement (e.g., with a partner) rather than over specific tasks: a socializer maximizes its partner’s influence on future states, similarly, in every environment, so the same human modeling applies across tasks. We believe that this task-independent approach is a key step toward making human-aligned ZSC practical across diverse tasks.
GAMMA human-aligned partner generation requires collecting actual human trajectories. This represents a different approach from ours: rather than modeling human behavior through learning objectives, GAMMA learns to reproduce diversity from collected human trajectories. Note that the human data is optional here as GAMMA can produce trajectories from non-human data as well, but this aspect of GAMMA, which uses encoder-based generation, differs substantially from our work.
Other baselines. Methods such as MEP, FCP, E3T, and CoMeDi are not designed to model human behavior, so we refer to the related work section for the discussion of their differences from our approach.
Regarding novelty of the Bartle taxonomy, Viewing the use of existing human trait taxonomies as a limitation of novelty somewhat overlooks the central idea behind our work. The paper's innovation lies in drawing on human-behaviour literature and deriving our traits from established, observed taxonomies that capture real human traits. This is essential to ensuring that our traits are meaningful and human-grounded. Conversely, introducing our own novel traits or taxonomy would lack human grounding and weaken the paper's central motivation.
We thank you for the valuable feedback. We will incorporate the corresponding clarifications into the revised paper within the next day or two, before the end of the discussion period. The changes will be marked in red.