- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors introduce MoltenPot and collect a large dataset of game trajectories from MeltingPot environments and evaluating four offline reinforcement learning baselines. Their work addresses the lack of support for offline reinforcement learning evaluation in MeltingPot. In the exprionments, the authors examine the four baseline performance under different experimental settings and the gap to the IPPO
Critical strengths
This work fills an important gap in MeltingPot by extending it to support offline reinforcement learning. And they provide a valuable platform for evaluating and developing algorithms that generalise across social dilemmas.
Compared with the baselines provided by Melting Pot, I believe collecting more data is a promising approach to improving the applicability of algorithms to social dilemmas.
Critical weaknesses
- The paper mainly evaluates offline RL methods in the existing MeltingPot environments. I think designing and implementing environments that capture meaningful and interesting social dilemmas is much more challenging than applying established algorithms to existing environments. While the collected datasets and evaluation protocols have value, but I don't think that is enough for a main contribution. Because the way to collect data is quite normal. Therefore, I find the contributions of this paper are kind of insufficient.
- The paper focuses on social dilemmas but provides very limited analysis of relevant social outcome metrics, such as the Gini coefficient or cooperation rates.
Questions for authors
- How do the authors ensure the quality of the collected datasets, particularly that they contain meaningful examples of cooperative behavior rather than only selfish behavior?
- What advantages does offline RL offer over online RL in this setting? What empirical or theoretical evidence motivates studying offline RL in MeltingPot, and what benefits does it offer over online RL in these environments?
3 comments
Q2: What advantages does offline RL offer over online RL in this setting? What empirical or theoretical evidence motivates studying offline RL in MeltingPot, and what benefits does it offer over online RL in these environments?
We believe this question highlights an important misunderstanding in the motivation and purpose of our work. Our objective is not to establish the benefits of offline RL over online RL, but to provide the inrastructure for methodological progress in an important setting that existing offline RL benchmarks have overlooked.
The motivation for offline RL is well established and extensively studied (Levine et al., 2020; Fu et al., 2020; Gulcehre et al., 2020). Its principal benefit is the ability to learn from previously collected data without requiring further environment interaction, which is particularly valuable when online exploration is costly, unsafe or impractical. Applications such as autonomous driving and healthcare illustrate these constraints (Gottesman et al., 2018).
Given these established motivations, the relevant research question is how to develop effective offline RL methods for mixed-motive multi-agent environments. Despite progress in offline RL, existing offline MARL research and benchmarks have concentrated overwhelmingly on fully cooperative settings (Formanek et al., 2023; 2026). This leaves a substantial gap in our ability to develop and evaluate methods for interactions involving agents with differing incentives.
Molten Pot addresses precisely this gap. By providing the first large-scale datasets, structured evaluation protocols and baselines for offline mixed-motive MARL, we establish the infrastructure needed for systematic methodological progress, just as OG-MARL helped enable progress in cooperative offline MARL.
Closing remarks
The central contribution of Molten Pot is to establish the datasets, evaluation protocols and taxonomy needed for systematic progress in offline mixed-motive RL. Just as OG-MARL helped provide the foundation for methodological advances in cooperative offline MARL, we believe Molten Pot can enable similar progress in this important and currently underexplored setting.
Thank you again for your feedback. We hope these clarifications address your concerns and would be grateful if you would reconsider your assessment.
References
Formanek, C., Jeewa, A., Shock, J., and Pretorius, A. Off-the-Grid MARL: Datasets and baselines for offline multi-agent reinforcement learning. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2023.
Formanek, J. C., Beyers, L., Tilbury, C. R., Shock, J. P., and Pretorius, A. Putting data at the centre of offline multi-agent reinforcement learning. Journal of Data-centric Machine Learning Research, 2026.
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv:2005.01643, 2020.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4RL: Datasets for deep data-driven reinforcement learning. arXiv:2004.07219, 2020.
Gulcehre, Caglar, et al. "Rl unplugged: A suite of benchmarks for offline reinforcement learning." Advances in neural information processing systems 33 (2020): 7248-7259.
Gottesman, O., et al. Evaluating reinforcement learning algorithms in observational health settings. arXiv:1805.12298, 2018.
Leibo, Joel Z., et al. "Scalable evaluation of multi-agent reinforcement learning with melting pot." International conference on machine learning. PMLR, 2021.
I absolutely dont think that Cooperation rates and Gini coefficients measure a different objective is a good response. I am not suggesting changing the benchmark objective, but reporting these quantities as diagnostics, which is exactly the role that the background return changes in Appendix G already partially play. You already mentioned that an increase in individual return may not come from cooperation and may instead come from exploiting co-players. Therefore, its become interesting and important to understand how and why the return increased but not just report the return itself.
I understand that Meltingpot does not report these metrics I said. However, Meltingpot's main contribution is collecting and building a large set of new environments. As the authors state in their response, the novelty of Molten Pot lies in the datasets and the evaluation rather than in new environment mechanics. Therefore, their absence in Meltingpot is no reason to omit them here, especially for a benchmark that evaluation should be very important.
On Q1, my question was not whether the dataset should contain only cooperative behavior. I would like to know whether it also contains diverse and meaningful cooperative behavior alongside other behaviors. Cooperative behavior or the behavior between cooperation and defection, is what makes mixed-motive settings interesting.
Thank you for reviewing our paper and for recognising the value of the datasets and evaluation framework. We would like to clarify an important aspect of the paper's motivation, which we believe may have been misunderstood.
The primary motivation for Molten Pot is not to fill a gap in Melting Pot, but to address a critical gap in offline mixed-motive RL research, where no established benchmarks currently exist.
Progress in RL methodology depends critically on strong benchmarks that expose meaningful challenges and provide a common foundation for algorithm development and comparison. We have seen this with OG-MARL (Formanek et al., 2023), which helped enable methodological progress in cooperative offline MARL by establishing standardised datasets and evaluations. Molten Pot aims to provide the equivalent foundation for mixed-motive offline MARL.
Our contribution extends well beyond applying existing algorithms to Melting Pot. It comprises three elements. First, scale: approximately 1 TB of skill-mixed trajectories across 47 scenarios, removing a substantial computational and engineering barrier to studying offline social learning. Second, evaluation protocols: three complementary settings that distinguish single-scenario learning, multi-scenario learning and zero-shot generalisation. Third, a taxonomy of social challenges: Tables 3 and 4 characterise distinct social configurations and 18 targeted generalisation shifts, enabling systematic evaluation of different aspects of social robustness.
Together, these contributions provide the infrastructure needed to develop and meaningfully compare new methods. Reusing established Melting Pot environments is deliberate: the novelty lies in the offline datasets, evaluation methodology and conceptual organisation of social challenges, rather than new environment mechanics.
An important distinction is between social robustness and prosocial behaviour. Following Melting Pot, we evaluate whether an agent can maximise its individual return while interacting with co-players whose incentives may differ. In mixed-motive environments, successful adaptation may involve cooperation, reciprocity, competition or exploitation. High individual return therefore does not necessarily imply socially beneficial behaviour.
We select to report changes in background-agent returns in Appendix G, allowing researchers to identify when focal-agent improvements benefit or disadvantage other agents. Cooperation rates and Gini coefficients would provide useful additional perspectives on collective welfare, but they measure a different objective from the individual-return competence our benchmark targets.
Dataset quality and pro-social cooperation are distinct considerations. We validate the converged independent-PPO behaviour policies against returns reported in the original Melting Pot work. Rather than collecting only expert trajectories, we log uniformly throughout training, producing skill-mixed datasets spanning near-random through converged behaviour (Figure 3 and Appendix E).
Importantly, our scenarios include cooperative, defective and reciprocal background populations (Table 3), including environments such as Clean Up and Coop Mining where cooperation can be mutually beneficial. The purpose is not to collect exclusively cooperative behaviour, but to capture the heterogeneous interactions that offline algorithms must learn to navigate.