Molten Pot: Evaluations & Datasets for Offline Social Reinforcement Learning

Many of the settings where reinforcement learning (RL) could matter most are both social and data-limited: agents must act in the presence of other decision-makers, yet cannot rely on online interaction to learn how to do so. Current evaluations do not target data-constrained social reasoning. Standard offline RL benchmarks treat the environment as non-social, while multi-agent benchmarks focus exclusively on fully cooperative settings. Thus the challenge of reasoning about partner identity and motivations under mixed incentives, and generalising across social structures from offline data alone, remains untested. We introduce Molten Pot, an evaluation protocol and datasets for offline mixed-motive social RL built on Melting Pot substrates. The protocol spans five substrates, 47 social scenarios, approximately one terabyte of trajectory data, and defines three complementary evaluation settings that each probe a different aspect of social robustness. Setting 1 tests offline RL in multi-agent, mixed-motive settings with fixed background populations. Setting 2 pools datasets across every scenario of a substrate with the scenario label withheld, so a single policy must learn from heterogeneous, unlabelled partner behaviour. Setting 3 evaluates zero-shot social generalisation through disjoint train/test splits that target specific social shifts. Finally, we provide four offline RL baselines (IQL, BC, BCQ, CQL) as a reliable starting point for future methods. They leave substantial headroom on our protocol, and we invite the community to adopt the benchmark and close the gap. The goal of Molten Pot is to establish offline social evaluation as a distinct and necessary target for RL research.