- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors introduce a benchmark called OpenSocialJax, designed to evaluate how well LLM agents perform in sequential social dilemmas. Extending two of the original MeltingPot environments, the authors design a procedural generator which makes millions of unseen environmental configurations agents must reason about while interacting with their partners. They find that even though models are able to maintain a memory of previous environment interactions and reason about being in the social dilemma, they still end up defecting and draining the shared commons.
Critical strengths
- The environment design makes fast simulation of diverse social dilemmas possible. This is a nice contribution because it removes some of the limits of prior work in the MARL literature where rules are fixed and it’s hard to understand whether the limits of agent behavior were due to a fundamental problem in how they were maximizing reward or because they were fixating on certain environmental queues.
- The cross-model evaluations had interesting findings. Currently most LLM evaluations center around swarms of agents all using the same model trying to accomplish a single task, but as we move towards more people having their own agents customized to them interacting and exchanging resources, these type of evals will be really important for understanding how systems might fail and what capabilities we should be looking to build into models.
Critical weaknesses
- It’s unclear how benchmarking LLMs in these 2d environments will transfer to settings they’re actually deployed in. I appreciate the need to evaluate them in sequential social dilemmas where they’re managing resources, but adding in these constraints where they have to perceive a 2D world and decide how to move around seems to add additional burden to their reasoning process that purely text based environments would not have. Could the authors motivate why 2D environments are the right choice here compared to adding the same types of procedural variation to an environment like GovSim?
- Relatedly, the Jax-based angle of the paper seems orthogonal to the main LLM evaluations. I think all of the LLM evals could be discussed without knowing the environment is made in Jax or that it’s a 2d SSD (presumably LLM inference would have been the main bottleneck of these evals even within the old MeltingPot environment). As an additional baseline, do the authors have results from training traditional Deep RL agents on the Jax-based environment? Does the speed of the environment combined with the procedural generator enable training policies at a large scale that unlocks new behavior beyond just defect? It would be nice to benchmark the LLM agent behaviors against Deep RL agents trained on both individual and global rewards so that the (sub)optimality of LLMs can be better contextualized.
Questions for authors
See weaknesses
4 comments
Thank you to the authors for their response.
For the response to weakness 1, I'm not entirely sold on the arguments raised. It seems unlikely to me that there is something uniquely expressible and open-ended about these Jax environments that text-based environments will not be able to account for. Features such as partial observability, discovering hidden tools, and an evolving social dilemma all seem like they could be built with a much richer combinatorial space than what is expressible from this discrete action and state space of the studied environments, while also being in distribution of what these models were actually trained on. I do acknowledge the point the authors raised about how models in text-environments may just try to map a solution to whatever the existing game theory literature said they should, so environments which require meaningful action are well motivated in this sense.
Despite my concerns, I think with further analysis of the difference between the Deep RL agents and the LLMs (the experiments run for weakness 2), I'd be happy to raise my score. For instance, why is the common reward producing 0 apples in cleanup? is this a bug from training? does it require longer runs? In what states would an LLM produce different decisions from the deep RL agent? For the Harvest example, it is nice to see a gap between individual vs common reward but what does the individual reward+behavior of the deep RL agents look like compared to the LLMs?
We sincerely thank the reviewer for the thoughtful and constructive feedback. Below, we explain our meta RL baseline in detail.
- Why does RL² score 0 apples in OpenCleanup?
It is not a bug, and longer training does not help. The RL² baseline still obtained 0 reward in OpenCleanup after a full training run of 10^9 timesteps. The environment is too complex for random actions to yield any reward at the start of training, leaving the network without a reward signal to learn from.
The river starts almost fully polluted, and apples grow only once it is clean enough. Cleaning requires three steps in order. The agent must pick up two tools in the right order according to the hidden rules, carry the resulting tool to the polluted river, and fire at waste of the matching level. A random policy earns 0 apples in 100% of episodes, so the policy has no reward signal to learn from throughout the episode.
We ran a small experiment to verify that the failure of the RL² baseline is not caused by a bug in our code. We simplified the OpenCleanup environment to make the reward easier to obtain as follows. Agents can directly collect the tool matching each waste type on the map (without having to explore tool combinations) and we relaxed the condition for apple growth (apples grow when less than 70% of the river is polluted, instead of less than 40% in the original environment). With these changes, the algorithm obtains the results below. Note that this only shows that our code has no bug that makes RL² baseline not work. This is the results of the simplified environment, and it cannot be compared directly with those of the LLM agents.
| Training steps | 1.0×10⁸ | 2.5×10⁸ | 5.0×10⁸ | 1.0×10⁹ | 1.5×10⁹ |
|---|---|---|---|---|---|
| Trial 1 | 65 | 121 | 181 | 279 | 304 |
| Trial 2 | 58 | 114 | 177 | 287 | 314 |
| Trial 3 | 57 | 116 | 177 | 283 | 314 |
| Trial 4 | 60 | 122 | 181 | 293 | 320 |
| Trial 5 | 62 | 123 | 180 | 289 | 319 |
| Total | 301 | 596 | 897 | 1,432 | 1,571 |
- OpenHarvest: how does RL policies compare with the LLM agents?
In OpenHarvest, the LLM agents and the RL policies behave in very different ways. The RL policy trained with individual reward eats apples as fast as possible. The policy trained with common reward always keeps some apples on the map to sustain the resource until the end of the episode.
Interestingly, the LLM agents can reason that they should leave apples in place to maximize the long term return, but once they observe other agents harvesting, they still deplete the orchard. For example, even the most capable model, Claude Opus 5.5, reasons that "There is a ripe cyan apple at (7,2) right next to me. Agent B, C and E all work this patch, so if I wait someone else will eat it." As a result, an LLM agent preserves the resource as long as it is not threatened by other agents, but exhausts it once it observes others approaching. Accordingly, under the RL policy with individual reward the apples are depleted after about 48 steps, and under common reward they last until the end of the episode (400 steps). The LLM agents (Claude Opus 5.5) deplete the resource after about 140 steps which is between individual reward policy and common reward policy.
The LLM agents nevertheless obtain higher returns, mainly because they are much better at eating each apple when its value is highest. For Claude Opus 5.5, the share of apples eaten at the most valuable stage reaches 0.73–0.86, and it earns up to 0.96 reward per apple, whereas the common reward policies obtain only 0.58 reward per apple.
In summary, the LLM agents are locally smart but collectively short-sighted. Each agent makes sensible decisions about which apple to eat and when. It even recognizes that restraint in using the resource would benefit everyone. But the agents' individually rational responses still exhaust the shared resource. The RL policies show the opposite profile. With common reward they sustain the resource, which suggests that RL policies or frameworks might have good potential to maintain the shared resource when suitable algorithms are designed (i.e., algorithms that let agents learn that restraint benefits everyone). At the same time, they struggle more than the LLM agents to infer a locally optimal strategy through trial and error.
Thanks for all of the analysis! This definitely makes the evals more interesting, I've raised my score!
We thank the reviewer for their feedback. Below, we respond to their comments and present the results of additional experiments.
Compared with text-based dilemmas such as GovSim, OpenSocialJax adds these three challenges to the evaluation of social dilemma in LLM agents.
First, in our OpenSocialJax, agents must discover the hidden rules governing the available tools and adapt how they use those tools to solve the task. As agents discover different rules at different times, their individual knowledge shapes how they act and which opportunities and incentives for cooperation arise. The dynamics of the social dilemma therefore evolve as agents learn and interact.
Second, compared with text-based environments that use predefined branches, our 2D environments let agents move and explore a shared space freely. Partial observability can make observations asymmetric which means that one agent may see another’s actions without being seen in return. This makes coordination more challenging because agents must act with incomplete and potentially different information about one another. This also gives agents mush larger state and action spaces for agents reasoning and behaviors.
At last, in a setting like GovSim, an LLM can frame the dilemma as a mathematical problem and use the resource regeneration dynamics to calculate a sustainable extraction rate. In OpenSocialJax, agents must put their reasoning into practice by coordinating actions such as cleaning pollution and harvesting under the rules of the current environment. The benchmark therefore tests whether agents can turn an understanding of the social dilemma into effective joint behavior.
Overall, OpenSocialJax combines richer environment dynamics with a broader range of agent behaviors, bringing the evaluation closer to real-world coordination settings where agents must act in a shared space under partial observability.
We added multiagent independent recurrent PPO (RPPO) as an RL² baseline for meta RL. In OpenHarvest, the agents’ returns increased across successive trials and showed the meta learning property. In OpenCleanup, the baseline failed to learn any signals. Rewards are sparse because agents must first clean up pollution before they can obtain any reward. It is almost impossible for agents using random actions to collect the right tools, clear enough waste, and obtain apples as rewards. This result illustrates why we focus on LLM agents. Their pretrained knowledge and reasoning abilities could help them identify such prerequisites and explore strategies in more complex social dilemmas.
We will also open-source our RL² implementation and welcome contributions that use meta learning to address sequential social dilemmas.
Cleanup (team apples per trial)
Harvest (team points per trial)