acceptodds
Under review as a conference paper at ICLR 2027

OpenSocialJax: Evaluating Social Adaptation in Open-ended Sequential Social Dilemmas

Abstract

Sequential social dilemma benchmarks capture diverse social interactions, but rarely test whether agents can infer hidden environmental rules through repeated interaction while adapting to other agents. We introduce OpenSocialJax, which extends Clean Up and Commons Harvest with procedurally generated layouts and hidden rules governing tool effects, rewards, and resource dynamics. Each episode samples one configuration from tens of millions of possibilities and holds it fixed across five trials. We evaluate eight large language models, including three closed source models and five open source models. Our evaluation tests whether models can discover hidden rules and adapt to other agents in homogeneous and mixed teams. We find that rule discovery often improves performance but does not reliably sustain cooperation, as the cleaning burden becomes concentrated among fewer agents in OpenCleanup and every team eventually exhausts the shared resource in OpenHarvest. In our mixed arena, one cooperative agent was able to sustain the public good in OpenCleanup at a cost to its own return, but could not prevent resource depletion in OpenHarvest. Together, these results highlight the difficulty of adapting jointly to unfamiliar environmental dynamics and the changing behavior of other agents. Code is available at https://anonymous.4open.science/r/OpenSocialJax-53B0.

Primary area:
datasets and benchmarks
Keywords:
Multi-agent, Social dilemmas, LLMs agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hLyEOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce a benchmark called OpenSocialJax, designed to evaluate how well LLM agents perform in sequential social dilemmas. Extending two of the original MeltingPot environments, the authors design a procedural generator which makes millions of unseen environmental configurations agents must reason about while interacting with their partners. They find that even though models are able to maintain a memory of previous environment interactions and reason about being in the social dilemma, they still end up defecting and draining the shared commons.

Critical strengths
  • The environment design makes fast simulation of diverse social dilemmas possible. This is a nice contribution because it removes some of the limits of prior work in the MARL literature where rules are fixed and it’s hard to understand whether the limits of agent behavior were due to a fundamental problem in how they were maximizing reward or because they were fixating on certain environmental queues.
  • The cross-model evaluations had interesting findings. Currently most LLM evaluations center around swarms of agents all using the same model trying to accomplish a single task, but as we move towards more people having their own agents customized to them interacting and exchanging resources, these type of evals will be really important for understanding how systems might fail and what capabilities we should be looking to build into models.
Critical weaknesses
  • It’s unclear how benchmarking LLMs in these 2d environments will transfer to settings they’re actually deployed in. I appreciate the need to evaluate them in sequential social dilemmas where they’re managing resources, but adding in these constraints where they have to perceive a 2D world and decide how to move around seems to add additional burden to their reasoning process that purely text based environments would not have. Could the authors motivate why 2D environments are the right choice here compared to adding the same types of procedural variation to an environment like GovSim?
  • Relatedly, the Jax-based angle of the paper seems orthogonal to the main LLM evaluations. I think all of the LLM evals could be discussed without knowing the environment is made in Jax or that it’s a 2d SSD (presumably LLM inference would have been the main bottleneck of these evals even within the old MeltingPot environment). As an additional baseline, do the authors have results from training traditional Deep RL agents on the Jax-based environment? Does the speed of the environment combined with the procedural generator enable training policies at a large scale that unlocks new behavior beyond just defect? It would be nice to benchmark the LLM agent behaviors against Deep RL agents trained on both individual and global rewards so that the (sub)optimality of LLMs can be better contextualized.
Questions for authors

See weaknesses

4 comments

We thank the reviewer for their feedback. Below, we respond to their comments and present the results of additional experiments.

  1. Weakness 1 (Why 2D environments rather than text-based ones like GovSim):

    W: It's unclear how benchmarking LLMs in these 2d environments will transfer to settings they're actually deployed in...Could the authors motivate why 2D environments are the right choice here compared to adding the same types of procedural variation to an environment like GovSim?

Compared with text-based dilemmas such as GovSim, OpenSocialJax adds these three challenges to the evaluation of social dilemma in LLM agents.

First, in our OpenSocialJax, agents must discover the hidden rules governing the available tools and adapt how they use those tools to solve the task. As agents discover different rules at different times, their individual knowledge shapes how they act and which opportunities and incentives for cooperation arise. The dynamics of the social dilemma therefore evolve as agents learn and interact.

Second, compared with text-based environments that use predefined branches, our 2D environments let agents move and explore a shared space freely. Partial observability can make observations asymmetric which means that one agent may see another’s actions without being seen in return. This makes coordination more challenging because agents must act with incomplete and potentially different information about one another. This also gives agents mush larger state and action spaces for agents reasoning and behaviors.

At last, in a setting like GovSim, an LLM can frame the dilemma as a mathematical problem and use the resource regeneration dynamics to calculate a sustainable extraction rate. In OpenSocialJax, agents must put their reasoning into practice by coordinating actions such as cleaning pollution and harvesting under the rules of the current environment. The benchmark therefore tests whether agents can turn an understanding of the social dilemma into effective joint behavior.

Overall, OpenSocialJax combines richer environment dynamics with a broader range of agent behaviors, bringing the evaluation closer to real-world coordination settings where agents must act in a shared space under partial observability.

  1. Weakness 2 (JAX angle orthogonal to LLM evals; missing Deep RL baselines):

    W: Relatedly, the Jax-based angle of the paper seems orthogonal to the main LLM evaluations. I think all of the LLM evals could be discussed without knowing the environment is made in Jax or that it's a 2d SSD (presumably LLM inference would have been the main bottleneck of these evals even within the old MeltingPot environment)... It would be nice to benchmark the LLM agent behaviors against Deep RL agents trained on both individual and global rewards so that the (sub)optimality of LLMs can be better contextualized.

We added multiagent independent recurrent PPO (RPPO) as an RL² baseline for meta RL. In OpenHarvest, the agents’ returns increased across successive trials and showed the meta learning property. In OpenCleanup, the baseline failed to learn any signals. Rewards are sparse because agents must first clean up pollution before they can obtain any reward. It is almost impossible for agents using random actions to collect the right tools, clear enough waste, and obtain apples as rewards. This result illustrates why we focus on LLM agents. Their pretrained knowledge and reasoning abilities could help them identify such prerequisites and explore strategies in more complex social dilemmas.

We will also open-source our RL² implementation and welcome contributions that use meta learning to address sequential social dilemmas.

Cleanup (team apples per trial)

RL²Trial 1Trial 2Trial 3Trial 4Trial 5Total
Individual reward000000
Common reward000000

Harvest (team points per trial)

RL²Trial 1Trial 2Trial 3Trial 4Trial 5Total
Individual reward39.838.237.835.037.2188.1
Common reward47.953.955.655.356.9269.7

Thank you to the authors for their response.

For the response to weakness 1, I'm not entirely sold on the arguments raised. It seems unlikely to me that there is something uniquely expressible and open-ended about these Jax environments that text-based environments will not be able to account for. Features such as partial observability, discovering hidden tools, and an evolving social dilemma all seem like they could be built with a much richer combinatorial space than what is expressible from this discrete action and state space of the studied environments, while also being in distribution of what these models were actually trained on. I do acknowledge the point the authors raised about how models in text-environments may just try to map a solution to whatever the existing game theory literature said they should, so environments which require meaningful action are well motivated in this sense.

Despite my concerns, I think with further analysis of the difference between the Deep RL agents and the LLMs (the experiments run for weakness 2), I'd be happy to raise my score. For instance, why is the common reward producing 0 apples in cleanup? is this a bug from training? does it require longer runs? In what states would an LLM produce different decisions from the deep RL agent? For the Harvest example, it is nice to see a gap between individual vs common reward but what does the individual reward+behavior of the deep RL agents look like compared to the LLMs?

We sincerely thank the reviewer for the thoughtful and constructive feedback. Below, we explain our meta RL baseline in detail.

  1. Why does RL² score 0 apples in OpenCleanup?

It is not a bug, and longer training does not help. The RL² baseline still obtained 0 reward in OpenCleanup after a full training run of 10^9 timesteps. The environment is too complex for random actions to yield any reward at the start of training, leaving the network without a reward signal to learn from.

The river starts almost fully polluted, and apples grow only once it is clean enough. Cleaning requires three steps in order. The agent must pick up two tools in the right order according to the hidden rules, carry the resulting tool to the polluted river, and fire at waste of the matching level. A random policy earns 0 apples in 100% of episodes, so the policy has no reward signal to learn from throughout the episode.

We ran a small experiment to verify that the failure of the RL² baseline is not caused by a bug in our code. We simplified the OpenCleanup environment to make the reward easier to obtain as follows. Agents can directly collect the tool matching each waste type on the map (without having to explore tool combinations) and we relaxed the condition for apple growth (apples grow when less than 70% of the river is polluted, instead of less than 40% in the original environment). With these changes, the algorithm obtains the results below. Note that this only shows that our code has no bug that makes RL² baseline not work. This is the results of the simplified environment, and it cannot be compared directly with those of the LLM agents.

Training steps1.0×10⁸2.5×10⁸5.0×10⁸1.0×10⁹1.5×10⁹
Trial 165121181279304
Trial 258114177287314
Trial 357116177283314
Trial 460122181293320
Trial 562123180289319
Total3015968971,4321,571
  1. OpenHarvest: how does RL policies compare with the LLM agents?

In OpenHarvest, the LLM agents and the RL policies behave in very different ways. The RL policy trained with individual reward eats apples as fast as possible. The policy trained with common reward always keeps some apples on the map to sustain the resource until the end of the episode.

Interestingly, the LLM agents can reason that they should leave apples in place to maximize the long term return, but once they observe other agents harvesting, they still deplete the orchard. For example, even the most capable model, Claude Opus 5.5, reasons that "There is a ripe cyan apple at (7,2) right next to me. Agent B, C and E all work this patch, so if I wait someone else will eat it." As a result, an LLM agent preserves the resource as long as it is not threatened by other agents, but exhausts it once it observes others approaching. Accordingly, under the RL policy with individual reward the apples are depleted after about 48 steps, and under common reward they last until the end of the episode (400 steps). The LLM agents (Claude Opus 5.5) deplete the resource after about 140 steps which is between individual reward policy and common reward policy.

The LLM agents nevertheless obtain higher returns, mainly because they are much better at eating each apple when its value is highest. For Claude Opus 5.5, the share of apples eaten at the most valuable stage reaches 0.73–0.86, and it earns up to 0.96 reward per apple, whereas the common reward policies obtain only 0.58 reward per apple.

In summary, the LLM agents are locally smart but collectively short-sighted. Each agent makes sensible decisions about which apple to eat and when. It even recognizes that restraint in using the resource would benefit everyone. But the agents' individually rational responses still exhaust the shared resource. The RL policies show the opposite profile. With common reward they sustain the resource, which suggests that RL policies or frameworks might have good potential to maintain the shared resource when suitable algorithms are designed (i.e., algorithms that let agents learn that restraint benefits everyone). At the same time, they struggle more than the LLM agents to infer a locally optimal strategy through trial and error.

Thanks for all of the analysis! This definitely makes the evals more interesting, I've raised my score!

Reviewer_YFLjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents two new multi-agent learning environments, OpenHarvest and OpenCleanup, designed to evaluate the ability of AI agents to cooperate in settings that present social dilemmas, while also having unknown transition dynamics. The two environments, collectively referred to as OpenSocialJax, are procedurally generated, with the mechanics and/or environment layouts varying between episodes. This forces agents to decide how, and whether, to coordinate at the same they are attempting to adapt to the current instance of the environment. The environments themselves are extensions of the existing "Clean Up" and "Common Harvest" scenarios from the Melting Pot suite of environments.

While the environments are suitable for training task-specific agents with multi-agent reinforcement learning agents, the paper evaluates agents based on eight existing LLMs, to determine how effectively they are able to learn the dynamics of these environments, and how well they are able to cooperate. Each agent in the environment is controlled via calls to an LLM (which may be the same or different for all agents).

These LLMs interact with the environment through a harness that asks the model that asks the model to provide an action in each state of the environment. The prompt for each action describes the current state, which entity in the environment the agent is controlling, the observed layout of the environment, and the know transition dynamics (such that the prompt tells the LLM what the agent does and does not know about the environment).

Experimental results show that while the LLM-based agents are able to explore the environments sufficiently to learn most of the details of the transition dynamics, in most cases they failed to sustain mutually beneficial cooperation. Analysis of the reasoning traces for these models suggests that at least some agents were able to correctly identify the social dilemma, and that optimal strategy is to defect rather than cooperate. Most models led to agents converging to mutual defection strategies, though Claude Opus 5.5 sustained mutual cooperation (sometimes to its disadvantage) more often than the other models tested.

Critical strengths
  1. The new environments could be useful in research on cooperation in multi-agent reinforcement learning. Tasks with fixed dynamics often allow agents to coordinate their behavior using predefined conventions. The introduction of procedurally-generated dynamics may force agents to identify strategies for coordinating their behavior online in each scenario.
  2. The experimental results do demonstrate that most LLM-based agents will identify that exploiting altruistic partners will be the most profitable strategy, and will not be biased towards supporting mutual cooperation.
Critical weaknesses
  1. The biggest weakness I see with this work is that it does not seem to evaluate the ability of LLM-based agents to actually learn about the dynamics of the environment. It appears that the actual learning mechanism is hard-coded, that is, observed parts of the layout or other environment rules are added to a separate model as soon as they are observed, and then this entire model is provided as part of the prompt for the LLM-based agent. Thus, from the LLM's perspective, they dynamics of the environment would appear to be revealed relatively quickly, without any need for the LLM itself to infer the dynamics from its own observations.
  2. The utility of the new environments as a benchmark for evaluating cooperation between LLM-based agents in social is also unclear. It is not immediately obvious that the environments provide either a means for any agent to enforce cooperation, nor a means for them to communicate to coordinate their behavior. Rather than traditional RL control tasks, more realistic cooperation scenarios (with opportunities for inter-agent communication, and long-term sanctions mechanisms) might be more useful in evaluating LLM behavior in social dilemmas.
Questions for authors
  1. A formal definition of "social dilemmas" is provided in section 3.1. Is there any guarantee that every possible generated environment will represent the same dilemma? For example, could there be layouts or crafting mechanics that make mutually beneficial cooperation impossible?
  2. While variable dynamics create a much more challenging problem for task-specific MARL agents, how much of an impact does it have on the performance of LLM-based agents, given that the (currently known) layout and other rules are provided directly as part of the prompt? In other words, what do the new environments tell us that the earlier Melting Pot environments could not?
  3. Do changes to the dynamics meaningfully affect how agents need to cooperate? That is, could the same cooperative strategy be implement in any version of the environment, or would the agents need to agree on qualitatively different joint strategies for different versions in order to cooperate?
  4. Are the LLM-based agents allowed to communicate with one another in any way?
  5. What would be the minimum individual return that would be expected if all agents learned to successfully cooperate? What would we expect to see if the LLM-based agents had "solved" the tasks?
11 comments

We sincerely thank the reviewers for their insightful comments and suggestions. Below, we clarify our approach and present results from additional experiments.

  1. Weakness 1 & Question 2 (Learning mechanism is hard-coded; dynamics revealed via prompt):

    W: The biggest weakness I see with this work is that it does not seem to evaluate the ability of LLM-based agents to actually learn about the dynamics of the environment...Thus, from the LLM's perspective, the dynamics of the environment would appear to be revealed relatively quickly, without any need for the LLM itself to infer the dynamics from its own observations. Q: While variable dynamics create a much more challenging problem for task-specific MARL agents, how much of an impact does it have on the performance of LLM-based agents... In other words, what do the new environments tell us that the earlier Melting Pot environments could not?

(a) Do agents infer the dynamics themselves? (W1)

The harness is primarily intended to support weaker agents, and we find that stronger agents such as Claude Opus 5.5 are less sensitive to its removal. To test this, we added an ablation that removes the memory harness entirely and limits the initial prompt to basic information (the map elements, how to earn rewards, what the tools do, and which actions agents can take). In this setting, agents must decide what information to retain, summarize it themselves, and infer the game rules from their own observations.

Claude Opus 5.5 (OpenCleanup):

SettingTrial 1Trial 2Trial 3Trial 4Trial 5Total
Full harness1633403283333501514
Minimal harness1774793022643211543

Claude Opus 5.5 (OpenHarvest):

SettingTrial 1Trial 2Trial 3Trial 4Trial 5Total
Full harness6279726363339
Minimal harness5575746556325

Qwen3.8-27B (OpenCleanup):

SettingTrial 1Trial 2Trial 3Trial 4Trial 5Total
Full harness6376407313265
Minimal harness02821211989

Qwen3.8-27B (OpenHarvest):

SettingTrial 1Trial 2Trial 3Trial 4Trial 5Total
Full harness4342444954232
Minimal harness4350444351231

Qwen3.5-4B (OpenCleanup):

SettingTrial 1Trial 2Trial 3Trial 4Trial 5Total
Full harness514294034122
Minimal harness000000

Qwen3.5-4B (OpenHarvest):

SettingTrial 1Trial 2Trial 3Trial 4Trial 5Total
Full harness4440434241210
Minimal harness4037423739195

(b) What the new environments add beyond Melting Pot (Q2)

Melting Pot evaluates cooperation under fixed rules. Our environments evaluate whether agents can discover initially unknown rules while cooperating and how those two processes affect each other. In OpenCleanup, the division of labor depends on which recipes each agent has discovered. In OpenHarvest, the maturity stage that yields the highest immediate reward can differ from the stage that allows the fastest regrowth, so agents must learn this trade-off as they coordinate when to harvest. Therefore, the dynamics of the social dilemma evolve as agents discover the hidden rules. Agents that know more of these rules can choose from a wider range of strategies.

  1. Weakness 2 & Question 4 (Enforcement and communication):

    W: The utility of the new environments...enforce cooperation, nor a means for them to communicate to coordinate their behavior. Rather than traditional RL control tasks, more realistic cooperation scenarios (with opportunities for inter-agent communication, and long-term sanctions mechanisms) might be more useful in evaluating LLM behavior in social dilemmas. Q: Are the LLM-based agents allowed to communicate with one another in any way?

In the experiments reported in the paper, agents interacted through their actions. Under partial observability, they could observe nearby agents’ behavior, infer whether others were contributing to the task, and use those observations and reasoning ability to decide whether to cooperate or defect.

For example, Claude Opus 5.5 wrote, “D never cleans. My best role is clearing G0.” In another instance, it wrote, “Others agent lighten waste C2→C1→C0. My role is sweeping the C0 they leave behind.” In both cases, the agent chose its role based on what it observed others doing.

We thank the reviewer for this suggestion about communication. In response, we added an optional communication feature. When enabled, at each time step an agent can send one message to and receive messages from other agents within its local observation range. Enabling communication improved Claude’s performance, while we observed no improvement for Qwen3.5-4B.

Claude Opus 5.5 Cleanup

SettingTrial 1Trial 2Trial 3Trial 4Trial 5Total
Full harness1633403283333501514
Full harness + Comm2924152933563321688

Claude Opus 5.5 Harvest

SettingTrial 1Trial 2Trial 3Trial 4Trial 5Total
Full harness6279726363339
Full harness + Comm861061199470475

Qwen3.5-4B Cleanup

SettingTrial 1Trial 2Trial 3Trial 4Trial 5Total
Full harness514294034122
Full harness + Comm00246143128

Qwen3.5-4B Harvest

SettingTrial 1Trial 2Trial 3Trial 4Trial 5Total
Full harness4440434241210
Full harness + Comm3835424247204

Even with communication enabled, agents tended to behave selfishly over successive trials. The social dilemma therefore persisted despite the performance gains observed.

  1. Question 1 (Do all generated environments represent the same dilemma?):

    Q: A formal definition of "social dilemmas"...Is there any guarantee that every possible generated environment will represent the same dilemma? ...

We do not verify each configuration individually, but the generators change the strength of the dilemma, not its existence and properties. In OpenCleanup, apples grow only when the river is clean and every rule set contains all cleaning effects, so cooperation is always possible. In OpenHarvest, hidden multipliers change regrowth rates but never remove regrowth.

The generator also constraints the map size and the proportions of map elements(such as numbers of waste, apples and clean water), keeping difficulty within a bounded range and avoiding unsolvable layouts. We also provide fully cooperative and fully selfish scripted oracle policies to test whether the generated environments exhibit a social dilemma schelling diagram.

Cleanup (apples per agent per trial):

Other cooperators01234
Cooperator0.018.756.486.1106.2
Defector0.052.9133.7175.9172.3

Harvest (points per agent per trial):

Other cooperators01234
Cooperator5.15.66.07.725.8
Defector8.410.212.115.922.3
  1. Question 3 (Do dynamics changes require different cooperative strategies?):

    Q: Do changes to the dynamics meaningfully affect how agents need to cooperate? That is, could the same cooperative strategy be implement in any version of the environment, or would the agents need to agree on qualitatively different joint strategies for different versions in order to cooperate?

Each environment is designed around a specific form of cooperation. In OpenCleanup, agents contribute to a public good by cleaning the river. In OpenHarvest, agents sustain a common pool resource by leaving some apples left. What the hidden rule changes is the concrete joint plan that carries this out. In OpenCleanup, the recipes decide which working tool each cleaner must craft and which waste it acts on, and therefore how the cleaning work should be divided. In OpenHarvest, the payoff and regrowth tables decide which ripeness stage of each colour to harvest and which cells recover fastest.

We added a rule changed experiment that changes the hidden rules between trials to test whether agents can adapt their cooperative strategies to the new rules. Trials 1 and 2 shared one environment configuration, and trials 3 and 4 shared a new configuration. Agents can still carry over their own summaries between trials (also from trial 2 to trial 3). After the rules changed, agents performed better in trial 3 than in trial 1. This suggests that their self-generated summaries captured transferable cooperative strategies for discovering and adapting to new rules.

Claude Opus 5.5 (OpenCleanup):

SettingTrial 1Trial 2Trial 3Trial 4
Rules resampled183305325303

Claude Opus 5.5 (OpenHarvest):

SettingTrial 1Trial 2Trial 3Trial 4
Rules resampled62827560
  1. Question 5 (Expected return under successful cooperation):

    Q: What would be the minimum individual return that would be expected if all agents learned to successfully cooperate? What would we expect to see if the LLM-based agents had "solved" the tasks?

The theoretical performance ceiling is difficult to derive, so we implemented selfish and cooperative oracle policies and used their empirical performance as a reference for what agents can achieve. The oracle policies are hand-coded policies that directly accesses the true hidden rules and the full map, then selects actions according to fixed decision rules. Even the best-performing LLM agent fell short of the performance achieved by the oracle policy.

OpenCleanup oracle (team apples per trial)

PolicyTrial 1Trial 2Trial 3Trial 4Trial 5Total
Selfish oracle policy0.00.00.00.00.00.0
Cooperative oracle policy543.3546.7510.3515.0538.72654.0
Claude163.3340.3328.3333.3350.31515.7

OpenHarvest oracle (team points per trial)

PolicyTrial 1Trial 2Trial 3Trial 4Trial 5Total
Selfish oracle policy42.742.644.838.840.6209.4
Cooperative oracle policy132.9122.3121.7125.4142.6644.9
Claude61.779.171.962.562.5337.7
Reviewer_ZebBOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a benchmark OpenSocialJax. This benchmark tests adaptibility of LLM based agents on adapting to unfamilia environments that contains hidden environmental rules and social dilemmas. The authors propose to extend the existing clean up and commons harvest environments from popular Melting pot. The paper proposes to generate many new layouts that do not exist in existing environments. Instead of hand designing a few fixed maps, authors wrote generators : a code that randomly creates a map , hidden rules, and agent dependencies. The spatial arrangements such as river, tool stations, etc is randomly generated. There are hidden mechanisms that agents must dicsover, since the rules and layout changes, the dependence of agents on each other changes too, agents are allowed to revise strategies. Hence the contribution of the paper is : per episode randomisation, millions of possible configurations, repeated trials, and agent protocols. The empirical evidence consists of frontier models : GPT-6 luna, GPT-6-sol, Opus 5.5, Qwen3.8-27B, etc. Validations are done for : Schelling diagrams, Homogeneous-team experiments, Rule discovery, Mixed-model arena.

Authors note that the current frontier models are able to discover many hidden rules and they improve after first trial, however, they are unable to resolve both dilemma.

Critical strengths
  • A novel tough benchmark that targets social adaptation for LLMs. The paper proposes to randomize map layouts, hidden mechanism rules, and inter agent dependencies using millions of configurations. It allows agents to infer from each configurations by keeping it fixed and allowing multiple trials so that agents may infer the latent rules from interactions and revisions. This becnhmark and evaluation clearly asks: can an agent infer hidden rules and adapt to co-players whose behavior changes as a result of those rules.

  • A good evaluation with interesting findings. A clean way to study how agents form and revise hypotheses over long horizons without a growing conversation history. The experiments are broad (8 frontier models, homogeneous teams + a mixed-model arena, 3 seeds, 5 trials), and the results yield a non-obvious, mechanistically explained finding. For example, it shows models do discover hidden rules, but this discovery may not lead to effective outcome.

Critical weaknesses
  • It seems that the generator output validation is assumed, it is not clear how the validation is done for sampled layouts, whether they are playable, in particular human validation will be useful. Also, I could not find reporting of the rejection rates or degenerate cases. It is not clear whether there are some protocols used for sampling quality, minimum spacing rules? non-overlap? region boundaries? connectivity? reachability checks? First, without stated constraints, "tens of millions of configurations" is a count of samples, not of valid, distinct, playable configurations.

  • The evaluation may be too narrow to support the generality claims, and the headline "adaptation" result is confounded by protocol design. The paper considers 3 seeds per environment, 5 trials of less than 400 steps and 5 agent homogeneous teams. There may be a confounding evaluations here: (a) 3-seed seems little for a distribution over millions of configurations, as we see the variance shown in figures 3-5 seem wide. In the multiple trial cases, improvement from trial 1 to trial 2 is partly just memory carry-over rather than genuine meta-adaptation? For example : the paper does not include a control (e.g., fresh agents each trial, or shuffled rules between trials) to separate "learning the latent configuration" from "benefiting from a persistent transcript." Prompt/role sensitivity is unexamined: the Schelling verification (Appendix C.1) relies on a single role-instruction phrasing with GPT-6-Luna?

Questions for authors
  1. What sampling quality generators enforce to keep the generator samples valid ?
  2. How many distinct valid configurations were actually used, and how were seeds chosen? You reported 3 seeds, were these selected uniformly, hand picked ?
  3. Is the headline combinatorial count (57,657,600 / 60,466,176) a count of valid configurations or of sampled rule assignments?
  4. Can you separate "learning the latent configuration" from "carrying over a persistent transcript"? Do you have a control condition e.g, fresh agents each trial, or rules resampled between trials that isolates genuine meta-adaptation from memory carry-over?
  5. Adding learning based baselines could be interesting if trained on same generators, to compare with LLM performance.
  6. Add more closely related papers that are missing.
7 comments

We thank the reviewer for the valuable comments and constructive feedback. Below, we clarify several aspects of our work and provide additional experimental results.

  1. Weakness 1 & Question 1 (Generator validation and sampling quality):

    It seems that the generator output validation is assumed, it is not clear how the validation is done for sampled layouts, whether they are playable... Also, I could not find reporting of the rejection rates or degenerate cases. It is not clear whether there are some protocols used for sampling quality... First, without stated constraints...not of valid, distinct, playable configurations. What sampling quality generators enforce to keep the generator samples valid?

Our environment generation pipeline has two components, rule generation and map generation. We describe each in turn.

Every combination of rules in our generation procedure is valid and yields a solvable task, because the composition depth is the same across configurations. In OpenCleanup, our generator assigns each of the seven effects to a different tool pair, so every setup has all the effects needed for cleanup. In OpenHarvest, our generator samples rewards and regrowth rules so that every setup is playable. It also samples map layouts and patch colors.

During map generation, we reject candidate layouts that violate explicit constraints on non-overlap, minimum spacing, distances from rivers and orchards, region connectivity, or reachability. The acceptance rates were 17.8% for OpenCleanup and 87% for OpenHarvest, and no duplicate layouts were accepted. We also constrain the map size and the proportions of resources and tools so that agents have a feasible way to obtain reward on every accepted map. Together with the layout constraints, these generation rules ensure that every accepted layout is solvable.

  1. Weakness 2 & Question 4 (Narrow evaluation and confounded adaptation result):

    W: The evaluation may be too narrow to support the generality claims...(a) 3-seed seems little for a distribution over millions of configurations... In the multiple trial cases, improvement from trial 1 to trial 2 is partly just memory carry-over...the Schelling verification (Appendix C.1) relies on a single role-instruction phrasing with GPT-6-Luna? Q: Can you separate "learning the latent configuration" from "carrying over a persistent transcript"? Do you have a control condition e.g., fresh agents each trial, or rules resampled between trials that isolates genuine meta-adaptation from memory carry-over?

(a) Number of seeds.

We acknowledge that 3 seeds cover only a small fraction of the possible configurations. We will evaluate the agents across more random seeds. Our generation procedure ensures that the rules are valid in every generated configuration, and we observed the similar social dilemma patterns across all 3 seeds. The evaluated configurations also yield clear performance differences among the models.

(b) Separating adaptation from memory carry-over (Q4).

Our LLM agents learn across trials through the context carried forward in their prompts. If this context is reset between trials, we would expect their behavior and performance to remain broadly similar to those in the first trial.

Control 1: Fresh agents each trial. We also evaluated a control condition in which we fully reset the prompt before each trial.

Claude Opus 5.5 (OpenCleanup):

SettingTrial 1Trial 2Trial 3
Reset prompt every trial177183160

Claude Opus 5.5 (OpenHarvest):

SettingTrial 1Trial 2Trial 3
Reset prompt every trial656164

Control 2: Rules resampled between trials. We then evaluated a condition in which trials 1 and 2 shared one environment configuration, and trials 3 and 4 shared a new configuration. Agents can still carry over their own summaries between trials, even from trial 2 to trial 3. After the rules changed, agents performed better in trial 3 than in trial 1. This suggests that their self-generated summaries captured transferable strategies for discovering and adapting to new rules, providing evidence of meta-adaptation across configurations.

Claude Opus 5.5 (OpenCleanup):

SettingTrial 1Trial 2Trial 3Trial 4
Rules resampled183305325303

Claude Opus 5.5 (OpenHarvest):

SettingTrial 1Trial 2Trial 3Trial 4
Rules resampled62827560

(c) Schelling verification.

To obtain a more reliable Schelling diagram, we measured payoffs across different mixtures of fully cooperative and fully selfish scripted oracle policies. The resulting payoff relationships satisfy the criteria for a sequential social dilemma.

Cleanup (apples per agent per trial):

Other cooperators01234
Cooperator0.018.756.486.1106.2
Defector0.052.9133.7175.9172.3

Harvest (points per agent per trial):

Other cooperators01234
Cooperator5.15.66.07.725.8
Defector8.410.212.115.922.3
  1. Question 2 (Number of configurations and seed selection):

    Q: How many distinct valid configurations were actually used, and how were seeds chosen? You reported 3 seeds, were these selected uniformly, hand picked?

    We evaluate 3 (seeds) distinct configurations per environment, one sampled by each seed, and share them across all experiments. The seeds were chosen at random before any evaluation, and no configuration was selected or discarded based on results.

  2. Question 3 (Combinatorial count: valid configurations or rule assignments):

    Q: Is the headline combinatorial count (57,657,600 / 60,466,176) a count of valid configurations or of sampled rule assignments?

These numbers count hidden rule sets, and every rule set is valid and distinct by construction. In OpenCleanup, our generator assigns the seven effects one-to-one to seven distinct tool pairs. Every generated rule set therefore includes all effects required for cleanup, ensuring that every rule assignment is playable. In OpenHarvest, every generated assignment of rewards and regeneration rules is likewise valid. Layouts and patch colors are sampled in addition.

  1. Question 5 (Learning-based baselines):

    Q: Adding learning based baselines could be interesting if trained on same generators, to compare with LLM performance.

We added multiagent independent recurrent PPO (RPPO) as an RL² baseline for meta RL. With individual rewards, each agent is trained using its own reward. With a common reward, all agents are trained using the same team level reward (average of all agents reward).

In OpenHarvest, the agents’ returns increased across successive trials. In OpenCleanup, the baseline failed to learn any signals. Rewards are sparse because agents must first clean up pollution before they can obtain any reward. This result illustrates why we focus on LLM agents. Their pretrained knowledge and reasoning abilities may help them identify such prerequisites and explore strategies in more complex social dilemmas.

OpenCleanup:

RL²Trial 1Trial 2Trial 3Trial 4Trial 5Total
Individual reward000000
Common reward000000

OpenHarvest:

RL²Trial 1Trial 2Trial 3Trial 4Trial 5Total
Individual reward39.838.237.835.037.2188.1
Common reward47.953.955.655.356.9269.7
  1. Question 6 (Missing related work):

    Q6: Add more closely related papers that are missing.

    We will add the following related papers. If there are other papers you consider relevant, we would be happy to discuss them in our work.

  • Saaket Agashe, Yue Fan, Anthony Reyna, and Xin Eric Wang. LLM-Coordination: Evaluating and analyzing multi-agent coordination abilities in large language models. arXiv preprint arXiv:2310.03903, 2023.

  • Nathan Cloos, Meagan Jens, Michelangelo Naim, Yen-Ling Kuo, Ignacio Cases, Andrei Barbu, and Christopher J. Cueva. Baba is AI: Break the rules to beat the benchmark. arXiv preprint arXiv:2407.13729, 2024.

  • Logan Cross, Violet Xiang, Agam Bhatia, Daniel L. K. Yamins, and Nick Haber. Hypothetical Minds: Scaffolding theory of mind for multi-agent tasks with large language models. In International Conference on Learning Representations, 2025.

  • Akshay Krishnamurthy, Keegan Harris, Dylan J. Foster, Cyril Zhang, and Aleksandrs Slivkins. Can large language models explore in-context? In Advances in Neural Information Processing Systems, volume 37, 2024.

  • Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, 2023.

  • Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.

Thanks for answering my questions, I have more clarity. I have few short questions below.

  • Acceptance rates 17.8% and 87%. Can we identify or breakdown further what leads to rejections ?
  • Can you explain a bit more: "no duplicate layouts were accepted" ?
  • control 1 do suggest that performance remains almost constant, and in control 2, agents performed better with trial. Here, do we learn that this is a form of "in-context learning with additional carry over memory" ? or do we see this as a form of meta adaptation?
  • RPPO gets 0 for opencleanup but RPPO with individual vs common reward gives zero, is this any hyper-parameter tuning issue? Can you check again?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.