acceptodds
Under review as a conference paper at ICLR 2027

Cooperative agents specialize more but explore less

Abstract

Cooperation is essential for multi-agent systems to succeed. Prior work has focused on making agents more cooperative, but we argue that cooperation is only the beginning of the problem. We study deep reinforcement learning and large language model agents in sequential social dilemmas, where agents face a tension between producing immediate rewards and maintaining shared resources that sustain future production. We find cooperative agents specialize more but explore less, which can eventually undermine group performance. Otherwise identical agents who work to maximize collective rewards are more cooperative and achieve higher performance by self-organizing into maintenance or production roles. Once agents specialize, maintenance agents explore significantly less than production agents, and their search narrows further over time. They then fail to replenish sufficient resources, leaving production agents with fewer resources to use. Group performance can consequently decline by up to 93% long after cooperation has been established. These results identify a potential failure mode that can emerge after successful cooperation and highlight the importance of preserving exploration as agents specialize in multi-agent systems.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multi-agent cooperation, specialization, exploration, sequential social dilemmas

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MosDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the relationship between cooperation, specialization, exploration, and long-term collective performance in multi-agent systems. The authors study sequential social dilemmas in which agents must balance immediate individual rewards against maintaining shared resources necessary for future production.

Using both deep reinforcement learning and large language model agents, the authors demonstrate that stronger collective incentives generally encourage cooperation and the emergence of specialized roles. However, maintenance-oriented agents tend to explore less than production-oriented agents. The authors further observe that reduced exploration can precede resource depletion and deterioration in collective performance.

The experiments cover CLEANUP, HARVEST, collaborative hiring, and research collaboration. The results suggest that successful cooperation and specialization do not necessarily guarantee sustainable long-term performance.

The reviewer finds the problem clearly defined, the assessment metrics mathematically well formulated, and the findings potentially meaningful beyond AI agent systems, including sociology, organizational behavior, and management science.

Critical strengths
  1. Clear and meaningful research question. The paper addresses an interesting problem: cooperation can initially improve collective performance through specialization, but the resulting specialization may reduce exploration and eventually undermine long-term efficiency. This is an important perspective that goes beyond the conventional objective of simply improving cooperation among agents.

  2. Clear formulation and presentation. The problem setting is easy to understand, and the evaluation criteria for efficiency, specialization, and exploration are mathematically well formulated. In particular, the use of spatial entropy provides a quantitative measure of differences in exploration behavior.

  3. Potential interdisciplinary relevance. The findings may be relevant not only to AI agent systems but also to fields such as sociology, organizational behavior, and management science, where specialization, division of labor, and organizational rigidity are important subjects.

  4. Experiments across different agent systems and contexts. The authors investigate both reinforcement-learning and LLM-based agents and extend the experiments beyond CLEANUP to other settings. This provides useful evidence that the observed behavioral patterns are not restricted to a single agent architecture or application context.

Critical weaknesses

The reviewer finds the paper interesting and generally well presented. However, several aspects of the experimental settings and the interpretation of the results require further clarification.

  1. Insufficient explanation of the environment dynamics

The dynamics of the CLEANUP environment are not entirely clear. Although the authors explain that apple regrowth depends on pollution and stops when the pollution rate exceeds a threshold, the temporal mechanisms governing the appearance of apples and pollution are not sufficiently described in the main text.

These mechanisms may substantially affect the incentives for cleaning, the emergence of specialized roles, and the subsequent decline in collective performance.

The HARVEST experiment provides an additional setting with different resource dynamics. However, the differences between the environments, particularly the implications of changing from explicit maintenance to restrained resource extraction, could be explained more clearly.

  1. Limited explanation of the relationship between reward incentives and efficiency

In the deep-RL experiments, the highest efficiency is observed at λ = 0.25. However, the reason for this result is not sufficiently explained.

The authors acknowledge that their reward formulation changes both the relative incentives and the reward scale. Consequently, it is unclear whether the observed optimum reflects the underlying cooperation mechanism or the specific reward formulation and experimental conditions.

Furthermore, the relationship between the reward parameter and efficiency differs between the deep-RL and LLM experiments, which deserves further discussion.

  1. Incomplete explanation of the mechanism underlying reduced exploration

One of the central findings is that cleaners explore less than harvesters, with the difference increasing over time.

The authors suggest that cleaners encounter pollution in relatively stable locations, whereas harvesters must search for apples across a broader area. They also provide an additional experiment in an environment without task-specific opportunities, which partially addresses this concern.

Nevertheless, the mechanism responsible for the progressive reduction in cleaners' exploration remains insufficiently established. It is unclear how much of this behavior results from environmental structure, reward incentives, or learned behavioral specialization.

  1. Limited analysis of individual roles and their transitions

The paper identifies cleaners, harvesters, and mixed-role agents based on their observed behavior. However, the analysis does not sufficiently explain how these roles emerge and evolve over time.

In particular, it is unclear whether agents maintain their specialized roles throughout an episode or switch roles in response to environmental changes.

Additional analysis of role-specific contributions and role transitions could provide useful insights into the mechanisms of cooperation and subsequent performance deterioration.

  1. Differences between deep-RL and LLM experimental settings

The experimental configurations differ substantially between deep-RL and LLM-based agents, including observation size, episode duration, incentive conditions, and environmental implementations.

Although different learning mechanisms naturally require different procedures, the paper does not sufficiently explain which environmental differences are necessary and how they may affect the observed behaviors.

This makes it difficult to determine whether differences in the experimental results arise from the agent architectures or from the experimental configurations.

  1. Limited causal evidence for the proposed explanation of performance deterioration

The authors show that reduced exploration precedes declining collective performance and provide temporal regression analyses supporting this relationship.

However, these results primarily establish temporal associations rather than demonstrating that reduced exploration is the direct cause of the subsequent decline.

An intervention that encourages maintenance agents to explore more broadly could provide stronger evidence for the proposed mechanism.

Questions for authors
  1. Temporal dynamics of the CLEANUP environment

Could the authors clarify how frequently pollution and apples appear in CLEANUP? In particular, how are their appearance probabilities determined, and how does the pollution level quantitatively affect apple regrowth below the threshold?

Since the paper focuses on the temporal evolution of cooperation and resource depletion, these dynamics seem important for interpreting the results.

  1. Differences between CLEANUP and HARVEST

Appendix D explains that HARVEST replaces explicit maintenance with a resource-extraction dilemma. Could the authors further clarify the differences in environmental structure and resource dynamics?

In particular, how does the absence of a separate maintenance action affect the interpretation of specialization and exploration compared with CLEANUP?

  1. Why does \lambda = 0.25 achieve the highest efficiency in deep RL?

Equation (1) gives each agent a combination of its individual reward and the team's total reward. Thus, even when λ is relatively small, agents have an incentive to improve collective performance.

Could the authors explain why \ lambda = 0.25 produces the highest efficiency in the deep-RL experiments?

Does this result persist when reward magnitudes are normalized? Could the initial spatial positions of the agents influence which agents become cleaners and consequently affect the optimal reward setting?

  1. Why do cleaners progressively explore less?

The authors suggest that cleaners repeatedly encounter pollution in relatively stable locations, while harvesters must search across a broader region.

Could the authors provide further analysis of the mechanism responsible for the progressive reduction in cleaners' exploration?

For example, is this primarily a consequence of the spatial distribution of pollution, the reward structure, or the learned specialization of the agents?

  1. Role-specific performance and contribution

Could the authors report additional statistics for cleaners, harvesters, and mixed-role agents, such as their average collected rewards, effective cleaning actions, and contributions to maintaining the shared resource?

Since cleaners do not directly receive rewards for cleaning, distinguishing individual rewards from contributions to collective performance would be particularly informative.

  1. Conditions for switching roles

Under what conditions do agents switch between cleaning and harvesting?

For example, when pollution approaches or exceeds the threshold that prevents apple regrowth, do harvesting agents begin cleaning more frequently?

An analysis of role transitions over time could help explain why initially successful cooperation sometimes deteriorates.

  1. Differences between RL and LLM experimental configurations

Could the authors clarify why the deep-RL and LLM experiments use different environmental configurations, particularly the observation sizes, interaction horizons, and incentive settings?

Which environmental properties are preserved across the two implementations, and to what extent might these differences affect the observed specialization and exploration patterns?

Suggestion (not a condition for acceptance)

  1. An interesting extension would be to introduce explicit costs associated with different actions, such as movement, cleaning, or harvesting. Although the current environment already incorporates opportunity costs because agents cannot clean and harvest simultaneously, additional action costs could provide a more realistic representation of economic and organizational decision-making. It would be interesting to investigate whether specialization and reduced exploration remain prevalent under different action-cost structures and whether these costs influence the emergence and stability of cooperative roles.

  2. Studies of social insects, particularly ants and honeybees, have shown that division of labor does not necessarily imply permanently fixed roles. Individual workers may switch tasks in response to changes in environmental conditions and colony needs (e.g., Gordon, 1996; Gordon, 2016). These observations may provide useful biological perspectives on the phenomenon investigated in this paper.

1 comment
AuthorsOpenReview ↗

Response to Reviewer MosD

We thank the reviewer for appreciating our research question, mathematical formulation, and cross-system experiments, as well as the helpful suggestions on action costs and social insect literature (Gordon, 1996, 2016), which we have added to our Discussion. In the General Response, we address: [Q4, Q6, Weakness 3, & W6] causal interventions and [Q5 & W4] (General 2); and [Q4, Q6, W3, & W6] possible mechanisms of narrowed cleaner exploration (General 3). We additionally clarify:

W1, Q1, Q2. Environment dynamics and HARVEST.

We will clarify the exact rules for each environment and distinguish what HARVEST can establish. In the deep RL Cleanup implementation, 79 of 167 river cells are polluted initially (47.31%); after step 50, eligible pollution appears with probability .5 per step, adding at most one cell. Apple regrowth in eligible empty cells occurs with probability .05×max⁡(0,1−q/.40).05 \times \max(0, 1 - q/.40), where qq is the pollution rate, and stops at q≥.40q \geq .40. HARVEST has no maintenance action: apple regrowth depends on neighboring apples, with probabilities 0, .001, .005, and .025 for 0, 1, 2, and ≥3 neighbors. Its specialization refers to differentiated production or extraction behavior, not cleaner–harvester roles. As the reviewer notes, spatial comparisons in HARVEST also depend heavily on spawn opportunities (approximately 9.9 versus 369.6 harvesting opportunities in the cited groups). We use HARVEST to support the incentive-to-differentiation finding, not the full exploration mechanism. We clarify this rationale on P. 2 and 20–21, L. 68–76 and 1057–1136.

W2, Q3. λ=.25\lambda = .25 and starting positions.

We do not claim that λ=.25\lambda = .25 is a universal efficiency optimum. Our deep RL reward combines individual reward and total team reward without normalizing their magnitudes, so λ\lambda changes both the incentive mixture and reward scale. The observed peak at λ=.25\lambda = .25 is specific to this formulation and should not be interpreted as a general optimum. Starting positions affect role formation but do not explain why λ=.25\lambda = .25 yields the highest efficiency. In spawn-matched comparisons, collective incentives still increase cleaner share by 38.66 percentage points in deep RL (95% CI [32.72, 44.54]) and 14.4 percentage points in GPT-5-mini (95% CI [5.6, 24.8]). We clarify the reward formulation and scope of this finding (Section 2.2, P. 3, L. 134–142; Discussion, P. 10, L. 524–528; Appendix H.3, P. 27, L. 1421–1432), without claiming a normalized-reward experiment was conducted.

W4, Q6. Role switching.

We examined sustained transitions between cleaning and harvesting roles in Group Cleanup trajectories. Switching was generally rare across models: deep RL (Cleaner → Harvester: 13/607 eligible windows; Harvester → Cleaner: 30/905), GPT-5-mini (2/374; 1/242), Luna (15/457; 11/327), MiniMax (19/515; 3/220), and DeepSeek (2/161; 0/67). Overall, agents tended to maintain their established roles rather than switch frequently (Appendix H.4, P. 27–28, L. 1446–1474).

Q5. Individual role contributions.

We conducted individual-level analysis to examine the relationship between each agent's spatial exploration and its own task contribution. Across 300 LLM collective-incentive trajectories (4 models × 3 contexts × 25 runs), a 0.10 increase in spatial entropy was associated with +0.315 more non-empty cleaning attempts for cleaning agents (95% CI [0.246, 0.384], p<.001p < .001) and +3.124 more apples collected for harvesting agents (95% CI [2.871, 3.378], p<.001p < .001). Similarly, in deep RL (30 seeds across 13 checkpoints), a 0.10 increase in spatial entropy was associated with a 5.58 percentage-point higher successful CLEAN rate for cleaners (95% CI [2.81, 8.36], p<.001p < .001) (Section 3.3, P. 9, L. 432–434; Appendix H.1, P. 25–26, L. 1342–1373).

W5, Q7. Deep RL versus LLM configurations.

We will explicitly report the differences and restrict primary comparisons to within-platform conditions. LLM agents receive a 7×7 local map, versus 11×11 for deep RL; larger ASCII views made LLM navigation difficult. LLM interactions use 100-round phases with a collective-incentive continuation to round 200, versus 1,000-step evaluation episodes for deep RL. Both retain the central maintenance–production trade-off and pollution-dependent apple regrowth, but implementation differences limit cross-platform attribution. We have not run an observation-size ablation, and the small context/horizon pilots do not establish configuration equivalence (Sections 3.1–3.2, P. 4–5, L. 183–196 and 252–264; Appendix C, P. 17, L. 878–886; Discussion, P. 10, L. 528–531).

Reviewer_EnDrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies cooperation in sequential social dilemmas (Cleanup, plus Harvest in an appendix) by varying a weight λ between individual and team reward. In IPPO agents and in LLM agents (GPT-5-mini, plus three more LLMs across Cleanup, collaborative hiring, and collaborative research), higher λ gives higher team output and a stronger production/maintenance division of labour. The authors then report that maintenance agents have lower spatial entropy (“explore less”) than production agents. They also report that in the LLM runs, team output declines late in a 200-round game, which they attribute to narrowing maintenance exploration. Validation is 30 IPPO teams per λ over 100M steps and 25 teams per LLM/context/condition, with bootstrap CIs and appendix probes (reward credit, frozen policy probes, an opportunity-free environment).

Critical strengths

The λ manipulation is clean, and the first link replicates widely. Collective incentives mostly raise efficiency and division of labour in IPPO and in 4 LLMs × 3 contexts (Table 1), and in a second dilemma (Harvest). This makes the paper a useful testbed whatever one makes of the later claims.

The reporting is unusually candid. It includes frozen-probe analysis (App. B), an opportunity-free probe with counterbalanced start positions (App. E), and reward-credit runs where the authors note a 5-seed sign-flip test cannot reach p<.05 (App. A). It also reports the MiniMax counterexample and a real Limitations section. These let a reader calibrate the claims, even though, as below, they weaken them.

Critical weaknesses

The causal chain (cooperation → specialization → narrower exploration → worse long-run output) is asserted more than shown, and the paper’s own controls cut against the middle links.

  • “Exploration” is spatial-occupancy entropy, which largely reflects the size of a role’s task region: cleaners work a river strip, harvesters an orchard. The authors concede this (Sec. 3.1 “we suspect…”; App. D, where low extractors are mostly agents spawned outside the orchard, with about 10 vs 370 harvesting opportunities per episode).

  • When the situation is held fixed, the effect largely vanishes. On frozen probes (App. B, Table 4) the role difference in exploration propensity is about 0 (−.021 to +.019).

  • In the reward-credit runs (App. A), λ=0, β=.1 reaches DoL .865 with an entropy gap of −.004, so strong specialization does not imply narrowed exploration. The other exploration gaps there are not significant (p ≥ .0625).

  • The title says cooperative agents “explore less”, but no cooperative-vs-selfish comparison is made, only cleaner-vs-harvester within teams. In Table 3, collective-incentive harvesters actually have higher entropy than λ=0 harvesters (.740 vs .702).

The “hidden cost” (late-stage decline) appears only in LLM agents and is not isolated from alternative explanations.

  • Both roles collapse in Phase 2 (cleaners .380→.074, harvesters .425→.114; Fig. 4d). Phase 2 exists only for the collective condition, so there is no 200-round control without specialization. This looks as much like a generic time or context effect.

  • Histories are retained. 11.29B tokens over 198,375 calls is about 57k tokens per call (calc), so long-context degradation or self-repetition is plausible. The App. C prompt also shows “Step: 1 / 100”, so end-of-horizon effects are possible in the compared windows.

  • The headline decline compares the end-of-Phase-1 peak (rounds 76–100) with the last window. Over whole phases (Table 2, 1–100 vs 101–200), only 1 of 12 model–context cells shows a decline with a CI excluding zero, and 5 show significant increases (calc).

  • No intervention that preserves exploration is tested, and MiniMax M3 as a “boundary case” is a single anecdote. Deep RL shows no decline at all (Fig. 2a), so “across all models and contexts” overstates.

Questions for authors

Q1. Can you report exploration in a way not tied to role geometry, such as entropy normalized to each role’s own task region, first-visit/novelty rate, or action-distribution entropy, with a spawn-matched comparison? Given Table 4 (about 0) and App. A (−.004), what evidence remains that specialization itself narrows exploration?

Q2. How much of role assignment is predicted by starting position, in both IPPO and LLM runs? The agents are “otherwise identical” only in policy. App. D already links high extraction to spawning inside the orchard.

Q3. What do the prompts say at rounds 101–200? Do agents know the horizon? Does the decline persist if end-of-horizon windows are excluded or the context is truncated?

2 comments
AuthorsOpenReview ↗

Response to Reviewer EnDr (1/2)

We thank the reviewer for appreciating our clean incentive manipulation, broad replication, and candid reporting. In the General Response, we address your concerns on: [Weakness 2.1, 2.2, & Q3] Phase 2 prompts, context rot, and horizon effects (General 1); [W 1, Q1, W1.4, & W 2.4] causal interventions (General 2); and [W1 & Q1] mechanisms of narrowed exploration (General 3). We additionally address your specific points:

Q1, W1.1. Role-specific exploration metrics.

Thank you for suggesting additional exploration metrics. First, we clarify that role-normalized spatial entropy (task-relevant spatial entropy, normalized to each role's own reachable task cells) is already defined in Equation 4 of the submission. In Cleanup, there are 203 reachable cleaning-stance cells for cleaners but only 122 reachable apple cells for harvesters (Appendix A.2, P. 16, L. 832–835), yet cleaners' role-normalized entropy remains lower by .226 (Section 3.1), showing that the gap cannot be explained by the size of the region alone. We also clarify that Appendix D describes HARVEST, a different game without the river-versus-orchard role structure of Cleanup. Second, following your suggestion, we computed the first-visit (novelty) rate across deep RL and all four LLMs . As shown below, all three metrics consistently show that cleaners explore less than harvesters (Appendix H.2, P. 26–27, L. 1375–1414).

System / ModelOverall entropy C / H / GapRole-normalized C / H / GapFirst-visit C / H / Gap
RL (IPPO, λ=1\lambda=1).581 / .730 / −.149.642 / .869 / −.226.095 / .162 / −.067
GPT-5-mini.286 / .429 / −.143.302 / .512 / −.210.295 / .570 / −.275
GPT-5.6-Luna.393 / .464 / −.071.329 / .529 / −.200.431 / .680 / −.249
MiniMax M3.353 / .437 / −.084.360 / .515 / −.155.406 / .638 / −.232
DeepSeek V4.369 / .457 / −.089.271 / .377 / −.106.383 / .604 / −.222

W1.2. Frozen probes and task preferences.

Although cleaners and harvesters show little difference in movement propensity on neutral probes (Δ\Delta expl. ∈ [-.021, +.019]), this observation does not contradict our findings: when presented with identical task-relevant observations in Table 4, cleaners are significantly more likely to clean and less likely to harvest than harvesters (all p<.01p < .01 for λ>0\lambda > 0), demonstrating distinct learned task preferences. During live environment interaction, these specialized preferences repeatedly direct agents toward familiar task activities and regions, resulting in narrower spatial exploration over time (Appendix B and Table 4, P. 16–17, L. 838–875).

W1.3. Reward-credit robustness.

We clarify that our primary focus is specialization emerging endogenously under collective incentives (λ>0\lambda > 0), whereas the cited λ=0\lambda=0, β=.1\beta=.1 condition relies on engineered individual cleaning rewards. The reward-credit experiments in Appendix A serve as robustness checks showing that role differentiation under collective incentives (λ∈{.25,1}\lambda \in \{.25, 1\}) is not an artifact of maintenance receiving zero direct credit, rather than our primary test of cooperation-induced specialization. We have clarified this distinction in the revised appendix (Appendix A, P. 15–16, L. 756–836).

AuthorsOpenReview ↗

Response to Reviewer EnDr (2/2)

W1.4. Cooperative versus individual incentives in Phase 2.

To compare individual and collective rewards over longer interactions, we extended individually rewarded (λ=0\lambda = 0) GPT-5-mini Cleanup teams through Phase 2 (rounds 101–200). Compared with individual incentives (λ=0\lambda = 0), collective incentives (λ=1\lambda = 1) produced stronger division of labor (ΔDoL=+.341\Delta \mathrm{DoL} = +.341, 95% CI [.184, .499], paired p<.001p < .001) and lower team-mean individual spatial entropy (Δ=−.071\Delta = -.071, 95% CI [−.106, −.035], paired p<.001p < .001) in Phase 2. This directly addresses the requested cooperative versus selfish comparison and indicates that the less specialized, individually incentivized agents explored more broadly in the extended interactions (Appendix I.4, P. 29, L. 1537–1561).

W2.3. Comparison window.

Our focus is on the late-stage performance trajectory after initial environment recovery and cooperation formation stabilize (Figure 4a–c). Because early rounds in Phase 1 reflect initial waste clearing (from 47.3% initial river pollution) and role emergence before apples can regrow, whole-phase averages (1–100 vs. 101–200) mix initial ramp-up with subsequent post-coordination dynamics. Comparing the final 25 rounds of each phase (76–100 vs. 176–200) isolates this late-stage downward trend, while full-phase comparisons are also reported in Table 2. We have clarified this rationale in the revised manuscript (P. 6, L. 315–319; Table 2, P. 8, L. 378–408).

W2.4. Scope of performance decline.

Regarding the different performance trajectories in deep RL and LLMs, the two experiments capture distinct temporal processes: LLM trajectories track within-episode adaptation over interaction histories, whereas deep RL curves track cross-checkpoint optimization as policies train to maximize cumulative team rewards (Figure 2a). Because deep RL curves reflect training progress rather than within-episode dynamics, we do not expect identical temporal patterns (Figs. 2 and 4, P. 5–6, L. 225–228 and 293–302).

We agree that “across all models and contexts” overgeneralizes our findings. We have removed it from the revised Introduction (P. 2, L. 95).

Q2, W1.1. Starting position.

Starting position influences initial role uptake in some models (e.g., in GPT-5-mini, the cleaner rate is 64.0% when spawning nearer the river vs. 33.3% nearer the orchard; seed-weighted difference +33.0 pp, 95% CI [15.0,48.7]), but does not explain specialization under collective incentives. Predicting roles from spawn position alone yields no gain over the majority baseline in deep RL (+0.0 pp), GPT-5-mini (−1.6 pp), MiniMax M3 (+0.0 pp), and DeepSeek V4 (−4.0 pp), and shows a positive gain only in GPT-5.6-Luna (+13.6 pp). After matching exact starting positions across seeds, collective incentives still substantially increase the share of cleaners in both deep RL (+38.66 pp, 95% CI [32.72,44.54]) and GPT-5-mini (+14.4 pp, 95% CI [5.6,24.8]) (Appendix H.3, P. 26–27, L. 1399–1443).

Reviewer_uBoaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper analyses the behavior of multi-agent systems putting particular emphasis on situations where we change traditionally mixed-motive scenarios into settings where cooperation is incentivized, e.g. by optimizing the common reward rather than the individual reward. The authors focus primarily on the Cleanup environment that was well established as a classical sequential social dilemma in the multi-agent reinforcement learning community. The authors consider two families of agents -- relatively small neural networks trained through reinforcement learning and SOTA LLM agents. They find that as these agents interact in the collaborative cleanup environment, they specialize to certain roles and that, in the case of LLM agents, the overall efficiency of the system drops over time. They attribute this failing to the limited exploration induced by specialization. The authors then perform further ablations using different families of LLM agents and different rephrasings of the cleanup environment.

Critical strengths
  • The paper considers the problem of cooperation in multi-agent systems. This is an important problem that has practical implications for current AI systems, and we need more research in this direction. The role of specialization is especially interesting and documenting any drawbacks related to it is of utmost importance.
  • The experiments are for the most part sound. The authors consistently provide confidence intervals, explain the technical setting clearly and in my opinion run experiments that well-designed as a starting point of investigation of the phenomena they want to investigate -- although as I will signal in the weaknesses section, I believe the experiment scope should be expanded.
Critical weaknesses

My main issue with the paper is that the main claim -- "Cooperative agents specialize more but explore less" is underinvestigated. I will break this point into smaller bulletpoints:

  • The authors suggest that the specialization is what causes the drop of exploration (and then, subsequently, the efficiency), but I would like to see more evidence and analysis for this claim. There is of course an analysis of temporal correlation between these phenomena, but I would appreciate a further study into the causal graph of this problem. For example, looking at the reasoning traces of the LLM agents might be of use -- why did they stop exploring later in the episode? Or looking at the problem from a different perspective -- could some of the issues be caused by the problem of more difficult credit assignment in collaborative settings [Sunehag 2018]? Additionally, it seems like in deep RL we don't see drops in efficiency after reduced exploration -- how do we reconcile these observations with the LLM results?
  • The authors only examine this behavior in the context of clean-up-like environments. In these cases it seems intuitive that specialization should emerge as it seems one of the most efficient solutions -- having agents run between the orchard and the river is much slower than having separate agents in these positions. The results might be vastly different in settings where there is weaker pressure towards specialization, and as such I think the authors should consider a wider array of environments. The HARVEST results in the appendix are a good first step towards it, but I would be more interested in further investigation of the LLM results which imply that more specialisation -> less exploration -> worse results.
  • The MiniMax results seem to suggest that decreasing the exploration does not automatically lead to lower efficiency. Why is it different in this case and what might be the underlying mechanism?
  • I think it would be useful to rule out certain simple explanations of these phenomena -- for example the performance of LLM agents after observing over 100 rounds of gameplay might fall due to "context rot", so a baseline that summarizes the results so far would be helpful. Additionally, being effective at the cleanup role inherently requires an agent to stay near the river and repeatedly clean rather than roam -- so lower measured exploration may simply be a necessary byproduct of effective role execution rather than the cause of the failure mode.

Finally, the authors miss some of the critical literature on cooperation problems. Prior work established that even in purely collaborative settings (i.e., all the agents share the same reward) the problem of effective collaboration remains and is highly non-trivial. The introduction states "These findings suggest that achieving cooperation in multi-agent systems is not the end goal." but I would argue this claim is not novel and has been seen in previous work, such as [Mahajan 2019, Li 2021, Carroll 2019].

Peter Sunehag et al. "Value-Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward." AAMAS 2018. Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. "MAVEN: Multi-Agent Variational Exploration." NeurIPS 2019 Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. "Celebrating Diversity in Shared Multi-Agent Reinforcement Learning." NeurIPS 2021 Micah Carroll, Rohin Shah, Mark K. Ho, Thomas L. Griffiths, Sanjit A. Seshia, Pieter Abbeel, and Anca Dragan. "On the Utility of Learning about Humans for Human-AI Coordination." NeurIPS 2019

Questions for authors

Please take a look at the issues mentioned in the Weaknesses section.

1 comment
AuthorsOpenReview ↗

Response to Reviewer uBoa

We thank the reviewer for recognizing the importance of our research question and the soundness of our experimental design. In the General Response, we address your concerns regarding: [Weakness 4] context rot and horizon effects (General 1); [W1 & W4] causal links between specialization, exploration, and performance (General 2); and [W1 & W3] LLM reasoning traces and mechanisms of narrowed cleaner exploration (General 3). We additionally clarify:

W1. Credit assignment and RL–LLM performance trajectories.

To test whether difficult credit assignment explains the performance decline, we examined LLM reasoning summaries and found that cleaners explicitly recognize that removing pollution sustains apple regrowth for the team. Thus, at least within our current experiments, reduced exploration seems to stem from adherence to familiar task routines rather than a failure to value maintenance (but see General 3).

RL and LLM trajectories capture different temporal scales: LLMs track within-episode adaptation over interaction histories (showing how initially successful specialization can degrade over time), whereas RL curves track cross-checkpoint optimization toward cumulative team rewards. While RL training curves do not mirror LLM within-episode decay, our exploration intervention confirms that converged RL policies still leave team output unrealized, with broader exploration raising output by 33.17 to 52.52 apples (General 2) (Figs. 2 and 4, P. 5–6, L. 225–228 and 293–302; Section 3.4, P. 9, L. 463–472).

W2. Generalizability beyond Cleanup.

We agree that extending our full causal chain to HARVEST (Appendix D) and environments with weaker specialization pressure is an important direction for future work. Our generalizability comes from varying semantic framing in Cleanup, collaborative hiring and research collaboration which activates distinct pretrained domain priors in a range of LLMs (Section 3.3). We now explicitly note this limitation in the revised Discussion (P. 10, L. 534–537).

W3. MiniMax M3 boundary case.

We believe that MiniMax M3's ability to maintain high performance despite specialization supports, rather than contradicts, our proposed mechanism. Unlike other models, specialized MiniMax M3 agents sustain active exploration alongside stable long-term output (Table 2), a pattern consistent with MiniMax's exploration-oriented post-training optimization (Liu et al., 2025; MiniMax, 2026). MiniMax M3 thus serves as a boundary case showing that specialization does not undermine performance when exploration is preserved, which further highlights the critical role of continued exploration in long-term collective outcomes (Section 3.3, P. 8–9, L. 429–432; Section 3.4, P. 9, L. 470–472; Discussion, P. 10, L. 517–523).

Literature.

Thank you for suggesting these relevant works. We have included these works in the revised paper on P. 9–10, L. 476–489. Briefly,

Sunehag et al. (2018) highlight a credit assignment problem in cooperative multi-agent settings: when agents share a team reward, it can be difficult to determine how much each agent’s actions contributed to the outcome. Credit assignment can be an interesting mechanism for our observation (General 3).

Mahajan et al. (2019) study coordinated exploration and Li et al. (2021) examine diversity in shared-policy multi-agent reinforcement learning. Both work provides possible solutions to increase exploration, which can be integrated into our work as interventions. However, our work differs from them by examining how emergent specialization relates to later information seeking and long-term performance, rather than proposing a new exploration algorithm.

Carroll et al. (2019) study human-AI coordination and the value of learning about human partners for cooperative play. In contrast, our focus is on endogenous role formation and adaptation over time among agents, not learning to coordinate with human partners.

References:

Liu et al., 2025. WebExplorer: Explore and Evolve for Training Long-Horizon Web Agents.

MiniMax, 2026. MiniMax M3. Official technical report. https://www.minimax.io/blog/minimax-m3

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response (3/3)

3. Why do cleaners explore less?

Inspecting LLM reasoning traces qualitatively, we observed that specialized cleaners repeatedly default to familiar maintenance routines even after noticing opportunities elsewhere (e.g., “it's probably better to clean now”), whereas harvesters more often reason about navigating the orchard to locate depleted or regrown apples. In terms of underlying mechanisms on why cleaners in particular explore less, we agree it is an intriguing extension of our work. Other than differences in action space and training environment, one intriguing hypothesis is credit assignment as pointed out by the reviewers. As maintenance benefits are indirect and often delayed, it may decrease cleaners’ incentives to explore broadly. Fully isolating these mechanisms is beyond the scope, but we agree it is a question that deserves much more careful investigation in the future, thus adding it to the revised conclusion on P. 10, L. 530–534.

Overall, we believe the reviewers’ input and our new data have further strengthened our argument. Below, we respond to individual comments that are not yet covered here.

Sincerely,

Authors

AuthorsOpenReview ↗

General response (2/3)

2a. Reducing specialization broadens cleaner exploration.

In deep RL, to test whether discouraging rigid specialization during training leads to broader exploration, we followed prior work on intrinsic-reward shaping (Hughes et al., 2018) to add an auxiliary minority-task bonus when agents completed the task it had performed less often across 30 training seeds. As compared to the baseline without such an intervention, both bonus levels (λaux=0.05\lambda_{\mathrm{aux}} = 0.05 and 0.100.10) substantially reduced specialization (ΔDoL=−0.73\Delta \mathrm{DoL} = -0.73 and −0.76-0.76), which leads to increased exploration across all metrics: overall spatial entropy (+0.164 and +0.176), cleaner-region entropy (+0.137 and +0.141), and first-visit rate (+0.152 and +0.159) (Section 3.4, P. 9, L. 453–461; Appendix F.1–F.2, P. 23–24, L. 1190–1250).

Minority-task bonusΔ DoL [95% CI]Δ Overall Entropy [95% CI]Δ Cleaner-Region Entropy [95% CI]Δ First-Visit Rate [95% CI]
λaux=.05\lambda_{\mathrm{aux}}=.05−0.729 [−0.766, −0.693]+0.164 [+0.142, +0.186]+0.137 [+0.115, +0.159]+0.152 [+0.135, +0.169]
λaux=.10\lambda_{\mathrm{aux}}=.10−0.764 [−0.795, −0.733]+0.176 [+0.155, +0.196]+0.141 [+0.119, +0.162]+0.159 [+0.141, +0.177]

On the other hand in LLMs, we tested multiple interventions to reduce specialization in GPT-5-mini agents, including explicit role-switching prompts, scheduled role rotation every 25 rounds, auxiliary incentives rewarding minority-task completion, balanced contributions, cross-role actions, or team flexibility. However, these interventions largely failed to reduce specialization: agents either resisted switching or quickly reverted to their initial roles. Consistent with this persistence, reasoning traces show that agents recognize alternative opportunities yet default to familiar routines (see examples in #3) (Section 3.4, P. 9, L. 458–461).

In short, reducing specialization broadens cleaner exploration among deep RL cleaners. Although the pattern for LLM agents is less conclusive because they resist prompt-level and incentive-level manipulations and stick to their in-context roles (Section 3.4, P. 9, L. 453–461; Appendix F.1–F.2, P. 23–24, L. 1190–1250).

2b. Increasing exploration improves group performance.

To test whether increasing spatial exploration improves collective performance, we applied a coverage-aware movement controller during the evaluation of trained deep RL agents to track how many times each cleaner had visited each grid cell, keeping learned policies frozen, rewards unchanged, and harvesters untouched. When a cleaner’s beam contained no pollution, the controller intervened with probability pint∈{0.05,0.10,0.20,0.30}p_{\mathrm{int}} \in \{0.05, 0.10, 0.20, 0.30\}, moving the cleaner to the least-visited reachable adjacent cell. Otherwise, the cleaner followed its original policy. Across 30 matched training seeds, cleaner spatial entropy rose monotonically with pintp_{\mathrm{int}}, which significantly increased team output output by+19.32 apples at 5%, +33.17 apples at 10% , +43.68 at 20% , and +52.52 at 30% , as compared to the baseline without this intervention. This result shows that guiding specialized cleaners toward less-visited cells during idle steps improves collective outcomes without retraining policies or altering rewards. This analysis is also consistent with our boundary-case observation in MiniMax M3, where preserved exploration naturally sustains high team performance alongside specialized roles (Section 3.4, P. 9, L. 463–472; Appendix G.1, P. 24–25, L. 1255–1304).

Coverage InterventionCleaner Spatial Entropy (Δ)Team Apples (Efficiency)Δ Team Apples [95% CI]p-value
Baseline (0%).585872.68——
5%.602 (+.0162)892.00+19.32 [−0.23,37.83].0543
10%.623 (+.0372)905.85+33.17 [12.95,52.77].0031
20%.633 (+.0475)916.37+43.68 [25.18,61.22]<.001
30%.644 (+.0589)925.20+52.52 [10.48,84.69].0068

We likewise applied an exploration prompt intervention in GPT-5-mini, instructing agents to seek information from areas not checked recently while keeping rewards unchanged. Across all 25 matched histories, the intervention significantly increased Phase 2 team apple output by +12.12 apples (+34.4%, 95% CI [5.56, 19.16], p=.003p = .003) (Section 3.4, P. 9, L. 467–472; Appendix G.2, P. 24–25, L. 1291–1339).

In short, intervening on agents’ actions by having them explore less-visited areas improved group output in both RL and LLM agents, suggesting exploration is one likely causal factor for performance (Section 3.4, P. 9, L. 463–472; Appendix G.1–G.2, P. 24–25, L. 1253–1339).

AuthorsOpenReview ↗

General response (1/3)

We thank all reviewers for their constructive feedback. We are glad that reviewers found our research question important (uBoa, MosD), experiments sound and clean (uBoa, EnDr, MosD), presentation clear and candid (uBoa, EnDr, MosD), with interdisciplinary impact (MosD). At the same time, reviewers asked us to (1) test whether the phenomena can be explained by simple alternatives such as context rot and the horizon effect, which our new experiments suggest are unlikely explanations; (2) establish causal evidence that specialized agents explore less (a), and reduced exploration decreases team performance (b), both of which our new experiments, mostly, support; and (3) dig deeper into why cleaners explore particularly little, a question we found interesting and included in Discussion with preliminary evidence, though a more thorough investigation is beyond the scope of this paper. Unless otherwise noted, our new experiments use deep RL and GPT-5-mini agents in Cleanup under collective incentives (λ=1\lambda = 1). We incorporated the results into the revised main text and appendix. In this meta response, we provide key experimental results, please see the updated paper for details (Section 3.4, P. 9, L. 445–472; Appendices F–I, P. 23–29).

1. Context rot and horizon effect are unlikely explanations for late-stage performance decline.

To test whether the late-stage performance decline in Phase 2 (rounds 101–200) is an artifact of long-context degradation ("context rot") or end-of-horizon effects, we conducted targeted ablations on context management and announced horizon length (5 matched histories per condition):

Context window ablations: Neither retaining only the latest 10 rounds of observations and decisions (Δ=+2.40\Delta = +2.40 apples, p=.813p = .813) nor replacing full histories with structured summary (Δ=−3.40\Delta = -3.40 apples, p=.750p = .750) significantly changed Phase 2 team apple production relative to the original full-context continuation (Section 3.2, P. 7, L. 371–374; Appendices I.1–I.2, P. 28–29, L. 1478–1521).

Time horizon ablations: First to clarify, in our original setup, both Phase 1 and Phase 2 used the exact same announced 100-round step counter (Step: 1/100 to 100/100, Appendix C.1), ensuring that any end-of-horizon effects applied equally to both phases. Second, explicitly informing agents at round 101 that the episode would continue through round 300 yielded no significant difference in team apple production over rounds 101–200 (Δ=−1.80\Delta = -1.80 apples, p=.875p = .875) (Section 3.2, P. 7, L. 374–377; Appendix I.3, P. 29, L. 1523–1535).

Combined with our exploration intervention (see below, which raised Phase 2 team output by 34.4%, p=.003p = .003) under the exact same context length and horizon), these results indicate that the late-stage decline is unlikely driven by context rot or horizon boundaries (Section 3.2, P. 7, L. 366–377; Appendices I.1–I.3, P. 28–29, L. 1478–1535).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.