- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the relationship between cooperation, specialization, exploration, and long-term collective performance in multi-agent systems. The authors study sequential social dilemmas in which agents must balance immediate individual rewards against maintaining shared resources necessary for future production.
Using both deep reinforcement learning and large language model agents, the authors demonstrate that stronger collective incentives generally encourage cooperation and the emergence of specialized roles. However, maintenance-oriented agents tend to explore less than production-oriented agents. The authors further observe that reduced exploration can precede resource depletion and deterioration in collective performance.
The experiments cover CLEANUP, HARVEST, collaborative hiring, and research collaboration. The results suggest that successful cooperation and specialization do not necessarily guarantee sustainable long-term performance.
The reviewer finds the problem clearly defined, the assessment metrics mathematically well formulated, and the findings potentially meaningful beyond AI agent systems, including sociology, organizational behavior, and management science.
Critical strengths
-
Clear and meaningful research question. The paper addresses an interesting problem: cooperation can initially improve collective performance through specialization, but the resulting specialization may reduce exploration and eventually undermine long-term efficiency. This is an important perspective that goes beyond the conventional objective of simply improving cooperation among agents.
-
Clear formulation and presentation. The problem setting is easy to understand, and the evaluation criteria for efficiency, specialization, and exploration are mathematically well formulated. In particular, the use of spatial entropy provides a quantitative measure of differences in exploration behavior.
-
Potential interdisciplinary relevance. The findings may be relevant not only to AI agent systems but also to fields such as sociology, organizational behavior, and management science, where specialization, division of labor, and organizational rigidity are important subjects.
-
Experiments across different agent systems and contexts. The authors investigate both reinforcement-learning and LLM-based agents and extend the experiments beyond CLEANUP to other settings. This provides useful evidence that the observed behavioral patterns are not restricted to a single agent architecture or application context.
Critical weaknesses
The reviewer finds the paper interesting and generally well presented. However, several aspects of the experimental settings and the interpretation of the results require further clarification.
- Insufficient explanation of the environment dynamics
The dynamics of the CLEANUP environment are not entirely clear. Although the authors explain that apple regrowth depends on pollution and stops when the pollution rate exceeds a threshold, the temporal mechanisms governing the appearance of apples and pollution are not sufficiently described in the main text.
These mechanisms may substantially affect the incentives for cleaning, the emergence of specialized roles, and the subsequent decline in collective performance.
The HARVEST experiment provides an additional setting with different resource dynamics. However, the differences between the environments, particularly the implications of changing from explicit maintenance to restrained resource extraction, could be explained more clearly.
- Limited explanation of the relationship between reward incentives and efficiency
In the deep-RL experiments, the highest efficiency is observed at λ = 0.25. However, the reason for this result is not sufficiently explained.
The authors acknowledge that their reward formulation changes both the relative incentives and the reward scale. Consequently, it is unclear whether the observed optimum reflects the underlying cooperation mechanism or the specific reward formulation and experimental conditions.
Furthermore, the relationship between the reward parameter and efficiency differs between the deep-RL and LLM experiments, which deserves further discussion.
- Incomplete explanation of the mechanism underlying reduced exploration
One of the central findings is that cleaners explore less than harvesters, with the difference increasing over time.
The authors suggest that cleaners encounter pollution in relatively stable locations, whereas harvesters must search for apples across a broader area. They also provide an additional experiment in an environment without task-specific opportunities, which partially addresses this concern.
Nevertheless, the mechanism responsible for the progressive reduction in cleaners' exploration remains insufficiently established. It is unclear how much of this behavior results from environmental structure, reward incentives, or learned behavioral specialization.
- Limited analysis of individual roles and their transitions
The paper identifies cleaners, harvesters, and mixed-role agents based on their observed behavior. However, the analysis does not sufficiently explain how these roles emerge and evolve over time.
In particular, it is unclear whether agents maintain their specialized roles throughout an episode or switch roles in response to environmental changes.
Additional analysis of role-specific contributions and role transitions could provide useful insights into the mechanisms of cooperation and subsequent performance deterioration.
- Differences between deep-RL and LLM experimental settings
The experimental configurations differ substantially between deep-RL and LLM-based agents, including observation size, episode duration, incentive conditions, and environmental implementations.
Although different learning mechanisms naturally require different procedures, the paper does not sufficiently explain which environmental differences are necessary and how they may affect the observed behaviors.
This makes it difficult to determine whether differences in the experimental results arise from the agent architectures or from the experimental configurations.
- Limited causal evidence for the proposed explanation of performance deterioration
The authors show that reduced exploration precedes declining collective performance and provide temporal regression analyses supporting this relationship.
However, these results primarily establish temporal associations rather than demonstrating that reduced exploration is the direct cause of the subsequent decline.
An intervention that encourages maintenance agents to explore more broadly could provide stronger evidence for the proposed mechanism.
Questions for authors
- Temporal dynamics of the CLEANUP environment
Could the authors clarify how frequently pollution and apples appear in CLEANUP? In particular, how are their appearance probabilities determined, and how does the pollution level quantitatively affect apple regrowth below the threshold?
Since the paper focuses on the temporal evolution of cooperation and resource depletion, these dynamics seem important for interpreting the results.
- Differences between CLEANUP and HARVEST
Appendix D explains that HARVEST replaces explicit maintenance with a resource-extraction dilemma. Could the authors further clarify the differences in environmental structure and resource dynamics?
In particular, how does the absence of a separate maintenance action affect the interpretation of specialization and exploration compared with CLEANUP?
- Why does \lambda = 0.25 achieve the highest efficiency in deep RL?
Equation (1) gives each agent a combination of its individual reward and the team's total reward. Thus, even when λ is relatively small, agents have an incentive to improve collective performance.
Could the authors explain why \ lambda = 0.25 produces the highest efficiency in the deep-RL experiments?
Does this result persist when reward magnitudes are normalized? Could the initial spatial positions of the agents influence which agents become cleaners and consequently affect the optimal reward setting?
- Why do cleaners progressively explore less?
The authors suggest that cleaners repeatedly encounter pollution in relatively stable locations, while harvesters must search across a broader region.
Could the authors provide further analysis of the mechanism responsible for the progressive reduction in cleaners' exploration?
For example, is this primarily a consequence of the spatial distribution of pollution, the reward structure, or the learned specialization of the agents?
- Role-specific performance and contribution
Could the authors report additional statistics for cleaners, harvesters, and mixed-role agents, such as their average collected rewards, effective cleaning actions, and contributions to maintaining the shared resource?
Since cleaners do not directly receive rewards for cleaning, distinguishing individual rewards from contributions to collective performance would be particularly informative.
- Conditions for switching roles
Under what conditions do agents switch between cleaning and harvesting?
For example, when pollution approaches or exceeds the threshold that prevents apple regrowth, do harvesting agents begin cleaning more frequently?
An analysis of role transitions over time could help explain why initially successful cooperation sometimes deteriorates.
- Differences between RL and LLM experimental configurations
Could the authors clarify why the deep-RL and LLM experiments use different environmental configurations, particularly the observation sizes, interaction horizons, and incentive settings?
Which environmental properties are preserved across the two implementations, and to what extent might these differences affect the observed specialization and exploration patterns?
Suggestion (not a condition for acceptance)
-
An interesting extension would be to introduce explicit costs associated with different actions, such as movement, cleaning, or harvesting. Although the current environment already incorporates opportunity costs because agents cannot clean and harvest simultaneously, additional action costs could provide a more realistic representation of economic and organizational decision-making. It would be interesting to investigate whether specialization and reduced exploration remain prevalent under different action-cost structures and whether these costs influence the emergence and stability of cooperative roles.
-
Studies of social insects, particularly ants and honeybees, have shown that division of labor does not necessarily imply permanently fixed roles. Individual workers may switch tasks in response to changes in environmental conditions and colony needs (e.g., Gordon, 1996; Gordon, 2016). These observations may provide useful biological perspectives on the phenomenon investigated in this paper.
Response to Reviewer MosD
We thank the reviewer for appreciating our research question, mathematical formulation, and cross-system experiments, as well as the helpful suggestions on action costs and social insect literature (Gordon, 1996, 2016), which we have added to our Discussion. In the General Response, we address: [Q4, Q6, Weakness 3, & W6] causal interventions and [Q5 & W4] (General 2); and [Q4, Q6, W3, & W6] possible mechanisms of narrowed cleaner exploration (General 3). We additionally clarify:
W1, Q1, Q2. Environment dynamics and HARVEST.
We will clarify the exact rules for each environment and distinguish what HARVEST can establish. In the deep RL Cleanup implementation, 79 of 167 river cells are polluted initially (47.31%); after step 50, eligible pollution appears with probability .5 per step, adding at most one cell. Apple regrowth in eligible empty cells occurs with probability .05×max(0,1−q/.40), where q is the pollution rate, and stops at q≥.40. HARVEST has no maintenance action: apple regrowth depends on neighboring apples, with probabilities 0, .001, .005, and .025 for 0, 1, 2, and ≥3 neighbors. Its specialization refers to differentiated production or extraction behavior, not cleaner–harvester roles. As the reviewer notes, spatial comparisons in HARVEST also depend heavily on spawn opportunities (approximately 9.9 versus 369.6 harvesting opportunities in the cited groups). We use HARVEST to support the incentive-to-differentiation finding, not the full exploration mechanism. We clarify this rationale on P. 2 and 20–21, L. 68–76 and 1057–1136.
W2, Q3. λ=.25 and starting positions.
We do not claim that λ=.25 is a universal efficiency optimum. Our deep RL reward combines individual reward and total team reward without normalizing their magnitudes, so λ changes both the incentive mixture and reward scale. The observed peak at λ=.25 is specific to this formulation and should not be interpreted as a general optimum. Starting positions affect role formation but do not explain why λ=.25 yields the highest efficiency. In spawn-matched comparisons, collective incentives still increase cleaner share by 38.66 percentage points in deep RL (95% CI [32.72, 44.54]) and 14.4 percentage points in GPT-5-mini (95% CI [5.6, 24.8]). We clarify the reward formulation and scope of this finding (Section 2.2, P. 3, L. 134–142; Discussion, P. 10, L. 524–528; Appendix H.3, P. 27, L. 1421–1432), without claiming a normalized-reward experiment was conducted.
W4, Q6. Role switching.
We examined sustained transitions between cleaning and harvesting roles in Group Cleanup trajectories. Switching was generally rare across models: deep RL (Cleaner → Harvester: 13/607 eligible windows; Harvester → Cleaner: 30/905), GPT-5-mini (2/374; 1/242), Luna (15/457; 11/327), MiniMax (19/515; 3/220), and DeepSeek (2/161; 0/67). Overall, agents tended to maintain their established roles rather than switch frequently (Appendix H.4, P. 27–28, L. 1446–1474).
Q5. Individual role contributions.
We conducted individual-level analysis to examine the relationship between each agent's spatial exploration and its own task contribution. Across 300 LLM collective-incentive trajectories (4 models × 3 contexts × 25 runs), a 0.10 increase in spatial entropy was associated with +0.315 more non-empty cleaning attempts for cleaning agents (95% CI [0.246, 0.384], p<.001) and +3.124 more apples collected for harvesting agents (95% CI [2.871, 3.378], p<.001). Similarly, in deep RL (30 seeds across 13 checkpoints), a 0.10 increase in spatial entropy was associated with a 5.58 percentage-point higher successful CLEAN rate for cleaners (95% CI [2.81, 8.36], p<.001) (Section 3.3, P. 9, L. 432–434; Appendix H.1, P. 25–26, L. 1342–1373).
W5, Q7. Deep RL versus LLM configurations.
We will explicitly report the differences and restrict primary comparisons to within-platform conditions. LLM agents receive a 7×7 local map, versus 11×11 for deep RL; larger ASCII views made LLM navigation difficult. LLM interactions use 100-round phases with a collective-incentive continuation to round 200, versus 1,000-step evaluation episodes for deep RL. Both retain the central maintenance–production trade-off and pollution-dependent apple regrowth, but implementation differences limit cross-platform attribution. We have not run an observation-size ablation, and the small context/horizon pilots do not establish configuration equivalence (Sections 3.1–3.2, P. 4–5, L. 183–196 and 252–264; Appendix C, P. 17, L. 878–886; Discussion, P. 10, L. 528–531).