acceptodds
Under review as a conference paper at ICLR 2027

Beyond Single Roles: Structured Role Discovery for Cooperative Multi-Agent Reinforcement Learning

Abstract

Role-based multi-agent reinforcement learning (MARL) learns roles to organize agents into specialized divisions of labor. Effective coordination requires balancing knowledge sharing with behavioral diversity. However, a central challenge is to move beyond current role assignments toward persistent role structures that capture inter-role associations and support reuse across episodes. To address this challenge, we propose SPAR, which constructs a role prototype space using semantic slots and represents each agent through a dual semantic assignment consisting of a primary role and an auxiliary role. Building on this representation, SPAR employs relation-preserving contrastive learning to optimize the geometric structure of the role space and introduces a performance-guided slot memory to reuse high-quality collaboration patterns. Experiments on SMAC and SMACv2 demonstrate that SPAR can learn dynamic yet structured roles and achieves competitive performance across multiple test scenarios.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, role discovery, semantic slots, relation-preserving contrastive learning, slot memory, cooperative MARL

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tiPyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SPAR, a framework for learning structured and reusable roles in cooperative MARL. Built on ACORM's role-conditioned QMIX framework, it uses recurrent semantic slots to assign each agent a primary and an auxiliary role. These assignments guide contrastive learning, preserving role distinctions while reducing repulsion between related roles. A performance-guided slot memory retrieves historical role structures based on contextual similarity and episode performance for reuse during training. Experiments on SMAC and SMACv2 report competitive performance, with ablations and qualitative analyses examining the proposed components and role dynamics.

Critical strengths

The paper is clearly written and well organized. The motivation for capturing relationships between roles and reusing role structures across episodes is clear. Throughout the manuscript, many aspects are not clearly or rigorously presented, which makes the paper confusing to read. Some details of the experimental setup are also insufficiently described. Overall, the manuscript requires another round of review.

Critical weaknesses

The method largely relies on established techniques, and the paper does not sufficiently distinguish its contribution from closely related work.

Questions for authors

The paper focuses on role learning, with primary-auxiliary role assignments and episodic slot memory as its main additions. At present, these seem mainly to combine established ideas, and the methodological advance is not sufficiently clear to me. I would appreciate clarification on the following points:

  1. What does the acronym “SPAR” stand for?
  2. The introduction identifies the challenge of connecting agents' distinct behaviors with their contributions to team coordination. What concrete problems arise when this connection is missing, and what evidence supports this motivation?
  3. The related-work discussion should consider more recent methods, such as GRDC (Gao et al., 2026). How does SPAR differ from its approach to prototype-based role discovery?
  4. What motivates the recurrent slot refinement in Eq. (7)? Figure 1 indicates that a GRU is used: are its parameters trained, and through which loss do they receive gradients, given the hard role assignments and detached weighting gates? What is the benefit of dynamically updating the slots, and could a comparison with fixed slots be provided?
  5. Zhang et al. (2025), in Role can be beneficial to mean field multi-agent reinforcement learning, already use contrastive learning to obtain role representations. Although their learning framework differs, what substantive advances does SPAR introduce over such approaches beyond primary-auxiliary role assignments?
  6. What explains SPAR’s lower mean win rates than the strongest baselines on all three SMACv2 10_vs_11 tasks?
  7. Please provide the SMACv2 learning curves and clarify the training budget for each scenario. Were these runs trained for two million environment steps, or a different budget, and were the budgets matched across methods?
  8. Beyond the single-trajectory visualization, what evidence shows that the primary and auxiliary assignments correspond to meaningful behaviors or coordination functions? In particular, how can we tell whether the auxiliary assignment represents a meaningful secondary role rather than simply the second-highest slot score?
4 comments
AuthorsOpenReview ↗

Response to Reviewer tiPy(1/2)

Response to Question 1

SPAR stands for Structured Primary Auxiliary Roles. The full name is given in the Abstract, line 017, and the Introduction, lines 069–071 of the revised manuscript, together with an explanation that primary and auxiliary roles represent the strongest and second-strongest slot associations, respectively.

Response to Question 2

The specific issue we address is that agents with different behaviors may still have coordination-relevant relationships that a single role label cannot adequately express. For example, an agent drawing enemy fire and another concentrating damage may have different primary roles while acting complementarily. If contrastive pairs are constructed solely according to whether their primary roles match, such cross-role pairs are also uniformly repelled as negatives. This example illustrates the design motivation. It is not a behavioral annotation of the learned roles.

We have specified the limitations of single-role representations and cross-episode structure retention in the Introduction, lines 061–074, and explained potentially related cross-role negatives and their treatment in Section 3.2, lines 246–265.

In Section 4.3 and Figure 4 of the main text, SPAR achieves a test win rate of 77.50%, compared with 60.38% for Uniform, which matches the mean weakening strength, and 55.63% for Shuffled, which preserves the weight multiset but shuffles its assignment. These results support the usefulness of role-conditioned weight allocation, but do not directly measure the causal contributions of individual behaviors to team return. We therefore interpret them as mechanistic evidence for cooperative representation learning, rather than as an attribution of individual contributions.

Response to Question 3

Thank you for this suggestion. We have added a discussion of GRDC (Gao et al., 2026) and its differences from SPAR in Section 2.1, lines 107–114.

GRDC constructs local interaction graphs from trajectory representations and uses prototype-based role discovery to organize intra-role communication. SPAR instead uses primary–auxiliary slot associations to adjust the contrastive weights of negative pairs with different primary roles, and reuses slot states during training according to context and historical episode performance. Both methods use prototype-based role representations, but use role information differently, focusing on role-driven communication and relation-aware representation learning with cross-episode structure reuse, respectively.

Response to Question 4

Recurrent refinement adjusts role prototypes using the current agent representations and historical slot states to accommodate changing behavioral features within an episode. In the current configuration, the GRU and other slot-module parameters remain at their initialized values. The slot states are updated. The contrastive loss learns representations through the trajectory encoder, query role encoder, and similarity matrix. The hard top-kk labels and detached weights do not transmit gradients to the slot module, and the TD loss does not backpropagate through it either. The non-differentiability of hard role labels does not mean that their values are fixed over time. They still change with the inputs and states.

We have clarified these distinctions in Section 3.1, lines 223–230, and Section 3.2, lines 276–279. Section 3.1 also cites the reservoir computing review by Lukoševičius and Jaeger (2009) to explain that fixed recurrent network weights can coexist with input-driven dynamic hidden states. This citation clarifies the distinction between state updates and parameter learning.

To examine the practical effect of dynamic updates, we have added a Fixed Slots control on 3s5z_vs_3s6z in Appendix D, Figure 6, showing both test win rate and evaluation return. Fixed Slots disables GRU-based state refinement while retaining slot-based role assignments, auxiliary-aware weighting, and SlotBank reuse. Role assignments remain input-dependent in both methods. The mean curves over five seeds show that the full SPAR method improves its win rate earlier and achieves higher late-training mean win rates and returns.

AuthorsOpenReview ↗

Response to Reviewer tiPy (2/2)

Response to Question 5

We agree that contrastive learning of role representations has been studied previously, and SPAR does not claim to introduce its first use in role discovery. We have added RMFQ (Zhang et al., 2025) and clarified its differences from SPAR in Section 2.1, lines 107–114.

RMFQ combines adaptive role assignment with a role-weighted mean-field approximation. SPAR's substantive extensions include selectively adjusting negative repulsion through primary–auxiliary associations and using performance-guided SlotBank to select and reuse historical slot states during training. Beyond storing historical representations, the latter filters candidates according to the current context and then selects content for reuse based on historical episode performance. The mechanisms are detailed in Sections 3.2 and 3.3.

We have added controls targeting these mechanisms in Section 4.3 and Figure 4 of the main text. Uniform, Shuffled, and the two memory-content controls achieve win rates of 60.38%, 55.63%, 66.88%, and 61.25%, respectively, all below the full SPAR method's 77.50%. These results support the contributions of weight allocation, performance-guided content selection, and contextual correspondence on this scenario, rather than merely supporting the addition of an auxiliary label or memory bank.

Response to Question 6

Thank you for highlighting this limitation. We believe one possible reason is that SMACv2 randomizes unit types, team compositions, and initial positions, requiring role inference to adapt to changing coordination configurations and making it harder to match historical slot structures to the current context. In the 10_vs_11 tasks, which involve larger teams and an enemy numerical advantage, these demands may further limit the effectiveness of role modeling and memory reuse.

ACORM also falls below the strongest baseline in mean win rate on all three tasks. SPAR improves over ACORM on each task, indicating that the proposed mechanisms remain useful under randomized configurations, but these gains are insufficient to close the gap to the strongest methods.

We have explicitly acknowledged this performance boundary in Section 4.1 of the main text, lines 399–402, and Appendix E.1, lines 1093–1098, and provided the corresponding learning curves in Appendix Figure 7.

Response to Question 7

We have added learning curves for seven methods across six SMACv2 scenarios in Appendix E.1, Figure 7, with a pointer in Section 4.1 of the main text, lines 401–402. Different scenarios use different cutoffs for curve display and result aggregation, while all methods within the same scenario use the same cutoff.

ScenarioLogged-step cutoff for result aggregation
Protoss 5_vs_52.0M
Terran 5_vs_52.0M
Zerg 5_vs_52.0M
Protoss 10_vs_113.0M
Terran 10_vs_112.5M
Zerg 10_vs_113.0M

To facilitate comparisons at common budgets, we have also added results for SPAR, ACORM, and QMIX at 0.5M, 1.0M, 1.5M, and 2.0M logged steps in Appendix E.2, Table 4. For each seed, we use the last recorded evaluation at or before the specified budget, then report the mean and sample standard deviation across five seeds. At 2.0M, SPAR achieves higher mean win rates than ACORM on five of the six tasks and higher means than QMIX on all six. ACORM remains stronger on Protoss 5v5, and the rankings also vary at earlier budgets.

Response to Question 8

The auxiliary role is indeed computed from the second-highest slot-matching score, and its additional function is to inform contrastive learning. When one agent's auxiliary role matches another agent's primary role, SPAR selectively weakens the negative repulsion between them. The relevant question is therefore whether this secondary association provides useful relational information.

To examine this, we have added Uniform and Shuffled controls in Section 4.3 and Figure 4 of the main text. Full SPAR achieves a win rate of 77.50%, compared with 60.38% and 55.63% for the two controls, respectively. In particular, Shuffled preserves the negative-weight multiset and only shuffles its assignment to sample pairs. Its lower mean win rate supports the practical learning value of allocating weights according to role associations.

However, this evidence does not establish that auxiliary slots correspond to distinct, nameable behavioral responsibilities, such as covering or supporting other agents. Section 4.2 of the main text, lines 465–474, reports 16 auxiliary-role switches and 12 primary-role switches in the displayed trajectory and explicitly identifies the visualization as illustrative. We therefore interpret auxiliary roles as secondary slot associations useful for cooperative representation learning. Whether they consistently correspond to distinct behavioral functions still requires validation through intervention analysis.

Thank you for your response. Your clarifications helped me better understand the paper, and I have raised my score to 3.

AuthorsOpenReview ↗

Thank You for Your Feedback

We sincerely thank the reviewer for the positive feedback and for reconsidering the assessment. We are glad that our clarifications and additional experiments helped address the concerns. We greatly appreciate your constructive suggestions.

Reviewer_2HUsOpenReview ↗
Rating
3
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SPAR to address the limitations of existing role-assignment methods in multi-agent reinforcement learning (MARL), which primarily rely on a single-role assumption and struggle to capture inter-role associations and reuse experience across episodes. SPAR constructs a role prototype space using semantic slots and provides each agent with a dual semantic assignment consisting of a primary role and an auxiliary role. It further introduces a main–auxiliary role-aware contrastive objective and a performance-guided slot memory module to downweight cross-role negatives and enable the cross-episode retrieval and reuse of slot structures associated with high-quality collaboration, respectively. Experiments on the SMAC and SMACv2 benchmarks demonstrate that SPAR can learn dynamic, structured roles and achieve highly competitive win rates across multiple test scenarios.

Critical strengths
  1. The paper has a certain degree of novelty. Unlike other Marl methods for role division, this method expands role allocation from the traditional "single hard label" to a dual soft allocation of "main role + auxiliary role". The paper also includes extensive theoretical analysis and proof in the main text and appendices and provides a rigorous discussion of the motivation and solution approach.
  2. Compared to other baseline methods, it achieves a performance improvement of significant practical magnitude. SPAR outperforms ACORM, QMIX, and other baselines across many evaluation tasks, demonstrating that the proposed method yields gains of genuine practical value.
Critical weaknesses
  1. The relationship between a "primary role" and an "auxiliary role" is not sufficiently well-defined; an auxiliary role is essentially one with a lower assignment probability compared to the primary role, and the text somewhat overstates the nature of this relationship. The second-ranked slot does not necessarily correspond to a complementary responsibility; it could reflect uncertainty between prototypes. The analysis in the QUALITATIVE ANALYSIS section only shows that there is a certain transitional relationship between the primary and auxiliary roles of the agent, but it still lacks experiments and analyses to illustrate other further dependencies, especially cross-agent relationships.
  2. The ablation study is extremely thin and lacks persuasiveness. The ablation experiment only performed ablation of the two components individually and together in the 3s5z_vs_3s6z task. Although the result curves show a decrease in performance, they lack certain analysis, making it impossible to confirm the actual contribution of each component, and the experiment did not demonstrate the role of the auxiliary role. furthermore, no experiments are provided to clarify the specific role of the auxiliary role. These limitations leave the paper’s main mechanisms insufficiently validated, despite the broader improvements reported for the complete method.
Questions for authors
  1. The experimental environment is rather limited; it is therefore unclear whether this optimization represents a general improvement in algorithm performance or if it is merely a specific tuning tailored for SMAC. Conducting experiments in more complex environments that involve distinct roles (such as Google Research Football) would provide more compelling evidence.
  2. The text and figures contain several writing and spelling errors. For example, “Role Encoder” is misspelled as “Role Enocder” in Figure 1, and there are missing spaces after punctuation marks in lines 056, 057 and 072 on the second page.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 2HUs

Response to Question 1

Thank you for this suggestion. We have extended the evaluation to three Google Research Football scenarios and compared SPAR with ACORM and QMIX. Section 4 of the main text now introduces this evaluation, while Appendix C and Figure 5 provide the learning curves, reward settings, and result analysis. Late-training test win rates across five seeds are summarized below.

ScenarioSPAR (%)ACORM (%)QMIX (%)
academy_3_vs_1_with_keeper32.00 ± 28.106.00 ± 8.981.63 ± 0.56
academy_counterattack_easy58.00 ± 14.2911.50 ± 15.802.00 ± 2.48
academy_counterattack_hard1.88 ± 3.160.38 ± 0.840.13 ± 0.28

On the first two scenarios, SPAR improves over ACORM by 26.00 and 46.50 percentage points, respectively, showing that the gains can extend beyond SMAC. Appendix C also explicitly reports the training rewards, including goal rewards, distance-checkpoint rewards, and a terminal penalty for episodes without a win.

These results support applicability after retraining in the target environment, rather than zero-shot transfer or universal superiority. All three methods still have very low absolute win rates on the hard scenario, and we explicitly acknowledge this limitation in Appendix C, lines 983–988.

Response to Question 2

Thank you for pointing out these issues. The spelling in Figure 1 has been corrected to “Role Encoder,” and the punctuation and spacing in the identified passages on the second page have been revised. We have also refined the related wording to improve the readability of the main text and framework diagram.

Thank you for the detailed response and additional experiments. After reading the other reviewers' comments and concerns, I also noticed that the performance on SMACv2 remains somewhat limited. As shown in Table 1 and Figure 7, SPAR does not consistently outperform the strongest baselines, particularly on the more challenging 10v11 scenarios, where the absolute win rates are relatively low. This raises some concerns about the effectiveness of the proposed method in more complex settings. I would also be interested in knowing how GoMARL performs on GRF, given its competitive performance on SMACv2. Nevertheless, I appreciate the authors' efforts to address the reviewers' concerns. I will maintain my original rating.

AuthorsOpenReview ↗

Thank You for Your Feedback

We sincerely thank the reviewer for the thoughtful follow-up and for acknowledging our efforts. We appreciate the remaining concerns regarding performance on challenging SMACv2 scenarios and the suggestion to compare with GoMARL on GRF. We will take these points into account in future work.

Reviewer_hqpNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SPAR extends ACORM with recurrent slot prototypes, primary and auxiliary role assignments, relation-weighted contrastive learning, and performance-guided SlotBank retrieval during training. Evaluation covers 12 SMAC and SMACv2 tasks; decentralized execution does not retrieve from SlotBank.

Critical strengths
  • SPAR improves over ACORM on 11 of 12 tasks, raising average win rate from 44.5% to 53.4%. Five-seed results and learning curves support practical value, despite substantial variability on some tasks.

  • Primary–auxiliary supervision and historical prototype reuse form a coherent extension while preserving decentralized execution. The paper appropriately identifies its single-trajectory visualization as illustrative.

Critical weaknesses
  • The experiments do not establish that secondary assignments capture cooperative relations rather than assignment ambiguity. Weakening related negatives is not compared with matched uniform or shuffled weakening, and SlotBank removal does not isolate contextual matching or win-based selection from prototype reuse.

  • The optimization path for the slot module remains unclear. Primary/auxiliary labels use hard ranking, gates are detached, and the auxiliary future-role loss has zero weight. Appendix B and Algorithm 1 describe slot-state updates but do not clearly identify which objectives train the projection, initial prototypes, and refinement GRU. This limits reproducibility.

Overall assessment: Weak rejection. The empirical gains are meaningful, but the structured-cooperation interpretation and slot-module optimization require stronger support.

Questions for authors
  • Clarification: Which slot-module parameters are optimized, by which losses and gradient paths? Please distinguish recurrent state updates from parameter learning and identify any fixed parameters.

  • Evidence: Can matched weakening and retrieval controls establish the benefit of role relationships and performance-guided reuse? Evidence beyond the principal ablation scenario would strengthen this interpretation.

2 comments
AuthorsOpenReview ↗

Response to Reviewer hqpN

Response to Question 1

Thank you for asking us to clarify this key distinction. In the current configuration, the slot input projection, normalization layers, initial prototypes, and GRU parameters remain at their initialized values. The slot states change dynamically. The contrastive loss updates the trajectory encoder, query role encoder, and bilinear similarity matrix WW. However, the hard top-kk role labels and detached negative weights do not transmit gradients to the slot module, and the TD loss does not backpropagate through it either.

The fixed-parameter GRU still performs recurrent refinement using the current agent representations and the preceding slot states, producing input-dependent role prototypes. Meanwhile, the upstream trajectory encoder remains trainable, so the overall mapping from raw trajectories to role assignments also changes. The reservoir computing methods reviewed by Lukoševičius and Jaeger (2009) use fixed recurrent network weights while allowing inputs to drive hidden-state evolution, providing a related precedent for fixed parameters with dynamic states. We cite this work to explain this mechanistic distinction, rather than as evidence of SPAR's performance advantage.

We have distinguished fixed parameters from dynamic states in Section 3.1, lines 223–230, and clarified the optimization targets and gradient paths in Section 3.2, lines 276–279. A controlled comparison of dynamic state updates is provided in Appendix D, Figure 6.

Response to Question 2

We have added matched weakening and retrieval-content controls in Section 4.3 and Figure 4 of the main text to distinguish the contributions of role relationships, overall negative weakening, and memory-content selection. The results on 3s5z_vs_3s6z are summarized below.

MethodTest win rate (%)
SPAR77.50 ± 16.28
Uniform weakening60.38 ± 21.97
Shuffled weakening55.63 ± 25.53
w/o performance-guided content selection66.88 ± 28.37
Shuffled context–slot alignment61.25 ± 30.88

Uniform assigns all negative pairs the mean of the original negative-pair weights within each episode and timestep. Shuffled preserves the weight multiset but randomly changes the correspondence between weights and negative pairs. SPAR achieves mean win rates approximately 17.1 and 21.9 percentage points higher, respectively, supporting the learning value of allocating weakening weights according to role relationships.

Both retrieval controls retain SlotBank and the original reference-gating calculation. The first replaces the retrieved content with the highest-similarity candidate. The second shuffles historical slot contents across retrieval contexts within the same binary episode-outcome category, namely win or non-win. SPAR achieves mean win rates approximately 10.6 and 16.3 percentage points higher, respectively, supporting the contributions of performance-guided content selection and contextual matching. The first control removes performance guidance only from content selection, rather than removing all performance information from the retrieval procedure.

We have also introduced additional evaluations on three GRF scenarios in Section 4 of the main text, with the full results in Appendix C, Figure 5. On two of these scenarios, SPAR improves over ACORM by 26.00 and 46.50 percentage points, respectively, providing additional performance evidence beyond SMAC.

AuthorsOpenReview ↗

Thank You for Your Consideration

We sincerely thank the reviewer for taking the time to reconsider our submission and revised manuscript. We greatly appreciate your constructive feedback and careful evaluation, which have helped us improve our work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Clarifications

We thank all reviewers for their constructive feedback and discussions. SPAR aims to improve cooperative multi-agent learning by modeling primary–auxiliary role relationships and reusing historical slot structures. In response to the reviews, we have revised the manuscript and conducted additional experiments. The main updates are summarized below.

  1. Slot optimization and dynamic updates: Clarified the gradient pathways and fixed slot-module parameters (Sections 3.1–3.2), and added a comparison with the Fixed Slots variant (Appendix D, Figure 6).

  2. Mechanism validation: Added matched negative-weighting and SlotBank controls to examine the effects of role-conditioned weight allocation, performance-guided content selection, and contextual alignment (Section 4.3, Figure 4).

  3. Evaluation beyond SMAC: Extended the evaluation to three Google Research Football (GRF) scenarios, providing additional empirical evidence of the method's performance across different environments (Appendix C, Figure 5).

  4. SMACv2 analysis: Added learning curves comparing seven methods across six scenarios, along with training-budget comparisons of SPAR, ACORM, and QMIX (Appendix E, Figure 7 and Table 4).

  5. Methodological contributions and clarifications: Further clarified SPAR's contributions relative to GRDC and RMFQ, corrected presentation and formatting issues, and more explicitly discussed the limitations of auxiliary-role interpretation and performance on challenging SMACv2 tasks.

Detailed responses and supporting evidence for the individual concerns have been provided in the corresponding reviewer discussions. We also appreciate the reviewers' engagement with our responses and their further consideration of our work.

Finally, we sincerely thank all reviewers and the Area Chair for their time, careful evaluations, and valuable feedback throughout the review and rebuttal process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.