acceptodds
Under review as a conference paper at ICLR 2027

CoMateEval: Benchmarking LLMs Across Team Roles in Dynamic Multi-Party Collaboration

Abstract

Large language models are increasingly integrated into everyday workflows as interactive assistants. However, real-world work is often carried out in teams rather than through isolated human–AI interactions. Recent studies have begun to evaluate LLMs in collaborative settings, but existing formulations of collaboration are often constrained to specific tasks, roles, or interaction paradigms, leaving broader team responsibilities and dynamic multi-party collaboration insufficiently characterized. To address these limitations, we introduce CoMateEval, a benchmark grounded in Belbin team role theory for evaluating LLMs across diverse team responsibilities in dynamic multi-party collaboration. CoMateEval covers 9 team roles through 18 collaborative scenarios and instantiates them into 370 evaluation cases spanning diverse forms of team contribution. We further design CoMateSim, a multi-party collaboration simulator supporting constrained adaptive interaction, where predefined event structures and branch conditions govern scenario progression while interaction trajectories evolve with model behavior. Model performance is assessed by how well it fulfills its assigned responsibilities. Experiments on frontier LLMs show that the strongest model achieves an average rubric pass rate of 45.47% across the selected scenarios, while different models exhibit distinct strengths across team responsibilities. These results identify substantial gaps in fulfilling the tested responsibilities and motivate further work toward reliable AI teammates.

Primary area:
datasets and benchmarks
Keywords:
large language models, team roles, multi-party collaboration, llm evaluation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_fyPqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CoMateEval evaluates an LLM’s contribution as one member of a simulated team, organizing responsibilities into nine Belbin roles and 18 scenarios. Its 370 cases combine background conversations, resources, conditional event graphs, and behavioral rubrics. CoMateSim controls scenario progression while allowing model actions to trigger different branches. Experiments show substantial differences across models and responsibilities, with the highest overall rubric pass rate reaching 45.47%. The central contribution is a benchmark and interaction protocol rather than a new collaboration algorithm.

Critical strengths
  1. Broader coverage of team responsibilities. CoMateEval evaluates explanation, resource identification, record maintenance, and replanning alongside mediation and emotional support. This extends the scope of mediation-focused evaluations such as ProMediate. The observed differences across roles also provide information beyond an aggregate leaderboard: the highest-scoring model does not dominate every responsibility.

  2. Inspectable scenario design and substantial quality assurance. Separating event specifications from orchestration makes the intended challenges explicit while retaining consequences for model actions. Independent case review, branch-coverage checks, and comparisons with human-reviewed scores across all recorded model–case trajectories provide meaningful support for the reported results.

Critical weaknesses
  1. Insufficient validation that the simulator accommodates reasonable alternative behavior. The scheduler advances to a mainline event when no branch matches, while NPCs may only paraphrase the selected instruction. A sensible clarification or alternative plan therefore depends on the designer having anticipated it. The worked example in Appendix B.2 makes this concern concrete: generic inquiries, refusals, and explicit noncommitments fall into a branch where the preceding response is nevertheless consumed as a closed transaction. This may be appropriate for that particular workflow, but its prevalence across the benchmark is unclear. It illustrates how difficulty can arise from a restrictive interaction contract. Branch reachability does not establish that valid alternative actions are handled adequately. An independent audit of unsuccessful trajectories would help distinguish failures of collaboration from limitations of the authored interaction paths. This is my main concern.

  2. Component reliability does not establish stability of complete interactions. Section 5.2 repeats component calls with fixed inputs and compares evaluators on existing trajectories. These checks are useful, but an early scheduling error can change the information available to the Bot and the branches it subsequently encounters. Rescoring the same transcript cannot reveal performance under a different, valid continuation. The reported scheduler agreement with humans is high but imperfect, making this a concrete source of uncertainty. Repeating complete sessions on a representative subset, with another simulator backend or human adjudication of consequential scheduling decisions, would provide stronger evidence that the reported rankings and role profiles are stable.

Questions for authors
  1. What happens when the Bot asks a relevant question or proposes a feasible alternative that matches no available branch? Can the simulator respond substantively without advancing past the opportunity to resolve it?
  2. Have independent reviewers examined unsuccessful trajectories specifically for reasonable actions that the event graph could not accommodate? How frequent are these cases, and how frequently do workflows impose irreversible consumption rules like those in the worked example?
  3. Do model rankings and role profiles remain stable when complete sessions are rerun or the scheduling/NPC backend changes? The fixed-input component repetitions and transcript rescoring do not answer this question.
7 comments
AuthorsOpenReview ↗

Response to Reviewer fyPq (1/3): Unmatched Actions and Design Trade-offs (W1, Q1)

Thank you for recognizing CoMateEval’s coverage of team responsibilities beyond mediation (S1) and highlighting the inspectability of its scenarios and quality assurance (S2). We address your review in three parts:

(1/3 · W1, Q1) Unmatched Actions and Design Trade-offs: Explains what happens when an action matches no branch and the trade-off we made in this design.

(2/3 · W1, Q2) Audit of Trajectory Limitations: Attributes and analyzes errors triggered by unmatched actions through an independent two-annotator statistical audit.

(3/3 · W2, Q3) Stability Across Cross-Backend Sessions: Replaces the simulator backend and reruns complete sessions to test ranking and role-profile stability.

Direct Answer (Q1): What Happens on Unmatched Actions

When the Bot asks a relevant question or proposes a feasible alternative that matches no currently unlocked branch condition, the simulator does not match another branch or improvise an off-graph reply. Instead, the scheduler advances to the next mainline node (Section 2.2), and the designated NPC only paraphrases that node's instruction. Accordingly, when no authored branch accommodates an inquiry, the simulator cannot respond substantively without advancing past that immediate opportunity—though mainline progression alone does not erase pre-authored branches that remain unlocked.

Design Trade-off: Freedom vs. Controllability

This behavior reflects a deliberate trade-off between interactive freedom and evaluation validity:

  • Unconstrained dialogue undermines validity and comparability: Letting simulated teammates converse without event-graph constraints risks distorting the evaluation—they can leak hidden clues prematurely, and because modern LLMs are highly accommodating, Bots and teammates tend to negotiate ad-hoc workarounds outside the intended task constraints, eliminating the designed bottlenecks and making runs non-comparable.

  • Fixed scripts remove interactive agency: Conversely, static-context benchmarks (Table 1) lock teammate utterances regardless of what the Bot asks or proposes.

We therefore adopted an Event Graph (6,661 mainline events and 4,860 conditional branches across 370 cases; Table 2) combining ordered mainlines with conditional branches. Key clues are placed on branches so that failing to trigger the right action causes the Bot to miss them—an intentional part of our evaluation signal.

Event-Graph Progression and Appendix B.2

On how opportunities remain open across turns, we distinguish the default progression rule from the case-specific gate in Appendix B.2:

  • Default branch persistence: Advancing along mainline nodes unlocks their attached branch nodes. In subsequent turns, the scheduler checks still-unlocked branch candidates before falling back to the mainline, so an untriggered branch that stays open and unskipped can still be selected later (once one child under a parent fires, its sibling branches are marked skipped; when the mainline finishes with no branch triggered, the session closes).

  • Appendix B.2 as a workflow-specific rule: In Alder Cove (Appendix B.2), root-mi-consume routes inquiries or refusals into child-mi-fallback because that specific closeout task explicitly mandates that "each register closes without late supplementation." This reflects the design of that specific case—spanning both its storyline progression and its evaluation rubrics—rather than a universal rule across all event graphs; specifically, in Alder Cove, the register closes after the Bot's reply even when that reply asks for clarification or explicitly withholds commitment, and we will make this consequence explicit in Appendix B.2.

AuthorsOpenReview ↗

Response to Reviewer fyPq (2/3): Audit of Trajectory Limitations (W1, Q2)

We strongly agree with your recommendation to conduct an independent trajectory audit (W1, Q2)—this check is genuinely important. When constructing each case, we designed every event graph to be as thorough as possible and worked hard to align the entire event logic; doing so is non-trivial and was the single hardest part of building our cases. Even so, occasional omissions cannot be completely avoided, which is precisely why we carried out the statistical audit below.

Audit Setup

Using a newly resampled 90-case balanced subset across all 18 scenarios (5 per scenario), we examined only unsuccessful trajectories (runs with missed rubric points) from GPT-5.5, Claude Opus 4.7, and Gemini 3.1 Pro Preview. Two independent human annotators (Annotator 1 and Annotator 2) each randomly sampled 500 runtime mainline node instances (n = 500) from these unsuccessful trajectories and classified unmatched-action errors into four categories:

Error CategoryAnnotator 1 (n = 500)Annotator 2 (n = 500)
Reasonable alternative left unfollowed7 (1.40%)4 (0.80%)
Action intent misconverted15 (3.00%)17 (3.40%)
Question or noncommitment treated as final submission9 (1.80%)13 (2.60%)
Valid correction or recovery request blocked11 (2.20%)7 (1.40%)
Total (Unmatched Actions)42 (8.40%)41 (8.20%)

Analysis

1. Whether a missed turn permanently closes an opportunity and our denominator scope?

Under CoMateSim's default scheduler, a single reply that fails to trigger an unlocked branch does not close it—while the unlocked branch remains eligible, its trigger condition stays open so the Bot can still select and trigger it in subsequent turns. Our n = 500 denominator measures the node-level incidence along unsuccessful trajectories where a node-level no-match event led to a downstream rubric evaluation error.

2. Consistent aggregate rates across independent samples.

Both annotators' independent random samples from unsuccessful trajectories yield closely aligned node-level rates (over 91% of sampled mainline nodes showed none of these unmatched-action errors). Because the two draws were sampled independently, this supports consistency in aggregate observed rates rather than item-level inter-annotator agreement.

3. Consistent category profile.

Both independent audits identify Action intent misconverted as the largest category and Reasonable alternative left unfollowed as the smallest.

AuthorsOpenReview ↗

Response to Reviewer fyPq (3/3): Stability Across Cross-Backend Sessions (W2, Q3)

We also shared your concern regarding the stability of our overall evaluation. CoMateSim is driven by three components: a script scheduler, user simulator, and evaluator. By design, we kept the scheduler and user simulator deliberately simple in logic to minimize model-reasoning variance, while still relying on LLMs for strong semantic judgment. Section 5.2 tested all three components separately for consistency, omitting a full-backend rerun due to cost. As you note (W2, Q3), component consistency does not directly guarantee full-state session stability, so we added a full-pipeline backend-swap experiment on a subset.

Experimental Setup

1. 90-case subset: We used the same newly resampled 90-case subset as Part 2/3 (5 per scenario across all 18 scenarios; 30 each for Thinking, Social, Action), preserving structure consistency with the full 370-case benchmark.

2. Full backend replacement: We replaced the full backend—scheduler, user simulator, and evaluator—with DeepSeek V4.1 Flash 1017 and reran full sessions from scratch.

3. Model coverage: Among the 9 benchmark models, Doubao Seed 2.0 Pro and Hunyuan-3 are omitted (—) as their legacy APIs are unavailable, leaving 7 common models.

Main Results

The table below compares all models horizontally across Thinking, Social, Action, and Overall, reporting score (%) (within-column rank) under Full Set (GPT-5.5, 370 cases), Subset (GPT-5.5, 90 cases), and Subset (DeepSeek V4.1 Flash 1017, abbreviated DS-v4.1, 90 cases).

ModelThinkSocialActionOverall
Full GPT-5.5Subset GPT-5.5Subset DS-v4.1Full GPT-5.5Subset GPT-5.5Subset DS-v4.1Full GPT-5.5Subset GPT-5.5Subset DS-v4.1Full GPT-5.5Subset GPT-5.5Subset DS-v4.1
GPT-5.542.17 (1)43.95 (1)50.60 (1)50.50 (1)50.82 (2)57.20 (1)43.99 (2)41.84 (2)46.80 (2)45.47 (1)45.54 (1)51.53 (1)
Claude Opus 4.740.62 (2)40.69 (2)49.60 (2)46.91 (2)51.63 (1)56.10 (2)37.24 (3)39.31 (4)46.30 (3)41.56 (2)43.88 (2)50.67 (2)
Gemini 3.1 Pro Preview37.89 (3)37.93 (3)48.20 (3)28.49 (7)31.36 (7)39.80 (5)55.81 (1)51.86 (1)61.80 (1)40.70 (3)40.38 (3)49.93 (3)
GLM-5.133.61 (5)36.23 (5)39.50 (5)44.14 (3)44.24 (3)45.60 (3)35.37 (4)39.95 (3)39.50 (4)37.60 (4)40.14 (4)41.53 (5)
DeepSeek V4 Pro34.96 (4)37.10 (4)42.06 (4)37.48 (4)39.39 (4)45.41 (4)34.67 (5)34.42 (5)39.18 (5)35.68 (5)36.97 (5)42.22 (4)
Kimi K2.633.34 (6)36.19 (6)36.86 (7)31.11 (6)32.48 (6)39.45 (6)22.00 (9)26.01 (9)37.28 (6)28.91 (6)31.56 (6)37.86 (6)
Doubao Seed 2.0 Pro31.29 (7)30.60 (7)—20.25 (9)20.52 (8)—29.21 (6)28.68 (7)—27.03 (7)26.60 (8)—
Hunyuan-322.99 (9)25.00 (9)—32.37 (5)33.85 (5)—25.11 (8)27.26 (8)—26.72 (8)28.70 (7)—
Qwen3.6 Plus31.07 (8)30.36 (8)37.20 (6)20.29 (8)19.78 (9)28.93 (7)27.62 (7)29.35 (6)35.91 (7)26.45 (9)26.50 (9)34.01 (7)

Consistency Analysis

Dim.Subset vs. Full (GPT-5.5: ρ\rho / Δ\Delta)Backend: DS-v4.1 vs. GPT-5.5 (Subset: ρ\rho / Δ\Delta)
Thinking1.000 / +1.12 pp0.964 / +5.94 pp
Social0.967 / +1.39 pp0.929 / +6.11 pp
Action0.967 / +0.85 pp0.929 / +6.29 pp
Overall0.983 / +1.13 pp0.964 / +6.11 pp

1. Subset vs. Full Benchmark

Under GPT-5.5, model rankings on the 90-case subset closely match the 370-case benchmark (ρ=0.983\rho = 0.983, 35/36 concordant pairs, +1.13 pp mean offset overall), with ρ=1.000\rho = 1.000 on Thinking and ρ=0.967\rho = 0.967 on both Social and Action, confirming high overall and category-level rank consistency across these nine models.

2. Backend Consistency: DS-v4.1 vs. GPT-5.5

Rerunning complete sessions with DeepSeek V4.1 Flash 1017 preserves overall rankings across the 7 common models (ρ=0.964\rho = 0.964, 20/21 pairs): the top 3 stay identical, with only GLM-5.1 and DeepSeek V4 Pro trading 4th/5th, and 6 of 7 models retain their strongest role category (Kimi K2.6 shifts from Thinking to Social).

3. Consistent Upward Score Trend

DeepSeek V4.1 Flash 1017 scores +6.11 pp higher overall than GPT-5.5, showing a consistent upward score trend alongside the +9.79 pp offset observed in Section 5.2 / Table 4 when using DeepSeek as the evaluator on fixed transcripts.

Reviewer_q1ZFOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a new concept and evaluation task to evaluate LLMs' abilities on 9 different team roles.

The authors introduce CoMateEval, a dataset and benchmark consisting of 18 scenarios and 370 cases, and CoMateSim, a simulation framework for simulating multi-party collaboration among LLM NPCs.

Critical strengths

Task & Concept: The task and concept introduced in this paper, evaluating LLMs' abilities to take on different team roles in multi-party interactions, is really interesting and increasingly important, as LLMs are becoming more situated in multi-party collaboration settings.

Design of CoMateEval and CoMateSim: The authors provide a well-grounded framework for selecting team roles, and the experimental settings are comprehensive. For example, they consider different conversation modes, including free-form, scripted, and collaboration sessions. This allows them to capture not only how LLMs interact with one another but also the downstream effects of these interactions.

Human Validation: The authors also conduct enough human reviews for both simulator validation and data construction, which shows reliability of their framework.

Presentation: The paper is very well-written and easy to follow, with clear visualizations.

Critical weaknesses

Despite the clear strengths of the task, the implications drawn from the dataset are not fully discussed in the paper. While the paper shows how LLMs perform across different team roles, It'd be valuable to the research field if authors can add more discussion of what these findings imply for designing LLM-based multi-party collaboration systems or for improving LLM training to better support multi-party collaboration.

Questions for authors

Could the authors take a look at the point in Critical Weaknesses, and discuss the potential implications of the results? In particular, it would be helpful to understand how the results could inform the design of LLM-based multi-party collaboration systems (e.g. how should they divide the roles) and future model training.

2 comments
AuthorsOpenReview ↗

Response to Reviewer q1ZF: Implications for System Design and Training

We sincerely thank you for your encouraging review and strong support, as well as for highlighting our grounded Belbin role formulation, comprehensive conversational modes, thorough human validation across both data curation and simulator checks, and clear presentation. We strongly agree that articulating how our empirical findings inform system design and future model training will substantially strengthen the paper's impact, and we will add a dedicated Discussion subsection covering two concrete directions:

1. Implications for Multi-Party Collaboration System Design

  • Role-aware candidate selection and team composition: Our evaluation reveals pronounced role specialization—no single model ranks first across all nine roles, and the highest overall score does not guarantee the top candidate for every responsibility. Practical multi-party systems should therefore avoid defaulting uniformly to a single leaderboard leader; instead, role-level scores across Thinking, Social, and Action categories can guide candidate shortlisting before empirically validating the composed team's performance.
  • Explicit state tracking and constraint-verification scaffolds: Because the strongest model achieves only 45.47% overall—and across the top three models, information gathering, constraint reasoning, and action coordination each account for over 10% of failed rubric points—system architectures can pair role specialization with structured shared-state trackers, cross-source conflict checks, and pre-commitment verification steps to test their gains on multi-source information integration in future team evaluations.

2. Implications for Future Model Training

  • Evidence-grounded rubric supervision on newly authored cases: Unlike open-ended dialogue setups that rely on holistic preference judgments, CoMateSim pairs controllable event graphs with observable binary evaluation points. After each session, CoMateSim scores each rubric item and cites supporting evidence from the trajectory. Future work could use these evidence-backed judgments to supervise models on newly authored training cases and test whether the cited evidence reliably supports process feedback on individual decisions.
  • Targeting multi-turn information synthesis and follow-through: Our error attribution identifies clear post-training priorities beyond single-turn helpfulness: training corpora and objectives should target gathering scattered facts from different teammates across turns, reconciling conflicting records prior to commitment, and sustaining multi-turn follow-through.
  • Curriculum construction and transfer testing over event graphs: Decoupling core mainlines from conditional branches allows authors to specify new training cases with varying role demands and branch structures. A testable training direction is to construct a curriculum of newly authored cases with progressively more information sources and constraints, and then evaluate whether targeted capabilities transfer to unseen scenarios and different teammates.

We will incorporate this structured discussion into the revised manuscript to better connect our empirical findings with multi-party system engineering and LLM post-training.

Thank you for the response! I hope authors incorporate this to the final paper.

Reviewer_XTzLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how well LLMs fulfill different team responsibilities in dynamic multi-party collaboration. Existing collaboration benchmarks are often limited to particular tasks, interaction paradigms, or a small set of agent roles, making it difficult to systematically characterize LLMs as general-purpose teammates.

The paper makes two main contributions. First, it introduces CoMateEval, a benchmark organized around nine Belbin-inspired team roles across Thinking, Social, and Action categories. These roles are instantiated into 18 collaborative scenarios and 370 evaluation cases. Second, it introduces CoMateSim, a multi-party collaboration simulator that uses predefined event graphs and conditional branches to constrain scenario progression while allowing model actions to affect subsequent interactions and NPC responses. Model behavior is evaluated using scenario-specific pass/fail rubrics corresponding to the responsibilities of its assigned role.

For validation, the authors evaluate nine frontier LLMs across all 370 cases. The strongest model achieves an overall rubric pass rate of 45.47%, and different models exhibit distinct strengths across team roles. The paper further analyzes recurring capability failures such as information gathering, constraint reasoning, and action coordination. The authors also conduct human and cross-model validation of the simulator components and rubric evaluator to assess evaluation reliability.

Critical strengths

1. A meaningful and relatively underexplored evaluation target.\textbf{1. A meaningful and relatively underexplored evaluation target.} I find the shift from evaluating isolated task performance to evaluating responsibility fulfillment within a team interesting. The nine-role formulation provides a more structured view of what it means for an LLM to act as a teammate than treating collaboration as a single aggregate capability.

Critical weaknesses

1. Dependence on the simulator backend may confound model comparisons.\textbf{1. Dependence on the simulator backend may confound model comparisons.} GPT-5.5 is used as the default backend for NPC response generation, event scheduling, and rubric evaluation, while GPT-5.5 is also one of the evaluated Bots. Because interaction trajectories themselves depend on scheduler decisions and generated NPC responses, model performance may partly reflect compatibility with this particular simulation backend rather than only the underlying collaboration ability being measured.

Questions for authors

Would the main model rankings and role-level conclusions remain similar if NPC generation and event scheduling were driven by a different simulator backend (e.g., DeepSeek) rather than GPT-5.5?

3 comments
AuthorsOpenReview ↗

Response to Reviewer XTzL: Cross-Backend Stability of Rankings and Roles

Thank you for your constructive feedback and for highlighting our shift from isolated task completion to role-specific responsibility fulfillment in teams, as well as our nine-role formulation (S1). To address your concern regarding potential self-backend bias (W1, Q1), building on Section 5.2—which already examined the consistency and effects of independently validating or swapping each pipeline component (user simulator, scheduler, and evaluator)—we further conducted an end-to-end backend replacement experiment, substituting DeepSeek across the entire pipeline and rerunning complete multi-turn sessions from scratch.

Experimental Setup

1. 90-case subset: We used the same newly resampled 90-case subset as Part 2/3 (5 per scenario across all 18 scenarios; 30 each for Thinking, Social, Action), preserving structure consistency with the full 370-case benchmark.

2. Full backend replacement: We replaced the full backend—scheduler, user simulator, and evaluator—with DeepSeek V4.1 Flash 1017 and reran full sessions from scratch.

3. Model coverage: Among the 9 benchmark models, Doubao Seed 2.0 Pro and Hunyuan-3 are omitted (—) as their legacy APIs are unavailable, leaving 7 common models.

Main Results

The table below compares all models horizontally across Thinking, Social, Action, and Overall, reporting score (%) (within-column rank) under Full Set (GPT-5.5, 370 cases), Subset (GPT-5.5, 90 cases), and Subset (DeepSeek V4.1 Flash 1017, abbreviated DS-v4.1, 90 cases).

ModelThinkSocialActionOverall
Full GPT-5.5Subset GPT-5.5Subset DS-v4.1Full GPT-5.5Subset GPT-5.5Subset DS-v4.1Full GPT-5.5Subset GPT-5.5Subset DS-v4.1Full GPT-5.5Subset GPT-5.5Subset DS-v4.1
GPT-5.542.17 (1)43.95 (1)50.60 (1)50.50 (1)50.82 (2)57.20 (1)43.99 (2)41.84 (2)46.80 (2)45.47 (1)45.54 (1)51.53 (1)
Claude Opus 4.740.62 (2)40.69 (2)49.60 (2)46.91 (2)51.63 (1)56.10 (2)37.24 (3)39.31 (4)46.30 (3)41.56 (2)43.88 (2)50.67 (2)
Gemini 3.1 Pro Preview37.89 (3)37.93 (3)48.20 (3)28.49 (7)31.36 (7)39.80 (5)55.81 (1)51.86 (1)61.80 (1)40.70 (3)40.38 (3)49.93 (3)
GLM-5.133.61 (5)36.23 (5)39.50 (5)44.14 (3)44.24 (3)45.60 (3)35.37 (4)39.95 (3)39.50 (4)37.60 (4)40.14 (4)41.53 (5)
DeepSeek V4 Pro34.96 (4)37.10 (4)42.06 (4)37.48 (4)39.39 (4)45.41 (4)34.67 (5)34.42 (5)39.18 (5)35.68 (5)36.97 (5)42.22 (4)
Kimi K2.633.34 (6)36.19 (6)36.86 (7)31.11 (6)32.48 (6)39.45 (6)22.00 (9)26.01 (9)37.28 (6)28.91 (6)31.56 (6)37.86 (6)
Doubao Seed 2.0 Pro31.29 (7)30.60 (7)—20.25 (9)20.52 (8)—29.21 (6)28.68 (7)—27.03 (7)26.60 (8)—
Hunyuan-322.99 (9)25.00 (9)—32.37 (5)33.85 (5)—25.11 (8)27.26 (8)—26.72 (8)28.70 (7)—
Qwen3.6 Plus31.07 (8)30.36 (8)37.20 (6)20.29 (8)19.78 (9)28.93 (7)27.62 (7)29.35 (6)35.91 (7)26.45 (9)26.50 (9)34.01 (7)

Consistency Analysis

Dim.Subset vs. Full (GPT-5.5: ρ\rho / Δ\Delta)Backend: DS-v4.1 vs. GPT-5.5 (Subset: ρ\rho / Δ\Delta)
Thinking1.000 / +1.12 pp0.964 / +5.94 pp
Social0.967 / +1.39 pp0.929 / +6.11 pp
Action0.967 / +0.85 pp0.929 / +6.29 pp
Overall0.983 / +1.13 pp0.964 / +6.11 pp

1. Subset vs. Full Benchmark

Under GPT-5.5, model rankings on the 90-case subset closely match the 370-case benchmark (ρ=0.983\rho = 0.983, 35/36 concordant pairs, +1.13 pp mean offset overall), with ρ=1.000\rho = 1.000 on Thinking and ρ=0.967\rho = 0.967 on both Social and Action, confirming high overall and category-level rank consistency across these nine models.

2. Backend Consistency: DS-v4.1 vs. GPT-5.5

Rerunning complete sessions with DeepSeek V4.1 Flash 1017 preserves overall rankings across the 7 common models (ρ=0.964\rho = 0.964, 20/21 pairs): the top 3 stay identical, with only GLM-5.1 and DeepSeek V4 Pro trading 4th/5th, and 6 of 7 models retain their strongest role category (Kimi K2.6 shifts from Thinking to Social).

3. Consistent Upward Score Trend

DeepSeek V4.1 Flash 1017 scores +6.11 pp higher overall than GPT-5.5, showing a consistent upward score trend alongside the +9.79 pp offset observed in Section 5.2 / Table 4 when using DeepSeek as the evaluator on fixed transcripts.

Dear Reviewer XTzL,

Could you please review the authors’ rebuttals and follow up with any further comments or questions?

Many thanks for your time and efforts.

Best regards, AC

Thank the authors for their detailed response. My concerns have been addressed, and I maintain my positive rating.

Reviewer_XWoXOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper builds a simulation framework and benchmark for agent collaboration under different team roles. Inspired by the Belbin team role theory, it creates an evaluation for 9 team roles in 18 collaborative scenarios and builds a simulator for measuring the performance of different LLMs agents. The authors then run experiments on 9 models and find their performance varies across different roles.

Critical strengths

I think overall the paper is clearly written and the experiments design is reasonable. Studying multi-agent collaboration is an important problem.

Critical weaknesses

While studying agent collaboration is an important and interesting problem, I am not sure if this work brings distinctive new insights from the prior work. Zhou et al 2024 has already proposed a similar setup for agent social simulation and evaluation. I understand the novelty claim from the proposed Belbin model, but isn’t the simulation and contribution a subset of that general framework in Sotopia and other similar work? I feel I do not find too many insights that set this work apart. While Zhou et al is cited and referred to in the related work section, I do encourage the authors to thoroughly investigate this line of work and precisely point out the main novelty.

I also think the technical contribution of this paper is a bit lacking. The simulation method is closer to other similar work, and analysis of the results is also straightforward and lacks a bit of technical depth. I appreciate that many models are tested in this work, but it’s hard for me to draw some generalizable conclusions or gain deeper insights based on the current paper.

Zhou, Xuhui, et al. "Sotopia: Interactive evaluation for social intelligence in language agents." International Conference on Learning Representations. Vol. 2024. 2024.

Questions for authors

I think the authors may have missed some related papers for evaluating Human–LLM Collaboration. To name a few,

  • Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration by Shao et al.
  • Completion ≠ Collaboration: Scaling Collaborative Effort with Agents by Shen et al.
  • PPP-Agent: Training Proactive and Personalized LLM Agents, by Sun et al.
  • CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks by Wang et al.
  • Tom-swe: User mental modeling for software engineering agents by Zhou et al.
  • Align AI to Dynamic Human-AI Workflows by Chen et al.

But I am not sure if that’s the right reference to draw from. I’d encourage the authors to consider studying multi-agent collaboration from a more principled and theoretical perspective. For example, can we connect some of the Belbin theory with some agent collaboration training algorithms like Cooperative Inverse Reinforcement Learning? I think that’d be a fruitful direction and can significantly increase the technical depth of this work.

Also can you articulate how this dataset is curated exactly? There are many vague descriptions like “Materials collected through broad web searches provide references for story backgrounds. Content processing includes anonymization and redaction of sensitive information.” (line 821) and I cannot have a clear picture of the data curation pipeline.

8 comments
AuthorsOpenReview ↗

Response to Reviewer XWoX (Overview): Acknowledgment and Response Structure

Thank you for recognizing the importance of studying multi-agent collaboration, as well as the clarity of our writing and the soundness of our experimental design. We address your concerns in three parts:

(1/3) Motivation, design rationale, and distinctive insights: Directly addressing your question on whether this work brings distinctive new insights beyond prior studies—we explain why we formulated this problem, what core methodological gaps our design resolves, and what non-obvious findings emerge from our evaluation.

(2/3) Positioning against related work and distinct value: We clarify how CoMateEval and CoMateSim differ from prior work and the human–LLM collaboration studies you mentioned, and spell out where our distinct value lies.

(3/3) Plain walkthrough of the dataset curation pipeline: We note that Sections 3.3–3.4 already detail the full pipeline (line 821 covers only preliminary background sourcing), and lay out step by step how cases are built and verified through human review.

AuthorsOpenReview ↗

Response to Reviewer XWoX (1/3): Motivation, Design, and Distinctive Insights

Motivation

Human–LLM collaboration in 1-on-1 settings is well studied; what motivated us is the natural next step: if we drop an LLM into a multi-party team as an ordinary member with a specific job to do, how well does it actually hold up?

Once an AI joins a real team, three things immediately distinguish the setting from solo assistance:

  • Information is scattered, and teammates talk to each other: Key facts sit with different members and shared files, and teammates converse directly without always addressing the AI.
  • The situation keeps shifting: Plans stall, records contradict one another, or members disagree mid-discussion.
  • Different roles call for different behaviors: Driving a stalled schedule, spotting defects in a proposal, and defusing team tension require very different interventions.

Establishing a reliable benchmark for this setting is challenging, as it requires satisfying five key requirements (as compared in Table 1 of the paper):

  1. Multi-turn follow-through: Single turns cannot show whether commitments hold.
  2. Proactive intervention: Teammates do not always address the model directly, so it must decide when to step in and whom to involve.
  3. Dynamic partner feedback: Under a frozen dialogue, teammate replies stay fixed, hiding whether the model's move changes anything.
  4. Controlled progression: Conversely, if teammates improvise freely, they may solve the bottleneck in one run and not another, giving models different tests.
  5. Objective evaluability: Scoring must rest on concrete evidence in the conversation rather than subjective taste.

Balancing dynamic feedback with controlled, comparable challenges is the core gap we bridge.

Design: CoMateEval and CoMateSim

To satisfy these five requirements, we built CoMateEval and CoMateSim around three concrete challenges:

  1. Task definition: Grounded in Belbin's team role theory (Section 3.2), we span nine functional roles across Thinking, Social, and Action categories rather than designing tasks ad hoc.
  2. Evaluation: Open-ended tasks like brainstorming invite subjective grading. Guided by representativeness and evaluability, we chose two scenarios per role with case-specific binary rubrics grounded in observable transcripts, ensuring objective grading without forcing one scripted path.
  3. Simulation framework: To balance dynamism and control, CoMateSim uses an Event Graph combining fixed mainlines with conditional branches. Mainlines keep core bottlenecks comparable across models, while the Bot freely chooses when to reply, use tools, or wait—triggering conditional branches and bounded NPC responses so initiative has visible consequences.

Distinctive Insights

Returning to our opening question, four takeaways stand out:

  1. Even top models fall short overall: The strongest models pass under half of assigned responsibility criteria, showing a wide gap between solo assistance and reliable teamwork.
  2. Role specialization is pronounced: No model leads all nine roles. Multi-agent systems should match models to roles rather than picking one overall winner.
  3. Core bottleneck—gathering key information and turning it into feasible action: Top models rarely fail on conversational polish; they struggle to extract dispersed facts, reason over constraints, and coordinate concrete actions.
  4. More importantly, we built and realized an end-to-end framework for studying this problem: Spanning task formulation, verifiable rubrics, and the interactive CoMateSim runtime, our solution can accommodate broader follow-up studies and extend directly to training models with live environmental feedback.
AuthorsOpenReview ↗

Response to Reviewer XWoX (2/3): Positioning Against Related Work and Distinct Value

In your review, you raised the following main concerns:

  • "I am not sure if this work brings distinctive new insights from the prior work."
  • "Zhou et al 2024 has already proposed a similar setup for agent social simulation and evaluation."
  • "I understand the novelty claim from the proposed Belbin model, but isn’t the simulation and contribution a subset of that general framework in Sotopia and other similar work?"
  • "I feel I do not find too many insights that set this work apart."
  • "I also think the technical contribution of this paper is a bit lacking. The simulation method is closer to other similar work, and analysis of the results is also straightforward and lacks a bit of technical depth."
  • "it’s hard for me to draw some generalizable conclusions or gain deeper insights based on the current paper."

We respectfully disagree with this claim. Three points explain why:

1. The works you cited differ by definition: 1-on-1 interaction vs. our extension to 1-to-N

Human–LLM collaboration spans diverse settings. By definition, your cited works focus on 1-on-1 interaction or social role-play, whereas we extend to 1-to-N multi-party teamwork—a strictly harder setting with non-directed chat, distributed facts, dynamic state shifts, and team roles:

  • Collaborative Gym (Shao et al.) & Completion ≠ Collaboration (Shen et al.): 1-on-1 shared workspace between one user (human or simulator) and one agent.
  • CollabSkill (Shao et al.): 1-on-1 worker–agent pairing on occupational tasks using Bayesian skill ratings.
  • PPP-Agent (Sun et al.) & ToM-SWE (Zhou et al.): both focus on 1-on-1 interaction. PPP-Agent studies proactive clarification and ToM-SWE models the user’s mental state.
  • Align AI to Dynamic Human-AI Workflows (Chen et al.): conceptual alignment agenda, not a simulator or benchmark.
  • SOTOPIA (Zhou et al., 2024): evaluates pairs of agents that take turns choosing open-ended actions in pursuit of private social goals. CoMateSim uses authored event conditions to control when teammates can reveal task information.

As you noted ("not sure if that's the right reference to draw from"), these ask how an agent aligns with one user or social goal—not 1-to-N team roles. On the contrary, 1-to-N collaboration is a further exploration building beyond 1-on-1 studies—which brings us directly to the 1-to-N literature itself.

2. Returning to our closest 1-to-N related work: can existing frameworks meet our need? No.

In Section 6 and Table 1, we targeted 1-to-N multi-party benchmarks—the closest prior work. Can they answer our research question? No:

  • Fixed-context benchmarks (DICE-Bench, ProACT) fix teammate replies and lack adaptive responses and multi-turn follow-through.
  • The human-participant studies in MUCA and HUMA examine how assistants participate in group conversations. MUCA also provides an LLM-based multi-user simulator (MUS).
  • Automated 1-to-N simulations miss key conditions—TheAgentCompany lacks constrained progression; ProMediate and SoCRATES evaluate mediators and guide progression mainly through participant prompts. SCENE studies group norms and repair after sanctions, with an orchestrator for turn order and explicit stopping rules. GroupTravelBench relies on travel-specific compromise rules.

3. Our distinct value: a general, compatible framework and benchmark

Because 1-on-1 work differs by definition and existing 1-to-N work cannot answer our core question—"if we drop an LLM into a multi-party team as an ordinary member with a specific job to do, how well does it actually hold up?"—our distinct value is creating CoMateSim and CoMateEval to make it measurable.

Moreover, our framework is not a subset of prior ones; conversely, in both task definition and format, it can accommodate and run many existing benchmarks (capturing 1-on-1, fixed-context, single-role, or free-form setups via participant counts, NPC scopes, and Event-Graph transitions), directly demonstrating its generality.

Finally, we clarify two scope points from your review:

  • On Belbin team-role theory: We do not claim Belbin theory as our novelty. It is an established management framework adopted purely as a principled scaffold for team roles and cases.
  • On training and CIRL: We appreciate the CIRL suggestion. Classic CIRL (Hadfield-Menell et al., 2016) is likewise formulated as a 1-on-1 (two-player) game around a shared reward, whereas our 1-to-N setting involves multiple teammates with distributed facts and distinct roles. Still, CoMateSim's runtime and rubrics can indeed support future collaborative training (e.g., multi-party extensions). As this paper is scoped as a benchmark study, training algorithms lie outside its necessary scope. We will add these clarifications and citations in the revision.

Comments on the paper

1 comment

Dear Reviewers,

As the AC–reviewer discussion phase is about to begin, please highlight any disagreements or inconsistencies across the reviews, as well as any concerns that remain unresolved after the author rebuttal.

Please share your perspectives on these points and engage with one another to help us reach a well-supported assessment of the paper.

Thank you for your time and contributions.

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.