- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CoMateEval evaluates an LLM’s contribution as one member of a simulated team, organizing responsibilities into nine Belbin roles and 18 scenarios. Its 370 cases combine background conversations, resources, conditional event graphs, and behavioral rubrics. CoMateSim controls scenario progression while allowing model actions to trigger different branches. Experiments show substantial differences across models and responsibilities, with the highest overall rubric pass rate reaching 45.47%. The central contribution is a benchmark and interaction protocol rather than a new collaboration algorithm.
Critical strengths
-
Broader coverage of team responsibilities. CoMateEval evaluates explanation, resource identification, record maintenance, and replanning alongside mediation and emotional support. This extends the scope of mediation-focused evaluations such as ProMediate. The observed differences across roles also provide information beyond an aggregate leaderboard: the highest-scoring model does not dominate every responsibility.
-
Inspectable scenario design and substantial quality assurance. Separating event specifications from orchestration makes the intended challenges explicit while retaining consequences for model actions. Independent case review, branch-coverage checks, and comparisons with human-reviewed scores across all recorded model–case trajectories provide meaningful support for the reported results.
Critical weaknesses
-
Insufficient validation that the simulator accommodates reasonable alternative behavior. The scheduler advances to a mainline event when no branch matches, while NPCs may only paraphrase the selected instruction. A sensible clarification or alternative plan therefore depends on the designer having anticipated it. The worked example in Appendix B.2 makes this concern concrete: generic inquiries, refusals, and explicit noncommitments fall into a branch where the preceding response is nevertheless consumed as a closed transaction. This may be appropriate for that particular workflow, but its prevalence across the benchmark is unclear. It illustrates how difficulty can arise from a restrictive interaction contract. Branch reachability does not establish that valid alternative actions are handled adequately. An independent audit of unsuccessful trajectories would help distinguish failures of collaboration from limitations of the authored interaction paths. This is my main concern.
-
Component reliability does not establish stability of complete interactions. Section 5.2 repeats component calls with fixed inputs and compares evaluators on existing trajectories. These checks are useful, but an early scheduling error can change the information available to the Bot and the branches it subsequently encounters. Rescoring the same transcript cannot reveal performance under a different, valid continuation. The reported scheduler agreement with humans is high but imperfect, making this a concrete source of uncertainty. Repeating complete sessions on a representative subset, with another simulator backend or human adjudication of consequential scheduling decisions, would provide stronger evidence that the reported rankings and role profiles are stable.
Questions for authors
- What happens when the Bot asks a relevant question or proposes a feasible alternative that matches no available branch? Can the simulator respond substantively without advancing past the opportunity to resolve it?
- Have independent reviewers examined unsuccessful trajectories specifically for reasonable actions that the event graph could not accommodate? How frequent are these cases, and how frequently do workflows impose irreversible consumption rules like those in the worked example?
- Do model rankings and role profiles remain stable when complete sessions are rerun or the scheduling/NPC backend changes? The fixed-input component repetitions and transcript rescoring do not answer this question.
7 comments
Response to Reviewer fyPq (2/3): Audit of Trajectory Limitations (W1, Q2)
We strongly agree with your recommendation to conduct an independent trajectory audit (W1, Q2)—this check is genuinely important. When constructing each case, we designed every event graph to be as thorough as possible and worked hard to align the entire event logic; doing so is non-trivial and was the single hardest part of building our cases. Even so, occasional omissions cannot be completely avoided, which is precisely why we carried out the statistical audit below.
Audit Setup
Using a newly resampled 90-case balanced subset across all 18 scenarios (5 per scenario), we examined only unsuccessful trajectories (runs with missed rubric points) from GPT-5.5, Claude Opus 4.7, and Gemini 3.1 Pro Preview. Two independent human annotators (Annotator 1 and Annotator 2) each randomly sampled 500 runtime mainline node instances (n = 500) from these unsuccessful trajectories and classified unmatched-action errors into four categories:
| Error Category | Annotator 1 (n = 500) | Annotator 2 (n = 500) |
|---|---|---|
| Reasonable alternative left unfollowed | 7 (1.40%) | 4 (0.80%) |
| Action intent misconverted | 15 (3.00%) | 17 (3.40%) |
| Question or noncommitment treated as final submission | 9 (1.80%) | 13 (2.60%) |
| Valid correction or recovery request blocked | 11 (2.20%) | 7 (1.40%) |
| Total (Unmatched Actions) | 42 (8.40%) | 41 (8.20%) |
Analysis
1. Whether a missed turn permanently closes an opportunity and our denominator scope?
Under CoMateSim's default scheduler, a single reply that fails to trigger an unlocked branch does not close it—while the unlocked branch remains eligible, its trigger condition stays open so the Bot can still select and trigger it in subsequent turns. Our n = 500 denominator measures the node-level incidence along unsuccessful trajectories where a node-level no-match event led to a downstream rubric evaluation error.
2. Consistent aggregate rates across independent samples.
Both annotators' independent random samples from unsuccessful trajectories yield closely aligned node-level rates (over 91% of sampled mainline nodes showed none of these unmatched-action errors). Because the two draws were sampled independently, this supports consistency in aggregate observed rates rather than item-level inter-annotator agreement.
3. Consistent category profile.
Both independent audits identify Action intent misconverted as the largest category and Reasonable alternative left unfollowed as the smallest.
Response to Reviewer fyPq (3/3): Stability Across Cross-Backend Sessions (W2, Q3)
We also shared your concern regarding the stability of our overall evaluation. CoMateSim is driven by three components: a script scheduler, user simulator, and evaluator. By design, we kept the scheduler and user simulator deliberately simple in logic to minimize model-reasoning variance, while still relying on LLMs for strong semantic judgment. Section 5.2 tested all three components separately for consistency, omitting a full-backend rerun due to cost. As you note (W2, Q3), component consistency does not directly guarantee full-state session stability, so we added a full-pipeline backend-swap experiment on a subset.
Experimental Setup
1. 90-case subset: We used the same newly resampled 90-case subset as Part 2/3 (5 per scenario across all 18 scenarios; 30 each for Thinking, Social, Action), preserving structure consistency with the full 370-case benchmark.
2. Full backend replacement: We replaced the full backend—scheduler, user simulator, and evaluator—with DeepSeek V4.1 Flash 1017 and reran full sessions from scratch.
3. Model coverage: Among the 9 benchmark models, Doubao Seed 2.0 Pro and Hunyuan-3 are omitted (—) as their legacy APIs are unavailable, leaving 7 common models.
Main Results
The table below compares all models horizontally across Thinking, Social, Action, and Overall, reporting score (%) (within-column rank) under Full Set (GPT-5.5, 370 cases), Subset (GPT-5.5, 90 cases), and Subset (DeepSeek V4.1 Flash 1017, abbreviated DS-v4.1, 90 cases).
| Model | Think | Social | Action | Overall | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Full GPT-5.5 | Subset GPT-5.5 | Subset DS-v4.1 | Full GPT-5.5 | Subset GPT-5.5 | Subset DS-v4.1 | Full GPT-5.5 | Subset GPT-5.5 | Subset DS-v4.1 | Full GPT-5.5 | Subset GPT-5.5 | Subset DS-v4.1 | |
| GPT-5.5 | 42.17 (1) | 43.95 (1) | 50.60 (1) | 50.50 (1) | 50.82 (2) | 57.20 (1) | 43.99 (2) | 41.84 (2) | 46.80 (2) | 45.47 (1) | 45.54 (1) | 51.53 (1) |
| Claude Opus 4.7 | 40.62 (2) | 40.69 (2) | 49.60 (2) | 46.91 (2) | 51.63 (1) | 56.10 (2) | 37.24 (3) | 39.31 (4) | 46.30 (3) | 41.56 (2) | 43.88 (2) | 50.67 (2) |
| Gemini 3.1 Pro Preview | 37.89 (3) | 37.93 (3) | 48.20 (3) | 28.49 (7) | 31.36 (7) | 39.80 (5) | 55.81 (1) | 51.86 (1) | 61.80 (1) | 40.70 (3) | 40.38 (3) | 49.93 (3) |
| GLM-5.1 | 33.61 (5) | 36.23 (5) | 39.50 (5) | 44.14 (3) | 44.24 (3) | 45.60 (3) | 35.37 (4) | 39.95 (3) | 39.50 (4) | 37.60 (4) | 40.14 (4) | 41.53 (5) |
| DeepSeek V4 Pro | 34.96 (4) | 37.10 (4) | 42.06 (4) | 37.48 (4) | 39.39 (4) | 45.41 (4) | 34.67 (5) | 34.42 (5) | 39.18 (5) | 35.68 (5) | 36.97 (5) | 42.22 (4) |
| Kimi K2.6 | 33.34 (6) | 36.19 (6) | 36.86 (7) | 31.11 (6) | 32.48 (6) | 39.45 (6) | 22.00 (9) | 26.01 (9) | 37.28 (6) | 28.91 (6) | 31.56 (6) | 37.86 (6) |
| Doubao Seed 2.0 Pro | 31.29 (7) | 30.60 (7) | — | 20.25 (9) | 20.52 (8) | — | 29.21 (6) | 28.68 (7) | — | 27.03 (7) | 26.60 (8) | — |
| Hunyuan-3 | 22.99 (9) | 25.00 (9) | — | 32.37 (5) | 33.85 (5) | — | 25.11 (8) | 27.26 (8) | — | 26.72 (8) | 28.70 (7) | — |
| Qwen3.6 Plus | 31.07 (8) | 30.36 (8) | 37.20 (6) | 20.29 (8) | 19.78 (9) | 28.93 (7) | 27.62 (7) | 29.35 (6) | 35.91 (7) | 26.45 (9) | 26.50 (9) | 34.01 (7) |
Consistency Analysis
| Dim. | Subset vs. Full (GPT-5.5: / ) | Backend: DS-v4.1 vs. GPT-5.5 (Subset: / ) |
|---|---|---|
| Thinking | 1.000 / +1.12 pp | 0.964 / +5.94 pp |
| Social | 0.967 / +1.39 pp | 0.929 / +6.11 pp |
| Action | 0.967 / +0.85 pp | 0.929 / +6.29 pp |
| Overall | 0.983 / +1.13 pp | 0.964 / +6.11 pp |
1. Subset vs. Full Benchmark
Under GPT-5.5, model rankings on the 90-case subset closely match the 370-case benchmark (, 35/36 concordant pairs, +1.13 pp mean offset overall), with on Thinking and on both Social and Action, confirming high overall and category-level rank consistency across these nine models.
2. Backend Consistency: DS-v4.1 vs. GPT-5.5
Rerunning complete sessions with DeepSeek V4.1 Flash 1017 preserves overall rankings across the 7 common models (, 20/21 pairs): the top 3 stay identical, with only GLM-5.1 and DeepSeek V4 Pro trading 4th/5th, and 6 of 7 models retain their strongest role category (Kimi K2.6 shifts from Thinking to Social).
3. Consistent Upward Score Trend
DeepSeek V4.1 Flash 1017 scores +6.11 pp higher overall than GPT-5.5, showing a consistent upward score trend alongside the +9.79 pp offset observed in Section 5.2 / Table 4 when using DeepSeek as the evaluator on fixed transcripts.
Response to Reviewer fyPq (1/3): Unmatched Actions and Design Trade-offs (W1, Q1)
Thank you for recognizing CoMateEval’s coverage of team responsibilities beyond mediation (S1) and highlighting the inspectability of its scenarios and quality assurance (S2). We address your review in three parts:
(1/3 · W1, Q1) Unmatched Actions and Design Trade-offs: Explains what happens when an action matches no branch and the trade-off we made in this design.
(2/3 · W1, Q2) Audit of Trajectory Limitations: Attributes and analyzes errors triggered by unmatched actions through an independent two-annotator statistical audit.
(3/3 · W2, Q3) Stability Across Cross-Backend Sessions: Replaces the simulator backend and reruns complete sessions to test ranking and role-profile stability.
Direct Answer (Q1): What Happens on Unmatched Actions
When the Bot asks a relevant question or proposes a feasible alternative that matches no currently unlocked branch condition, the simulator does not match another branch or improvise an off-graph reply. Instead, the scheduler advances to the next mainline node (Section 2.2), and the designated NPC only paraphrases that node's instruction. Accordingly, when no authored branch accommodates an inquiry, the simulator cannot respond substantively without advancing past that immediate opportunity—though mainline progression alone does not erase pre-authored branches that remain unlocked.
Design Trade-off: Freedom vs. Controllability
This behavior reflects a deliberate trade-off between interactive freedom and evaluation validity:
Unconstrained dialogue undermines validity and comparability: Letting simulated teammates converse without event-graph constraints risks distorting the evaluation—they can leak hidden clues prematurely, and because modern LLMs are highly accommodating, Bots and teammates tend to negotiate ad-hoc workarounds outside the intended task constraints, eliminating the designed bottlenecks and making runs non-comparable.
Fixed scripts remove interactive agency: Conversely, static-context benchmarks (Table 1) lock teammate utterances regardless of what the Bot asks or proposes.
We therefore adopted an Event Graph (
6,661mainline events and4,860conditional branches across370cases; Table 2) combining ordered mainlines with conditional branches. Key clues are placed on branches so that failing to trigger the right action causes the Bot to miss them—an intentional part of our evaluation signal.Event-Graph Progression and Appendix B.2
On how opportunities remain open across turns, we distinguish the default progression rule from the case-specific gate in Appendix B.2:
Default branch persistence: Advancing along mainline nodes unlocks their attached branch nodes. In subsequent turns, the scheduler checks still-unlocked branch candidates before falling back to the mainline, so an untriggered branch that stays open and unskipped can still be selected later (once one child under a parent fires, its sibling branches are marked skipped; when the mainline finishes with no branch triggered, the session closes).
Appendix B.2 as a workflow-specific rule: In Alder Cove (Appendix B.2),
root-mi-consumeroutes inquiries or refusals intochild-mi-fallbackbecause that specific closeout task explicitly mandates that "each register closes without late supplementation." This reflects the design of that specific case—spanning both its storyline progression and its evaluation rubrics—rather than a universal rule across all event graphs; specifically, in Alder Cove, the register closes after the Bot's reply even when that reply asks for clarification or explicitly withholds commitment, and we will make this consequence explicit in Appendix B.2.