acceptodds
Desk rejected at ICLR 2027

Can Agents Compete as a Team? Benchmarking Long-Horizon Multi-Agent Collaboration with Realistic Contests

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model (LLM)-based multi-agent systems (MASs) have demonstrated extraordinary performance in numerous tasks and domains, but their collaborative capability in challenging real-world tasks is still largely underexplored. To bridge this gap, we propose ContestWorld, a benchmark dedicated to evaluating agent collaboration with diverse team-based human contests. It consists of 321 contests from different years with 39 categories, such as MCM, ICPC, and ARML, spanning 9 domains, ranging from mathematics to computer science, business, and linguistic contests. ContestWorld is deployed on a carefully designed realistic contest simulator that closely aligns with human settings, with an action interface comprising 27 common actions, three optional memory actions, and contest-specific tools, filtered according to contest rules and protocol settings. For instance, in an ICPC contest, three teammates share a single compiler, so only one of them can use it at a time. In ContestWorld, agents must allocate tool calls within the limited turns, interacting with the teammates and environments to achieve the best performance. With ContestWorld, we have conducted a comprehensive evaluation of four primary baselines and six backbone LLMs. The best-performing GPT-6 model achieves only 63.4% on average, demonstrating the challenge of the proposed dataset. Our analysis shows that teams primarily benefit from improved task coverage and selection, while their remaining failures stem from weak verification and poor feedback integration.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-agent LLM, Large Language Models, Benchmark, Evaluation, Contests

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_GcCfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a benchmark comprising 321 competitions across 39 competition categories and nine domains to evaluate multi-agent collaboration under competition-specific rules. The benchmark features challenging tasks, many at the Olympiad level, diverse domains, support for multiple collaboration protocols, and a relatively comprehensive evaluation framework that includes reference answers and LLM-based judges. The experimental analysis suggests that multi-agent systems do not consistently outperform single-agent systems, that their benefits primarily come from broader task coverage, and that memory modules can improve collaborative performance.

Critical strengths
  1. The benchmark offers a more challenging task collection than most existing benchmarks for multi-agent collaboration. The empirical results show that even leading proprietary models cannot successfully complete all tasks.
  2. The paper evaluates state-of-the-art models on the benchmark and reveals performance gaps among them.
Critical weaknesses
  1. Task difficulty does not necessarily imply greater demands on collaboration. Although the tasks are challenging, it is unclear whether their difficulty translates into meaningful challenges for multi-agent collaboration. The paper aims to evaluate collaboration in complex real-world settings, yet the difficulty of this benchmark appears to arise primarily from demanding reasoning problems, rather than from the need for effective communication or division of labor. As shown in Table 1, the single-agent (SA) baseline outperforms multi-agent baselines in several settings. This raises the possibility that performance depends more on the underlying models’ reasoning capabilities than on their collaboration strategies. The finding in Section 4.4 that “Teams cover the task better” is also consistent with improved coverage rather than improved reasoning quality on individual subtasks. Such gains could arise because distributing work reduces the risk of a single agent becoming stuck on one subtask, or because the team has access to more execution turns. Overall, the paper provides limited evidence that the benchmark captures the benefits of collaboration strategies themselves. For example, a more compelling demonstration would show that individually weaker agents can, through collaboration and discussion, achieve better reasoning outcomes than an agent working alone. As presented, the benchmark seems better suited to evaluating reasoning and knowledge capabilities, and its suitability for evaluating multi-agent collaboration remains insufficiently established.
  2. Potential data contamination undermines the reliability of model comparisons. Many tasks are drawn from publicly available competition materials spanning 2009–2026, which may already have been included in the pretraining corpora of the evaluated models. Some models may therefore have memorized reference answers. Differences in exposure to these materials could confound the results and compromise the fairness of comparisons across models.
Questions for authors
  1. Would smaller models benefit more from multi-agent collaboration? The evaluation covers several state-of-the-art models, but it remains unclear whether collaboration would yield larger gains for smaller models, such as those in the 7B–72B parameter range commonly studied in related work. The limited model selection restricts the breadth and generalizability of the paper’s conclusions.

  2. Could the analysis include quantitative process-level metrics? Sections 4.4 and 4.5 present several observations, but the explanations are largely descriptive. For example, the statement “For a team, therefore, additional turns can simply mean that different members are doing more useful work” is neither sufficiently precise nor convincingly supported. Quantitative analysis of process-level metrics would help substantiate these explanations and make the conclusions clearer and more persuasive.

5 comments
AuthorsOpenReview ↗

Author Response (Part 1)

Thank you for the insightful suggestions. We address all your weakness points and questions below.

For example, a more compelling demonstration would show that individually weaker agents can, through collaboration and discussion, achieve better reasoning outcomes than an agent working alone. (also Q1)

This is an insightful suggestion! To investigate whether smaller models achieve larger gains from collaboration, we further evaluate Qwen3-4B under the same experimental setting and compare its performance with larger models.

Model / MethodMathCompNatSciLawBizPhysInterModel.LangAvg.
Qwen3-4B / OTC22.05.370.956.227.431.753.516.051.037.1
Qwen3-4B / SA17.29.952.749.639.630.338.00.048.931.8
GLM-5.3 / OTC43.030.331.462.068.347.366.582.059.254.4
GLM-5.3 / SA31.727.526.483.782.555.068.042.548.051.7
DeepSeek-V3.1 / OTC24.926.429.454.373.540.068.277.067.251.2
DeepSeek-V3.1 / SA13.817.227.556.466.647.763.682.020.343.9
Kimi-K2.6 / OTC46.527.237.281.969.360.067.179.062.158.9
Kimi-K2.6 / SA33.822.433.365.470.428.768.042.868.248.1

As shown in the table, OTC improves Qwen3-4B from 31.8 to 37.1 (+5.3 points, +16.7% relative), demonstrating that collaboration can also benefit smaller LLMs. Compared with larger models, GLM-5.3 improves from 51.7 to 54.4 (+2.7 points, +5.2%), DeepSeek-V3.1 from 43.9 to 51.2 (+7.3 points, +16.6%), and Kimi-K2.6 from 48.1 to 58.9 (+10.8 points, +22.5%). Interestingly, Qwen3-4B achieves a larger relative improvement than GLM-5.3 and a comparable improvement to DeepSeek-V3.1. These results suggest that smaller models can benefit more from collaboration than some larger models. Kimi-K2.6 exhibits even larger gains. We speculate that this may be related to its training for multi-agent communication or swarm-based collaboration, potentially enabling more effective multi-agent coordination.

Task difficulty does not necessarily imply greater demands on collaboration.

We agree that task difficulty alone does not necessarily imply a need for collaboration. To better understand the demands of collaboration in ContestWorld, we conduct three in-depth and large scale analyses. First, we measure the widespread existence of collaboration by examining two types of interdependence: task interdependence (whether solving one subtask requires the result of another) and agent interdependence (whether one agent's execution relies on another agent's output). Inspired by the CCE metric [1], we construct dependency graphs from agent trajectories and propose metrics to quantify collaboration. Second, we evaluate the effectiveness of collaboration through budget-controlled experiments with non-communicating parallel agents and other baselines, demonstrating the importance of the collaboration in ContestWorld. Third, we analyze successful and unsuccessful collaboration chains and investigate their real contributions to task performance.

Part 1: Measuring Interdependence in Multi-Agent Collaboration

We further investigate how agents collaborate sequentially, beyond parallel task coverage, by examining task and agent interdependence in ContestWorld. Specifically, we carefully measure two types of interdependence: task interdependence, where solving subtask i requires the output of subtask j, showing the natural dependency of the sub-questions in ContestWorld, and agent interdependence, where Agent B's execution depends on the output of Agent A.

Following the CCE metric [1], we construct dependency graphs by tracing the contributions of individual actions and analyze the chain of inter-agent collaboration.


Agent Interdependence.

We find that 98.61% of trajectories contain cross-agent dependencies, and 70.83% contain collaboration chains involving at least three agents. These results demonstrate that inter-agent dependencies are widespread in ContestWorld. The most frequent dependency transitions include:

  • speak → speak: 98
  • work → review_answer: 98
  • speak → edit_work: 83
  • direct_message → work: 79
  • direct_message → direct_message: 75
  • work → work: 69

We further provide two representative examples of sequential agent interactions:

AuthorsOpenReview ↗

Author Response (Part 2)

Example 1: Long Agent Interaction Chain (CTF 2022 Matrix lab2).

Agent 1 initializes the coding environment and assigns tasks. Agent 3 unzips the packages. Agent 2 searches the unpacked directory, while Agent 1 lists the files. Agent 2 checks for decompilation tools, Agent 1 scans the files again, and Agent 4 examines bytecode headers and strings. 

Example 2: Sequential Agent Handoffs.

Agent 5 calculates the correct flag and writes it to a public draft. Agent 2 reviews and approves the same version, and Agent 1 submits the approved draft. This sequence contains three dependency edges and two agent switches, illustrating how agents sequentially contribute to the final submission.

Task Interdependence.

We further examine dependencies between contest problems, identifying cases where solving one problem requires results from another. The following table summarizes the dependency chains and their average lengths across domains.

MetricMathCompNatSciLawBizPhysInterModel.Lang
Verified Chains9238722318141469
Mean Length (Edges)1.571.821.262.222.392.431.793.333.67

These results indicate that task dependencies also exist across diverse contest domains, with longer dependency chains observed in Biz and Phys domains.

Together, the task-level and agent-level analyses provide evidence that ContestWorld contains meaningful opportunities for multi-agent collaboration beyond task coverage and selection.

Example 2: Sequential Agent Handoffs.

Agent 5 calculates the correct flag and writes it to a public draft. Agent 2 reviews and approves the same version, and Agent 1 submits the approved draft. This sequence contains three dependency edges and two agent switches, illustrating how agents sequentially contribute to the final submission.

Task Interdependence. We further examine dependencies between contest problems, identifying cases where solving one problem requires results from another. The following table summarizes the verified dependency chains and their average lengths across domains.

MetricMathCompNatSciLawBizPhysInterModel.Lang
Verified Chains9238722318141469
Mean Length (Edges)1.571.821.262.222.392.431.793.333.67

These results indicate that task dependencies also exist across diverse contest domains, with longer dependency chains observed in Biz and Phys domains. Together, the task-level and agent-level analyses demonstrate that ContestWorld involves and requires a number of sequential multi-agent collaboration beyond task coverage and selection.

Part 2: Evaluating the Effectiveness of Collaboration

To isolate the benefits of collaboration from task coverage and answer selection, we further conduct budget-controlled experiments with GPT-6 across nine domains. We compare OTC against several baselines, including independent parallel agents without communication, Best-of-N, self-consistency, and a controlled single-agent setting.

SettingMathCompNatSciLawBizPhysInterModel.LangAvg.
OTC69.646.672.183.576.663.572.783.260.469.8
Controlled SA 2×73.559.666.682.080.859.254.583.850.667.8
Parallel Agent35.748.263.981.376.464.354.578.047.861.1
Best-of-N48.543.772.284.277.268.854.577.048.463.8
Self-consistency37.543.573.980.776.566.072.273.061.365.0

As shown in the table, OTC achieves an average NT score of 69.8, substantially outperforming independent parallel agents (61.1, +8.7 points), Best-of-N (63.8, +6.0 points), and self-consistency (65.0, +4.8 points). Notably, OTC also outperforms the controlled single-agent setting with twice the budget of OTC (67.8, +2.0 points). The improvements are particularly pronounced in Mathematics (+33.9 points), Interdisciplinary (+18.2 points), and Language (+12.6 points) compared with independent parallel agents.

These results suggest that OTC's effectiveness cannot be fully explained by independent parallel execution or sampling-based answer selection alone. They provide additional evidence that structured inter-agent interactions offer benefits beyond those of non-communicating parallel baselines.

AuthorsOpenReview ↗

Author Response (Part 3)

Part 3: Analyzing Successful and Unsuccessful Collaboration Chains

We further investigate whether collaboration chains contribute to task performance. We manually verify 286 outcome chains, among which 224 (78%) contribute to scoring points, while 62 (22%) are associated with final answer errors.

TypeNumber of ChainsPercentage
Good22478%
Bad6222%

Interestingly, we observe a complementary pattern using the CCE metric from AgentWorld, which measures the proportion of agent actions that meaningfully contribute to task progress.

MethodCCEContributing AgentsAgent Nodes
Single Agent-0.91.0
Decentralized0.191.94.8
Centralized0.322.44.7
OTC0.552.83.9

As shown in the table, Decentralized and Centralized achieve CCE scores of only 0.19 and 0.32, respectively, compared with 0.39 for the single-agent baseline. In contrast, OTC achieves a CCE score of 0.55, with 2.8 contributing agents out of 3.9 agent nodes on average.

These findings also help explain why multi-agent methods sometimes fail to outperform single-agent baselines. The challenge is not necessarily that the tasks lack opportunities for collaboration, but that existing multi-agent systems often fail to collaborate effectively and efficiently. This distinction further motivates ContestWorld as a benchmark for evaluating the effectiveness of multi-agent collaboration.

To better understand the underlying mechanisms, we further examine representative successful and unsuccessful collaboration chains.

Good Chains. We identify two representative patterns of effective collaboration:

  1. Solving dependent problems using teammates' results. In the 2018 WMTC Junior round, Q12 explicitly requires the answer to Q8. Agent 5 solves Q8 and obtains 4. Agent 4 then uses this value to determine the side ratio of 4:1 in Q12 and calculates the area as 20 cm². Finally, Agent 2 verifies the calculation and submits the correct answer.

  2. Incorporating teammates' feedback to improve solutions. In IIOT 2021's All the Emojis!, Agent 2 identifies that Agent 1's initial solution could consume approximately 1.7 GB of memory, exceeding the 256 MB limit. Agent 4 proposes finding all code matches first and counting valid decodings afterward. Building on this suggestion, Agent 1 redesigns the solution to allocate only the necessary memory and store match lengths, reducing the estimated memory consumption of the main data structures to 48 MB. Agent 2 verifies that the revised solution correctly handles overlapping matches, and Agent 4 submits the code, achieving 100/100 in the benchmark's code review.

Bad Chains. We also identify two representative failure patterns:

  1. Failure to preserve cross-problem dependencies and contest constraints. In the WMTC Advanced round, Q13 explicitly requires the value of T from Q5. However, Agent 3 incorrectly uses T = 50, derived from Q2, and obtains an answer of 75, which is subsequently submitted. A similar constraint violation occurs in the Jessup competition: although the applicant requests a declaration that the denial of consular access did not violate international law, the team instead drafts and submits a document requesting a declaration that it did violate international law, even after the review process.

  2. Failure to act on teammates' error feedback. In 91 runs (12%), an agent correctly identifies a problem, but the team fails to address it. For example, in an ARML problem, two agents reject the same answer four times for the same reason, yet the issue remains unresolved. Similarly, in an HMMT problem where the correct answer is 7, Agent 7 proposes 6, and Agent 1 subsequently accepts the incorrect answer during the review process.

Overall, these three analyses demonstrate that ContestWorld involves substantial task and agent interdependence, while effective collaboration remains challenging for existing multi-agent systems. The performance gains from collaboration, together with the observed failures in ineffective coordination, highlight the importance of evaluating not only task outcomes but also how agents collaborate to achieve them.

Reviewer_9H6uOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ContestWorld to test multi agent collaboration using 321 contests from 39 categories across nine domains.

Critical strengths

The benchwork is very exhaustive, spanning math, programming, science and other domains. The scale and diversity of the collection are significant strengths. The paper also has realistic task constraints and communication protocols which are potentially a good addition to the community for evaluating multi agents

Critical weaknesses

My primary concern with the paper is that there is limited analytical depth especially given the scale of the benchmark.After going through the paper, relatively little is known about why teams fail. Like 4.3 samples 100 trajectories and Table 2 reports 44 issues, from one protocol, one annotator, no agreement statistics, and nothing per domain or per model. Given the diversity of the benchmark, we do not get useful information on where LLMs lack, why these behaviours arise, and which conditions (team size, budget, deliverable type, judge type) affect them. There are other failure taxonomy for multi-agent systems like Cemri et al., “Why Do Multi-Agent LLM Systems Fail?”, NeurIPS 2025. They are not cited or used or such similar analysis is largely lacking.

Many tasks do not really require collaboration, and the paper never tests whether they do, unless I missed it. Table 7 labels tracks with “parallel problem solving”, “task allocation”, “parallel solution design”, i.e. plainly parallel. Appendix A.2 also shows that the collaborative part of the original contests is mostly stripped out: no opposition in debates, etc. What remains is a problem set with a shared answer sheet. The authors could characterize tasks by interdependence (does solving problem i require output from problem j, or a role another agent holds) so we can clearly see where performance comes from and where agents actually lack.

Going more into that makes the eval break further. The team’s advantage is not compute-normalized or cost-controlled. Multi-agent teams consume significantly more inference tokens and compute due to message history, memory scratchpads, and peer drafts. Without an N independent agents baseline with no communication, “teams cover the task better” may just be having multiple independent search threads rather than collaboration. Table 3 then says what one expects: SA zero score runs have 24.1% coverage because a single agent gets stuck on early problems and runs out of turns. This is a major confound for the coverage finding and for the NT gains. A fair baseline is cost-controlled SA, or best-of-N / self-consistency, or N independent agents with no communication. Without it, things in Table 1 cannot be attributed to collaboration. I think Kapoor et al., “AI Agents That Matter”, has a cost-controlled agent evaluation for reference.

Multi-agent coordination is not reliably better than one agent, and the framing doesnt make this clear. Decentralized is below SA on three of four models, centralized is below SA on DeepSeek, and the only consistent gains come from OTC, which bundles coach, memory and a mandatory non-author review gate. So another read is that plain teams do not help and a review gate plus memory does. This also matters for Section 4.4’s “the coach makes the team more selective”: I didnt see review gate ablated, so the selectivity is as plausibly from mandatory peer approval as from the coach.

The paper needs and can really use more analysis and ablations.

Questions for authors

How many of the 100 sampled trajectories had no identified issue, and what is the inter-annotator agreement on Table 2? Also what temperature was used, and were any runs repeated?

6 comments
AuthorsOpenReview ↗

Author Response (Part 1)

Thank you for the insightful suggestions. We address all your weakness points and questions below.

(The current analysis) relies on one protocol and one annotator, with no agreement statistics or breakdowns by domain or model. Given the diversity of the benchmark, it provides limited insight into where LLMs fail, why these behaviors arise, and which conditions (team size, budget, deliverable type, and judge type) affect them. There are other failure taxonomy for multi-agent systems like Cemri et al., “Why Do Multi-Agent LLM Systems Fail?”, NeurIPS 2025. They are not cited or used or such similar analysis is largely lacking.

To provide a more comprehensive understanding of collaboration failures, we carefully examine 703 traces across all models, nine task domains, and different multi-agent methods, using the same experienced annotator as in our NT evaluation. We identify four representative failure types covering different stages of multi-agent collaboration. Following MAST, we categorize F1 as team-level failures, F2 as inter-agent failures, and F3 as verification or submission failures.

F1 — Incorrect Role or Work Allocation. The team fails to follow the assigned task allocation, resulting in duplicated work or neglected tasks. For example, in the WUDC debate competition, Agent 1 was assigned to write the PM speech, while Agent 2 was responsible for the DPM speech. However, Agent 1 subsequently wrote both speeches, while Agent 2 only reviewed the drafts.

F2a — Missed or Ignored Peer Input. An agent overlooks messages from teammates or fails to incorporate their feedback. For example, in an IIOT coding problem, Agent 3 used max(time1, time2) to calculate the execution time of two subtrees, which Agent 1 explicitly rejected. However, Agent 3 continued using the same code in subsequent turns without addressing Agent 1's feedback.

F2b — Misalignment Between Words and Actions. An agent deviates from the agreed task or takes actions inconsistent with its stated decisions. For example, in an ARML mathematics competition, Agent 2 submitted a solution for Question 2, and Agent 3 explicitly stated in its review that "the draft's answer of 1/3 is incorrect." However, within the same review turn, Agent 3 ultimately approved the incorrect answer.

F3 — Failed Team Verification or Premature Submission. The team fails to conduct proper verification or submits a draft before resolving inconsistencies. For example, in the APPE Intercollegiate Ethics Bowl, Agent 2 prepared a four-section draft, including a Q&A section containing only three question-answer pairs. During the review, Agent 1 incorrectly claimed that the draft included four Q&A pairs and even described a fourth question about how lawmakers could reach an agreement, which was absent from the actual document. Despite this inconsistency, Agent 1 submitted the draft without modification.


Model-Level Analysis

ModelTotal Error LabelsF1 AllocationF2a Missed InputF2b Action MismatchF3 Verification
Claude Sonnet 4.65632.2%48.2%10.7%8.9%
DeepSeek V3.11915.8%57.9%15.8%10.5%
Qwen3.6-35B-A3B2313.1%26.1%30.4%30.4%
GPT-6-Luna128.3%50.0%8.3%33.4%
GLM-5.3825.0%25.0%25.0%25.0%
Kimi-K2.62317.4%26.1%26.1%30.4%
Qwen3-4B-Instruct238.7%43.5%21.7%26.1%
Qwen3-30B-Instruct2516.0%44.0%4.0%36.0%
Overall18919.6%41.8%16.4%22.2%

As shown in the table, F2a (Missed or Ignored Peer Input) is the most prevalent collaboration failure, accounting for 41.8% of all error labels. This reveals a fundamental challenge in multi-agent collaboration: agents frequently fail to incorporate information and feedback from their teammates. F3 (Verification Failures) is the second most common error type (22.2%), suggesting that teams also struggle to verify intermediate results and reach agreement before final submission.

Interestingly, F2a becomes more dominant failure type even among stronger models, such as GPT-6-Luna (50.0%) and Claude Sonnet 4.6 (48.2%). This suggests that stronger individual capabilities do not necessarily make them ignore peers more. These findings highlight the importance of improving inter-agent communication, feedback integration, and team-level verification.


AuthorsOpenReview ↗

Author Response (Part 2)

Domain-Level Analysis

DomainTotal Error LabelsF1 AllocationF2a Missed InputF2b Action MismatchF3 Verification
Mathematics355.7%34.3%28.6%31.4%
Computing4914.3%40.8%24.5%20.4%
Natural Sciences3821.0%60.5%5.3%13.2%
Law2133.3%28.6%9.5%28.6%
Business1020.0%60.0%0.0%20.0%
Physics825.0%37.5%12.5%25.0%
Interdisciplinary1241.7%41.7%16.6%0.0%
Modeling812.5%25.0%0.0%62.5%
Language837.5%25.0%25.0%12.5%
Overall18919.6%41.8%16.4%22.2%

Across different domains, we observe distinct patterns of collaboration failures. F2a is the most prevalent failure type across most domains, particularly in Natural Sciences (60.5%) and Business (60.0%), highlighting the difficulty of effectively incorporating information from science and business teammates. In contrast, F1 is more prominent in open-ended domains, such as Law (33.3%), Language (37.5%), and Interdisciplinary tasks (41.7%) because in open-ended domains, the responsibilities and task boundaries may be harder to define.


Influence of Team Size

Team SizeTotal Error LabelsF1 AllocationF2a Missed InputF2b Action MismatchF3 Verification
2–49124.2%43.9%14.3%17.6%
5–77213.9%41.7%18.0%26.4%
8+2619.2%34.6%19.2%27.0%
Overall18919.6%41.8%16.4%22.2%

Interestingly, as team size increases, F2a becomes less prevalent, while F2b and F3 account for larger proportions of collaboration failures. F1 also remains less prevalent in larger teams than in the smallest group. This suggests a potential trade-off in scaling multi-agent collaboration: larger teams may benefit from increased information sharing and peer feedback, but also introduce greater coordination and verification complexity.

In particular, the increasing volume of inter-agent communication may make it harder to maintain consistency between decisions and actions (F2b) and ensure that all necessary checks are completed before submission (F3). These findings suggest that scaling multi-agent systems requires not only effective information sharing but also stronger mechanisms for maintaining action consistency and reliable team-level verification.


Influence of Evaluation Type

Evaluation TypeTotal Error LabelsF1 AllocationF2a Missed InputF2b Action MismatchF3 Verification
Rule-based Eval9716.5%45.4%17.5%20.6%
LLM Rubric Eval9222.8%38.1%15.2%23.9%
Overall18919.6%41.8%16.4%22.2%

Across the two evaluation types, we observe distinct patterns of collaboration failures. Compared with rule-based contests, rubric-based contests exhibit higher proportions of F1 (22.8% vs. 16.5%) and F3 (23.9% vs. 20.6%) errors. This may be attributed to the open-ended nature of rubric-based tasks, such as debates, where task responsibilities are more difficult to define and final outputs require more subjective evaluation and team-level verification.

In contrast, rule-based contests exhibit a higher proportion of F2a errors (45.4% vs. 38.1%), suggesting that agents may struggle more with incorporating precise feedback and intermediate results from teammates in tasks with well-defined objectives. These findings further highlight how different question types characteristics introduce distinct challenges for multi-agent collaboration.

AuthorsOpenReview ↗

Author Response (Part 3)

A fair baseline is cost-controlled SA, or best-of-N / self-consistency, or N independent agents with no communication.

SettingMathCompNatSciLawBizPhysInterModel.LangAvg.
OTC69.646.672.183.576.663.572.783.260.469.8
Controlled SA73.559.666.682.080.859.254.583.850.667.8
Parallel Agent35.748.263.981.376.464.354.578.047.861.1
Best-of-N48.543.772.284.277.268.854.577.048.463.8
Self-consistency37.543.573.980.776.566.072.273.061.365.0

To better isolate the benefits of collaboration from task coverage and answer selection, we further conduct budget-controlled experiments with GPT-6, comparing OTC against several baselines, including independent parallel agents without communication, Best-of-N, self-consistency, and a controlled single-agent setting.

As shown in the table, OTC achieves an average score of 69.8, outperforming independent parallel agents (61.1, +8.7 points), Best-of-N (63.8, +6.0 points), and self-consistency (65.0, +4.8 points). Notably, OTC also outperforms the controlled single-agent setting with twice the budget of OTC (67.8, +2.0 points). The improvements are particularly pronounced in Math (+33.9 points), Interdisciplinary (+18.2 points), and Language (+12.6 points) compared with independent parallel agents.

These results suggest that OTC's effectiveness cannot be fully explained by independent parallel execution or sampling-based answer selection alone. We provide strong evidence that inter-agent interactions offer benefits beyond those of non-communicating parallel baselines.

To understand how agents collaborate beyond parallel running, we further present examples of sequential collaboration chains in which agents contribute to task completion through sequential interactions. For instance:

Example 1: Long Collaboration Chain

Agent 1 initializes the coding environment and assigns tasks.
Agent 3 unzips the packages.
Agent 2 searches the unpacked directory.
Agent 1 lists the files.
Agent 2 checks for decompilation tools.
Agent 1 scans the files again.
Agent 4 examines bytecode headers and strings.

This chain contains 7 steps with 4 different agents.

Example 2: Agent Switching

Agent 5 calculates the correct flag and writes it to a public draft.
Agent 2 reviews and approves the same draft.
Agent 1 submits the approved version.

Agent 2 approves the exact draft subsequently submitted by Agent 1. 

Agent 2 carefully checks the submissions to ensure the correctness.

The authors could characterize tasks by interdependence (does solving problem i require output from problem j, or a role another agent holds) so we can clearly see where performance comes from and where agents actually lack.

This is a good idea! We further investigate how agents collaborate sequentially, beyond parallel task coverage, by examining task and agent interdependence in ContestWorld. Specifically, we carefully measure two types of interdependence: task interdependence, where solving subtask i requires the output of subtask j, showing the natural dependency of the sub-questions in ContestWorld, and agent interdependence, where Agent B's execution depends on the output of Agent A.

Following the CCE metric [1], we construct dependency graphs by tracing the contributions of individual actions and analyze the chain of inter-agent collaboration.


Agent Interdependence.

We find that 98.61% of trajectories contain cross-agent dependencies, and 70.83% contain collaboration chains involving at least three agents. These results demonstrate that inter-agent dependencies are widespread in ContestWorld. The most frequent dependency transitions include:

  • speak → speak: 98
  • work → review_answer: 98
  • speak → edit_work: 83
  • direct_message → work: 79
  • direct_message → direct_message: 75
  • work → work: 69

We further provide two representative examples of sequential agent interactions:

Example 1: Long Agent Interaction Chain (CTF 2022 Matrix lab2).

Agent 1 initializes the coding environment and assigns tasks. Agent 3 unzips the packages. Agent 2 searches the unpacked directory, while Agent 1 lists the files. Agent 2 checks for decompilation tools, Agent 1 scans the files again, and Agent 4 examines bytecode headers and strings. 

Example 2: Sequential Agent Handoffs.

Agent 5 calculates the correct flag and writes it to a public draft. Agent 2 reviews and approves the same version, and Agent 1 submits the approved draft. This sequence contains three dependency edges and two agent switches, illustrating how agents sequentially contribute to the final submission.
Reviewer_6gobOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ContestWorld, a benchmark for long-horizon multi-agent collaboration based on 321 real-world team contests across nine domains. It evaluates different collaboration strategies and identifies task coverage, verification, and feedback integration as key factors affecting performance.

Critical strengths
  • Diverse and challenging benchmark. ContestWorld covers a broad range of realistic contest tasks with team coordination and resource constraints.

  • Useful empirical findings. The analysis reveals meaningful limitations in multi-agent coordination, particularly verification failures and poor feedback integration.

Critical weaknesses
  • Limited isolation of genuine collaboration. The observed multi-agent gains mainly come from better task coverage and selection, rather than demonstrated collaborative reasoning. Comparisons with independent parallel agents under matched budgets are needed.

  • Limited realism of simulated contests. Many real-world contests are reduced to static written tasks without physical execution or live interaction, potentially weakening the benchmark's validity.

  • Insufficient evaluation validation. The LLM-based evaluation is validated using only 14 trajectories and one human annotator, which is limited given the diversity of tasks.

Questions for authors
  • Can the authors compare multi-agent collaboration against independent parallel agents with the same total inference budget?

  • How reliably do the benchmark scores reflect actual contest performance, especially for tasks requiring physical execution or interactive competition?

3 comments
AuthorsOpenReview ↗

Author Response (Part 1)

Thank you for the thoughtful suggestions. We address all your weakness points and questions below.

Comparisons with independent parallel agents under matched budgets are needed. (also Q1)

We further conduct budget-controlled experiments with GPT-6, comparing OTC against several baselines, including independent parallel agents without communication, Best-of-N, self-consistency, and a budget-matched single-agent setting.

SettingMathCompNatSciLawBizPhysInterModel.LangAvg.
OTC69.646.672.183.576.663.572.783.260.469.8
Controlled SA73.559.666.682.080.859.254.583.850.667.8
Parallel Agent35.748.263.981.376.464.354.578.047.861.1
Best-of-N48.543.772.284.277.268.854.577.048.463.8
Self-consistency37.543.573.980.776.566.072.273.061.365.0

As shown in the table, OTC achieves an average score of 69.8, outperforming independent parallel agents (61.1, +8.7 points), Best-of-N (63.8, +6.0 points), and self-consistency (65.0, +4.8 points). Notably, OTC also outperforms the controlled single-agent setting with twice the budget of OTC (67.8, +2.0 points). The improvements are particularly pronounced in Math (+33.9 points), Interdisciplinary (+18.2 points), and Language (+12.6 points) compared with independent parallel agents.

These results suggest that OTC's effectiveness is not merely from independent parallel execution or sampling-based answer selection alone, demonstrating that that collaborative reasoning is needed for ContestWorld to achieve a better performance.

Limited isolation of genuine collaboration. The observed multi-agent gains mainly come from better task coverage and selection, rather than demonstrated collaborative reasoning.

In addition to the experiments presented in our previous response, we further investigate how agents collaborate sequentially, beyond parallel task coverage, by examining task and agent interdependence in ContestWorld. Specifically, we carefully measure two types of interdependence: task interdependence, where solving subtask i requires the output of subtask j, showing the natural dependency of the sub-questions in ContestWorld, and agent interdependence, where Agent B's execution depends on the output of Agent A.

Following the CCE metric [1], we construct dependency graphs by tracing the contributions of individual actions and analyze the chain of inter-agent collaboration.


Agent Interdependence.

We find that 98.61% of trajectories contain cross-agent dependencies, and 70.83% contain collaboration chains involving at least three agents. These results demonstrate that inter-agent dependencies are widespread in ContestWorld. The most frequent dependency transitions include:

  • speak → speak: 98
  • work → review_answer: 98
  • speak → edit_work: 83
  • direct_message → work: 79
  • direct_message → direct_message: 75
  • work → work: 69

We further provide two representative examples of sequential agent interactions:

Example 1: Long Agent Interaction Chain (CTF 2022 Matrix lab2).

Agent 1 initializes the coding environment and assigns tasks. Agent 3 unzips the packages. Agent 2 searches the unpacked directory, while Agent 1 lists the files. Agent 2 checks for decompilation tools, Agent 1 scans the files again, and Agent 4 examines bytecode headers and strings. 

Example 2: Sequential Agent Handoffs.

Agent 5 calculates the correct flag and writes it to a public draft. Agent 2 reviews and approves the same version, and Agent 1 submits the approved draft. This sequence contains three dependency edges and two agent switches, illustrating how agents sequentially contribute to the final submission.

Task Interdependence.

We further examine dependencies between contest problems, identifying cases where solving one problem requires results from another. The following table summarizes the dependency chains and their average lengths across domains.

MetricMathCompNatSciLawBizPhysInterModel.Lang
Verified Chains9238722318141469
Mean Length (Edges)1.571.821.262.222.392.431.793.333.67

These results indicate that task dependencies also exist across diverse contest domains, with longer dependency chains observed in Biz and Phys domains.

Together, the task-level and agent-level analyses demonstrate that ContestWorld contains and requires a number of sequential multi-agent collaboration beyond task coverage and selection.

[1] AgentWorld: Benchmarking Long-Horizon Collaboration of Multi-agent LLMs

AuthorsOpenReview ↗

Author Response (Part 2)

Insufficient evaluation validation. The LLM-based evaluation is validated using only 14 trajectories and one human annotator, which is limited given the diversity of tasks.

We further conduct a large-scale human annotation on the NT scores, covering all 7 models, with all 28 competitions that use NT scores (the rest of competitions are rule-based). We first hire 3 annotators with computer science background and conduct a pilot test with 5 contest traces and let them annotate the original score and then we compute the NT scores.

In the primary analysis of five submissions scored by the same three annotators, Krippendorff's α was 0.8214 (95% CI [0.3083, 0.9224]). The point estimates indicate a significantly high agreement in the rating among annotators. Then we conduct the full annotation and compute the correlation between the human annotated score and LLM-as-a-judge.

Spearman ρLinearly weighted κQuadratically weighted κMAE
0–100 (NT)0.71780.58730.822612.3114
0–50.72150.60720.80440.5767
0–1 float0.71780.58730.82260.1231

As shown in the table, on the NT score (0–100 scale), results show that Spearman’s ρ = 0.7178 with a mean absolute error of 12.3114 points. The generalized weighted κ was 0.5873 with linear weights and 0.8226 with quadratic weights. The two sets of scores showed a highly positive rank correlation, demonstrating the effectiveness of the LLM-as-a-judge. Furthermore, the rank correlation was similar on the discrete 0–5 scale (ρ = 0.7215) and 0-1 float (ρ = 71.78), demonstrating our choice of normalization is also acceptable.

All the scores are graded by GPT-6-Luna, and to test whether our scores depend on the single judge model, we ran a two-judge alignment experiment on Inter-judge reliability. Every model trace was scored independently by two judges: Claude-Sonnet-4.6 and GPT-6-Luna and computed the NT score separately. Over 1,090 traces across seven models, the judges agree strongly (Pearson r = 0.865, Spearman ρ = 0.868, quadratic-weighted κ = 0.826), supporting the reliability and robustness of our LLM evaluation.

Limited realism of simulated contests. Many real-world contests are reduced to static written tasks without physical execution or live interaction, potentially weakening the benchmark's validity. (also Q2)

Simplifying physical execution and live interaction does not undermine the validity of ContestWorld, for two reasons.

First, these abstractions preserve the core problem-solving requirements of the contests. For competitions involving physical execution, we evaluate model-generated plans against reference solutions or using LLM-as-a-judge, allowing us to assess the quality of the proposed solutions without requiring physical execution. For live competitions, solving the tasks generally does not depend on opponents completing their tasks, and live interaction primarily contributes to the competitive atmosphere rather than the underlying problem-solving requirements. Second, our human annotation results provide empirical support for the validity of the automated evaluation. As discussed in our previous response, on the NT scale (0–100), automated scores achieve a Spearman correlation of ρ = 0.7178 with human judgments, demonstrating a strong positive rank correlation between human and automated evaluations, supporting the reliability of our LLM-as-a-judge evaluation despite the simplifications.

On the other hand, the simplified the design can reduce the confounding factors introduced by adversarial settings, or unstable live, allowing us to more clearly observe, isolate, and evaluate multi-agent collaboration.

Dear Reviewer,

As the discussion period is approaching its end, we would like to kindly follow up on our rebuttal. We hope our responses have addressed your concerns. We would greatly appreciate hearing your thoughts.

Thank you again for your time and consideration.

Authors

Reviewer_inZROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that LLM multi-agent collaboration is rarely tested on realistic tasks, since existing benchmarks favour games, narrow pseudo-tasks or individual competition problems. CONTESTWORLD converts human team competitions—from ARML and ICPC to moot courts and MCM—into agent tasks, reported as 321 contests in 39 tracks and nine domains. Contests run in a turn-based simulator described as closely aligned with real-world settings: at most 90 turns, one rule-filtered action per agent turn, and scoring by gold answers, a programming judge, a rubric-conditioned LLM judge or a deck rubric. The authors compare a single agent, decentralized and centralized teams, and Open Table Coach (OTC: a problem-blind coach brief, persistent memory and peer approval) on four open-weight backbones, plus GPT-6 and Claude Sonnet 4.6 under OTC only, using a normalized task score (NT) and an LLM-judged collaboration score (CS).

Critical strengths

The task pool is unusually broad for a multi-agent benchmark, adding moot courts, ethics bowls, equity research, modeling and linguistics to mathematics and programming, with teams of 2 to 15. Deliverables range from answer sheets and proof packets to memorials, slide decks, LaTeX modeling papers and CTF flags.

Critical weaknesses
  1. My primary concern is the practical value of this benchmark. Harnesses such as Claude Code and Codex are already standard and support multi-agent and sub-agent workflows, yet none is evaluated: all protocols run in the authors' simulator, non-OTC agents "retain only short-term memory of the previous turn" (Sec. 4.1), and the frontier models run only under OTC. Would OTC's lead over the single agent survive if that agent were such a harness with its own sub-agents, and would OTC's coach brief, memory and approval gate improve these harnesses?

  2. Most of NT comes from an unnamed LLM judge with thin validation: of the 31 session-level tracks in App. D (321 sessions), 25 (267 sessions) are assigned rubric_llm_v1 and only five (49 sessions) gold-answer matching. Sec. 4.1 names GPT-6 only as the CS judge, and the NT judge's model and prompt are never given. The sole validation (Sec. 3.5) uses "14 contest traces generated by GPT and Claude" and one annotator, and reports an MAE "on the 1–5 scale", which matches the CS ratings rather than NT's point-based rubrics (e.g., DebateBench, "separately out of 100, total 200"). Whether NT itself was validated is unclear, and no open-model trace (16 of 18 Table 1 rows) is included. I therefore cannot determine whether the Table 1 differences reflect task performance rather than the scoring behaviour of an unidentified judge.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Author Response

Thank you for the valuable suggestions. We provide answers to all your weakness points and questions below.

Would OTC's lead over the single agent survive if that agent were such a harness with its own sub-agents, and would OTC's coach brief, memory and approval gate improve these harnesses?

Yes. Our additional experiments show that OTC maintains its performance advantage even when compared against strong agent harnesses. Specifically, we evaluate the Codex harness under three configurations within the ContestWorld framework: (1) Codex Single, where a single agent operates with the Codex harness; (2) Subagent Codex, where multiple Codex harness instances operate within the framework; and (3) OTC + Codex, which integrates OTC's coordination mechanisms with the Codex harness.

SettingsMathCompNatSciLawBizPhysInterModelLangAvg.
Codex Single45.178.746.385.853.463.345.586.546.761.3
Subagent Codex48.570.278.874.829.861.590.976.046.564.1
OTC + Codex57.776.381.178.861.265.790.978.044.870.5

As shown in the table, OTC + Codex achieves the highest overall performance among the three configurations, with an average score of 70.5, outperforming Subagent Codex (64.1) and Codex Single (61.3) by 6.4 and 9.2 points, respectively. These results reveal several important findings. First, OTC's advantage persists even with a state-of-the-art agent harness. Compared with Subagent Codex, OTC + Codex achieves significant improvements. This suggests that OTC provides additional benefits beyond enabling multi-agent execution through an advanced harness. We further analyze the importance of each components, and found the coach plays a more important role than review gates, while the effectiveness of the memory becomes smaller due to that the harness provides duplicated harness machismis. Second, multi-agent collaboration remains beneficial with harness. Subagent Codex outperforms Codex Single, demonstrating the potential benefits of multi-agent collaboration. However, the further gains achieved by OTC indicate that simply adding subagents is insufficient to fully realize this potential. Third, ContestWorld remains challenging even with advanced agent harnesses. Despite the improvements achieved by OTC + Codex, its overall score reaches only 70.5, with particularly limited performance in Language (44.8), Mathematics (57.7), and Business (61.2), indicating the multi-agent collaboration still have considerable room for improvement.

Whether NT itself was validated is unclear, and no open-model trace (16 of 18 Table 1 rows) is included.

To assess how closely NT scores align with human judgments, we further conduct a large-scale human annotation covering all five open-source models and two closed-source models across all 28 contests with LLM-based judging (the remaining contests use rule-based evaluation).

We first recruit three annotators with computer science backgrounds and conduct a pilot study on five contest traces. The annotators independently assign scores, which are subsequently normalized into NT scores. In this pilot study, Krippendorff's α reaches 0.8214 (95% CI [0.3083, 0.9224]), indicating high inter-annotator agreement in the point estimate. We then conduct the full annotation study and measure the correlation between human judgments and LLM-as-a-judge scores.

ScaleSpearman ρLinearly Weighted κQuadratically Weighted κMAE
0–100 (NT)0.71780.58730.822612.3114
0–50.72150.60720.80440.5767
0–1 float0.71780.58730.82260.1231

As shown in the table, on the NT scale (0–100), the LLM-based scores achieve a Spearman correlation of 0.7178 with human judgments, with a mean absolute error of 12.3114 points. The linearly and quadratically weighted κ values reach 0.5873 and 0.8226, respectively. These results demonstrate a strong positive rank correlation between human and automated evaluations, supporting the effectiveness of LLM-as-a-judge. Furthermore, similar rank correlations are observed on the discrete 0–5 scale (ρ = 0.7215) and normalized 0–1 scale (ρ = 0.7178), suggesting that our normalization does not substantially affect the ranking consistency.

Since our NT scores in the paper are evaluated by GPT-6-Luna, we further investigate whether the evaluation depends on the choice of judge model. Specifically, we uses two judges, Claude-Sonnet-4.6 and GPT-6-Luna, and compute their NT scores separately. Across 1,090 traces from seven models, the two judges exhibit strong agreement (Pearson r = 0.865, Spearman ρ = 0.868, and quadratically weighted κ = 0.826), further supporting the inter-judge reliability and robustness of our LLM evaluation.

Dear Reviewer,

As the discussion period is approaching its end, we would like to kindly follow up on our rebuttal. We hope our responses have addressed your concerns. We would greatly appreciate hearing your thoughts.

Thank you again for your time and consideration.

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision of PDF

Dear Reviewers,

We sincerely thank you for your constructive feedback and insightful suggestions. We have uploaded a revised version of our paper, with all changes highlighted in blue for clarity.

We have carefully addressed all reviewer comments and included detailed point-by-point responses in the appendix. We have also revised the main paper to address several common concerns, including human annotation, validation of the collaboration evaluation, in-depth analyses, and additional experiments across harness and small model settings.

We greatly appreciate your time, effort, and thoughtful feedback, which have helped us improve the quality and clarity of our work.

Sincerely, All Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.