- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a benchmark comprising 321 competitions across 39 competition categories and nine domains to evaluate multi-agent collaboration under competition-specific rules. The benchmark features challenging tasks, many at the Olympiad level, diverse domains, support for multiple collaboration protocols, and a relatively comprehensive evaluation framework that includes reference answers and LLM-based judges. The experimental analysis suggests that multi-agent systems do not consistently outperform single-agent systems, that their benefits primarily come from broader task coverage, and that memory modules can improve collaborative performance.
Critical strengths
- The benchmark offers a more challenging task collection than most existing benchmarks for multi-agent collaboration. The empirical results show that even leading proprietary models cannot successfully complete all tasks.
- The paper evaluates state-of-the-art models on the benchmark and reveals performance gaps among them.
Critical weaknesses
- Task difficulty does not necessarily imply greater demands on collaboration. Although the tasks are challenging, it is unclear whether their difficulty translates into meaningful challenges for multi-agent collaboration. The paper aims to evaluate collaboration in complex real-world settings, yet the difficulty of this benchmark appears to arise primarily from demanding reasoning problems, rather than from the need for effective communication or division of labor. As shown in Table 1, the single-agent (SA) baseline outperforms multi-agent baselines in several settings. This raises the possibility that performance depends more on the underlying models’ reasoning capabilities than on their collaboration strategies. The finding in Section 4.4 that “Teams cover the task better” is also consistent with improved coverage rather than improved reasoning quality on individual subtasks. Such gains could arise because distributing work reduces the risk of a single agent becoming stuck on one subtask, or because the team has access to more execution turns. Overall, the paper provides limited evidence that the benchmark captures the benefits of collaboration strategies themselves. For example, a more compelling demonstration would show that individually weaker agents can, through collaboration and discussion, achieve better reasoning outcomes than an agent working alone. As presented, the benchmark seems better suited to evaluating reasoning and knowledge capabilities, and its suitability for evaluating multi-agent collaboration remains insufficiently established.
- Potential data contamination undermines the reliability of model comparisons. Many tasks are drawn from publicly available competition materials spanning 2009–2026, which may already have been included in the pretraining corpora of the evaluated models. Some models may therefore have memorized reference answers. Differences in exposure to these materials could confound the results and compromise the fairness of comparisons across models.
Questions for authors
-
Would smaller models benefit more from multi-agent collaboration? The evaluation covers several state-of-the-art models, but it remains unclear whether collaboration would yield larger gains for smaller models, such as those in the 7B–72B parameter range commonly studied in related work. The limited model selection restricts the breadth and generalizability of the paper’s conclusions.
-
Could the analysis include quantitative process-level metrics? Sections 4.4 and 4.5 present several observations, but the explanations are largely descriptive. For example, the statement “For a team, therefore, additional turns can simply mean that different members are doing more useful work” is neither sufficiently precise nor convincingly supported. Quantitative analysis of process-level metrics would help substantiate these explanations and make the conclusions clearer and more persuasive.
5 comments
Author Response (Part 2)
Example 1: Long Agent Interaction Chain (CTF 2022 Matrix lab2).
Agent 1 initializes the coding environment and assigns tasks. Agent 3 unzips the packages. Agent 2 searches the unpacked directory, while Agent 1 lists the files. Agent 2 checks for decompilation tools, Agent 1 scans the files again, and Agent 4 examines bytecode headers and strings.
Example 2: Sequential Agent Handoffs.
Agent 5 calculates the correct flag and writes it to a public draft. Agent 2 reviews and approves the same version, and Agent 1 submits the approved draft. This sequence contains three dependency edges and two agent switches, illustrating how agents sequentially contribute to the final submission.
Task Interdependence.
We further examine dependencies between contest problems, identifying cases where solving one problem requires results from another. The following table summarizes the dependency chains and their average lengths across domains.
| Metric | Math | Comp | NatSci | Law | Biz | Phys | Inter | Model. | Lang |
|---|---|---|---|---|---|---|---|---|---|
| Verified Chains | 92 | 38 | 72 | 23 | 18 | 14 | 14 | 6 | 9 |
| Mean Length (Edges) | 1.57 | 1.82 | 1.26 | 2.22 | 2.39 | 2.43 | 1.79 | 3.33 | 3.67 |
These results indicate that task dependencies also exist across diverse contest domains, with longer dependency chains observed in Biz and Phys domains.
Together, the task-level and agent-level analyses provide evidence that ContestWorld contains meaningful opportunities for multi-agent collaboration beyond task coverage and selection.
Example 2: Sequential Agent Handoffs.
Agent 5 calculates the correct flag and writes it to a public draft. Agent 2 reviews and approves the same version, and Agent 1 submits the approved draft. This sequence contains three dependency edges and two agent switches, illustrating how agents sequentially contribute to the final submission.
Task Interdependence. We further examine dependencies between contest problems, identifying cases where solving one problem requires results from another. The following table summarizes the verified dependency chains and their average lengths across domains.
| Metric | Math | Comp | NatSci | Law | Biz | Phys | Inter | Model. | Lang |
|---|---|---|---|---|---|---|---|---|---|
| Verified Chains | 92 | 38 | 72 | 23 | 18 | 14 | 14 | 6 | 9 |
| Mean Length (Edges) | 1.57 | 1.82 | 1.26 | 2.22 | 2.39 | 2.43 | 1.79 | 3.33 | 3.67 |
These results indicate that task dependencies also exist across diverse contest domains, with longer dependency chains observed in Biz and Phys domains. Together, the task-level and agent-level analyses demonstrate that ContestWorld involves and requires a number of sequential multi-agent collaboration beyond task coverage and selection.
Part 2: Evaluating the Effectiveness of Collaboration
To isolate the benefits of collaboration from task coverage and answer selection, we further conduct budget-controlled experiments with GPT-6 across nine domains. We compare OTC against several baselines, including independent parallel agents without communication, Best-of-N, self-consistency, and a controlled single-agent setting.
| Setting | Math | Comp | NatSci | Law | Biz | Phys | Inter | Model. | Lang | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| OTC | 69.6 | 46.6 | 72.1 | 83.5 | 76.6 | 63.5 | 72.7 | 83.2 | 60.4 | 69.8 |
| Controlled SA 2× | 73.5 | 59.6 | 66.6 | 82.0 | 80.8 | 59.2 | 54.5 | 83.8 | 50.6 | 67.8 |
| Parallel Agent | 35.7 | 48.2 | 63.9 | 81.3 | 76.4 | 64.3 | 54.5 | 78.0 | 47.8 | 61.1 |
| Best-of-N | 48.5 | 43.7 | 72.2 | 84.2 | 77.2 | 68.8 | 54.5 | 77.0 | 48.4 | 63.8 |
| Self-consistency | 37.5 | 43.5 | 73.9 | 80.7 | 76.5 | 66.0 | 72.2 | 73.0 | 61.3 | 65.0 |
As shown in the table, OTC achieves an average NT score of 69.8, substantially outperforming independent parallel agents (61.1, +8.7 points), Best-of-N (63.8, +6.0 points), and self-consistency (65.0, +4.8 points). Notably, OTC also outperforms the controlled single-agent setting with twice the budget of OTC (67.8, +2.0 points). The improvements are particularly pronounced in Mathematics (+33.9 points), Interdisciplinary (+18.2 points), and Language (+12.6 points) compared with independent parallel agents.
These results suggest that OTC's effectiveness cannot be fully explained by independent parallel execution or sampling-based answer selection alone. They provide additional evidence that structured inter-agent interactions offer benefits beyond those of non-communicating parallel baselines.
Author Response (Part 3)
Part 3: Analyzing Successful and Unsuccessful Collaboration Chains
We further investigate whether collaboration chains contribute to task performance. We manually verify 286 outcome chains, among which 224 (78%) contribute to scoring points, while 62 (22%) are associated with final answer errors.
| Type | Number of Chains | Percentage |
|---|---|---|
| Good | 224 | 78% |
| Bad | 62 | 22% |
Interestingly, we observe a complementary pattern using the CCE metric from AgentWorld, which measures the proportion of agent actions that meaningfully contribute to task progress.
| Method | CCE | Contributing Agents | Agent Nodes |
|---|---|---|---|
| Single Agent | - | 0.9 | 1.0 |
| Decentralized | 0.19 | 1.9 | 4.8 |
| Centralized | 0.32 | 2.4 | 4.7 |
| OTC | 0.55 | 2.8 | 3.9 |
As shown in the table, Decentralized and Centralized achieve CCE scores of only 0.19 and 0.32, respectively, compared with 0.39 for the single-agent baseline. In contrast, OTC achieves a CCE score of 0.55, with 2.8 contributing agents out of 3.9 agent nodes on average.
These findings also help explain why multi-agent methods sometimes fail to outperform single-agent baselines. The challenge is not necessarily that the tasks lack opportunities for collaboration, but that existing multi-agent systems often fail to collaborate effectively and efficiently. This distinction further motivates ContestWorld as a benchmark for evaluating the effectiveness of multi-agent collaboration.
To better understand the underlying mechanisms, we further examine representative successful and unsuccessful collaboration chains.
Good Chains. We identify two representative patterns of effective collaboration:
-
Solving dependent problems using teammates' results. In the 2018 WMTC Junior round, Q12 explicitly requires the answer to Q8. Agent 5 solves Q8 and obtains 4. Agent 4 then uses this value to determine the side ratio of 4:1 in Q12 and calculates the area as 20 cm². Finally, Agent 2 verifies the calculation and submits the correct answer.
-
Incorporating teammates' feedback to improve solutions. In IIOT 2021's All the Emojis!, Agent 2 identifies that Agent 1's initial solution could consume approximately 1.7 GB of memory, exceeding the 256 MB limit. Agent 4 proposes finding all code matches first and counting valid decodings afterward. Building on this suggestion, Agent 1 redesigns the solution to allocate only the necessary memory and store match lengths, reducing the estimated memory consumption of the main data structures to 48 MB. Agent 2 verifies that the revised solution correctly handles overlapping matches, and Agent 4 submits the code, achieving 100/100 in the benchmark's code review.
Bad Chains. We also identify two representative failure patterns:
-
Failure to preserve cross-problem dependencies and contest constraints. In the WMTC Advanced round, Q13 explicitly requires the value of T from Q5. However, Agent 3 incorrectly uses T = 50, derived from Q2, and obtains an answer of 75, which is subsequently submitted. A similar constraint violation occurs in the Jessup competition: although the applicant requests a declaration that the denial of consular access did not violate international law, the team instead drafts and submits a document requesting a declaration that it did violate international law, even after the review process.
-
Failure to act on teammates' error feedback. In 91 runs (12%), an agent correctly identifies a problem, but the team fails to address it. For example, in an ARML problem, two agents reject the same answer four times for the same reason, yet the issue remains unresolved. Similarly, in an HMMT problem where the correct answer is 7, Agent 7 proposes 6, and Agent 1 subsequently accepts the incorrect answer during the review process.
Overall, these three analyses demonstrate that ContestWorld involves substantial task and agent interdependence, while effective collaboration remains challenging for existing multi-agent systems. The performance gains from collaboration, together with the observed failures in ineffective coordination, highlight the importance of evaluating not only task outcomes but also how agents collaborate to achieve them.
Author Response (Part 1)
Thank you for the insightful suggestions. We address all your weakness points and questions below.
This is an insightful suggestion! To investigate whether smaller models achieve larger gains from collaboration, we further evaluate Qwen3-4B under the same experimental setting and compare its performance with larger models.
As shown in the table, OTC improves Qwen3-4B from 31.8 to 37.1 (+5.3 points, +16.7% relative), demonstrating that collaboration can also benefit smaller LLMs. Compared with larger models, GLM-5.3 improves from 51.7 to 54.4 (+2.7 points, +5.2%), DeepSeek-V3.1 from 43.9 to 51.2 (+7.3 points, +16.6%), and Kimi-K2.6 from 48.1 to 58.9 (+10.8 points, +22.5%). Interestingly, Qwen3-4B achieves a larger relative improvement than GLM-5.3 and a comparable improvement to DeepSeek-V3.1. These results suggest that smaller models can benefit more from collaboration than some larger models. Kimi-K2.6 exhibits even larger gains. We speculate that this may be related to its training for multi-agent communication or swarm-based collaboration, potentially enabling more effective multi-agent coordination.
We agree that task difficulty alone does not necessarily imply a need for collaboration. To better understand the demands of collaboration in ContestWorld, we conduct three in-depth and large scale analyses. First, we measure the widespread existence of collaboration by examining two types of interdependence: task interdependence (whether solving one subtask requires the result of another) and agent interdependence (whether one agent's execution relies on another agent's output). Inspired by the CCE metric [1], we construct dependency graphs from agent trajectories and propose metrics to quantify collaboration. Second, we evaluate the effectiveness of collaboration through budget-controlled experiments with non-communicating parallel agents and other baselines, demonstrating the importance of the collaboration in ContestWorld. Third, we analyze successful and unsuccessful collaboration chains and investigate their real contributions to task performance.
Part 1: Measuring Interdependence in Multi-Agent Collaboration
We further investigate how agents collaborate sequentially, beyond parallel task coverage, by examining task and agent interdependence in ContestWorld. Specifically, we carefully measure two types of interdependence: task interdependence, where solving subtask i requires the output of subtask j, showing the natural dependency of the sub-questions in ContestWorld, and agent interdependence, where Agent B's execution depends on the output of Agent A.
Following the CCE metric [1], we construct dependency graphs by tracing the contributions of individual actions and analyze the chain of inter-agent collaboration.
Agent Interdependence.
We find that 98.61% of trajectories contain cross-agent dependencies, and 70.83% contain collaboration chains involving at least three agents. These results demonstrate that inter-agent dependencies are widespread in ContestWorld. The most frequent dependency transitions include:
speak → speak: 98work → review_answer: 98speak → edit_work: 83direct_message → work: 79direct_message → direct_message: 75work → work: 69We further provide two representative examples of sequential agent interactions: