acceptodds
Under review as a conference paper at ICLR 2027

MECoBench: A Systematic Study of Multimodal Agent Collaboration in Embodied Environments

Abstract

Recent multimodal large language models (MLLMs) have strong potential as embodied agents, but their ability to collaborate in visually grounded environments remains underexplored. To address this gap, we introduce **MECoBench**, a multimodal embodied cooperation benchmark with a controlled evaluation platform spanning two distinct task domains and diverse task and team settings. Through extensive experiments across various MLLMs, we summarize three key findings: (i) Most models benefit from collaboration, but individual capability alone does not determine collaborative performance. (ii) Communication is essential for collaboration gains, with reliable information sharing and coordinated execution distinguishing effective collaboration from failure. (iii) Appropriate team size, organization, and information sharing can further improve collaborative performance. Generally, MECoBench provides a systematic testbed for understanding the mechanisms and limits of multimodal embodied collaboration.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Agent Systems, Multimodal Large Language Models, Embodied AI, Multi-Agent Collaboration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Wz1GOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates multimodal agent collaboration across household manipulation and outdoor navigation and exploration, with shared and partitioned workspaces. The experiments compare models, communication methods, team sizes, and memory mechanisms, showing variable collaboration gains and recurring coordination failures.

Critical strengths
  • The configurable platform spans indoor and outdoor tasks with different coordination requirements, which provides a useful resource for studying embodied collaboration.
  • The model coverage is braod and macthed total action budgets provide informative comparisons between individual and collabortive perfromance.
  • Communication ablations, sensitivity tests, and additional experiments strengthen the evidence that communication can improve task completion.
Critical weaknesses
  • There is limited statistical support. For example, the outdoor evaluation does not include confidence intervals or repeated runs.
  • The mechanistic claims exceed the evidence. Tables 3-4 largely report pooled associations from keyword or rule-based detection, but model capability and task difficulty can confound these relationships.
  • Table 22 documents simultaneous grabs of the same object succeeding and a decoder changing the proposed object. These artifacts complicate attribution of failures to agent coordination rather than the execution pipeline.
  • There are incomplete resource controls, since equal action budgets do not equalize inference cost across communication protocols. And comparisons at matched total tokens and latency are needed to prove efficiency advantages.
Questions for authors
  1. Do the principal model rankings and protocol improvements remain reliable across repeated runs and paired confidence intervals, particularly on the small outdoor subsets?
  2. How accurate are the behavior detectors against human annotations, and do the reported associations persist after controlling for model identity and task difficulty?
  3. How often do decoder errors and inconsistent simultaneous actions occur, and do the main conclusions hold after correcting or excluding these cases?
Reviewer_w9KtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper designs a benchmark MECoBench with a controlled evaluation platform to systematically evaluate collaboration among multimodal AI agents in simulated embodied environments. It supports one to five agents and investigates different coordination and communication mechanisms. To evaluate collaboration, the authors use task success and completion metrics, with three collaboration specific metrics including Division of Labor (DOL), Conflict Action Rate (CAR), and Handover Failure Rate (HFR). While the paper addresses an important research problem and provides a comprehensive evaluation of multimodal agent collaboration, the demonstrated novelty seems to be limited. The paper provides useful empirical observations. The distinctive scientific contributions need convincingly demonstrate sufficiently or rigorous experimental evidence.

Critical strengths
  1. The paper addresses an important research problem in multimodal embodied multi-agent collaboration. It provides a unified evaluation framework covering different team sizes, cooperation structures, and communication mechanisms, enabling systematic comparisons under a common experimental setting.
  2. The experimental evaluation is relatively comprehensive, involving multiple multimodal language models and different collaboration configurations. The inclusion of both parallel and sequential tasks helps examine how different task structures affect collaboration performance.
  3. Beyond conventional task success and completion rates, the paper analyzes agent interactions through metrics such as DOL, CAR, and HFR, along with communication behaviors and failure cases. The additional analysis provides useful diagnostic information about coordination inefficiencies and collaboration failures.
Critical weaknesses
  1. The three research questions are broad and have been extensively studied in multi-agent systems, including the effects of team size, communication mechanisms, and incomplete information. Most of the conclusions, such as communication improving collaboration and excessive coordination introducing additional overhead, are consistent with existing findings. The authors should more clearly identify the new observations, scientific insights, or conclusions derived from this study and explain how these findings advance our understanding of multi-agent collaboration beyond prior work. This would help better highlight the paper's contributions.

  2. The motivation for introducing a new benchmark is not sufficiently convincing. The paper claims that multimodal agent collaboration in visually grounded embodied environments remains underexplored and therefore calls for a new benchmark. This claim appears somewhat overstated. Several existing studies, including CoELA, TeamCraft, PARTNR, and VIKI-Bench, have already investigated embodied multi-agent collaboration involving visual perception, communication, coordination, and task dependencies. In particular, TeamCraft already provides a multimodal multi-agent benchmark in a visually rich 3D environment. It’s good to more clearly identify which specific research gaps remain unaddressed by existing benchmarks and explain why MECoBench is necessary to address them. The current motivation is too broad and does not sufficiently distinguish the proposed work from prior studies.

  3. Table 1 provides a useful overview of existing multi-agent benchmarks, but the comparison does not sufficiently establish the novelty of MECoBench. Several prior works already support capabilities closely related to those emphasized in this paper. For example, TeamCraft supports multimodal observations, variable team sizes, and centralized/decentralized multi-agent control, while PARTNR investigates embodied collaboration involving spatial and temporal constraints, task dependencies, and coordination failures. The current comparison focuses primarily on available features rather than the collaborative capabilities and research challenges addressed by existing benchmarks. Although MECoBench integrates several evaluation dimensions, it remains unclear what new collaboration behaviors or scientific insights this integration enables beyond prior work. A more detailed comparison, particularly with TeamCraft and PARTNR, would help better establish the significance and novelty of the proposed benchmark.

  4. The paper does not sufficiently justify whether these metrics accurately reflect collaboration quality. For example, a higher DOL score indicates a more equal distribution of completed subgoals, but equal task distribution does not necessarily imply better collaboration, particularly when agents have different capabilities or tasks have different complexities. Similarly, CAR captures only certain simultaneous action conflicts, while HFR measures failed handover attempts without necessarily reflecting overall coordination effectiveness.

  5. The benchmark uses a fixed total action budget for parallel tasks, distributing approximately 60 action steps among the agents. As team size increases, each agent receives fewer available actions. Consequently, declining performance with larger teams may be caused by insufficient per-agent action budgets rather than increasing coordination complexity. The authors should consider evaluating team-size scaling under both fixed total action budgets and fixed per-agent budgets.

  6. Insufficient evidence supporting explanations of communication effectiveness. The paper analyzes communication behaviors such as action delegation, location sharing, and coordination queries, and associates them with differences in task success. However, the relationships are primarily observational. For example, successful teams may naturally exhibit more effective action delegation, while difficult tasks may generate more coordination queries. Therefore, the reported correlations do not establish that particular communication behaviors directly cause improvements or degradation. Controlled experiments isolating specific communication mechanisms would strengthen these claims.

Questions for authors
  1. What is the justification for using DOL, CAR, and HFR to characterize collaboration quality? In particular, DOL measures the evenness of completed-subgoal distribution, but a more balanced workload does not necessarily imply more effective collaboration. Could the authors provide evidence that these metrics meaningfully reflect collaboration effectiveness across the evaluated tasks?
  2. Several existing benchmarks, including TeamCraft and PARTNR, have investigated similar multi-agent collaboration problems. Could the authors clarify the key differences and explain what new scientific insights MECoBench provides beyond prior work?
  3. The fixed total action budget reduces the available steps per agent as team size increases. How do the authors distinguish performance degradation caused by coordination overhead from that caused by reduced per-agent action budgets? Would the conclusions remain consistent under a fixed per-agent budget?
Reviewer_emyUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MECoBench, a benchmark and evaluation platform to evaluate individual capability and collaboration among MLLM-based embodied agents. It covers two domains:

  • MECo-House: 240 long-horizon rearrangement tasks in VirtualHome, each with 2–7 target objects.
  • MECo-City: 60 object-goal navigation tasks (MECo-Nav) and 40 exploration tasks (MECo-Exp) in Virtual-Community.

House and Nav tasks each come in two versions. In the Shared Workspace, all agents can reach all regions. In the Partitioned Workspace, agents have disjoint accessible regions and must hand objects over. The platform varies team size (1–5), team organization (broadcast, discussion, leader–worker), and communication mechanism (none, text, shared memory, vision-augmented leader). The total action budget is held fixed across team sizes.

The authors evaluate 15 MLLMs on a 226-task subset and report three findings:

  1. Most models benefit from collaboration. Individual capability correlates with collaborative performance but does not determine it.
  2. Communication is essential. Information sharing and task coordination are associated with success. Conflicts, execution mismatches and unreliable information are associated with failure.
  3. The best collaboration mode depends on both team size and model capability.
Critical strengths

1. A controllable testbed that isolates collaboration variables for current MLLMs. MECoBench puts team size (1–5), organization, communication mechanism, and shared vs. partitioned access into one closed-loop harness. It compares teams with single agents under the same total action budget and evaluates 15 current proprietary and open MLLMs. Each task family has precedents, but having these controls in one place, with matched budgets and current models, is what may make the benchmark useful to the community.

Critical weaknesses

1. The paper claims to evaluate multimodal embodied cooperation, but its tasks largely bypass vision. Agents receive egocentric images, but almost all the information needed to act is supplied as text or by the simulator. At most, vision is left with the final check that an already-localized object is in view:

  • House goals state the possible location (room and furniture) and the appearance of every target object (Fig. 6). The main text never mentions this (Sec. 3.2).
  • The MECo-Nav map given to agents marks target cars and their distances in unexplored areas in the example input (Fig. 12(b)). MECo-Exp success is defined as geometric map coverage.

Consistent with this, no experiment removes or replaces the visual input. The only visual manipulation, vision-augmented leadership, changes SR by +3.1 and −1.0 points on the two models tested (Fig. 5(c)). The benchmark therefore does not support its central claim to measure multimodal embodied cooperation, which is what is meant to distinguish it from text-based benchmarks.

2. The setting is not new, and the findings offer little new insight.

  • The task families are standard. House follows the WAH-style rearrangement setup in VirtualHome, as the paper itself states (Sec. 3.1), and Nav and Exp are conventional object-goal navigation and exploration problems (Sec. 3.1). The paper does not show what these settings can measure that existing multi-agent embodied benchmarks cannot.
  • The most robust results (communicating teams beat a single agent; handover tasks fail without communication) are largely predictable. The claims that could be new, namely the benefits of team organization and shared memory, mostly rest on differences of a few tasks out of 96 (often 1–5).
  • Because a benchmark built from familiar settings must justify itself through new and reliable findings, this weighs decisively in my assessment.
Questions for authors

1. Task selection. How were the 226 evaluated tasks (96 House, 12 Nav, 10 Exp) selected from the 640 constructed tasks? Sec. 4.1 calls them a "fixed subset" but does not describe the procedure.

2. Heterogeneous teams. Beyond the strong-leader experiment mentioned in Sec. 5.2, why are heterogeneous teams combining different model families (e.g., a proprietary model with an open model) not evaluated? Such mixed teams seem central to realistic multi-agent deployment, and they could reveal whether cross-model communication creates failure modes absent in homogeneous teams.

3. Choice of VirtualHome. Why was VirtualHome chosen for the household domain? Its high-level actions (e.g., moving directly to a room) and limited physical interaction seem to reduce the embodied and visual challenges that the paper uses as motivation. Did you consider simulators with richer physics or low-level egocentric navigation, and would you expect the conclusions to transfer?

4. What exactly is serialized into the agents' input block? Fig. 10 lists the observation views, object location and appearance priors, memory and dialogue, but the input block in the prompts is not shown. Is there any textual rendering of what the agent currently sees, as opposed to the raw images? If so, how much of the visibility check is done from text rather than from the image?

5. The 89.6% configuration. What exactly is the "3-agent shared memory" configuration behind the 89.6% SR in Fig. 1(d) and Sec. 5.3? Shared-memory results are otherwise reported only for 2 and 4 agents (Fig. 5(a)), and it is unclear whether this configuration also uses leader–worker organization. Please report its full results (SR, CR, AUC, tokens).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.