- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper evaluates multimodal agent collaboration across household manipulation and outdoor navigation and exploration, with shared and partitioned workspaces. The experiments compare models, communication methods, team sizes, and memory mechanisms, showing variable collaboration gains and recurring coordination failures.
Critical strengths
- The configurable platform spans indoor and outdoor tasks with different coordination requirements, which provides a useful resource for studying embodied collaboration.
- The model coverage is braod and macthed total action budgets provide informative comparisons between individual and collabortive perfromance.
- Communication ablations, sensitivity tests, and additional experiments strengthen the evidence that communication can improve task completion.
Critical weaknesses
- There is limited statistical support. For example, the outdoor evaluation does not include confidence intervals or repeated runs.
- The mechanistic claims exceed the evidence. Tables 3-4 largely report pooled associations from keyword or rule-based detection, but model capability and task difficulty can confound these relationships.
- Table 22 documents simultaneous grabs of the same object succeeding and a decoder changing the proposed object. These artifacts complicate attribution of failures to agent coordination rather than the execution pipeline.
- There are incomplete resource controls, since equal action budgets do not equalize inference cost across communication protocols. And comparisons at matched total tokens and latency are needed to prove efficiency advantages.
Questions for authors
- Do the principal model rankings and protocol improvements remain reliable across repeated runs and paired confidence intervals, particularly on the small outdoor subsets?
- How accurate are the behavior detectors against human annotations, and do the reported associations persist after controlling for model identity and task difficulty?
- How often do decoder errors and inconsistent simultaneous actions occur, and do the main conclusions hold after correcting or excluding these cases?