- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces Embodied Agent Arena, a 1,000-case evaluation suite for frontier VLM agents covering five capability domains: Geometry (370), Spatial Reasoning (220), Affordance (70), Task Planning (157), and Manipulation (183). The cases are drawn from 32 existing benchmarks (e.g., InFlux, Map-free, TraceSpatial, 3DSRBench, MindCube, VSI-Bench, UMD, ALFRED, ScienceWorld, CALVIN, RLBench, RoboTwin, ManiSkill, BEHAVIOR-1K) plus GeoProbe, a new 168-query geometric-estimation benchmark built mostly from Blender renders, with four real photo pairs. A "minimal harness" uses a persistent Python runtime for fixed-input and planning tasks and each source's native agent loop and helpers for manipulation. Five closed-source models (Astra, Sol, Fable, Qwen-Max, Gemini) and two open-source models are evaluated, with separate metrics for task success, geometric precision, and affordance grounding. The main findings are as follows:
- Astra leads overall: lowest pose error, highest contact validity, 80.3% Planning success, and 41.8% Manipulation success.
- Agents frequently produce accurate local estimates (e.g., close trajectory paths) but miss the endpoints and constraints that decide success.
- Under matched budgets, adding RGB at every decision can reverse rankings (e.g., Astra's CALVIN success drops from 30% to 0, while Fable's rises from 0% to 40%).
- Multi-round review with specialist tools gives mixed, entry-dependent gains.
The paper concludes that coordinated, goal-directed completion is the main gap to robot generalism.
Critical strengths
-
The arena is broad, and its protocol is careful and transparent. It combines fixed-input probes and interactive episodes across five domains in one harness. Continuous metrics (pose error, AbsRel, trajectory RMSE, IoU) are kept separate from binary terminal success, and the paper reports run-to-run SD, valid-submission coverage, budgets, sensitivity to source composition (Table A10), equal-source weighting, and common-valid-cohort checks. This level of accounting is uncommon in VLM-agent evaluations, and it makes the results interpretable instead of reducing them to a single leaderboard number. It is the main argument for acceptance.
-
Several diagnostic findings are useful beyond the leaderboard. On TraceSpatial, trajectory error and success disagree: most of Astra's tracing failures come from where the path starts or ends (endpoint placement), not from the overall path error. In GeoProbe, the remaining depth error is mostly a matter of global scale, and pose estimation fails completely under combined rotation and translation. Rankings reverse under the RGB-expanded protocol, with the reversal traced to when each agent builds a reusable feedback controller (Fig. A3). Agents also locate valid contact points without covering the functional region. These observations provide concrete evidence that local competence does not imply task completion. They also show that the interface and observation protocol are part of what is being measured.
Critical weaknesses
-
The contribution is mostly a re-packaging of existing benchmarks, and the analysis is too narrow and descriptive to answer the title question. Except for GeoProbe (168 queries, almost all synthetic, with only four real photo pairs), every case comes from an existing benchmark, and the paper's own Table 1 shows close overlap with BEAR, EmbodiedBench, EmbodiedEval, Embodied Arena, and VLABench. Most of Section 4.3 and Appendix D is a "deep dive of Astra," a single closed model, which makes the paper read more like a model capability report than a general study. The stated conclusion is close to an observation already familiar in this literature: local estimation is better than complete goal-directed execution. The paper also offers no calibration against non-VLM references, such as specialist geometry estimators (e.g., the GeoCalib or LightGlue pipelines the authors use as tools), VLA policies, or published task-specific methods on the same sources. Without such references, it is hard to judge what "ready" means. With no physical-robot evaluation either, the paper does not answer whether frontier VLM agents are ready to be robot generalists. It describes how seven models rank on a curated mix of simulated and fixed-input tasks.
-
The interactive results, which matter most for the generalist claim, rest on small, single-run samples and confounded protocols. Planning and Manipulation use a single episode per case, and many sources contribute only 5–10 tasks, some goal families just one. As a result, a single episode can shift a source score by 10–20 points, and many headline contrasts (e.g., CALVIN at 30% vs. 0%, ManiSkill at 55.6% vs. 0%, or HumanCLAW sitting) cannot be separated from noise. The evaluation is also not uniform across models:
- reasoning effort differs (maximum for Sol, high for Astra and Fable, unset for API models);
- Fable receives three repetitions instead of five;
- some records are missing and are counted as zeros (e.g., Fable's baseline ManiSkill);
- each source exposes different native helpers.
The RGB-expanded experiment shows that rankings can reverse completely under matched budgets. Taken together, these issues make the Manipulation and Planning rankings fragile. They also raise the question of whether the observed gap reflects model capability or harness design. The paper acknowledges protocol dependence but still draws model-level conclusions from these numbers.
Questions for authors
Can you report confidence intervals or rerun the interactive domains (Planning, Manipulation) with at least three seeds? Which of the reported model differences in those domains are statistically significant?
Follow-up: remaining questions on analysis and reliability
We have now posted both parts of our response. Part 1 presents the complete seven-model failure distributions. Part 2 adds cluster-aware uncertainty, paired CALVIN executions, missing-record sensitivity, and clarification of the specialist and deployment scope.
Could you let us know whether any concern remains unresolved, especially about the contribution beyond individual benchmarks or the reliability of the interactive findings? We would be happy to discuss the relevant evidence or clarify our planned revisions.
If this evidence addresses your main concerns, we would appreciate your reconsideration of the overall assessment and rating.