acceptodds
Under review as a conference paper at ICLR 2027

Are Frontier VLM Agents Ready to Be Robot Generalists? An Empirical Study with the Embodied Agent Arena

Abstract

Frontier vision-language models (VLMs) combine scene estimation, interaction grounding, and executable actions. Understanding how these abilities support complete robotic tasks is central to evaluating their readiness as robot generalists. We introduce Embodied Agent Arena to examine where local competence supports, or falls short of, complete task success across Geometry, Spatial Reasoning, Affordance, Task Planning, and Manipulation. The arena contains 1,000 cases drawn from 32 established sources and GeoProbe, our new benchmark for geometric estimation on Blender renders and real-scene images. A minimal harness preserves source observations and operations while separating metric precision, functional grounding, and native goal completion. We evaluate seven VLMs, analyze Astra's task-specific advantages, and compare richer-observation execution protocols and multi-round review. Across the arena, Astra's advantage is strongest in precise estimation and usable-contact localization; completing coordinated, goal-directed actions remains the key gap to robot generalism.

Primary area:
datasets and benchmarks
Keywords:
Vision-Language Agents, Robot Generalists, Embodied Evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fChgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces Embodied Agent Arena, a 1,000-case evaluation suite for frontier VLM agents covering five capability domains: Geometry (370), Spatial Reasoning (220), Affordance (70), Task Planning (157), and Manipulation (183). The cases are drawn from 32 existing benchmarks (e.g., InFlux, Map-free, TraceSpatial, 3DSRBench, MindCube, VSI-Bench, UMD, ALFRED, ScienceWorld, CALVIN, RLBench, RoboTwin, ManiSkill, BEHAVIOR-1K) plus GeoProbe, a new 168-query geometric-estimation benchmark built mostly from Blender renders, with four real photo pairs. A "minimal harness" uses a persistent Python runtime for fixed-input and planning tasks and each source's native agent loop and helpers for manipulation. Five closed-source models (Astra, Sol, Fable, Qwen-Max, Gemini) and two open-source models are evaluated, with separate metrics for task success, geometric precision, and affordance grounding. The main findings are as follows:

  • Astra leads overall: lowest pose error, highest contact validity, 80.3% Planning success, and 41.8% Manipulation success.
  • Agents frequently produce accurate local estimates (e.g., close trajectory paths) but miss the endpoints and constraints that decide success.
  • Under matched budgets, adding RGB at every decision can reverse rankings (e.g., Astra's CALVIN success drops from 30% to 0, while Fable's rises from 0% to 40%).
  • Multi-round review with specialist tools gives mixed, entry-dependent gains.

The paper concludes that coordinated, goal-directed completion is the main gap to robot generalism.

Critical strengths
  1. The arena is broad, and its protocol is careful and transparent. It combines fixed-input probes and interactive episodes across five domains in one harness. Continuous metrics (pose error, AbsRel, trajectory RMSE, IoU) are kept separate from binary terminal success, and the paper reports run-to-run SD, valid-submission coverage, budgets, sensitivity to source composition (Table A10), equal-source weighting, and common-valid-cohort checks. This level of accounting is uncommon in VLM-agent evaluations, and it makes the results interpretable instead of reducing them to a single leaderboard number. It is the main argument for acceptance.

  2. Several diagnostic findings are useful beyond the leaderboard. On TraceSpatial, trajectory error and success disagree: most of Astra's tracing failures come from where the path starts or ends (endpoint placement), not from the overall path error. In GeoProbe, the remaining depth error is mostly a matter of global scale, and pose estimation fails completely under combined rotation and translation. Rankings reverse under the RGB-expanded protocol, with the reversal traced to when each agent builds a reusable feedback controller (Fig. A3). Agents also locate valid contact points without covering the functional region. These observations provide concrete evidence that local competence does not imply task completion. They also show that the interface and observation protocol are part of what is being measured.

Critical weaknesses
  1. The contribution is mostly a re-packaging of existing benchmarks, and the analysis is too narrow and descriptive to answer the title question. Except for GeoProbe (168 queries, almost all synthetic, with only four real photo pairs), every case comes from an existing benchmark, and the paper's own Table 1 shows close overlap with BEAR, EmbodiedBench, EmbodiedEval, Embodied Arena, and VLABench. Most of Section 4.3 and Appendix D is a "deep dive of Astra," a single closed model, which makes the paper read more like a model capability report than a general study. The stated conclusion is close to an observation already familiar in this literature: local estimation is better than complete goal-directed execution. The paper also offers no calibration against non-VLM references, such as specialist geometry estimators (e.g., the GeoCalib or LightGlue pipelines the authors use as tools), VLA policies, or published task-specific methods on the same sources. Without such references, it is hard to judge what "ready" means. With no physical-robot evaluation either, the paper does not answer whether frontier VLM agents are ready to be robot generalists. It describes how seven models rank on a curated mix of simulated and fixed-input tasks.

  2. The interactive results, which matter most for the generalist claim, rest on small, single-run samples and confounded protocols. Planning and Manipulation use a single episode per case, and many sources contribute only 5–10 tasks, some goal families just one. As a result, a single episode can shift a source score by 10–20 points, and many headline contrasts (e.g., CALVIN at 30% vs. 0%, ManiSkill at 55.6% vs. 0%, or HumanCLAW sitting) cannot be separated from noise. The evaluation is also not uniform across models:

    • reasoning effort differs (maximum for Sol, high for Astra and Fable, unset for API models);
    • Fable receives three repetitions instead of five;
    • some records are missing and are counted as zeros (e.g., Fable's baseline ManiSkill);
    • each source exposes different native helpers.

    The RGB-expanded experiment shows that rankings can reverse completely under matched budgets. Taken together, these issues make the Manipulation and Planning rankings fragile. They also raise the question of whether the observed gap reflects model capability or harness design. The paper acknowledges protocol dependence but still draws model-level conclusions from these numbers.

Questions for authors

Can you report confidence intervals or rerun the interactive domains (Planning, Manipulation) with at least three seeds? Which of the reported model differences in those domains are statistically significant?

1 comment
AuthorsOpenReview ↗

Follow-up: remaining questions on analysis and reliability

We have now posted both parts of our response. Part 1 presents the complete seven-model failure distributions. Part 2 adds cluster-aware uncertainty, paired CALVIN executions, missing-record sensitivity, and clarification of the specialist and deployment scope.

Could you let us know whether any concern remains unresolved, especially about the contribution beyond individual benchmarks or the reliability of the interactive findings? We would be happy to discuss the relevant evidence or clarify our planned revisions.

If this evidence addresses your main concerns, we would appreciate your reconsideration of the overall assessment and rating.

Reviewer_YZPXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Embodied Agent Arena, a 1,000-case evaluation covering geometry, spatial reasoning, affordances, task planning, and manipulation. It combines existing benchmarks with GeoProbe, the new geometric-estimation benchmark, and evaluates seven VLM. Its main finding is that strong intermediate capabilities do not consistently transfer into complete robotic task success.

Critical strengths
  1. Evaluating the same models across five capability domains provides a useful view of their strengths and limitations.
  2. The paper separates accurate estimates, valid contacts, intermediate execution, and complete goal satisfaction.
  3. The RGB-expanded experiments show that observation delivery can substantially change model rankings.
Critical weaknesses
  1. Interactive tasks use one adopted run. The CALVIN ranking reversal is based on only ten sequences, so its stability is unclear.
  2. Additional images increase token use and reduce executed actions for Astra. The experiment does not separate difficulties using visual information from reduced execution opportunities.
  3. Tool-assisted review allows six additional responses, while self-review and crop review allow two. This makes the benefit of tools difficult to isolate.
Questions for authors
  1. Do the main manipulation findings persist across repeated paired runs?
  2. Does the CALVIN ranking reversal remain when additional budget accommodates the extra images?
  3. How does tool-assisted review compare with self-review when both receive the same number of responses?
3 comments
AuthorsOpenReview ↗

Follow-up: paired CALVIN and matched-review controls

Both parts of our response are now available. The 24 paired CALVIN executions on two sequences examine RGB effects without cumulative-token stopping. The 240-record review study matches all review arms at six responses and includes Tool-masked to separate returned specialist feedback from additional deliberation.

Could you let us know whether these controls address your questions about repeated execution, image-token budgets, and the tool comparison? We would be happy to discuss any remaining concern about the settings, results, or their interpretation.

If the new evidence resolves your main concerns, we would appreciate your reconsideration of the overall assessment and rating.

W2 and W3 resolved. The paired CALVIN repetitions also help, although only two of the ten sequences covered, so I would still suggesting to avoid making a claim about a stable CALVIN ranking reversal unless 10 sequences are done.

AuthorsOpenReview ↗

Revision to the CALVIN comparison

Thank you for confirming that W2 and W3 are resolved. We followed your suggestion on W1 and removed the general CALVIN ranking claim from the Introduction and Conclusion. Section 4.4 (p. 8) and the caption of Table 3b (p. 9) now specify one run per sequence and condition. Astra completed 3/10 sequences under Baseline and 0/10 under RGB-expanded. Fable completed 0/10 and 4/10, respectively.

We also clarified a budget detail in Appendix E.1. The 1.28M-token ceiling in our earlier response applies to RGB-expanded. The available Baseline traces for Astra and Fable record 640k. In the separate matched repeats on 0557 and 0626, both conditions had the same response and action limits and no cumulative-token stop. Adding per-round RGB increased Fable’s completions from 1/3 to 2/3 on 0557 and decreased them from 3/3 to 2/3 on 0626, giving 4/6 in each condition. Stability across repeated runs on all ten sequences remains open.

Reviewer_5qRZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Embodied Agent Arena, an empirical evaluation framework for assessing whether frontier vision-language model (VLM) agents are ready to serve as general-purpose embodied agents. The framework combines 32 existing data sources with a newly introduced GeoProbe benchmark, covering five capability domains: geometry, spatial reasoning, affordance grounding, task planning, and manipulation. It evaluates seven VLM agents across 1,000 cases, including both fixed-input and interactive settings. The paper aims to distinguish performance on intermediate capabilities from successful completion of end-to-end embodied tasks. Its findings suggest that strong local geometric or manipulation performance does not necessarily translate into reliable task completion, and that observation protocols can substantially affect model performance and relative rankings.

Critical strengths
  1. Important research question. The paper investigates a timely and practically relevant question: whether current frontier VLM agents possess the broad and reliable capabilities required for general-purpose embodied tasks. Evaluating multiple capabilities rather than relying on a single benchmark is a useful direction.

  2. Broad empirical coverage. The framework covers five capability domains and combines existing datasets with GeoProbe. The inclusion of both fixed-input evaluations and interactive tasks provides a broader view of agent capabilities than evaluations restricted to static visual question answering.

  3. Beyond single aggregate metrics. The paper examines multiple aspects of performance, including geometric errors, trajectory endpoints, contact localization, and task-level success. This helps reveal failure modes that may be hidden by conventional average-error metrics.

  4. Useful observations about the gap between local competence and task success. The results indicate that low geometric trajectory error does not necessarily imply correct endpoints, collision avoidance, or satisfaction of terminal conditions. Similarly, competent local manipulation does not guarantee successful completion of household tasks. These findings are relevant to the design and evaluation of embodied agents.

  5. Evidence of sensitivity to observation protocols. The reported changes in manipulation performance under different observation settings suggest that evaluation protocols can materially affect measured capability and model rankings. This is an important empirical observation that merits further investigation.

Critical weaknesses
  1. The methodological novelty of the unified arena needs clearer justification. Much of the evaluation framework is assembled from existing data sources. The paper should more clearly establish what methodological capabilities the unified framework enables that cannot be obtained by evaluating the constituent benchmarks independently. The contribution would be stronger if the integration yielded new, validated insights beyond a consolidated comparison of existing results.

  2. Potential confounding factors in cross-model comparisons. The evaluation interfaces and execution mechanisms are not fully homogeneous across domains. Several domains use a Python-based runtime, whereas manipulation retains native agent loops, and source-specific interfaces and budgets remain relevant. Differences in tool access, execution budgets, action limits, input formatting, or agent scaffolding may influence the results. The paper should better separate differences attributable to the underlying models from those caused by the evaluation harness.

  3. Uncertainty and statistical reliability are insufficiently established. The nominal number of cases does not necessarily correspond to the same number of independent observations, because multiple queries can share scenes. In addition, the number of repeated runs varies across models and tasks, with some interactive evaluations relying on a single run. These factors make it difficult to assess the robustness of model rankings and reported performance differences. Scene-level or task-level confidence intervals, cluster-aware statistical analysis, and additional repeated interactive evaluations would strengthen the evidence.

  4. The aggregate capability index is sensitive to design choices. The five-domain index applies manually selected transformations to heterogeneous metrics and then weights the domains equally. These choices may substantially affect the aggregate scores and model rankings. The index should not be interpreted as a definitive measure of general-purpose robotic capability without sensitivity analyses and a stronger justification for the normalization and weighting schemes. Per-domain results should remain central to the paper's conclusions.

  5. GeoProbe's external validity is not sufficiently established. GeoProbe contributes new evaluation cases, but its evaluation is predominantly based on synthetic scenes, with limited real-scene validation. It remains unclear how well its results predict performance in realistic, physically grounded environments. More extensive validation on real images or physical robot tasks would help establish the benchmark's practical relevance.

  6. The claims about robot generality may exceed the evidence. The benchmark covers several important capabilities, but success across these tasks does not by itself establish readiness for general-purpose robotic deployment. The paper should more carefully distinguish benchmark performance, robustness across task distributions, and reliable operation in real-world environments.

Questions for authors
  1. Novelty: What new evaluation capability or scientific insight does Embodied Agent Arena provide beyond the independent evaluation of its constituent benchmarks? Can the authors identify findings that specifically depend on the unified framework?

  2. Fairness of comparisons: How are model-specific prompts, tool access, execution budgets, action limits, and agent scaffolding controlled across models and domains? To what extent could these factors explain the observed performance differences or ranking changes?

  3. Statistical robustness: Can the authors report confidence intervals and significance analyses that account for queries sharing the same scene? For tasks with stochastic outcomes, how stable are the reported rankings when interactive evaluations are repeated multiple times?

  4. Aggregate score sensitivity: How robust are the overall rankings to alternative metric normalizations and domain weights? Would the main conclusions remain unchanged if the aggregate index were replaced by per-domain comparisons or alternative reasonable aggregation schemes?

  5. Real-world validity: How well do GeoProbe results transfer from synthetic scenes to real-world visual inputs and physical manipulation tasks? Can the authors provide additional real-scene or real-robot validation?

  6. Failure and invalid-output accounting: For metrics computed over valid predictions, how are invalid outputs and prediction failures incorporated into the final results? Can the authors report prediction coverage alongside conditional error metrics to ensure that models with higher failure rates are not inadvertently favored?

1 comment
AuthorsOpenReview ↗

Follow-up: remaining questions on evidence and evaluation controls

All three parts of our response are now available. They cover the seven-model failure analyses, normalization and source-weighting sensitivity, valid-prediction coverage, clustered uncertainty, and the new CALVIN and matched-review controls. We also clarify the distinction between benchmark capability and physical-robot deployment.

Are there any remaining concerns about the contribution, comparison protocol, statistical evidence, or scope that you would like us to address? Please let us know which point would benefit from further evidence or clarification; we would welcome the discussion.

If the responses resolve your main concerns, we would appreciate your reconsideration of the overall assessment and rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.