- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents EmboScope, a benchmark for evaluating embodied perception, understanding, and planning in VLMs. It introduces a reusable, schema-driven QA construction pipeline and comprises 4000+ questions across multiple diverse tasks and interaction sources. Through extensive evaluation of different VLMs, the authors identify limitations in cross-view reasoning, temporal interaction understanding, and goal-oriented planning, and observe that textual chain-of-thought reasoning yields limited improvements on embodied tasks.
Critical strengths
- This paper is clearly written and easy to follow.
- The integration of human egocentric, real-robot, and simulated data offers diverse evaluation scenarios.
- The shared interaction schema enables efficient reuse of annotations across different evaluation tasks.
- The fine-grained task evaluations offer a detailed characterization of model strengths and weaknesses.
Critical weaknesses
- The reliability of the benchmark's ground truth is not sufficiently validated. Since the QA pipeline relies heavily on automated annotations, potential errors may propagate into the generated questions and answers. Although the authors describe a quality-control process, quantitative evidence of annotation accuracy and human verification is limited.
- The designed tasks in this paper may not adequately reflect actual planning capabilities. In particular, the demonstrated next action is not necessarily the only valid action, as multiple strategies may lead to the same goal. Also, evaluating task progress does not necessarily imply an ability to plan and execute actions. Therefore, it remains unclear to what extent these tasks measure genuine embodied planning.
- The conclusion that textual reasoning provides limited benefits for embodied tasks is not sufficiently supported. The effectiveness of reasoning in embodied VLMs can depend heavily on the inference pipeline, prompting strategy, and reasoning configuration. Without systematically controlling these factors, it is difficult to determine whether the limited improvements reflect fundamental limitations of textual reasoning or simply suboptimal evaluation settings.
Questions for authors
- The authors introduce a unified interaction schema to organize different embodied capabilities. However, it remains unclear why such a unified schema is necessary for benchmark design. Could these capabilities be evaluated independently without relying on a shared schema? The paper would benefit from a clearer justification of what fundamental limitations of existing benchmarks this design addresses and what additional value it brings to capability evaluation.
- The relatively poor performance of embodied VLMs may not necessarily indicate weaker embodied understanding or planning capabilities. Since the benchmark relies heavily on language-based QA, these models may underperform simply because they are less capable of following language instructions or producing answers in the expected format. The current evaluation does not clearly disentangle instruction-following ability from embodied reasoning ability. It is possible that these models could outperform general-purpose VLMs when coupled with action decoders in actual robotic tasks. How do the authors account for this distinction when interpreting the results?