acceptodds
Under review as a conference paper at ICLR 2027

EmboScope: A Multi-Source Comprehensive Benchmark for Embodied Perception, Understanding, and Planning

Abstract

Vision-language models (VLMs) show promise for embodied tasks, yet understanding actions, state changes, and goals remains challenging. Existing benchmarks often struggle to combine broad interaction coverage, fine-grained capability diagnosis, and scalable data construction. We introduce **EmboScope**, a multi-source embodied benchmark with a reusable construction framework. It decouples interaction annotation from rule-based QA generation through a shared schema of objects, actions, and state changes, enforcing task-specific observation constraints while enabling reuse of annotations and task rules. EmboScope integrates human egocentric, real-robot, and simulated interactions across indoor and outdoor settings in everyday and industrial domains. Its 4,467 questions cover 14 tasks at three levels: perception, embodied semantics, and planning and reasoning. We evaluate 23 general-purpose and embodied VLMs, with four additional thinking configurations. The results reveal a gap between local recognition and coherent interaction understanding, alongside complementary strengths across models. Our findings suggest that scaling textual chain-of-thought alone may be insufficient for substantial gains on embodied tasks, motivating stronger visual grounding and interaction understanding.

Primary area:
datasets and benchmarks
Keywords:
Vison-Language Model, Embodied Intelligence, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EYtPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EmboScope, a benchmark for evaluating embodied perception, understanding, and planning in VLMs. It introduces a reusable, schema-driven QA construction pipeline and comprises 4000+ questions across multiple diverse tasks and interaction sources. Through extensive evaluation of different VLMs, the authors identify limitations in cross-view reasoning, temporal interaction understanding, and goal-oriented planning, and observe that textual chain-of-thought reasoning yields limited improvements on embodied tasks.

Critical strengths
  1. This paper is clearly written and easy to follow.
  2. The integration of human egocentric, real-robot, and simulated data offers diverse evaluation scenarios.
  3. The shared interaction schema enables efficient reuse of annotations across different evaluation tasks.
  4. The fine-grained task evaluations offer a detailed characterization of model strengths and weaknesses.
Critical weaknesses
  1. The reliability of the benchmark's ground truth is not sufficiently validated. Since the QA pipeline relies heavily on automated annotations, potential errors may propagate into the generated questions and answers. Although the authors describe a quality-control process, quantitative evidence of annotation accuracy and human verification is limited.
  2. The designed tasks in this paper may not adequately reflect actual planning capabilities. In particular, the demonstrated next action is not necessarily the only valid action, as multiple strategies may lead to the same goal. Also, evaluating task progress does not necessarily imply an ability to plan and execute actions. Therefore, it remains unclear to what extent these tasks measure genuine embodied planning.
  3. The conclusion that textual reasoning provides limited benefits for embodied tasks is not sufficiently supported. The effectiveness of reasoning in embodied VLMs can depend heavily on the inference pipeline, prompting strategy, and reasoning configuration. Without systematically controlling these factors, it is difficult to determine whether the limited improvements reflect fundamental limitations of textual reasoning or simply suboptimal evaluation settings.
Questions for authors
  1. The authors introduce a unified interaction schema to organize different embodied capabilities. However, it remains unclear why such a unified schema is necessary for benchmark design. Could these capabilities be evaluated independently without relying on a shared schema? The paper would benefit from a clearer justification of what fundamental limitations of existing benchmarks this design addresses and what additional value it brings to capability evaluation.
  2. The relatively poor performance of embodied VLMs may not necessarily indicate weaker embodied understanding or planning capabilities. Since the benchmark relies heavily on language-based QA, these models may underperform simply because they are less capable of following language instructions or producing answers in the expected format. The current evaluation does not clearly disentangle instruction-following ability from embodied reasoning ability. It is possible that these models could outperform general-purpose VLMs when coupled with action decoders in actual robotic tasks. How do the authors account for this distinction when interpreting the results?
Reviewer_g1KAOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

EmboScope is a multi-source benchmark for embodied perception, interaction understanding, and planning-related judgments. Its construction pipeline combines frame-level spatial annotation and segment-level interaction annotation in a shared schema, from which a rule-based engine generates questions under task-specific observation constraints. The benchmark contains 4,467 questions covering 14 tasks. Evaluation of 23 VLMs, with four additional thinking configurations, reveals heterogeneous capability profiles, complementary errors, and inconsistent accuracy gains from additional thinking. The paper contributes both an evaluation dataset and a reusable construction methodology.

Critical strengths
  1. Reusable construction with explicit observation boundaries. EmboScope brings human egocentric videos, real-robot interactions, and simulated interactions into a common annotation and question-generation workflow (§3.1; Figure 1). Separating interaction records from task rules makes the construction process easier to inspect and reuse. Explicit observation cutoffs for prediction tasks are also a sensible safeguard against future-information leakage. These are useful engineering choices that improve the organization of the evaluation resource.
  2. Informative reporting beyond a single aggregate score. The evaluation of 23 VLMs across 14 tasks provides a useful overview of performance variation (§§4.2–4.5). Task-level results and error-complementarity analyses offer more information than an aggregate ranking: on the closed-response subset, Kimi correctly answers 46.36% of the questions that Gemini gets wrong. These findings are useful for understanding where models differ and which errors they share, giving the dataset empirical value even though this does not by itself establish methodological novelty.
Critical weaknesses
  1. The paper’s innovation is not sufficiently distinguished from existing benchmark construction and evaluation practices. While the unified construction framework has practical value, the paper does not sufficiently establish what new evaluation insight it enables beyond combining existing sources, task families, and construction practices. Table 1 and Appendix C already identify reusable construction pipelines in RoboVQA and EgoPlan-Bench2, while the related-work discussion describes existing benchmarks for action, state, and planning judgments.
  2. The capability interpretation remains confounded by task and response-format differences. Some central diagnostic interpretations rely on comparisons that change several factors simultaneously. For example, §4.3 contrasts Gemini’s 95.17% on action between frames with 68.81% on action sequence description, but the latter requires a more extensive answer and uses open-response judging rather than multiple-choice scoring.
Questions for authors
  1. What is the novel contribution of the proposed question-generation framework beyond recombining existing datasets, tasks, or evaluation protocols? Which design choices introduce genuinely new forms of reasoning or measurement, and how do you distinguish innovation from superficial variation?
  2. Do the proposed questions reveal capabilities or failure modes that cannot be detected by existing benchmarks? Can you provide controlled comparisons showing that the new task formulation, rather than increased difficulty, ambiguity, or stricter scoring, is responsible for the observed diagnostic value?
  3. How does the framework support the discovery of previously untested reasoning strategies, and can its questions generate new hypotheses about model behavior? Please report whether the resulting insights generalize across datasets, models, and task types rather than reflecting artifacts of a particular construction process.
Reviewer_9ptgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces many questions across 14 tasks covering perception, embodied semantics, and planning. Its pipeline combines spatial annotations, VLM-generated interaction schemas, and rule-based question generation. Evaluation of different VLMs and four additional thinking configurations identifies differences in task performance and inconsistent benefits from thinking.

Critical strengths
  • The paper provides broad evaluation coverage, since it combines human, robot, and simulated interactions across 14 tasks. This provides a useful resource for comparing embodied understanding beyond a single aggregate score.
  • It provides a reusable construction design because it separates interaction annotation from question generation, which can reduce repeated annotation effort when extending tasks or sources.
  • Evaluation has detailed task results.
Critical weaknesses
  • There is insufficient quantitative reliability evidence. For instance, Appendix B describes extensive quality control but reports neither human verification coverage nor agreement and residual error rates. Open responses also rely on a single VLM judge without reported human agreement.
  • The validation of capability diagnosis is limited. Sections 4.2- 4.3 interpret task differences as capability gaps, but tasks differ in observations, answer formats, and scoring. Without text-only or matched episode controls, the results do not prove whether these gaps reflect the intended capabilities rather than task construction or evaluation effects.
  • Human verification details, such as coverage and residual error rates, are unreported.
  • The claimed construction scalability and reuse lack quantitative evidence on annotation costs and human effort.
Questions for authors
  1. What fraction of questions received human verification, what residual error and ambiguity rates were observed, and how closely does the VLM judge agree with human ratings?
  2. Do the main capability gaps persist under text-only and shuffled-frame controls?
  3. Can you quantify construction costs and demonstrate the effort saved when extending the benchmark?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.