acceptodds
Under review as a conference paper at ICLR 2027

PLAN-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?

Abstract

Vision-Language Models (VLMs) have achieved strong performance in visual perception and spatial reasoning, leading to growing interest in using them as high-level planners for embodied AI.However, physical tasks require models to integrate spatial information, reason over temporal states, and generate multi-step plans under various constraints, raising the question of whether current VLMs can reliably support such reasoning and planning independently of execution.We introduce Plan-Bench, a benchmark for evaluating the high-level reasoning and multi-step planning capabilities of VLMs relevant to embodied worlds, while isolating the influence of action execution.It covers three complementary dimensions, spanning 16 subclasses and 699 curated samples.Extensive evaluation of 28 VLMs reveals room for improvement, with even the strongest model achieving only 75.0% accuracy.We further find that model scaling generally improves these capabilities.Ablation studies also indicate larger models show increasing potential as high-level planners.Overall, Plan-Bench provides a lightweight probe for evaluating VLMs' reasoning and planning capabilities relevant to embodied environments and assessing their potential as decision makers for physical tasks.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Embodied Reasoning and Planning, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pcmbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Plan-Bench, an offline visual question-answering benchmark designed to assess VLMs' spatial cognition, temporal reasoning, and multi-step planning in embodied settings independently of low-level action execution. It comprises three capability dimensions, 16 subtasks, and 699 samples, including 183 multiple-choice questions with a single correct answer and 516 open-ended questions. Visual inputs are drawn from controlled scene generation, simulated environments, and existing datasets, with manual annotation and independent review. Open-ended responses are evaluated by an LLM judge using reference answers and explicit answer-equivalence rules, with human verification of 200 model responses. The authors evaluate 28 VLMs and report a best overall accuracy of 75.0%, substantial variation across subtasks, performance degradation from misleading views, and substantial gains from Thinking mode on multi-step planning. These experiments directly characterize task-specific performance on the offline benchmark and provide indirect evidence of the models' potential as practical embodied planners.

Critical strengths
  1. A unified diagnostic benchmark with substantive data quality controls. Plan-Bench covers cross-view information integration, distinguishing reflections from physical objects, temporal tracking, and complete plan generation under ordering and dependency constraints (§3.2, Figure 3, Appendix Tables 2 and 5). Each retained item requires unanimous approval from three reviewers who did not participate in the initial annotation. Filtering precedes model evaluation and checks visual sufficiency, answer correctness, and alignment with the intended capabilities, alongside scene-level deduplication (§3.1; Appendix A.3, Tables 6-7, pp. 16-17). These safeguards support its value as a lightweight, reusable diagnostic resource.

  2. The analyses provide useful information beyond aggregate rankings. Figure 4 shows that Kimi-K3 achieves 70.2% overall accuracy but only 10.0% on Rearrangement Planning, while Qwen3.8-Max scores only 38.0% on Persistent Counting. Figure 5 shows that additional misleading views degrade spatial judgments across all models tested in this analysis, and Figure 7 breaks down the gains from Thinking mode across capability dimensions (§4.3, pp. 7-8). These results help identify specific model weaknesses and caution against inferring comprehensive planning ability from a single overall score.

Critical weaknesses
  1. The contribution relative to closely related embodied planning benchmarks is not sufficiently established. Section 1 and Figure 1 (p. 2) broadly frame prior evaluations as either static reasoning tests or embodied tasks that couple planning with execution. However, existing benchmarks already assess visual planning separately from low-level motor control: PhyBlock evaluates one-shot complete assembly plans (§4.1), RoboVQA includes offline, open-ended questions asking for five remaining planning steps (Figure 1; §IV-A), and VidEgoThink evaluates video-conditioned symbolic action sequences (§3.2). None of these three direct precedents is cited in the supplied manuscript. Execution decoupling alone therefore does not establish the additional contribution of Plan-Bench.

    The comparison with cited benchmarks is also insufficiently precise. EgoPlan-Bench2 is mentioned in §2.1 (p. 2) primarily for multi-image reasoning, without discussing its explicit planning objective or its Horizon-k extension for predicting multiple future actions from candidate choices (§4.5.2, Figure 9, Table 4). EmbodiedBench is discussed in §2.2 (p. 3), but its EB-ALFRED and EB-Habitat tasks already expose high-level skill interfaces (§4.1-4.3). A more informative distinction is interactive skill planning with environment feedback versus fixed-observation offline scoring. These omissions affect the rationale for the benchmark, rather than merely citation completeness: Plan-Bench may offer useful additions through its task coverage, explicit constraints, and treatment of equivalent plans, but their additional diagnostic value relative to the closest prior work has not been clearly established.

  2. The headline results contain a verifiable aggregation inconsistency. Reconstructing Qwen3.8-Max's correct-answer counts from the six Spatial Cognition subtask accuracies in Figure 4 and the sample counts in Appendix Table 2 yields 169/259, or 65.3%, whereas Table 1 reports 66.4%. The corresponding counts for Temporal Reasoning and Multi-Step Planning are 164/210 and 188/230, respectively, yielding 521/699, or 74.5%, overall, rather than the 75.0% reported in Table 1 and the abstract (pp. 1, 6-7, 15). These differences cannot be explained by rounding to one decimal place. This inconsistency limits the verifiability of a central reported result and requires clarification as to whether it reflects transcription errors or different evaluation or aggregation protocols.

Questions for authors
  1. Result aggregation and inference settings. For each model, do Table 1, Figure 4, and Appendix Tables 8-10 use the same samples and underlying evaluation outputs? Please provide consistent correct-answer counts and denominators, and clarify any averaging across repeated runs or weighting. If the numerical differences are transcription errors, the condensation step preserves the original answers, and the main trends remain unchanged, this would substantially alleviate the reliability concern. Undisclosed differences in evaluated subsets or generation procedures would instead affect the fairness of model comparisons and the interpretation of the ablations.

  2. Performance of VLMs trained specifically for embodied tasks. Table 1 mainly emphasizes general-purpose VLMs. Could you compare representative VLMs with dedicated embodied reasoning or VQA training, such as embodied-task-aligned models evaluated in Embodied Arena, with suitable general-purpose baselines under the same evaluation protocol? If such models are already included, please identify them and clarify their training focus. Do they show gains or losses across Spatial Cognition, Temporal Reasoning, and Multi-Step Planning? This comparison would clarify how the benchmark responds to embodied specialization and whether it reveals improvements or trade-offs across the capabilities it measures, providing substantive insight beyond the current rankings.

  3. Interpretation of the navigation-manipulation performance gap. Section 4.3 and Figure 6 report 69.3% on Manipulation versus 54.6% on Navigation for Qwen3.8-27B, and a gap of approximately 15 percentage points for Step3-VL-10B. However, Claude-Opus-4.8 shows an even larger gap: 67.3% versus 46.2%, or 21.1 percentage points. Which subtasks or error types account for these mismatches, and what evidence supports the subsequent claim that larger models maintain more consistent performance across embodied tasks?

  4. Stability of the principal findings. Given 699 questions across 16 subtasks, with typically only 30-50 questions per subtask, how stable are the main rankings and conclusions about model scale, misleading views, and Thinking gains? Please distinguish uncertainty due to the finite set of questions from sensitivity to prompt wording or stochastic decoding. Relevant evidence could include stratified resampling of existing predictions, for example repeated subsets of roughly 350 questions, or targeted sensitivity analyses on representative models using reasonable prompt variations or different decoding seeds where sampling is enabled. Stable rankings and effect directions would support the reliability of the findings.

I would consider revising my rating upward if the authors clarify the benchmark’s added value relative to prior work and satisfactorily address the concerns about result consistency, robustness, and the evidence supporting the main conclusions.

Reviewer_QvxkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Plan-Bench, a benchmark for evaluating whether VLMs can perform high-level reasoning and multi-step planning relevant to embodied tasks when low-level action execution is factored out. It contains 699 curated questions (183 single-choice, 516 open-ended) organized into 16 subtasks under three dimensions: Spatial Cognition (259), Temporal Reasoning (210), and Multi-Step Planning (230). Items are produced by a three-stage pipeline: (i) capability taxonomy and task definition; (ii) dual-source visual acquisition, combining controlled synthetic generation (Three.js, OmniGibson) with existing datasets (CLEVR, CaptainCook4D, RxR, MEVIS, PMD/MSD, RoboMME, SpatialQA); and (iii) manual annotation of 2,580 candidate QA pairs followed by multi-criteria filtering and triple review, yielding 699 items (27.1% retention). Open-ended items are graded by an LLM judge (DeepSeek-V4-Flash) that receives hand-written answer hints enumerating valid alternatives; the judge agrees with human labels on all 200 sampled responses. Twenty-eight VLMs (23 open-weight, 5 commercial) are evaluated; the best model (Qwen3.8-Max) reaches 75.0% overall. Analyses report scaling trends, gains from Thinking mode (largest on planning), degradation under misleading views, navigation/manipulation alignment, and reflection-induced hallucination failures.

Critical strengths
  1. A well-motivated and genuinely useful problem framing. The paper correctly identifies that existing embodied benchmarks entangle high-level decision errors with low-level control errors, and that multimodal reasoning benchmarks do not target embodied planning. An execution-decoupled, lightweight QA probe for the "planner" component is a sensible artifact that practitioners could use to triage VLM planners before wiring them into a hierarchy. This is what makes the work potentially valuable — its value is entirely contingent on the probe being predictive.
  2. Transparent construction and broad, fine-grained evaluation. The pipeline is documented with explicit filtering criteria and before/after statistics (Tables 6–7, Figure 9), scene-level redundancy control, 16-subtask result tables, and 28 models spanning the open/commercial and scale spectrum. This transparency makes the resource credible, and the per-subtask breakdown is the most informative part of the paper.
Critical weaknesses
  1. The benchmark's central construct — "execution-decoupled high-level planning" — is not validated. No evidence is provided that Plan-Bench scores predict how well a VLM actually plans in an embodied system. Because the title, abstract, third contribution, and the concluding claim about "potential as high-level planners" all depend on this, the omission is consequential: as presented, Plan-Bench may largely measure multi-image perception and instruction-following rather than planning, and the reader cannot tell which.
  2. The empirical evidence is too thin and too unevenly controlled to support the strength of the claims. Small per-subtask samples without confidence intervals or significance testing; no human ceiling; judge validation covering only 29% of the data with a perfect-agreement result; an evaluation protocol that differs across models (temperature, greedy vs. sampling, an extra condensation pass applied only to Qwen models, undocumented configs for three models); and no contamination analysis despite heavy reuse of public pretraining-scale datasets. Any of these could plausibly reorder the model ranking or flip the "scaling helps / larger models are better planners" conclusion, which is the paper's main takeaway.
Questions for authors
  1. Do you have any evidence that Plan-Bench accuracy predicts embodied planning performance? For example, does a model's Plan-Bench score correlate with its success rate when used as the high-level planner in an existing hierarchical/agentic benchmark (EmbodiedBench, RoboCerebra, LongAct)? Without this, the framing of "potential as high-level decision makers" remains an assumption.
  2. Is there a human baseline on the full benchmark or a stratified subset? If humans also score, say, ~80%, how would you characterize the residual gap between that ceiling and Qwen3.8-Max's 75.0%?
Reviewer_t2iuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose Plan-Bench which comprises 699 instances, including 183 single-choice and 516 open-ended questions, covering 16 subtasks across 3 primary categories of:

  1. Spatial Cognition -- reflecting settings where VLMs may observe the same scene through multiple views, and it must integrate the observations (from different views) to form a cohere understanding of the enviornment
  2. Temporal Reasoning -- evaluating whether VLMs can understand temporal dependencies across sequential observations
  3. Multi-Step Planning -- for which the VLMs must reason about task goals, scene constraints, and action dependencies to generate plans prior to execution.

Their experiments evaluate 28 VLMs, including 23 open-source and 5 commercial models, and they share their main and ablation results.

Critical strengths
  • The paper is well organized and easy to follow.
  • The authors claim working on a problem that has not been studied so far, i.e., physical tasks require integrating spatial information, reasoning over temporal states, and generating multi-step plans, and whether VLMs can show promising reasoning and planning with respect to those tasks.
  • The work shows that the strongest VLMs can fail 25% of times.
Critical weaknesses
  • I appreciate the fact that the authors propose Plan-Bench as a new benchmark for evaluation of embodied VLM-based reasoning and planning; however, this work lacks novelty
  • I wish the authors would've dig deeper and provided in-depth analysis on why VLMs may perform poorly for some tasks while strongly for other tasks. In other words, the paper lacks strong discussion section
  • I could hardly learn anything new from the ablation study section
Questions for authors
  • Line 248-249: I'm wondering how 200 samples are randomly selected that the judge and human achieve 100% agreement? Does this challenge the difficulty of the tasks provided by the benchmark?
  • Line 370-371: Could you add more details about "More views do not necessarily lead to better spatial cognition"? What's the performance humans in that setting, 100%?
  • Are all tasks formed as QA?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.