- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Plan-Bench, an offline visual question-answering benchmark designed to assess VLMs' spatial cognition, temporal reasoning, and multi-step planning in embodied settings independently of low-level action execution. It comprises three capability dimensions, 16 subtasks, and 699 samples, including 183 multiple-choice questions with a single correct answer and 516 open-ended questions. Visual inputs are drawn from controlled scene generation, simulated environments, and existing datasets, with manual annotation and independent review. Open-ended responses are evaluated by an LLM judge using reference answers and explicit answer-equivalence rules, with human verification of 200 model responses. The authors evaluate 28 VLMs and report a best overall accuracy of 75.0%, substantial variation across subtasks, performance degradation from misleading views, and substantial gains from Thinking mode on multi-step planning. These experiments directly characterize task-specific performance on the offline benchmark and provide indirect evidence of the models' potential as practical embodied planners.
Critical strengths
-
A unified diagnostic benchmark with substantive data quality controls. Plan-Bench covers cross-view information integration, distinguishing reflections from physical objects, temporal tracking, and complete plan generation under ordering and dependency constraints (§3.2, Figure 3, Appendix Tables 2 and 5). Each retained item requires unanimous approval from three reviewers who did not participate in the initial annotation. Filtering precedes model evaluation and checks visual sufficiency, answer correctness, and alignment with the intended capabilities, alongside scene-level deduplication (§3.1; Appendix A.3, Tables 6-7, pp. 16-17). These safeguards support its value as a lightweight, reusable diagnostic resource.
-
The analyses provide useful information beyond aggregate rankings. Figure 4 shows that Kimi-K3 achieves 70.2% overall accuracy but only 10.0% on Rearrangement Planning, while Qwen3.8-Max scores only 38.0% on Persistent Counting. Figure 5 shows that additional misleading views degrade spatial judgments across all models tested in this analysis, and Figure 7 breaks down the gains from Thinking mode across capability dimensions (§4.3, pp. 7-8). These results help identify specific model weaknesses and caution against inferring comprehensive planning ability from a single overall score.
Critical weaknesses
-
The contribution relative to closely related embodied planning benchmarks is not sufficiently established. Section 1 and Figure 1 (p. 2) broadly frame prior evaluations as either static reasoning tests or embodied tasks that couple planning with execution. However, existing benchmarks already assess visual planning separately from low-level motor control: PhyBlock evaluates one-shot complete assembly plans (§4.1), RoboVQA includes offline, open-ended questions asking for five remaining planning steps (Figure 1; §IV-A), and VidEgoThink evaluates video-conditioned symbolic action sequences (§3.2). None of these three direct precedents is cited in the supplied manuscript. Execution decoupling alone therefore does not establish the additional contribution of Plan-Bench.
The comparison with cited benchmarks is also insufficiently precise. EgoPlan-Bench2 is mentioned in §2.1 (p. 2) primarily for multi-image reasoning, without discussing its explicit planning objective or its Horizon-k extension for predicting multiple future actions from candidate choices (§4.5.2, Figure 9, Table 4). EmbodiedBench is discussed in §2.2 (p. 3), but its EB-ALFRED and EB-Habitat tasks already expose high-level skill interfaces (§4.1-4.3). A more informative distinction is interactive skill planning with environment feedback versus fixed-observation offline scoring. These omissions affect the rationale for the benchmark, rather than merely citation completeness: Plan-Bench may offer useful additions through its task coverage, explicit constraints, and treatment of equivalent plans, but their additional diagnostic value relative to the closest prior work has not been clearly established.
-
The headline results contain a verifiable aggregation inconsistency. Reconstructing Qwen3.8-Max's correct-answer counts from the six Spatial Cognition subtask accuracies in Figure 4 and the sample counts in Appendix Table 2 yields 169/259, or 65.3%, whereas Table 1 reports 66.4%. The corresponding counts for Temporal Reasoning and Multi-Step Planning are 164/210 and 188/230, respectively, yielding 521/699, or 74.5%, overall, rather than the 75.0% reported in Table 1 and the abstract (pp. 1, 6-7, 15). These differences cannot be explained by rounding to one decimal place. This inconsistency limits the verifiability of a central reported result and requires clarification as to whether it reflects transcription errors or different evaluation or aggregation protocols.
Questions for authors
-
Result aggregation and inference settings. For each model, do Table 1, Figure 4, and Appendix Tables 8-10 use the same samples and underlying evaluation outputs? Please provide consistent correct-answer counts and denominators, and clarify any averaging across repeated runs or weighting. If the numerical differences are transcription errors, the condensation step preserves the original answers, and the main trends remain unchanged, this would substantially alleviate the reliability concern. Undisclosed differences in evaluated subsets or generation procedures would instead affect the fairness of model comparisons and the interpretation of the ablations.
-
Performance of VLMs trained specifically for embodied tasks. Table 1 mainly emphasizes general-purpose VLMs. Could you compare representative VLMs with dedicated embodied reasoning or VQA training, such as embodied-task-aligned models evaluated in Embodied Arena, with suitable general-purpose baselines under the same evaluation protocol? If such models are already included, please identify them and clarify their training focus. Do they show gains or losses across Spatial Cognition, Temporal Reasoning, and Multi-Step Planning? This comparison would clarify how the benchmark responds to embodied specialization and whether it reveals improvements or trade-offs across the capabilities it measures, providing substantive insight beyond the current rankings.
-
Interpretation of the navigation-manipulation performance gap. Section 4.3 and Figure 6 report 69.3% on Manipulation versus 54.6% on Navigation for Qwen3.8-27B, and a gap of approximately 15 percentage points for Step3-VL-10B. However, Claude-Opus-4.8 shows an even larger gap: 67.3% versus 46.2%, or 21.1 percentage points. Which subtasks or error types account for these mismatches, and what evidence supports the subsequent claim that larger models maintain more consistent performance across embodied tasks?
-
Stability of the principal findings. Given 699 questions across 16 subtasks, with typically only 30-50 questions per subtask, how stable are the main rankings and conclusions about model scale, misleading views, and Thinking gains? Please distinguish uncertainty due to the finite set of questions from sensitivity to prompt wording or stochastic decoding. Relevant evidence could include stratified resampling of existing predictions, for example repeated subsets of roughly 350 questions, or targeted sensitivity analyses on representative models using reasonable prompt variations or different decoding seeds where sampling is enabled. Stable rankings and effect directions would support the reliability of the findings.
I would consider revising my rating upward if the authors clarify the benchmark’s added value relative to prior work and satisfactorily address the concerns about result consistency, robustness, and the evidence supporting the main conclusions.