acceptodds
Under review as a conference paper at ICLR 2027

VA-Bench: From Spatial Reasoning to Embodied Action in General-Purpose VLMs

Abstract

Spatial intelligence in vision-language models is commonly evaluated through question answering over pre-specified visual observations. Robot manipulation offers a more demanding test, but existing evaluations typically assess learned visuomotor policies rather than the spatial intelligence of general-purpose VLMs. We introduce VA-Bench, which uses closed-loop embodied manipulation to evaluate whether general VLMs can turn spatial understanding into physical action. Given a task goal and visual demonstrations of the task procedure, the model must infer action-relevant geometry and generate metric robot actions without relying on a task-specific action policy or privileged spatial information. During execution, it can perform active perception by choosing additional viewpoints when needed and revise subsequent actions from interaction feedback. We refer to this overall capability as self-directed metric spatial grounding. VA-Bench further tests geometric shifts by preserving the task procedure while changing the action-relevant geometry. Across 14 tasks, existing VLMs show near-perfect target identification and task understanding, yet remain substantially weaker at spatial grounding and online correction; the highest overall task success rate is only 53.9%. Active perception improves success by 7.86–23.93% over controls given predefined views, while geometric shifts cause drops of up to 32.14%. Overall, these results show that for general VLMs, knowing what to do does not yet translate into knowing where and how to act reliably.

Primary area:
datasets and benchmarks
Keywords:
Spatial Intelligence, VLM, Robotics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nYQeOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

S1. Genuine gap, sharply framed. The core question—whether a general VLM’s own spatial intelligence suffices for physical action, without learned action heads or privileged geometry—is well-motivated and distinct from both spatial QA and robot-policy benchmarks. Table 1’s “spatial responsibility allocation” framing (model-selected views / executed actions / interleaved control / no learned head) crisply positions the work against ManipBench, Theory of Space, ESPIRE, etc.

S2. Exceptional construction rigor. The seed pipeline (Appendix C.4) is well above benchmark norms: 2,000-step settling with ≤3° orientation drift, 318 probed → 280 retained (an honestly reported 88.1% conditional yield), and every retained instance certified by a verified expert execution end-to-end. This kind of physical feasibility verification is rarely done.

S3. Statistical care unusual for this area. Three-run means with sample SD, explicit refusal to rank the top-3 models given overlapping ranges (L319–322), and a diagnostics protocol with three independent annotators, majority voting, Fleiss κ = 0.942 plus Gwet AC1 for near-ceiling criteria, and an aggregation-rule robustness analysis (Appendix C.10.1). The non-circularity check on FG vs. checker verdicts (82.08% agreement with asymmetric error patterns, L1886–1896) is a convincing defense against “annotators copied the outcome.”

S4. Well-designed controlled interventions. The active/passive/sequential observation comparison (Table 3), the matched geometric-shift track, and the summary-source ablation (Table 4) isolate mechanisms rather than merely reporting leaderboard numbers. The finding that human-authored summaries improve all readers (+13.33) while Qwen summaries hurt others (−31.25) is genuinely informative about where failures live.

Critical strengths

Honest scoping throughout. The paper repeatedly refrains from overclaiming: it notes the active-vs-passive comparison does not isolate viewpoint selection alone (L414–415), that the geometric-shift track mixes target-selection and re-grounding demands (B.2), that diagnostics are behavioral not causal (B.1), and that simulation-only results say nothing about real-robot safety (Ethics). This discipline raises trust in the headline findings.

Critical weaknesses

W1. Headline causal claim is stronger than the evidence. The abstract/intro state “active perception improves success by 7.86–23.93%,” yet >99% of camera actions are semantic viewpoint selections and incremental adjustments account for <1% (L411–415, C.6); the paper itself concedes the comparison does not isolate the causal effect of viewpoint selection (L414–415). Since both passive controls also receive more images per control point (5 vs. 1), the gap may partly reflect context/attention burden from multi-image payloads rather than “active perception.” The main text acknowledges this, but the abstract-level framing (“active perception”) promises more than is tested.

W2. No difficulty calibration. There is no oracle upper bound (e.g., a model given privileged object pose/depth) and no human study through the same interface. Without either, 53.9% top success is hard to interpret: is this a ceiling imposed by the interface/action space, or a real spatial-reasoning deficit? The expert-execution verification shows the instances are solvable, but not how hard they are for a competent agent under this protocol.

W3. Single-run controls with no variance estimates. The passive controls, geometric-shift track, and summary ablation all use matched single runs. Given API-level nondeterminism, headline gaps like the 32.14-point geometric drop (single run) carry no uncertainty bounds. The three-run protocol exists for the base task; extending it to even one model per control would materially strengthen the claims.

W4. Budget sensitivity unexamined. The 0.00% strict long-horizon success alongside 61/100 object placements (Qwen3.8-max) strongly suggests budget/termination sensitivity, yet no experiment varies Bτ. Without it, one cannot separate “cannot do the task” from “needs more steps.”

Questions for authors

Can you add an oracle-conditioned upper bound (privileged object pose or depth supplied to the same interface) for at least the top model? This would calibrate W2 directly. Can you run the passive/sequential controls three times for one model to bound the variance of the 7.86–23.93 point gaps (W3)? Does success change materially at 2× budget for the top model, on the base suite and the long-horizon track (W4)? Can you probe metric estimation directly—ask models to report predicted object displacement before acting and compare to ground truth—to separate “misestimates geometry” from “misemits action”? In the geometric-shift track, can you disentangle target selection (distractors) from re-grounding by adding distractor-free variants (B.2 admits the confound)? Please name the simulator and embodiment explicitly in the main text (W5).

3 comments

We thank the reviewer for the careful assessment and constructive suggestions. We address each concern below.

W1. Active-Perception Claims

We added random, cyclic, and heuristic view-selection controls for GPT-5.6-sol across the base tasks. All four conditions follow the original base evaluation protocol, with the same total action budget, single-image input format, and other evaluation settings. The model still decides when to request an observation, while each baseline replaces only the selected viewpoint with its corresponding rule. Model-selected views achieve 56.07% success, compared with 23.57% for random, 27.14% for cyclic, and 35.71% for heuristic selection. These results show a consistent advantage over rule-based viewpoint selection under the same evaluation protocol. We will revise the active-perception claims in the abstract and introduction accordingly and include the new controls in the revision.

W2. Privileged Geometry Reference

We added an oracle-conditioned evaluation with Qwen3.8-max, the strongest model in our main evaluation, by providing ground-truth object poses alongside RGB observations. Both conditions were freshly evaluated across the base tasks, with demonstration summaries, action interfaces, executors, and action budgets held fixed. Providing ground-truth object poses increases success from 56.43% to 68.57% (+12.14 percentage points). This oracle-pose condition provides an upper-bound reference for evaluating model performance when object-pose estimation is no longer required, while retaining the model's responsibility for contact reasoning, action planning, and metric command generation. We will include the overall and per-task results in the revision and extend oracle-conditioned evaluations to more models to further characterize their performance upper bounds.

W3. Single-Run Controls and Variance Estimates

We extended the passive observation controls, geometric-shift evaluation, and summary-source ablation to three runs for GPT-5.6-sol. Each run uses 20 seeds per task, and all evaluations use the same API model version. Base and geometric-shift conditions use matched seeds. We report mean success ± sample standard deviation.

EvaluationThree-run success (Mean ± SD, %)
Observation (14 tasks)Active: 51.55 ± 4.31; Bundled: 32.38 ± 1.83; Sequential: 35.83 ± 2.89
Geometric shift (7 tasks)Base: 71.67 ± 7.23; Shift: 43.33 ± 4.06
Summary source (3 tasks)Own: 63.33 ± 10.93; Qwen: 23.89 ± 2.55; Opus: 77.78 ± 1.92; Human: 90.00 ± 1.67

The results support all three original findings. Active observation remains better than both passive conditions. Geometric shift reduces mean success by 28.33 ± 3.38 percentage points. Summary quality also has a clear effect: human-authored summaries improve success by 26.67 points over the model's own summary. These repeated evaluations provide variance estimates for all three analyses. We will include detailed per-run and per-task results in the revision.

W4. Long-Horizon Action Budget

We evaluated Qwen3.8-max on the long-horizon task using twice the original action budget. The model completed 1 out of 20 episodes, compared with 0/20 under the original budget. This preliminary result suggests that the action budget may be one factor affecting task completion. We will report this result, further clarify this limitation, and include additional tasks and experiments in the revision.

W5. Robot Embodiment and Motion Planner

VA-Bench uses the ALOHA-AgileX embodiment in RoboTwin2.0. All models share a fixed execution layer responsible for inverse kinematics, collision checking, and trajectory generation. No task-specific learned action head is used.

Additional Diagnostics: Metric Estimation and Distractor Effects

We conducted two targeted experiments to examine these issues. Metric estimation and command encoding: We tested GPT-5.6-sol on 10 cube-grasping displacement trials. The model predicted the correct axis and sign in all 10 trials, and 9 estimates had errors below 10 mm. In the remaining case, the ground-truth displacement was −50 mm, while the model estimated −62 mm. The model correctly encoded its −62 mm estimate into the action command, indicating that the error came from metric estimation rather than command encoding.

Distractor effects: We compared distractor-present and distractor-free cube-grasping settings using 10 matched seeds. Success increased from 50% (5/10) to 70% (7/10) without distractors, indicating that distractors contribute to task difficulty. We will include both experiments and their detailed settings in the revision, and extend these evaluations to more models and task families.

Thank you again for your time and consideration. We would like to briefly follow up on our rebuttal. We hope the additional experiments and clarifications have addressed your concerns. Please let us know if anything remains unclear. We would be happy to provide further clarification. Thank you again for your valuable feedback.

Dear Reviewer,

Thank you again for your time and consideration. As the discussion period is coming to an end, we would like to briefly follow up on our rebuttal.

We hope our additional experiments and clarifications have addressed your concerns. We would greatly appreciate your final feedback, particularly whether any concerns remain or your assessment of our work has changed.

Thank you again for your valuable time and feedback.

Best regards, The Authors

Reviewer_2fPmOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VA-BENCH, an embodied benchmark designed to evaluate whether general-purpose VLMs can transform spatial understanding into successful physical action within 14 task families. Conventional spatial reasoning benchmarks use fixed observations and text-based answers, VA-BENCH evaluates models through closed-loop manipulation. Models receive task instructions, RGB observations, and demonstration-derived procedural summaries, and have to actively acquire additional views, infer robot-object geometry, generate metric robot actions, and revise behavior using execution feedback. A shared executor performs trajectory generation while all spatial decisions remain the responsibility of the evaluated VLM. Active perception consistently outperforms the two passive observation settings, and geometric shifts reduce success rates. None of the tested models completes the long-horizon composition task, although some make substantial partial progress.

Critical strengths

S1: The central contribution is conceptually strong. The paper identifies an important gap between existing spatial-reasoning evaluations and embodied-manipulation benchmarks and proposes a benchmark that directly evaluates whether general-purpose VLMs can convert visual observations into successful physical interaction. This addresses a meaningful and timely research question.

S2: In addition to terminal success, it introduces subtask progress metrics and nine trajectory-level diagnostics covering perception, manipulation, and recovery. These analyses provide useful insight into where failures arise.

S3: The active-perception, geometric-shift, demonstration-summary, and long-horizon composition experiments meaningfully strengthen the paper. These studies help separate different failure sources and increase confidence that the observed weaknesses are not artifacts of a single benchmark configuration.

Critical weaknesses

W1: The benchmark does not clearly separate spatial-grounding limitations from difficulties with planning and the metric-control interface. Section 3.4 and Appendix B acknowledge this, but the paper still interprets the failures mainly through spatial grounding. The behavioral annotations describe what happens without establishing why. This matters because the benchmark’s central contribution depends on what its scores actually measure.

W2: The main model comparison also depends on the quality of each model’s demonstration summary. Table 4 shows substantial sensitivity: GPT-5.6-sol achieves 73.33% with its own summary but 21.67% with Qwen’s on the evaluated subset. The summary ablation is useful, but does not establish whether the main rankings persist under matched procedural information. This limits how confidently the differences can be interpreted as differences in spatial-grounding ability.

Questions for authors

Q1: Could you provide more evidence separating spatial-grounding errors from difficulty using the direct metric-control interface? I appreciate that Section 3.4 and Appendix B discuss this limitation. What remains unclear to me is how much of the observed failure can be attributed to each.

Q2: Section 5.1 explicitly acknowledges that the active/passive comparison does not isolate viewpoint selection alone. Could you further separate the effect of choosing when and which view to request from differences in observation count and presentation? A budget-matched random or fixed-order view-request baseline might help here.

Q3: Given the large summary-source effects in Table 4, how should we interpret the main model rankings? Do you have evidence that the differences persist when procedural information is held more closely matched?

Q4: Could you unpack the near-ceiling spatial-perception scores under geometric shift in Table 16? In particular, please report TL, AE, and SR separately, with their denominators and corresponding scores on the same base-task subset. This would also help distinguish target-selection problems from geometric re-grounding failures.

Q5 (minor): Figure 1, Example 2 asks about grasping a bottle, but the answer describes moving a ball over a bowl. Please correct the example.

Suggestions that would benefit the paper (not evaluation-critical to me):

  • A small cross-embodiment study would be useful to see whether the rankings and failure patterns persist. I do not consider this necessary for the current contribution, but the conclusions should remain tied to the evaluated embodiment and interface.
  • A spatially specialized model or VLA-based system could provide useful context. I recognize that a learned action policy changes the evaluation setting, so I would treat this as a complementary comparison.
  • The main paper already points to Appendix B. A short statement of the most important limitations directly in the main text would still help.
  • An appendix roadmap would make the supplementary material easier to navigate.
3 comments

We thank the reviewer for the careful assessment and constructive suggestions. We address each concern below.

W1 / Q1. Spatial Grounding and Metric Control

To examine the role of geometric information, we added a paired RGB vs. RGB + GT pose experiment with Qwen3.8-max. Both conditions were freshly evaluated across the base tasks, while summaries, action interfaces, executors, and action budgets were held fixed. Providing accurate object poses increases success from 56.43% to 68.57% (+12.14 percentage points), suggesting that object-pose grounding contributes to the observed performance gap. However, accurate object poses alone do not guarantee successful manipulation: the model must still reason about grasp and contact geometry, plan actions, and generate precise metric commands. Together with the nine behavioral diagnostics, these results provide complementary evidence from different perspectives for understanding the challenges of spatially grounded manipulation. We will further disentangle the individual contributions of perception, planning, and action generation in future work.

W2 / Q3. Demonstration Summary and Model Rankings

Our main evaluation measures the complete pipeline, including demonstration understanding and action execution. Summary quality therefore affects performance. Table 4 shows that model differences persist under shared procedural information. With the same human-authored summary, Qwen3.8-max, Opus-5, GPT-5.6-sol, GPT-5.6-terra, and Gemini-3.6 achieve 95.00%, 91.67%, 88.33%, 63.33%, and 38.33% success, respectively. The model ranking remains unchanged when all five models receive the same human-authored summary. This shows that summary quality affects performance but does not fully explain the differences between models on the evaluated three-task subset. We will extend this controlled comparison to more tasks and models and report the results in the revision.

Q2. Active Viewpoint Selection

We added random, cyclic, and heuristic view-selection controls for GPT-5.6-sol across the base tasks. All four conditions follow the original base evaluation protocol, sharing the same total action budget, single-image input format, and other evaluation settings. The model retains control over when to request an observation, while each baseline replaces only the selected viewpoint with its corresponding rule. Model-selected views achieve 56.07% success, compared with 23.57% for random, 27.14% for cyclic, and 35.71% for heuristic selection, outperforming the strongest rule-based baseline by 20.36 percentage points. These results show that model-directed viewpoint selection outperforms rule-based alternatives under the same action budget. Since observation requests remain model-directed, their actual number and timing may vary across conditions. We will clarify this distinction and report the implementation details in the revision.

Q4. Spatial-Perception Scores under Geometric Shift

We report TL, AE, and SR separately for the seven matched Base and Generalization tasks (140 episodes per model and condition; %, Base / Generalization). Qwen3.8-max: TL 100.00/99.29, AE 91.43/100.00, SR 90.00/100.00; GPT-5.6-sol:TL 100.00/100.00, AE 86.43/100.00, SR 90.71/97.86; GPT-5.6-terra: TL 100.00/100.00, AE 62.14/100.00, SR 68.57/100.00; GPT-5.6-luna: TL 100.00/100.00, AE 43.57/100.00, SR 77.14/100.00. TL remains near ceiling, indicating that models generally identify the correct target, but this does not guarantee accurate geometric grounding. Generalization scenes introduce more occlusion, geometric changes, and distractors. These ambiguities prompt models to acquire useful views and use new evidence (AE), as well as make alignment adjustments that reduce gripper–target offsets (SR). These scores reflect observable local behaviors, not guaranteed task success. We will include representative trajectories in the revision and develop more targeted diagnostics for held-out settings in future work.

Q5. Figure 1 Correction

We will correct Figure 1, Example 2, so that the answer matches the bottle-grasping question.

Additional Suggestions

We will add a brief limitations statement to the main text and an appendix roadmap. Our conclusions remain specific to the evaluated embodiment and control interface. We will explore cross-embodiment evaluation and comparisons with spatially specialized models or VLA systems in future work.

The rebuttal substantially addresses my concerns. The new active-view controls provide convincing evidence that model-selected viewpoints outperform random, cyclic, and heuristic alternatives under a matched action budget. The shared human-summary experiment also shows that the ranking of the five evaluated models persists. The RGB vs. RGB + GT pose experiment provides useful evidence that object-pose grounding contributes to the observed failures, while also showing that pose information alone does not resolve planning and metric-control challenges.

My remaining concern is primarily about interpretation rather than the overall value of the benchmark. The benchmark still combines several capabilities, and the behavioral diagnostics identify observable local behaviors rather than fully disentangling their underlying causes. The near-ceiling AE and SR scores under geometric shift, despite lower terminal success, suggest that these metrics may capture whether models attempt exploration or alignment rather than whether they do so accurately enough to solve the task. Could you clarify the diagnostic criteria and their episode-level relationship to terminal success?

Overall, the rebuttal strengthens the paper and increases my confidence in acceptance. I maintain my rating of Weak Accept, now with a more positive assessment.

We sincerely thank the reviewer for the thoughtful feedback and positive reassessment of our work. We greatly appreciate the recognition of our additional experiments and the constructive suggestions throughout the review process. We will further clarify the interpretation and scope of the behavioral diagnostics in the revised manuscript. Thank you again for your time and valuable feedback.

Reviewer_Te5jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VA-Bench evaluates general-purpose VLMs through closed-loop manipulation in RoboTwin. Across 14 tasks, models use demonstration-derived procedural summaries, RGB observations, and robot proprioception to select viewpoints and generate metric robot actions. The benchmark also tests geometric shifts and long-horizon composition. Results show that strong task recognition does not translate into reliable execution, with the best overall success rate reaching 53.9%.

Critical strengths
  1. The benchmark goes beyond spatial question answering by requiring models to generate metric manipulation commands and complete tasks through closed-loop interaction, without a learned task-specific action policy.
  2. Clear separation of model decisions and execution support. Models must infer task-relevant object geometry from RGB observations and feedback, while a shared executor handles inverse kinematics and trajectory generation. Privileged object coordinates, depth, and grasp annotations are withheld.
  3. Useful controlled comparisons and behavioral analysis. Active versus passive observation comparisons, geometric-shift tests, demonstrationsummary controls, and trajectory annotations provide complementary evidence about execution failures beyond aggregate success rates.
Critical weaknesses
  1. Incremental novelty and insufficiently established added value. The benchmark primarily integrates existing RoboTwin manipulation tasks with a VLM interface. This combination does not by itself establish a substantial conceptual advance.
  2. Confounded active-perception comparisons. The active and passive conditions differ in observation selection, presentation format, and observation count, making it difficult to attribute the performance gains specifically to intelligent viewpoint selection. Moreover, over 99% of camera actions select predefined semantic viewpoints. Budget-matched random, cyclic, and heuristic view-selection baselines would better establish the value of model directed perception.
  3. Limited isolation of spatial grounding from other system components. Task success depends jointly on visual interpretation, numerical action generation, planning, context management, and the fixed executor. Behavioral annotations describe observed failures but do not establish their causes. Component-specific interventions would help determine whether spatial grounding is the principal bottleneck.
  4. Simplified sensing interface and no real world validation. Camera actions directly update the simulated viewpoint without accounting for camera-arm reachability, collision constraints, or motion cost beyond a discrete action charge. All experiments are conducted in simulation, leaving the practical relevance of the results in the realworld.
Questions for authors

Please see Critical Weaknesses.

3 comments

We thank the reviewer for the constructive feedback and valuable suggestions. We address each concern below.

W1. Novelty and Added Value

RoboTwin serves as the simulation environment for VA-Bench. Our contribution is a distinct evaluation protocol that tests whether VLMs can turn spatial understanding into reliable robot actions. Unlike conventional spatial understanding benchmarks, VA-Bench requires models to select viewpoints, estimate metric spatial relations, generate actions, and correct errors through feedback. We also select, reorganize, and design 14 task families, together with geometric-shift and long-horizon evaluations.

Our results reveal a clear gap between understanding and execution. Models achieve 100% target localization and at least 99.6% manipulation-semantics understanding, yet their spatial-relation scores are only 70.4–78.9%. The best three-run mean task success rate is 53.93 ± 3.17%. These findings highlight the value of evaluating actionable spatial intelligence beyond spatial recognition. We will clarify these contributions in the revision.

W2. Active-Perception Comparisons

We added random, cyclic, and heuristic view-selection baselines for GPT-5.6-sol across the base tasks. All four conditions follow the original base evaluation protocol, with identical total action budgets, single-image input formats, and other evaluation settings. The model still decides when to request an observation, while each baseline replaces only the selected viewpoint with its corresponding rule. Model-selected views achieve 56.07% success, compared with 23.57% for random, 27.14% for cyclic, and 35.71% for heuristic selection, outperforming the strongest rule-based baseline by 20.36 percentage points. These results demonstrate the advantage of model-directed viewpoint selection over rule-based alternatives under the same evaluation protocol and action budget. We will include these results and implementation details in the revision.

W3. Spatial Grounding and Component Attribution

To further investigate the role of spatial information, we added a paired RGB versus RGB + GT pose experiment with Qwen3.8-max, one of our top-performing models. Both conditions were freshly evaluated across the base tasks. Summaries, action interfaces, executors, and budgets were held fixed.

Providing ground-truth object poses increases success from 56.43% to 68.57% (+12.14 percentage points), indicating that object-pose grounding contributes to the performance gap. However, pose information alone does not guarantee successful manipulation. The model must still infer grasp and contact geometry, plan the manipulation, and generate precise metric actions. This intervention therefore shows that spatial grounding accounts for part of the challenge, while downstream planning and action generation remain important. We will include these results and investigate individual components further in future work.

W4. Real-World Validation

To provide a preliminary hardware evaluation, we conducted a cube-grasping experiment using a Songling Piper robotic arm and 11 Hikvision DS-E12a cameras, with the camera layout designed to approximate the viewpoints in the simulation. GPT-5.6-sol achieved 2/20 successes without a demonstration summary, while the success count increased to 8/20 when provided with a summary generated from a simulation video. We also qualitatively observed that the model selected additional viewpoints more frequently when the summary was provided. These results provide preliminary evidence that our evaluation pipeline can be implemented on physical hardware. We will include the hardware setup and experimental results in the revision, along with further evaluation settings and results.

Thank you again for your time and consideration. We would like to briefly follow up on our rebuttal. We hope the additional experiments and clarifications have addressed your concerns. Please let us know if anything remains unclear. We would be happy to provide further clarification. Thank you again for your valuable feedback.

Dear Reviewer,

Thank you again for your time and consideration. As the discussion period is coming to an end, we would like to briefly follow up on our rebuttal.

We hope our additional experiments and clarifications have addressed your concerns. We would greatly appreciate your final feedback, particularly whether any concerns remain or your assessment of our work has changed.

Thank you again for your valuable time and feedback.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.