- Rating
- 3
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
S1. Genuine gap, sharply framed. The core question—whether a general VLM’s own spatial intelligence suffices for physical action, without learned action heads or privileged geometry—is well-motivated and distinct from both spatial QA and robot-policy benchmarks. Table 1’s “spatial responsibility allocation” framing (model-selected views / executed actions / interleaved control / no learned head) crisply positions the work against ManipBench, Theory of Space, ESPIRE, etc.
S2. Exceptional construction rigor. The seed pipeline (Appendix C.4) is well above benchmark norms: 2,000-step settling with ≤3° orientation drift, 318 probed → 280 retained (an honestly reported 88.1% conditional yield), and every retained instance certified by a verified expert execution end-to-end. This kind of physical feasibility verification is rarely done.
S3. Statistical care unusual for this area. Three-run means with sample SD, explicit refusal to rank the top-3 models given overlapping ranges (L319–322), and a diagnostics protocol with three independent annotators, majority voting, Fleiss κ = 0.942 plus Gwet AC1 for near-ceiling criteria, and an aggregation-rule robustness analysis (Appendix C.10.1). The non-circularity check on FG vs. checker verdicts (82.08% agreement with asymmetric error patterns, L1886–1896) is a convincing defense against “annotators copied the outcome.”
S4. Well-designed controlled interventions. The active/passive/sequential observation comparison (Table 3), the matched geometric-shift track, and the summary-source ablation (Table 4) isolate mechanisms rather than merely reporting leaderboard numbers. The finding that human-authored summaries improve all readers (+13.33) while Qwen summaries hurt others (−31.25) is genuinely informative about where failures live.
Critical strengths
Honest scoping throughout. The paper repeatedly refrains from overclaiming: it notes the active-vs-passive comparison does not isolate viewpoint selection alone (L414–415), that the geometric-shift track mixes target-selection and re-grounding demands (B.2), that diagnostics are behavioral not causal (B.1), and that simulation-only results say nothing about real-robot safety (Ethics). This discipline raises trust in the headline findings.
Critical weaknesses
W1. Headline causal claim is stronger than the evidence. The abstract/intro state “active perception improves success by 7.86–23.93%,” yet >99% of camera actions are semantic viewpoint selections and incremental adjustments account for <1% (L411–415, C.6); the paper itself concedes the comparison does not isolate the causal effect of viewpoint selection (L414–415). Since both passive controls also receive more images per control point (5 vs. 1), the gap may partly reflect context/attention burden from multi-image payloads rather than “active perception.” The main text acknowledges this, but the abstract-level framing (“active perception”) promises more than is tested.
W2. No difficulty calibration. There is no oracle upper bound (e.g., a model given privileged object pose/depth) and no human study through the same interface. Without either, 53.9% top success is hard to interpret: is this a ceiling imposed by the interface/action space, or a real spatial-reasoning deficit? The expert-execution verification shows the instances are solvable, but not how hard they are for a competent agent under this protocol.
W3. Single-run controls with no variance estimates. The passive controls, geometric-shift track, and summary ablation all use matched single runs. Given API-level nondeterminism, headline gaps like the 32.14-point geometric drop (single run) carry no uncertainty bounds. The three-run protocol exists for the base task; extending it to even one model per control would materially strengthen the claims.
W4. Budget sensitivity unexamined. The 0.00% strict long-horizon success alongside 61/100 object placements (Qwen3.8-max) strongly suggests budget/termination sensitivity, yet no experiment varies Bτ. Without it, one cannot separate “cannot do the task” from “needs more steps.”
Questions for authors
Can you add an oracle-conditioned upper bound (privileged object pose or depth supplied to the same interface) for at least the top model? This would calibrate W2 directly. Can you run the passive/sequential controls three times for one model to bound the variance of the 7.86–23.93 point gaps (W3)? Does success change materially at 2× budget for the top model, on the base suite and the long-horizon track (W4)? Can you probe metric estimation directly—ask models to report predicted object displacement before acting and compare to ground truth—to separate “misestimates geometry” from “misemits action”? In the geometric-shift track, can you disentangle target selection (distractors) from re-grounding by adding distractor-free variants (B.2 admits the confound)? Please name the simulator and embodiment explicitly in the main text (W5).
3 comments
Thank you again for your time and consideration. We would like to briefly follow up on our rebuttal. We hope the additional experiments and clarifications have addressed your concerns. Please let us know if anything remains unclear. We would be happy to provide further clarification. Thank you again for your valuable feedback.
Dear Reviewer,
Thank you again for your time and consideration. As the discussion period is coming to an end, we would like to briefly follow up on our rebuttal.
We hope our additional experiments and clarifications have addressed your concerns. We would greatly appreciate your final feedback, particularly whether any concerns remain or your assessment of our work has changed.
Thank you again for your valuable time and feedback.
Best regards, The Authors
We thank the reviewer for the careful assessment and constructive suggestions. We address each concern below.
W1. Active-Perception Claims
We added random, cyclic, and heuristic view-selection controls for GPT-5.6-sol across the base tasks. All four conditions follow the original base evaluation protocol, with the same total action budget, single-image input format, and other evaluation settings. The model still decides when to request an observation, while each baseline replaces only the selected viewpoint with its corresponding rule. Model-selected views achieve 56.07% success, compared with 23.57% for random, 27.14% for cyclic, and 35.71% for heuristic selection. These results show a consistent advantage over rule-based viewpoint selection under the same evaluation protocol. We will revise the active-perception claims in the abstract and introduction accordingly and include the new controls in the revision.
W2. Privileged Geometry Reference
We added an oracle-conditioned evaluation with Qwen3.8-max, the strongest model in our main evaluation, by providing ground-truth object poses alongside RGB observations. Both conditions were freshly evaluated across the base tasks, with demonstration summaries, action interfaces, executors, and action budgets held fixed. Providing ground-truth object poses increases success from 56.43% to 68.57% (+12.14 percentage points). This oracle-pose condition provides an upper-bound reference for evaluating model performance when object-pose estimation is no longer required, while retaining the model's responsibility for contact reasoning, action planning, and metric command generation. We will include the overall and per-task results in the revision and extend oracle-conditioned evaluations to more models to further characterize their performance upper bounds.
W3. Single-Run Controls and Variance Estimates
We extended the passive observation controls, geometric-shift evaluation, and summary-source ablation to three runs for GPT-5.6-sol. Each run uses 20 seeds per task, and all evaluations use the same API model version. Base and geometric-shift conditions use matched seeds. We report mean success ± sample standard deviation.
The results support all three original findings. Active observation remains better than both passive conditions. Geometric shift reduces mean success by 28.33 ± 3.38 percentage points. Summary quality also has a clear effect: human-authored summaries improve success by 26.67 points over the model's own summary. These repeated evaluations provide variance estimates for all three analyses. We will include detailed per-run and per-task results in the revision.
W4. Long-Horizon Action Budget
We evaluated Qwen3.8-max on the long-horizon task using twice the original action budget. The model completed 1 out of 20 episodes, compared with 0/20 under the original budget. This preliminary result suggests that the action budget may be one factor affecting task completion. We will report this result, further clarify this limitation, and include additional tasks and experiments in the revision.
W5. Robot Embodiment and Motion Planner
VA-Bench uses the ALOHA-AgileX embodiment in RoboTwin2.0. All models share a fixed execution layer responsible for inverse kinematics, collision checking, and trajectory generation. No task-specific learned action head is used.
Additional Diagnostics: Metric Estimation and Distractor Effects
We conducted two targeted experiments to examine these issues. Metric estimation and command encoding: We tested GPT-5.6-sol on 10 cube-grasping displacement trials. The model predicted the correct axis and sign in all 10 trials, and 9 estimates had errors below 10 mm. In the remaining case, the ground-truth displacement was −50 mm, while the model estimated −62 mm. The model correctly encoded its −62 mm estimate into the action command, indicating that the error came from metric estimation rather than command encoding.
Distractor effects: We compared distractor-present and distractor-free cube-grasping settings using 10 matched seeds. Success increased from 50% (5/10) to 70% (7/10) without distractors, indicating that distractors contribute to task difficulty. We will include both experiments and their detailed settings in the revision, and extend these evaluations to more models and task families.