acceptodds
Withdrawn from ICLR 2027

RoboApproachBench: Diagnosing Spatial Understanding during Real-Robot Approach

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) are increasingly explored as perception and reasoning modules for embodied agents. Physical approach provides a concrete setting for evaluating their spatial understanding: as a robot moves toward a persistent target, target-relative geometry, ego-motion, and visibility change together. We introduce RoboApproachBench, a real-robot benchmark for diagnosing visual spatial understanding during recorded approaches. The benchmark contains 9,491 questions across 14 tasks, constructed from a corpus of 2,650 real-robot approach trajectories. Its tasks cover goal grounding, target geometry and approach progress, ego-motion, turn-sequence structure, and target visibility and bearing. Models receive task-specific monocular RGB observations and explicitly stated task information, while synchronized sensor measurements and target annotations support reference-answer construction. We evaluate 28 open- and closed-source MLLMs and observe uneven task-level performance. Across 28 MLLMs, peak-turn characterization exceeds temporal localization in 27 of the 28 models, while leading models’ strengths in approach ordering and visibility-event recognition coexist with weaker target-relative geometry. Across same-scale Qwen3-VL pairs, Thinking variants improve aggregate scores but exhibit task-specific regressions. Controlled bearing diagnostics on Qwen3.8-Flash yield higher accuracy with post-reappearance than pre-reappearance prefixes, highlighting the importance of observation endpoints when estimating earlier invisible states. RoboApproachBench provides a trajectory-grounded testbed for diagnosing task-specific strengths and limitations in MLLMs’ visual spatial understanding during real-robot approaches.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Models, Embodied AI, Spatial Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nhgYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces RoboApproachBench, a multimodal spatial understanding benchmark for real-robot target-approach processes. The benchmark contains 2,650 real-robot trajectories, 9,491 questions, and 14 task types, covering target identification, target depth and approach progress, robot ego-motion, turn sequences, target visibility, and relative bearing estimation during occlusion. The authors evaluate 28 open- and closed-source MLLMs and observe substantial performance differences across tasks, identifying bottlenecks in current models’ real-robot spatial reasoning capabilities.

Critical strengths
  1. The paper focuses on the practically meaningful setting of a robot approaching a persistent target and organizes the transition from navigation to target interaction into a unified evaluation framework. The task taxonomy is relatively clear, distinguishing capabilities including target identification, metric geometry, ego-motion, turn structure, visibility events, and hidden-target state estimation.

  2. The data are collected using real robots, with reference answers constructed from synchronized sensor measurements and human annotations. This provides a stronger embodied context than purely synthetic or static-image evaluations.

Critical weaknesses

1. Limited data diversity makes it difficult to support strong generalization claims

Although the benchmark contains 2,650 trajectories and 9,491 questions, all data are collected from only three rooms and are restricted to static targets, teleoperated trajectories, and successfully completed approaches. A large number of trajectories from the same environments increases within-domain sampling density rather than independent environment-level diversity. Therefore, the current model rankings are insufficient to support conclusions about general real-robot approach spatial reasoning.

2. The results do not yet demonstrate new insights beyond existing benchmarks

Section 4.3 compares visibility-event recognition with hidden-target bearing estimation, despite substantial differences in their input conditions and task difficulty. The resulting performance gap is largely expected from the task design, making this analysis closer to a sanity check than a new spatial-reasoning finding. Section 4.4 further controls the observation range and query time, but ultimately reports only that the observation endpoint may affect accuracy, that performance does not vary monotonically across query times, and that joint accuracy across multiple query times is low. Without further explanation or analysis, these results lack a substantive takeaway. Overall, Sections 4.3 and 4.4 provide some useful benchmark-design observations, but they do not yet establish a scientific insight with sufficient explanatory power, constructiveness, or novelty.

3. Missing data-quality checks and human-performance evaluation

The authors do not describe whether the final benchmark samples underwent human checking or report the corresponding pass rate. They also do not evaluate human performance as a baseline.

Questions for authors

Regarding Weakness 1, while it may be difficult to expand the benchmark to additional scenes, could the authors report and compare model performance across the three existing environments? I am particularly interested in whether the scores and model rankings differ substantially across scenes.

Regarding Weakness 3, could the authors provide details and results for the relevant data-quality checks, as well as a human-performance baseline?

I would be inclined to accept the paper if my concerns are adequately addressed.

Reviewer_TvkbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces a real-robot benchmark for diagnosing spatial understanding during target approach. Built from 2,650 trajectories, the benchmark contains 9,491 questions spanning 14 task categories: target recognition, depth and approach progress, ego-motion, heading structure, target disappearance, and bearing estimation for occluded targets. Models receive monocular RGB observations as specified per task. Reference answers are constructed from stereo matching, pose logging and human target annotations.

Critical strengths
  1. Complementary spatial tasks are organised around a unified real interaction process. The benchmark separates travelled distance from net approach, steering magnitude from extremal localization, and target visibility from occluded bearing estimation. This discourages generalizing success on one sub-task as evidence of holistic spatial competence.
  2. The authors conduct careful audits for input conditions and evaluation shortcuts. The appendix transparently discusses pseudo-ground-truth, answer margin tolerances, text-only ablations and option construction artefacts. Bearing diagnosis further separates causal prefixes from conditions with re-appearing observations. These details improve auditability of the benchmark.
Critical weaknesses
  1. In-scene accuracy of reference labels has not been directly measured. Depth values are sampled at the centre of target bounding boxes from stereo matching. For non-rigid, slender or partially occluded objects, the centre pixel may not correspond to the target surface. Error metrics measured on Middlebury cannot be directly transferred to the collected robot scenes. While the appendix defines sufficient conditions for stable labels and margin buffers, the empirical error bounds within the collected data are not measured. Low model accuracy may therefore conflate true capability deficits, label noise and inherent monocular measurement limits.
  2. Unified ranking and cross-task diagnosis suffer from comparability confounds. Closed-source models are evaluated on 300 questions per task while open-source models use the full benchmark, yet all models are ranked together. Differences in test set distribution complicate ranking interpretation. Tasks also vary in option count, answer format, time horizon and availability of future information, so raw accuracy gaps cannot be equated directly to spatial ability gaps. Text ablations are helpful, but human or geometric baseline performance under identical monocular inputs is missing to establish task solvability.
Questions for authors
  1. Can the authors independently measure depth, approach distance, velocity and occluded bearing error within their capture environment, and report label flip rates or results filtered to a high-confidence subset?
  2. If all models are re-evaluated on an identical 4,200-question subset, do rankings and open/closed-source gaps persist? Please report paired difference confidence intervals.
  3. What performance levels can humans and suitable geometric estimators achieve given identical monocular inputs? Do cross-task gaps remain after controlling for answer space and observation conditions?
Reviewer_5C1gOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RoboApproachBench, an offline benchmark for evaluating multimodal large language models’ spatial understanding during real-robot approaches to stationary targets. It comprises 9,491 questions across 14 tasks, constructed from 2,650 teleoperated trajectories collected in three indoor rooms. The authors evaluate 28 multimodal models and report uneven performance across tasks: stronger approach ordering, turn characterization, and visibility-event recognition coexist with weaker metric geometry and motion-extremum localization. The main contribution is a trajectory-grounded diagnostic benchmark and empirical analysis.

Critical strengths
  1. Focused and coherent evaluation setting. The benchmark organizes complementary spatial tasks around real-robot approaches to persistent targets, connecting goal-relative geometry, ego-motion, and visibility changes within a shared setting.
  2. Traceable reference-answer construction. Human target annotations, stereo-derived depth, and synchronized pose records support reproducible answer generation, while construction signals are withheld from evaluated models.
  3. Broad model coverage and targeted analyses. Evaluation of 28 multimodal models, matched Instruct or Thinking comparisons, and controlled bearing experiments provide evidence of task-specific performance differences and highlight the importance of observation boundaries.
Critical weaknesses
  1. Limited environmental diversity and insufficient support for broad generalization. Although the benchmark contains 9,491 questions, these are generated from only 2,650 trajectories collected in only three indoor rooms. Multiple questions can share the same trajectory, while different trajectories may share backgrounds and target objects. The question count therefore overstates the breadth of independent evaluation conditions. This narrow coverage limits the extent to which the reported findings support broader claims about spatial understanding in real-robot settings.
  2. Narrow evaluation scope and insufficiently demonstrated practical relevance. The benchmark evaluates offline visual question answering rather than action selection, closed-loop navigation, or successful target approach. While the queried quantities are relevant to robotics, the paper does not establish that performance on these tasks predicts downstream robotic competence or informs practical system design. Consequently, the application value of the benchmark remains insufficiently validated. As a result, the robotics framing feels imposed on a collection of general visual-spatial tasks.
  3. Incremental novelty relative to existing spatial and video benchmarks. Depth estimation, distance-change reasoning, ego-motion understanding, temporal localization, and spatial-state inference have already been studied in prior benchmarks. The principal distinction here is their organization around recorded robot approaches to persistent targets. The paper does not sufficiently demonstrate what additional diagnostic information this organization provides beyond existing evaluations, for example through matched-model comparisons or complementary error analyses. 4.Presentation and experimental reporting require tighter quality control. The disclosed use of generative AI is not itself a weakness; the concern is whether the resulting presentation has been adequately checked for clarity and consistency. For example, the title 3.2 in Figure 2 is incorrect and AI generated.
Questions for authors

Please see the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.