- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces RoboApproachBench, a multimodal spatial understanding benchmark for real-robot target-approach processes. The benchmark contains 2,650 real-robot trajectories, 9,491 questions, and 14 task types, covering target identification, target depth and approach progress, robot ego-motion, turn sequences, target visibility, and relative bearing estimation during occlusion. The authors evaluate 28 open- and closed-source MLLMs and observe substantial performance differences across tasks, identifying bottlenecks in current models’ real-robot spatial reasoning capabilities.
Critical strengths
-
The paper focuses on the practically meaningful setting of a robot approaching a persistent target and organizes the transition from navigation to target interaction into a unified evaluation framework. The task taxonomy is relatively clear, distinguishing capabilities including target identification, metric geometry, ego-motion, turn structure, visibility events, and hidden-target state estimation.
-
The data are collected using real robots, with reference answers constructed from synchronized sensor measurements and human annotations. This provides a stronger embodied context than purely synthetic or static-image evaluations.
Critical weaknesses
1. Limited data diversity makes it difficult to support strong generalization claims
Although the benchmark contains 2,650 trajectories and 9,491 questions, all data are collected from only three rooms and are restricted to static targets, teleoperated trajectories, and successfully completed approaches. A large number of trajectories from the same environments increases within-domain sampling density rather than independent environment-level diversity. Therefore, the current model rankings are insufficient to support conclusions about general real-robot approach spatial reasoning.
2. The results do not yet demonstrate new insights beyond existing benchmarks
Section 4.3 compares visibility-event recognition with hidden-target bearing estimation, despite substantial differences in their input conditions and task difficulty. The resulting performance gap is largely expected from the task design, making this analysis closer to a sanity check than a new spatial-reasoning finding. Section 4.4 further controls the observation range and query time, but ultimately reports only that the observation endpoint may affect accuracy, that performance does not vary monotonically across query times, and that joint accuracy across multiple query times is low. Without further explanation or analysis, these results lack a substantive takeaway. Overall, Sections 4.3 and 4.4 provide some useful benchmark-design observations, but they do not yet establish a scientific insight with sufficient explanatory power, constructiveness, or novelty.
3. Missing data-quality checks and human-performance evaluation
The authors do not describe whether the final benchmark samples underwent human checking or report the corresponding pass rate. They also do not evaluate human performance as a baseline.
Questions for authors
Regarding Weakness 1, while it may be difficult to expand the benchmark to additional scenes, could the authors report and compare model performance across the three existing environments? I am particularly interested in whether the scores and model rankings differ substantially across scenes.
Regarding Weakness 3, could the authors provide details and results for the relevant data-quality checks, as well as a human-performance baseline?
I would be inclined to accept the paper if my concerns are adequately addressed.