- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces EmbodimentSemantic, a spatial scene-graph dataset and benchmark built from 10 LIBERO-Spatial manipulation tasks (500 demonstrations, 62,250 dual-view timesteps) along with a 257-episode real-robot SO101 dataset. Pairwise spatial relations among seven objects are automatically extracted from MuJoCo state in the simulated split, and 10 VLMs are evaluated on exact triplet F1 alongside diagnostic metrics such as relation-type coverage and directional reversal rate.
Critical strengths
- Evaluating VLMs on structured, object-grounded spatial scene graphs over manipulation trajectories is a well-motivated step beyond standard multiple-choice embodied VQA.
- Contrasting relation-type coverage against exact triplet F1 and directional reversal rate provides a useful diagnostic for separating plausible spatial vocabulary usage from accurate object-pair relational binding.
Critical weaknesses
- Flaws in the ground-truth coordinate frame and evaluation setup make it impossible to interpret low VLM F1 scores as genuine spatial grounding failures. Spatial relations (left/right, front/behind) are defined in the fixed MuJoCo world frame rather than the camera frame, and this world-frame label is reused directly for the moving wrist camera. Inferring fixed world-frame directions from a single moving wrist image without camera extrinsics is geometrically ill-posed, which directly explains the low F1 and high reversal rates on wrist views. Furthermore, the paper notes that the 180-degree rotation needed to fix LIBERO's upside-down agentview rendering is applied only for visualization rather than to stored images, meaning left/right and front/behind may be inverted relative to the input images. Combined with 128×128 image resolution, task prompts that explicitly leak initial spatial relations (e.g., "the bowl on the cookie box"), and the absence of a trivial static-graph baseline on scenes where only one object moves, the benchmark scores cannot be reliably interpreted.
- The benchmark scope is narrow, and the real-robot dataset cannot serve as an evaluation benchmark because it lacks ground-truth annotations. All simulated data come from a single LIBERO-Spatial task family with the same seven objects and pick-and-place goal, sampled densely every five steps over largely static scenes. Meanwhile, although the 257-episode real-robot SO101 dataset is presented as a primary contribution, its scene graphs are pseudo-labeled by Gemini-3.1-pro without human verification—despite Gemini-3.1-pro achieving only 0.567 (agentview) and 0.377 (wrist) F1 on the paper's own simulated benchmark. This leaves the real-robot split unusable for quantitative evaluation and substantially limits the paper's contribution as a benchmark resource.
Questions for authors
- Were the LIBERO agentview images rotated right-side up before being fed to the VLMs, is the global world-frame convention explained in the prompt, and how do F1 and reversal rates change if relations are defined in each camera's own coordinate frame?
- Because only the target bowl moves during a LIBERO-Spatial episode, what F1 score does a trivial baseline achieve (e.g., predicting the first-frame graph or per-task majority graph at every step), and how does VLM performance change when the task instruction is removed from the prompt and images are rendered at 256×256 or higher?
- Do the authors plan to provide human-verified ground truth for the real-robot SO101 split, and where do the segmentation masks and wrist-view object identities in the SamGraph baseline come from?
Official Comment by Reviewer 1C97
Thank you for your thoughtful response and for the work you put into addressing my questions during the rebuttal period. After considering your response, I’ve decided to keep my original score.