acceptodds
Under review as a conference paper at ICLR 2027

EmbodimentSemantic: A Spatial Scene-Graph Dataset and Benchmark for Vision-Language Models on Embodied Manipulation Trajectories

Abstract

Spatial grounding remains a key limitation of vision-language-action (VLA) systems for robotic manipulation. While current models can recognize objects and follow language instructions, they often lack an explicit representation of how objects are arranged in space, including support, containment, ordering, occlusion, and depth-sensitive relations. We introduce EmbodimentSemantic, a spatial scene-graph dataset and benchmark for evaluating relational grounding in embodied manipulation. EmbodimentSemantic represents scenes as directed object-relation-object triplets, where each triplet specifies a spatial relation between an ordered pair of objects using a fixed set of relations. This representation enables direct evaluation of object binding, relation prediction, and spatial consistency. The dataset includes real-world manipulation observations collected with the low-cost SO101 robot arm, together with generated scene graphs for studying spatial grounding in practical robotic settings. To provide controlled validation, we also introduce a simulator-grounded LIBERO benchmark with over 60K manipulation frames and more than 120K camera-specific scene graphs across paired third-person and wrist views, where ground-truth relations are derived automatically from MuJoCo geometry, world coordinates, camera projections, and visibility constraints. Experiments across open-source and commercial VLMs show that current models often predict plausible relations but struggle with exact depth-aware and viewpoint-dependent spatial structure. EmbodimentSemantic provides a unified framework for diagnosing spatial grounding in VLM perception.

Primary area:
datasets and benchmarks
Keywords:
scene graphs, embodied AI, vision-language models, spatial reasoning, robot manipulation, multimodal benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1C97OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces EmbodimentSemantic, a spatial scene-graph dataset and benchmark built from 10 LIBERO-Spatial manipulation tasks (500 demonstrations, 62,250 dual-view timesteps) along with a 257-episode real-robot SO101 dataset. Pairwise spatial relations among seven objects are automatically extracted from MuJoCo state in the simulated split, and 10 VLMs are evaluated on exact triplet F1 alongside diagnostic metrics such as relation-type coverage and directional reversal rate.

Critical strengths
  1. Evaluating VLMs on structured, object-grounded spatial scene graphs over manipulation trajectories is a well-motivated step beyond standard multiple-choice embodied VQA.
  2. Contrasting relation-type coverage against exact triplet F1 and directional reversal rate provides a useful diagnostic for separating plausible spatial vocabulary usage from accurate object-pair relational binding.
Critical weaknesses
  1. Flaws in the ground-truth coordinate frame and evaluation setup make it impossible to interpret low VLM F1 scores as genuine spatial grounding failures. Spatial relations (left/right, front/behind) are defined in the fixed MuJoCo world frame rather than the camera frame, and this world-frame label is reused directly for the moving wrist camera. Inferring fixed world-frame directions from a single moving wrist image without camera extrinsics is geometrically ill-posed, which directly explains the low F1 and high reversal rates on wrist views. Furthermore, the paper notes that the 180-degree rotation needed to fix LIBERO's upside-down agentview rendering is applied only for visualization rather than to stored images, meaning left/right and front/behind may be inverted relative to the input images. Combined with 128×128 image resolution, task prompts that explicitly leak initial spatial relations (e.g., "the bowl on the cookie box"), and the absence of a trivial static-graph baseline on scenes where only one object moves, the benchmark scores cannot be reliably interpreted.
  2. The benchmark scope is narrow, and the real-robot dataset cannot serve as an evaluation benchmark because it lacks ground-truth annotations. All simulated data come from a single LIBERO-Spatial task family with the same seven objects and pick-and-place goal, sampled densely every five steps over largely static scenes. Meanwhile, although the 257-episode real-robot SO101 dataset is presented as a primary contribution, its scene graphs are pseudo-labeled by Gemini-3.1-pro without human verification—despite Gemini-3.1-pro achieving only 0.567 (agentview) and 0.377 (wrist) F1 on the paper's own simulated benchmark. This leaves the real-robot split unusable for quantitative evaluation and substantially limits the paper's contribution as a benchmark resource.
Questions for authors
  1. Were the LIBERO agentview images rotated right-side up before being fed to the VLMs, is the global world-frame convention explained in the prompt, and how do F1 and reversal rates change if relations are defined in each camera's own coordinate frame?
  2. Because only the target bowl moves during a LIBERO-Spatial episode, what F1 score does a trivial baseline achieve (e.g., predicting the first-frame graph or per-task majority graph at every step), and how does VLM performance change when the task instruction is removed from the prompt and images are rendered at 256×256 or higher?
  3. Do the authors plan to provide human-verified ground truth for the real-robot SO101 split, and where do the segmentation masks and wrist-view object identities in the SamGraph baseline come from?
1 comment
Reviewer_1C97OpenReview ↗

Official Comment by Reviewer 1C97

Thank you for your thoughtful response and for the work you put into addressing my questions during the rebuttal period. After considering your response, I’ve decided to keep my original score.

Reviewer_EpSrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EmbodimentSemantic, a spatial scene-graph dataset and benchmark for robotic manipulation trajectories. Scenes are represented as directed triplets (objectA, relation, objectB) over a fixed set of eight relations. The dataset has two parts. The first is real-world manipulation data collected with an SO101 arm, with scene graphs generated by gemini-3.1-pro. The second is a LIBERO-Spatial benchmark, where ground-truth graphs for the third-person and wrist views are derived by rules from MuJoCo geometry, world coordinates, and camera projections. The authors evaluate ten VLMs and a segmentation-based baseline, SamGraph. The best model, gemini-3.1-pro, reaches a mean per-task F1 of 0.567 on the third-person view and 0.377 on the wrist view, and the authors conclude that current VLMs often predict plausible relations but fail to bind them to the correct ordered object pairs.

Critical strengths
  1. The problem is relevant. Evaluating directed triplets directly is more fine-grained than VQA-style spatial benchmarks and exposes errors such as wrong object binding and reversed relations.
  2. The simulator labels are reproducible. Ground truth is generated deterministically from simulator state for two camera views, and the evaluation protocol, including the fixed vocabulary and the handling of the two identical bowls, is clearly described.
  3. The paper evaluates a fair number of open-source VLMs and reports several diagnostic metrics, including hallucination rate, reversal rate, direction consistency, and per-relation F1.
Critical weaknesses
  • The contribution is not clearly positioned, the novelty is limited, and the writing needs revision. The introduction presents the SO101 real-world data as the main contribution, but this part has no ground truth, its scene graphs are generated by gemini-3.1-pro, and it is not used for any quantitative evaluation, while the conclusion describes the work as a simulator-grounded benchmark. Deriving spatial relations from simulator or 3D geometry by rules is common, for example in CLEVR, 3DSSG, and Rel3D, and closely related work such as Action Genome, PSG, What'sUp, and ConceptGraphs is not discussed. MomaGraph-R1 is used as a baseline in Table 1, but MomaGraph is not cited. In addition, model names differ across the text, tables, and figures, the number of real-world episodes is given as both 250 and 257, and the appendix figures are not discussed in the text.
  • The labels are not validated, and the task definition does not match some of the conclusions. Each ordered pair receives only one relation based on the dominant axis, on-top-of depends on 2D box overlap, containment is hardcoded for a single task, and the third-person graph includes all 42 ordered pairs in every frame without visibility filtering. There is no human check of the labels and no human performance on the same 128×128 inputs. Left/right and front/behind are defined in the world frame, and it is unclear whether the models are told this convention, while the wrist graph reuses the same world-frame relations with visibility filtering only. The low scores may therefore partly reflect label conventions, and the wrist-view results do not support the claim that VLMs lack egocentric spatial grounding.
  • The baselines and experiments are not sufficient. Gemini is the only commercial model, and there are no spatially specialized VLMs such as SpatialRGPT or RoboBrain, nor scene-graph generation models trained on PSG or Visual Genome. All models use a single prompt, without variants that state the frame convention or give few-shot examples, and the fraction of discarded invalid outputs is not reported. SamGraph reaches 0.82 F1 on the third-person view, but its inputs and method are not described, and its wrist evaluation uses predictions from the external camera. There is also no train/test split and no fine-tuning or downstream manipulation experiment to show how the dataset would be used.
  • The scope is narrow and the metrics are of limited help. The simulator part covers one LIBERO-Spatial suite with seven objects and ten similar layouts, and nearby sampled frames are highly correlated, so the counts of 60K timesteps and 120K graphs overstate its size. Coverage only checks whether a reference predicate type appears anywhere in the prediction, which is close to 1 for almost any output with only eight predicates, so the main finding of high coverage and low F1 largely follows from the metric itself. The per-relation differences are also small (0.278 vs. 0.233) and are reported without confidence intervals or error analysis by visibility, distance, or occlusion.
Questions for authors
  1. What is the main novelty compared with prior work that derives spatial relations or scene graphs from geometry or annotations, such as CLEVR, 3DSSG, Rel3D, Action Genome, and PSG? Why is MomaGraph not cited?
  2. Why is the SO101 data presented as the main contribution when it has no ground truth? Does it contain 250 or 257 episodes, and could the recorded depth stream or manual labels for a subset provide ground truth?
  3. How accurate are the automatic labels? Can the authors check a subset manually and report human performance? How are pairs with close |Δx| and |Δy| handled, and could a pair have more than one relation?
  4. Does the prompt state the world-frame convention? Do the models receive the 128×128 images or upsampled ones, and in which orientation? Do the results change when the convention is given or higher-resolution images are used?
  5. Are the SamGraph masks from SAM or from the simulator, and does it use depth, camera parameters, or ground-truth positions? Can the authors add other commercial models such as GPT or Claude, spatial VLMs, and scene-graph generation models, and report the invalid output rate and the variance across runs?
  6. Can the authors provide a train/test split and show the effect of fine-tuning on this data, or the relation between scene-graph accuracy and downstream manipulation success?
Reviewer_Qf57OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EmbodimentSemantic, a dataset and benchmark for predicting directed spatial scene graphs from robot manipulation observations. It augments 500 LIBERO-Spatial demonstrations with simulator-derived annotations covering 62,250 paired timesteps and 124,500 images from external and wrist cameras. The benchmark uses seven object identities and eight spatial predicates, evaluating exact triplet recovery and diagnostic metrics. The authors also contribute 257 SO101 demonstrations with generated scene graphs and a segmentation-based baseline, SamGraph. Evaluated VLMs struggle with exact relational grounding, especially in wrist views. The strongest VLM achieves mean per-task F1 scores of 0.5674 and 0.3773 on external and wrist views, respectively; SamGraph reaches 0.8196 on the external view. The real-world component lacks verified reference graphs and is presented as a resource rather than a quantitative benchmark.

Critical strengths

The evaluation targets a useful failure mode: correctly assigning a directed spatial relation to a specific object pair. Exact triplet matching and complementary diagnostics expose errors that relation-word coverage alone cannot characterize. Permutation-invariant scoring for visually indistinguishable bowls also removes an arbitrary naming penalty.

The simulator annotation pipeline provides dense, paired-view observations throughout manipulation demonstrations. This controlled resource supports reproducible investigation of spatial grounding under stable and moving cameras. The substantial gap between SamGraph and prompted VLMs on the external view is informative and motivates closer investigation of explicit geometric representations.

Critical weaknesses

The semantic validity of the reference labels is insufficiently established. Support is inferred from projected-box overlap and relative height, while containment uses a task-specific exception (Sections 4.1–4.2). These rules do not directly verify physical contact or geometric containment. Selecting one predicate per ordered pair also excludes simultaneously valid relations. Furthermore, wrist-view directions remain world-frame based, but the evaluation description does not specify how models are informed of that convention (Sections 4.3–5). Consequently, errors may partly reflect annotation heuristics or coordinate mismatches. An independent label audit and the complete prompting protocol are necessary to substantiate the interpretation of the scores.

Validation and reproducibility of the broader resource remain limited. Quantitative evidence covers one LIBERO task family with seven objects; the SO101 component has no verified reference graphs, leaving its annotation quality unmeasured. SamGraph is promising, but its mask acquisition, identity assignment, and relation-generation procedure are insufficiently documented. Its wrist evaluation additionally uses fewer episodes and external-camera predictions for identity maintenance. These differences prevent a clean comparison under matched conditions and limit conclusions about which information or processing explains its performance.

Questions for authors

Label-validation request: How accurate are the generated labels against independently verified annotations, particularly for support, containment, and relation transitions? Please report a stratified audit and sensitivity to the overlap threshold. Would correcting disputed labels change the model rankings or main conclusions?

Protocol clarification: What coordinate-frame information is supplied to each VLM? Please provide the complete prompt and specify any axis definitions, camera poses, or reference examples. How is alignment with world-frame labels ensured for the moving wrist camera?

Baseline clarification and comparison request: How does SamGraph obtain masks, object identities, and spatial predicates, and does it require manual initialization or calibration? Please compare it with VLMs on identical frames and clarify the effect of external-camera information on wrist-view results.

Resource-validation request: Has any subset of the SO101 generated graphs been independently checked? If so, please report the sampling procedure and relation-level error rates. Such evidence would materially strengthen the claimed real-world resource contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.