acceptodds
Under review as a conference paper at ICLR 2027

SpaVLA: From Explicit 3D Spatial Understanding to Robotic Manipulation

Abstract

Vision–language–action (VLA) models inherit rich semantic knowledge from pretrained vision–language models (VLMs), yet developing 3D spatial understanding and using it to guide action generation remain challenging. We propose SpaVLA, a spatially enhanced VLA model that connects geometric perception to action generation through explicit, manipulation-relevant 3D spatial understanding. To enhance the 3D spatial capabilities, we first construct ManiSpatial-565K, which pairs robot manipulation data with 3D spatial question-answering (QA) annotations to provide fine-grained, manipulation-relevant supervision. To support reasoning about these spatial relationships, we introduce a 3D-aware module that incorporates geometric representations into SpaVLA's VLM backbone. We then introduce a progressive training strategy to align this enhanced spatial understanding with robotic control. By conditioning the action expert on fine-grained spatial reasoning generated by the VLM, SpaVLA connects geometric perception to action generation, enabling the VLA model to act on manipulation-relevant 3D spatial relationships. To evaluate how well this spatial understanding supports manipulation, we further introduce SimplerEnv-Spatial, a 20-task benchmark for spatial referring and reasoning. SpaVLA achieves 48.6% on MMSI-Bench and 95.6% on MindCube, with manipulation success rates of 84.4% on Google Robot under SimplerEnv's Visual Matching protocol and 65.6% on SimplerEnv-Spatial. Real-world experiments further demonstrate spatial instruction following under object rearrangement, destination relocation, and illumination changes. The dataset and code will be publicly available.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, Spatial Intelligence

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bdeKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SpaVLA, a vision-language-action model designed to improve manipulation-relevant 3D spatial understanding and use that understanding to guide action generation. The method combines VGGT geometric features, the ManiSpatial-565K spatial question-answering dataset, and progressive training. Its action expert can condition on spatial answers generated by the model. The authors also introduce SimplerEnv-Spatial, a benchmark of 20 tasks. SpaVLA achieves a 65.6% average success rate on this benchmark and demonstrates adaptation to changes such as object rearrangement in real-robot experiments.

Critical strengths
  1. The paper distinguishes between answering spatial questions and successfully acting on spatial instructions. SimplerEnv-Spatial directly evaluates the latter. The spatial questions in the dataset are also aligned with manipulation demonstrations, connecting the training objective to robot tasks.

  2. Beyond reporting benchmark scores, the authors replace the spatial answer while holding the scene and instruction fixed. Correct answers improve task success, while answers naming another object often redirect the robot toward that object. This supports the claim that spatial answers influence action selection.

Critical weaknesses
  1. Training strategy and inference efficiency. The ManiSpatial-565K dataset plausibly improves the VLM backbone’s embodied spatial understanding. However, Stage 3 trains the action expert to condition on a generated spatial answer. At inference time, this introduces an answer-generation step before action prediction, potentially increasing latency. What motivates this design choice?

  2. (Minor) Some ablations do not isolate the sources of the final performance gain. In Table 3, the full method and the variant without scene-level QA also differ in training history, robot-data mixture, and inference mode, as the paper acknowledges. The controlled comparison of answer conditioning is primarily conducted on a checkpoint trained for 4,000 updates. More closely matched, single-factor ablations under the final training configuration would clarify each component’s contribution to the 65.6% success rate.

Questions for authors

See Weaknesses 1.

2 comments

We thank the reviewer for recognizing the connection between spatial understanding and action, and the evidence that spatial answers influence target selection.

W1. Motivation and inference cost

The generated answer resolves spatial constraints into an explicit target or placement description, which the action expert combines with current visual observations.

We test this mechanism under the final 20k training configuration, using the same parent checkpoint, data, QA supervision, and batch size. The ablated action expert is masked from answer tokens and all subsequent tokens during both training and inference. QA supervision is retained, and both variants still generate a first-frame answer under the answer-conditioned evaluation protocol.

Removing the connection reduces Visual Matching from 82.79% to 77.15%, Variant Aggregation from 74.95% to 74.18%, and SimplerEnv-Spatial from 65.25% to 58.25% (a decrease of 7.00 points). MMSI and MindCube remain similar: 48.60% versus 48.50%, and 95.14% versus 94.86%. These matched retraining results support the answer-to-action connection in the final policy.

The final checkpoint’s native interface also achieves 65.63%, versus 61.74% for using the same generated answer as an ordinary replacement instruction over 1,440 paired episodes (exact McNemar p = 0.001).

In answer-conditioned mode, answer generation occurs once per evaluated episode. Subsequent predictions reuse the first-frame answer with updated observations, without further language decoding. The direct instruction-to-action reference omits answer conditioning. On one A100-80GB, with bf16, batch size 1, original image resolutions, and ten flow steps, image-to-action timing is:

PredictionAnswer usageLatency (ms)p95 (ms)
Initial answer-conditioned predictionGenerate an answer and condition actions on it211.00280.00
Subsequent answer-conditioned predictionsReuse the first-frame answer; no further language decoding96.8098.90
Direct instruction-to-action predictionNo generated-answer condition96.2098.70

Timing includes preprocessing and returning the 16 × 7 action chunk to the CPU, after compilation and warm-up. The implementation uses compilation, CUDA graphs, fused projections, and image-feature reuse without changing the architecture, input resolutions, or flow-step count. Steady-state answer-conditioned latency is comparable to direct instruction-to-action prediction.

W2. Ablations under the final configuration

The new 20k answer-conditioning experiment directly extends the controlled comparison to the final configuration. We also retrained a variant without geometric inputs to either the VLM or scene tokenizer, matching its parent checkpoint, data schedule, batch size, and update count.

For the original scene-QA variant, direct instruction-to-action prediction is its trained mode, achieving 43.61% versus 65.63% on the same task configurations. We distinguish this co-training recipe comparison from the new matched ablations that isolate geometric inputs and answer conditioning.

Thanks for the authors' rebuttal. Most of my questions and concerns have been solved. As I have gived a positive score, I would like to maintain it.

Reviewer_WLo8OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

SpaVLA combines Qwen3-VL, VGGT geometric features, a scene tokenizer, and a flow-based action expert. Progressive training integrates spatial QA and action supervision, with generated task-resolving answers conditioning action prediction. The paper introduces ManiSpatial-565K, with spatial annotations derived from robot demonstrations, and SimplerEnv-Spatial, a 20-task benchmark for spatially constrained manipulation. Reported results include improved spatial understanding, 65.6% success on SimplerEnv-Spatial, and real-world pick-and-place performance. Answer interventions probe how generated answers influence target selection.

Critical strengths

1.SimplerEnv-Spatial requires unambiguous targets, checks ambiguity geometrically, and evaluates models with identical initial layouts and instructions. SpaVLA’s 65.6% success versus 37.3% for the strongest adapted baseline provides stronger evidence that spatial information supports manipulation than language-only QA. 2.Across 504 paired episodes controlling scene, instruction, and seed, replacing answers with correct ones raises success from 50.0% to 55.0%. Counterfactual answers redirect manipulation to the named alternative in 61.9% of episodes and reduce true-target success to 0.2%. Statistical testing supports the answer interface’s role in control.

Critical weaknesses

1.Geometry ablations assess only spatial QA, while other comparisons are confounded or use a 4k-update model, leaving geometry’s contribution to the final policy unclear. Real-world experiments also do not validate generated-answer conditioning because the policy acts directly from instructions. 2.Robo2VLM-1 derives spatial QA from real robot trajectories, while InstructVLA’s VLA-IT integrates spatial QA and relational instruction rewriting with action learning, suggesting limited novelty for ManiSpatial’s broad supervision approach. Although metric temporal scene graphs and target-preserving rewrites may distinguish ManiSpatial, their added value lacks systematic coverage comparisons or matched comparisons against existing supervision. The paper also reports no independent, stratified annotation audit, leaving semantic correctness, reference uniqueness, and metric accuracy insufficiently established. Robo2VLM-1 (Chen et al., 2025), InstructVLA (Yang et al., 2025)

Questions for authors

1.Please train a full policy without geometric inputs to either VLM fusion or the scene tokenizer, holding data, training budget, and answer-conditioned inference fixed. Also isolate scene QA’s contribution under matched training history and inference. 2.Please report independent QA correctness and ambiguity rates by source and spatial skill, along with metric distance errors, the proportion passing independent scale verification, and audit sample sizes. 3.Please specify VQA sources, sampling weights, and checkpoint lineage for Tables 1–3, clarify the relationship between 10k local updates (Table 6) and 20k updates (Appendix D.3), and identify checkpoints used in matched-budget comparisons. 4.Please isolate manipulation gains from ManiSpatial’s metric scene graphs and task-resolving rewrites through comparisons with existing trajectory-grounded QA and instruction rewriting, controlling the backbone, demonstrations, supervision amount, training budget, and inference mode.

4 comments

We thank the reviewer for recognizing the benchmark's spatial disambiguation design and the answer-intervention evidence. We add final-stage ablations and annotation audits, and clarify the training and dataset design below.

Q1. Geometry and answer conditioning in the final policy

The new geometry and answer-conditioning variants share the full model’s Stage-2 initialization, training-data schedule, global batch size of 768, and 20k final-stage updates.

Geometric inputs. We remove both VLM-side geometric fusion/DeepStack inputs and the VGGT branch of the scene tokenizer, retaining the full model’s inference protocol.

ModelGoogle Robot VMGoogle Robot VASimplerEnv-SpatialMMSIMindCubeSPAR
SpaVLA (matched control)83.6775.7364.7948.6095.1487.15
Without geometric inputs81.3774.8063.5444.2093.8182.50

Scores are percentages. VM = Visual Matching; VA = Variant Aggregation. The geometry ablation and its control use the same evaluation setting.

Removing geometry decreases Visual Matching, Variant Aggregation, and SimplerEnv-Spatial by 2.30, 0.93, and 1.25 points, respectively. It also reduces MMSI by 4.40 points, MindCube by 1.33 points, and SPAR by 4.65 points. This complements the Stage-1 comparison with evidence for geometric inputs in the final policy and spatial understanding.

Answer conditioning. We mask the action expert from answer tokens and all subsequent tokens during both training and inference, while retaining image–instruction context and QA supervision. Under the answer-conditioned evaluation protocol, both variants still generate an answer at the first frame.

ModelGoogle Robot VMGoogle Robot VASimplerEnv-SpatialMMSIMindCube
SpaVLA82.7974.9565.2548.6095.14
Without answer-to-action conditioning77.1574.1858.2548.5094.86

Removing the connection reduces Visual Matching, Variant Aggregation, and SimplerEnv-Spatial by 5.64, 0.77, and 7.00 points, respectively, while MMSI and MindCube change little. This matched retraining comparison isolates the contribution of the answer-to-action connection with QA supervision held fixed.

On the final checkpoint’s original 1,440 paired episodes, the native inference interface achieves 65.63%, versus 61.74% when the same generated answer is supplied as an ordinary replacement instruction (exact McNemar p = 0.001). This inference comparison complements the matched retraining evidence for the answer-to-action interface.

Scene QA. Table 3(b) compares co-training recipes with and without robot-scene QA and aligned answer–action windows. In its trained direct instruction-to-action mode, the variant without scene QA achieves 43.61%, versus SpaVLA’s 65.63% on the same task configurations. The new matched ablation above isolates answer access with QA supervision held fixed.

Real-world scope. Franka evaluates direct instruction-to-action generalization. The paired interventions and new 20k ablation provide the evidence for the answer–action interface.

Q2. Annotation correctness, ambiguity, and metric validation

We audited 1,406 QA pairs from 150 randomly sampled trajectories. Two human reviewers independently assessed answer semantics, reference uniqueness, and demonstrated-target preservation; a third reviewer adjudicated disagreements. Numerical distances received plausibility checks, with metric accuracy evaluated separately against measurement references.

QA categoryQA pairsAccepted (%)Ambiguous (%)
Without explicit distance values1,06093.681.04
With explicit distance values34693.060.58
Total1,40693.530.92

Percentages are computed within each row under the stated audit criteria. A complementary model-based review of 74,444 QA pairs from 7,500 trajectories accepts 92.72%, without access to human judgments. For metric validation, we apply our annotation pipeline, including focal-scale correction, to DROID RGB observations and compare its estimated object-to-object distances with reference distances computed from DROID’s sensor depth. Across 12,312 object pairs from 3,224 trajectories, the median absolute distance error is 4.10 cm, with a relative error of 14.20%; 82.00% of the evaluated pairs are within 10 cm of the reference. The median ratio of annotated to reference distances is 0.918. Directional relations agree with sensor-depth-derived references in 97.30% of 24,970 judgments. On Bridge, distances estimated separately from two views have a median discrepancy of 2.00 cm over 1,609 matched pairs from 450 trajectories.

Q3. VQA sampling and checkpoint correspondence

Stage 1 uses SenseNova-SI-800K, SPAR-7M-train, MindCube-train, EmbSpatial-SFT, and LLaVA-OneVision. Co-training additionally includes VST, RefCOCO/RefCOCOg, and SAT-v2; joint fine-tuning adds ManiSpatial scene QA and task-resolving instruction–answer–action supervision. We use designated training partitions where corresponding evaluations exist.

The full-VLM schedule mixes VQA and robot exposures at 3:1. For the final joint-training segment, the configured sampling shares within the VQA stream are:

VQA sourceShare of VQA (%)
SPAR39.77
SenseNova27.18
LLaVA (general VQA)16.70
VST3.67
EmbSpatial2.75
MindCube2.75
RefCOCO1.41
RefCOCOg1.40
SAT1.04
ManiSpatial QA3.31

Shares are rounded; batch-alignment padding accounts for less than 0.01%. The ManiSpatial row covers QA-only exposures; aligned instruction–answer–action examples are sampled through the robot stream.

The main understanding, Google Robot, and SimplerEnv-Spatial results use the same final 20k Stage-3 checkpoint, also used for Full in Table 3(b). The 10k entry in the submitted supplementary Table 6 is a typographical error and should read 20k.

Table 3(a) evaluates Stage-1 models, (b) compares training recipes, and (c,d) evaluate training schedules and answer access at 4k updates. Baseline checkpoint sources are described in the submitted Appendix D.3, now Appendix D.1. The new Q1 ablations share the full model’s parent checkpoint, data schedule, batch size, and 20k final-stage budget.

Q4. Relationship to Robo2VLM and InstructVLA

Robo2VLM and InstructVLA are closely related. Robo2VLM constructs spatial and interaction QA from robot sensory information and manipulation phases. InstructVLA’s VLA-IT combines scene descriptions, QA, relational references, command rewriting, and contextual instruction following for action learning.

ManiSpatial’s contribution is an explicit construction protocol linking object-level metric temporal scene graphs, target disambiguation, and demonstration-aligned action supervision. For discrete target selection, it requires:

  1. Multiple admissible candidates when the spatial constraint is removed.
  2. A unique candidate when the constraint is retained.
  3. Preservation of the demonstrated target and action intent.

The question, resolving answer, and demonstration are then paired for action learning, while scene-level QA provides complementary geometric supervision. This construction makes the spatial constraint necessary for identifying the intended target and keeps the supervision aligned with recorded actions. We will clarify these construction requirements in the related-work and dataset sections.

Thank you for the rebuttal. The matched 20k-update ablations address my main concern. The checkpoint discrepancy is also resolved. I am raising my rating to 3 (Weak Acceptance).

Thank you for your positive feedback and for acknowledging our additional ablation results and clarifications. We appreciate your constructive comments, which have helped us strengthen the paper. We will incorporate the matched-budget ablations and checkpoint clarifications into the revised manuscript.

Reviewer_KvrrOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SpaVLA is a vision-language-action model designed to improve 3D spatial understanding for robotic manipulation. It introduces ManiSpatial-565K, which pairs manipulation trajectories with fine-grained spatial question-answer annotations, and SimplerEnv-Spatial, a 20-task benchmark for spatial referring and reasoning. The model incorporates geometric features from a frozen VGGT into the VLM backbone and uses a dual-stream Mixture-of-Transformers to condition actions on scene tokens and VLM-generated spatial answers. Training proceeds through spatial QA pretraining, action adaptation, and joint fine-tuning. Evaluations on MMSI-Bench, MindCube, SimplerEnv, SimplerEnv-Spatial, and real-world Franka tasks show improvements over π₀.₅ and InstructVLA.

Critical strengths
  1. Jointly training the VLM and DiT through MoT cross-attention preserves the VLM’s spatial understanding capabilities while improving action generation.

  2. The new ManiSpatial-565K dataset provides a valuable resource for the research community.

  3. SpaVLA significantly outperforms the baselines on SimplerEnv-Spatial and demonstrates robust generalization in real-world scenarios.

Critical weaknesses
  1. The failure-case analysis lacks a systematic assessment. The authors should report the frequency of each error type and analyze how spatial reasoning errors and action execution errors respectively affect overall performance.

  2. The annotation quality of ManiSpatial-565K and its impact remain unclear. The dataset relies heavily on automatic annotation and filtering, while some geometric relations are retained as unverified hypotheses, making the reliability of the semantic annotations difficult to assess. The authors should clarify the QA verification procedure, provide quality statistics from human verification or a verified subset, and explain whether controlled comparisons have been conducted to assess how annotation quality affects downstream manipulation performance.

  3. The necessity of the three-stage progressive training strategy has not been directly established. The paper lacks a comparison that skips Stage 1, such as training the full model jointly end-to-end from the outset. Consequently, it is unclear whether strengthening the VLM’s spatial understanding first is necessary. The authors should provide this ablation and analyze its effects on both spatial understanding and manipulation performance.

Questions for authors

see weakness

4 comments

We thank the reviewer for recognizing ManiSpatial’s value and SpaVLA’s improvements in spatially constrained manipulation. We address the three concerns with quantitative failure analysis, annotation audits, and a direct joint-training comparison.

W1. Systematic failure analysis

We analyze all 495 failures among the final checkpoint’s 1,440 SimplerEnv-Spatial episodes. Simulator states, generated answers, and task-completion conditions define three mutually exclusive categories:

Failure categoryEpisodesShare of failures
Answer names a non-target object377.47%
Wrong-target interaction, excluding the cases above22144.65%
Execution failure without the preceding target errors23747.88%
Total495100.00%

Answer/target-selection failures and execution failures account for 17.92 and 16.46 percentage points of the overall 34.38% failure rate, respectively. Thus, the analysis distinguishes resolving and grounding the intended target from completing the physical interaction.

In the 648 episodes with name-identifiable candidates, generated answers identify the correct target in 93.98% of episodes. Success is 72.58% with a correct answer versus 5.13% with an incorrect answer. In 35 of the 37 failed episodes with an incorrect answer, the robot interacts with the object named by that answer. This connects answer correctness to observed target selection. We will include this quantitative breakdown in the failure analysis.

W2. Annotation verification and quality

Annotations use initial, middle, and final RGB observations, instance masks, the original demonstration instruction, and geometric candidates. Geometric provenance and scale-verification status are recorded separately from QA acceptance. Verification checks object identity, reference uniqueness, the demonstrated target, and the placement outcome. Metric candidates additionally pass visibility and geometric-confidence checks; distance-based disambiguation requires non-overlapping uncertainty intervals. Ambiguous or unsupported QA candidates are rejected, followed by structural and temporal consistency checks. In 2,500 sampled trajectories, 6,358 of 40,617 candidate slots were rejected (15.65%).

We conducted a human audit of 1,406 QA pairs from 150 randomly sampled trajectories. Two human reviewers independently reviewed answer semantics, reference uniqueness, and target preservation, with disagreements adjudicated by a third reviewer. The table distinguishes QA containing explicit numerical distances in the question, answer, or options. Numerical values received plausibility checks; metric accuracy was measured separately.

QA categoryQA pairsAccepted (%)Ambiguous (%)
Without explicit distance values1,06093.681.04
With explicit distance values34693.060.58
Total1,40693.530.92

Acceptance refers to the stated audit criteria. A complementary GPT-6 Astra review of 74,444 QA pairs from 7,500 trajectories accepts 92.72%, without access to human judgments.

For metric validation, we apply our annotation pipeline, including focal-scale correction, to DROID RGB observations and compare its estimated object-to-object distances with reference distances computed from DROID’s sensor depth. Across 12,312 object pairs from 3,224 trajectories, the median absolute distance error is 4.10 cm, with a relative error of 14.20%; 82.00% of the evaluated pairs are within 10 cm of the reference. The median ratio of annotated to reference distances is 0.918. Directional relations agree with sensor-depth-derived references in 97.30% of 24,970 judgments. On Bridge, distances estimated separately from two views have a median discrepancy of 2.00 cm over 1,609 matched pairs from 450 trajectories.

Recorded robot actions remain unchanged, and demonstrations without an accepted task-resolving pair retain their original instructions. The audits and geometric checks directly quantify annotation reliability. Annotation quality is not varied as a separate policy-training factor; the matched policy ablations keep annotations fixed to isolate geometry and answer access.

W3. Progressive training versus joint training from the outset

Following your suggestion, we train the full model jointly from the original Qwen3-VL-4B, with newly initialized geometry-fusion and action modules. This bypasses Stages 1 and 2 and uses the final-stage data, batch size, objectives, and inference protocol.

Training strategyJoint-training updatesGoogle Robot VMGoogle Robot VASimplerEnv-SpatialMMSIMindCube
Progressive SpaVLA20k82.7974.9565.2548.6095.14
Direct joint training20k61.5246.7333.5447.7093.71
Direct joint training40k72.6257.8235.9246.6093.14
Direct joint training50k71.3255.6236.8847.0094.00

Scores are percentages. VM = Visual Matching; VA = Variant Aggregation. Update counts refer to the joint-training stage.

Extending direct joint training does not close the manipulation gap. At 20k updates, direct training achieves 61.52% on Visual Matching, 46.73% on Variant Aggregation, and 33.54% on SimplerEnv-Spatial, compared with SpaVLA’s 82.79%, 74.95%, and 65.25%, respectively. From 40k to 50k updates, Visual Matching decreases from 72.62% to 71.32% and Variant Aggregation from 57.82% to 55.62%, while SimplerEnv-Spatial improves only modestly, from 35.92% to 36.88%. All three remain substantially below the progressive model. SpaVLA also retains higher MMSI and MindCube scores in these comparisons. These results support the progressive recipe for connecting spatial understanding to reliable manipulation.

Thank you for the detailed rebuttal and additional experiments. The systematic failure analysis, annotation quality verification, and direct joint-training comparisons have substantially addressed my major concerns. Although some limitations remain, I appreciate the authors' efforts and find the new evidence convincing. I have therefore decided to raise my rating.

Thank you for your thoughtful feedback and for recognizing our additional experimental evidence and analyses. We appreciate your constructive suggestions throughout the discussion. We will incorporate the new results, failure analyses, and annotation quality verification into the revised paper to further strengthen its clarity and completeness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.