- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents SpaVLA, a vision-language-action model designed to improve manipulation-relevant 3D spatial understanding and use that understanding to guide action generation. The method combines VGGT geometric features, the ManiSpatial-565K spatial question-answering dataset, and progressive training. Its action expert can condition on spatial answers generated by the model. The authors also introduce SimplerEnv-Spatial, a benchmark of 20 tasks. SpaVLA achieves a 65.6% average success rate on this benchmark and demonstrates adaptation to changes such as object rearrangement in real-robot experiments.
Critical strengths
-
The paper distinguishes between answering spatial questions and successfully acting on spatial instructions. SimplerEnv-Spatial directly evaluates the latter. The spatial questions in the dataset are also aligned with manipulation demonstrations, connecting the training objective to robot tasks.
-
Beyond reporting benchmark scores, the authors replace the spatial answer while holding the scene and instruction fixed. Correct answers improve task success, while answers naming another object often redirect the robot toward that object. This supports the claim that spatial answers influence action selection.
Critical weaknesses
-
Training strategy and inference efficiency. The ManiSpatial-565K dataset plausibly improves the VLM backbone’s embodied spatial understanding. However, Stage 3 trains the action expert to condition on a generated spatial answer. At inference time, this introduces an answer-generation step before action prediction, potentially increasing latency. What motivates this design choice?
-
(Minor) Some ablations do not isolate the sources of the final performance gain. In Table 3, the full method and the variant without scene-level QA also differ in training history, robot-data mixture, and inference mode, as the paper acknowledges. The controlled comparison of answer conditioning is primarily conducted on a checkpoint trained for 4,000 updates. More closely matched, single-factor ablations under the final training configuration would clarify each component’s contribution to the 65.6% success rate.
Questions for authors
See Weaknesses 1.
2 comments
Thanks for the authors' rebuttal. Most of my questions and concerns have been solved. As I have gived a positive score, I would like to maintain it.
We thank the reviewer for recognizing the connection between spatial understanding and action, and the evidence that spatial answers influence target selection.
W1. Motivation and inference cost
The generated answer resolves spatial constraints into an explicit target or placement description, which the action expert combines with current visual observations.
We test this mechanism under the final 20k training configuration, using the same parent checkpoint, data, QA supervision, and batch size. The ablated action expert is masked from answer tokens and all subsequent tokens during both training and inference. QA supervision is retained, and both variants still generate a first-frame answer under the answer-conditioned evaluation protocol.
Removing the connection reduces Visual Matching from 82.79% to 77.15%, Variant Aggregation from 74.95% to 74.18%, and SimplerEnv-Spatial from 65.25% to 58.25% (a decrease of 7.00 points). MMSI and MindCube remain similar: 48.60% versus 48.50%, and 95.14% versus 94.86%. These matched retraining results support the answer-to-action connection in the final policy.
The final checkpoint’s native interface also achieves 65.63%, versus 61.74% for using the same generated answer as an ordinary replacement instruction over 1,440 paired episodes (exact McNemar p = 0.001).
In answer-conditioned mode, answer generation occurs once per evaluated episode. Subsequent predictions reuse the first-frame answer with updated observations, without further language decoding. The direct instruction-to-action reference omits answer conditioning. On one A100-80GB, with bf16, batch size 1, original image resolutions, and ten flow steps, image-to-action timing is:
Timing includes preprocessing and returning the 16 × 7 action chunk to the CPU, after compilation and warm-up. The implementation uses compilation, CUDA graphs, fused projections, and image-feature reuse without changing the architecture, input resolutions, or flow-step count. Steady-state answer-conditioned latency is comparable to direct instruction-to-action prediction.
W2. Ablations under the final configuration
The new 20k answer-conditioning experiment directly extends the controlled comparison to the final configuration. We also retrained a variant without geometric inputs to either the VLM or scene tokenizer, matching its parent checkpoint, data schedule, batch size, and update count.
For the original scene-QA variant, direct instruction-to-action prediction is its trained mode, achieving 43.61% versus 65.63% on the same task configurations. We distinguish this co-training recipe comparison from the new matched ablations that isolate geometric inputs and answer conditioning.