- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents SpatialWAM, a world action model that introduces explicit 3D geometric learning on top of a pre-trained video generation backbone. The premise is that video models are trained for pixel-level plausibility, which does not guarantee the geometric accuracy required for fine-grained manipulation. The method couples three experts (video, spatial, and action) through joint attention. The spatial expert holds spatial queries arranged on a regular 3D grid together with dynamics queries instantiated per prediction time; the STGM module fuses the two and decodes voxel-anchored Gaussian scenes for the current and future times through a shared decoder, then renders each scene into multiple calibrated views for depth supervision, thereby constraining a common geometry across views. Training proceeds in two stages: geometric pre-training requires no action annotations and relies only on calibrated RGB-D videos, while post-training adds the action expert and retains the video and depth objectives. STGM is dropped at inference. The authors evaluate on RoboCasa, LIBERO, LIBERO-Plus, and three real-world tasks, and argue for the value of geometric learning and shared 3D scene modeling through Video-Only and 2D Depth variants trained on the same pre-training corpus.
Critical strengths
-
The cross-view rendering constraint on a shared 3D scene is a substantive mechanism, not a terminological distinction. Existing geometry-aware methods largely supervise per-view depth maps or geometric latents, so geometric consistency is only implicitly constrained along the temporal axis. This paper parameterizes the scene as voxel-anchored Gaussians in a common workspace coordinate frame, so the same set of primitives can be rendered into any calibrated view and cross-view inconsistency is penalized directly.
-
Separating spatial from dynamics queries, combined with bidirectional joint attention, allows geometric supervision to reshape the video representation rather than merely reading geometry out of video features. More importantly, the queries attend to both current and future video tokens during joint denoising, so predicting future geometry can draw on the evolving features of future frames. The authors' use of this point to distinguish GeoPredict is accurate and pertinent.
-
The cost of geometric supervision is confined to training, and it requires no action annotations. Camera intrinsics and extrinsics enter only the rendering path and STGM is dropped entirely at inference, so deployment needs no calibration parameters and incurs no additional inference overhead; the absence of an action-annotation requirement during pre-training extends the data boundary for representation learning from robot demonstrations to calibrated RGB-D videos. The gap between the Simulation-Only variant and the full model also offers preliminary evidence that real data contributes complementary information.
Critical weaknesses
-
The paper lacks discussion of and quantitative comparison against the closest geometry-aware work, which leaves its characterization of the prior-work boundary inaccurate. The paper summarizes the supervision used by existing geometry-aware WAMs as "targeting depth maps or latent features without explicit multi-view rendering from a shared 3D scene," yet GaussianDream (arXiv:2605.20752) overlaps substantially with this submission at the recipe level: it likewise uses learnable queries to extract geometric features, likewise decodes both current and future 3D Gaussians through a shared decoder, likewise drops the geometric heads at inference, and likewise adopts two-stage training in which the geometric stage uses no action annotations. That work does predict a shared set of renderable primitives; the sharing simply occurs along the temporal axis rather than across views, so the summary above does not hold once it is taken into account. I do not consider this to negate the novelty of the submission. Relatedly, GWM (ICCV 2025) is an earlier combination of Gaussian supervision with RoboCasa and goes uncited, and X-WAM, WAM4D, and WSA1 are discussed in the related work but none appear in the comparison tables, leaving the claim that this paper's form of geometric supervision improves on existing geometry-aware WAMs without same-protocol numerical support.
-
Several key settings of the real-world experiments are left unspecified, which affects reproducibility and limits any assessment of what geometric supervision contributes in real scenes. The paper is quite clear about the provenance of geometric information in the pre-training corpus, but the corresponding details for the authors' own platform are entirely absent: whether real-world depth is sensed or estimated by a monocular model (no camera model or depth estimator is named), the extrinsic calibration procedure and its accuracy, the definition of the real-world workspace coordinate frame, and how invalid depth is handled. These are not mere presentation oversights, because the geometric loss feeds camera parameters directly into rendering, so calibration error shifts the supervision signal as a whole, and real-world calibration is far less accurate than the exactly known parameters available in simulation. The tension is more concrete still given that two of the three real-world tasks involve a transparent lid, a reflective pot, and a slender straw, which are precisely the canonical failure cases for depth sensors, while the paper's mask only selects pixels whose back-projections fall inside the workspace and mentions no filtering of invalid depth. The real-world section also contains no visualization on the geometric side, in contrast to the qualitative video-prediction comparison provided for simulation, so what geometric supervision actually constrains on such tasks cannot be judged. In addition, the real-world comparison includes only one external baseline without geometric supervision, so that contrast confounds "with or without geometric learning" and "with or without pre-training on real-world data," and the authors' explanation for their own simulation-pre-trained variant (that noise in real data yields robustness) does not match what actually differs between the variants, namely whether the pre-training corpus includes real-world data.
Questions for authors
-
Could GaussianDream be included in the related work and the comparison tables, and could the characterization of prior supervision be revised to distinguish geometric representations shared along the temporal axis from those shared across views? That formulation is both accurate and still supportive of this paper's positioning. In addition, could those of X-WAM, WAM4D, and WSA1 that report RoboCasa or LIBERO results be added to the comparison tables? The paper's central claim concerns improving the form of geometric supervision used by geometry-aware WAMs, and the absence of any geometry-aware WAM from the comparison tables weakens that positioning.
-
Could the real-world geometric setup be specified: the source of depth (sensor model or depth estimation model), the extrinsic calibration procedure and its accuracy, the definition of the workspace coordinate frame, and how invalid depth pixels are handled (in particular, the proportion of valid depth in the transparent-lid and slender-straw scenes, and whether failed pixels assigned zero depth enter the geometric loss)? Providing sample real-world depth maps alongside renderings of the predicted Gaussian scenes would be the most direct evidence that the paper's claimed mechanism holds in real scenes.
3 comments
I thank the authors for the detailed response, which fully addresses my concerns. The revised related work now distinguishes temporal sharing from cross-view sharing, which clarifies the contribution relative to GaussianDream, and the expanded comparison tables support the central claim. The added real-world details on depth acquisition, calibration, and the workspace definition, together with the depth visualizations, make the setup reproducible. The additional control that pre-trains Cosmos Policy on the same data goes beyond what I asked for and helps separate the effect of geometric learning from that of the pre-training data. I will keep my positive score.
Thanks for your positive feedback!
Thank you very much for acknowledging our revisions and additional experiments and for your positive feedback! Your constructive comments and suggestions have helped us improve the quality of our paper. Thanks!
Response to Reviewer DkJ1
We sincerely thank the reviewer for the thoughtful review and constructive suggestions. We address the comments and questions below.
W1 & Q1: Related Work
Answer: We thank the reviewer for highlighting the need for a more complete discussion and comparison with closely related methods. Below, we provide further discussion of related geometry-aware works and present additional quantitative comparisons.
Discussion of Related Methods: As you mentioned, GaussianDream (Zhang et al., 2026) and our SpatialWAM share similarities, such as using learnable queries and predicting current and future Gaussian scenes for geometric learning. However, GaussianDream propagates a Gaussian template along the temporal axis and uses the learned prefix to condition a VLA policy. As a WAM policy, SpatialWAM emphasizes cross-view sharing: at each current or future time, one Gaussian scene is rendered across views for depth supervision, imposing a common geometric constraint. Our spatial and dynamics queries attend to video tokens through bidirectional attention, allowing geometric supervision to shape video representations for action generation. We have added discussions and citations of GaussianDream and GWM (Lu et al., 2025) in Section 2.2 and revised the characterization to distinguish temporal sharing from cross-view sharing.
Quantitative Comparisons: As requested, we have added GaussianDream, GWM, and X-WAM to the RoboCasa comparison in Table 1. We have also added GaussianDream and WSA₁ to the LIBERO comparison in Table 2. WAM4D does not report comparable RoboCasa or LIBERO results and is therefore discussed in the related work without a numerical entry.
W2 & Q2: Real-World Evaluation
Answer: We thank the reviewer for pointing out the missing details in our real-world experiments. Below, we clarify depth acquisition and processing, detail camera calibration and the workspace definition, present real-world depth visualizations, and discuss the factors contributing to the performance gains. We have added the missing details of our real-world experiments to Appendix D.
Depth acquisition and processing: The two external cameras and the wrist-mounted camera are RealSense D435i cameras. For each external camera, we use FoundationStereo (Wen et al., 2025) to estimate metric depth from stereo infrared images using known stereo baseline and intrinsics. We then align depth to RGB using known intrinsics and infrared-to-color extrinsics. The depth loss applies the workspace mask and excludes zero-depth pixels. Across all real-world training data, zero-depth rates are 0.77% overall, 2.78% in transparent-lid regions, and 2.43% in slender-straw regions. The processed depth maps (columns 2 and 5 of Figure 7 in Appendix D.3) capture the main scene structure and provide accurate geometric supervision.
Camera calibration and workspace definition: We calibrate the external cameras to the robot base using Tsai–Lenz eye-to-hand calibration from 30 poses of a checkerboard rigidly attached to the end effector. The right-handed workspace frame originates at the robot base, with x forward, y left, and z upward. Its bounds are x∈[0,1.6] m, y∈[−0.8,0.8] m, and z∈[0,1.0] m. To assess extrinsic accuracy, we reconstruct point clouds from 100 two-view training frame pairs using supervision depth and calibrated camera parameters. Mean two-view F1 scores are 72.52% at 5 mm and 85.21% at 20 mm, supporting the accuracy of our calibrated extrinsics.
Real-world depth visualization: As requested, we provide real-world supervision depth alongside depth rendered from SpatialWAM’s predicted Gaussian scenes in Figure 7 of Appendix D.3. These visualizations provide direct qualitative evidence that explicit 3D geometric supervision enables SpatialWAM to learn and reconstruct real-world scene geometry. Together with the real-world task results, they support the contribution of this mechanism to successful robotic manipulation.
Analysis of performance gains: To address the confounding between geometric learning and pre-training data, we additionally pre-train Cosmos Policy on the same data as SpatialWAM and post-train it on Task 3. It achieves 12% success versus 38% for SpatialWAM under matched pre-training data, supporting the contribution of explicit 3D geometric learning to our performance gains on this task. Regarding SpatialWAM-Sim, we agree that this comparison does not isolate noise effects. We have revised the manuscript to state that including real-world data in pre-training improves real-world task performance.