acceptodds
Under review as a conference paper at ICLR 2027

SpatialWAM: Spatially Grounded World Action Models via Explicit 3D Geometric Learning

13 views

Abstract

World action models inherit spatio-temporal priors from video models, but visual plausibility does not ensure geometric accuracy for fine-grained manipulation. We present **SpatialWAM**, which learns spatially grounded video and spatial representations through explicit 3D geometric learning for action generation. During video–spatial pre-training, a spatial expert extracts features from current and future video tokens using spatial and dynamics queries. Spatio-Temporal Gaussian Modeling (STGM) then fuses these features to predict current and future 3D Gaussian scenes shared across views. Multi-view depth supervision in turn shapes both experts’ representations through bidirectional attention, using calibrated RGB-D videos without action annotations. During post-training, a randomly initialized action expert accesses the learned representations through joint attention, while all three experts are fully updated with continued video and depth supervision. At inference, all three experts remain active, while STGM is omitted. SpatialWAM achieves 73.4% success on RoboCasa, 97.8% on LIBERO, and 85.9% on LIBERO-Plus, with 72.0% average success across three real-world tasks. Comparisons using identical pre-training data show gains over video-only and 2D depth pre-training, supporting geometric learning and shared 3D scene modeling.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Visuomotor Learning, World Action Model, 3D Gaussian Splatting

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DkJ1OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents SpatialWAM, a world action model that introduces explicit 3D geometric learning on top of a pre-trained video generation backbone. The premise is that video models are trained for pixel-level plausibility, which does not guarantee the geometric accuracy required for fine-grained manipulation. The method couples three experts (video, spatial, and action) through joint attention. The spatial expert holds spatial queries arranged on a regular 3D grid together with dynamics queries instantiated per prediction time; the STGM module fuses the two and decodes voxel-anchored Gaussian scenes for the current and future times through a shared decoder, then renders each scene into multiple calibrated views for depth supervision, thereby constraining a common geometry across views. Training proceeds in two stages: geometric pre-training requires no action annotations and relies only on calibrated RGB-D videos, while post-training adds the action expert and retains the video and depth objectives. STGM is dropped at inference. The authors evaluate on RoboCasa, LIBERO, LIBERO-Plus, and three real-world tasks, and argue for the value of geometric learning and shared 3D scene modeling through Video-Only and 2D Depth variants trained on the same pre-training corpus.

Critical strengths
  1. The cross-view rendering constraint on a shared 3D scene is a substantive mechanism, not a terminological distinction. Existing geometry-aware methods largely supervise per-view depth maps or geometric latents, so geometric consistency is only implicitly constrained along the temporal axis. This paper parameterizes the scene as voxel-anchored Gaussians in a common workspace coordinate frame, so the same set of primitives can be rendered into any calibrated view and cross-view inconsistency is penalized directly.

  2. Separating spatial from dynamics queries, combined with bidirectional joint attention, allows geometric supervision to reshape the video representation rather than merely reading geometry out of video features. More importantly, the queries attend to both current and future video tokens during joint denoising, so predicting future geometry can draw on the evolving features of future frames. The authors' use of this point to distinguish GeoPredict is accurate and pertinent.

  3. The cost of geometric supervision is confined to training, and it requires no action annotations. Camera intrinsics and extrinsics enter only the rendering path and STGM is dropped entirely at inference, so deployment needs no calibration parameters and incurs no additional inference overhead; the absence of an action-annotation requirement during pre-training extends the data boundary for representation learning from robot demonstrations to calibrated RGB-D videos. The gap between the Simulation-Only variant and the full model also offers preliminary evidence that real data contributes complementary information.

Critical weaknesses
  1. The paper lacks discussion of and quantitative comparison against the closest geometry-aware work, which leaves its characterization of the prior-work boundary inaccurate. The paper summarizes the supervision used by existing geometry-aware WAMs as "targeting depth maps or latent features without explicit multi-view rendering from a shared 3D scene," yet GaussianDream (arXiv:2605.20752) overlaps substantially with this submission at the recipe level: it likewise uses learnable queries to extract geometric features, likewise decodes both current and future 3D Gaussians through a shared decoder, likewise drops the geometric heads at inference, and likewise adopts two-stage training in which the geometric stage uses no action annotations. That work does predict a shared set of renderable primitives; the sharing simply occurs along the temporal axis rather than across views, so the summary above does not hold once it is taken into account. I do not consider this to negate the novelty of the submission. Relatedly, GWM (ICCV 2025) is an earlier combination of Gaussian supervision with RoboCasa and goes uncited, and X-WAM, WAM4D, and WSA1 are discussed in the related work but none appear in the comparison tables, leaving the claim that this paper's form of geometric supervision improves on existing geometry-aware WAMs without same-protocol numerical support.

  2. Several key settings of the real-world experiments are left unspecified, which affects reproducibility and limits any assessment of what geometric supervision contributes in real scenes. The paper is quite clear about the provenance of geometric information in the pre-training corpus, but the corresponding details for the authors' own platform are entirely absent: whether real-world depth is sensed or estimated by a monocular model (no camera model or depth estimator is named), the extrinsic calibration procedure and its accuracy, the definition of the real-world workspace coordinate frame, and how invalid depth is handled. These are not mere presentation oversights, because the geometric loss feeds camera parameters directly into rendering, so calibration error shifts the supervision signal as a whole, and real-world calibration is far less accurate than the exactly known parameters available in simulation. The tension is more concrete still given that two of the three real-world tasks involve a transparent lid, a reflective pot, and a slender straw, which are precisely the canonical failure cases for depth sensors, while the paper's mask only selects pixels whose back-projections fall inside the workspace and mentions no filtering of invalid depth. The real-world section also contains no visualization on the geometric side, in contrast to the qualitative video-prediction comparison provided for simulation, so what geometric supervision actually constrains on such tasks cannot be judged. In addition, the real-world comparison includes only one external baseline without geometric supervision, so that contrast confounds "with or without geometric learning" and "with or without pre-training on real-world data," and the authors' explanation for their own simulation-pre-trained variant (that noise in real data yields robustness) does not match what actually differs between the variants, namely whether the pre-training corpus includes real-world data.

Questions for authors
  1. Could GaussianDream be included in the related work and the comparison tables, and could the characterization of prior supervision be revised to distinguish geometric representations shared along the temporal axis from those shared across views? That formulation is both accurate and still supportive of this paper's positioning. In addition, could those of X-WAM, WAM4D, and WSA1 that report RoboCasa or LIBERO results be added to the comparison tables? The paper's central claim concerns improving the form of geometric supervision used by geometry-aware WAMs, and the absence of any geometry-aware WAM from the comparison tables weakens that positioning.

  2. Could the real-world geometric setup be specified: the source of depth (sensor model or depth estimation model), the extrinsic calibration procedure and its accuracy, the definition of the workspace coordinate frame, and how invalid depth pixels are handled (in particular, the proportion of valid depth in the transparent-lid and slender-straw scenes, and whether failed pixels assigned zero depth enter the geometric loss)? Providing sample real-world depth maps alongside renderings of the predicted Gaussian scenes would be the most direct evidence that the paper's claimed mechanism holds in real scenes.

3 comments
AuthorsOpenReview ↗

Response to Reviewer DkJ1

We sincerely thank the reviewer for the thoughtful review and constructive suggestions. We address the comments and questions below.


W1 & Q1: Related Work

Answer: We thank the reviewer for highlighting the need for a more complete discussion and comparison with closely related methods. Below, we provide further discussion of related geometry-aware works and present additional quantitative comparisons.

  • Discussion of Related Methods: As you mentioned, GaussianDream (Zhang et al., 2026) and our SpatialWAM share similarities, such as using learnable queries and predicting current and future Gaussian scenes for geometric learning. However, GaussianDream propagates a Gaussian template along the temporal axis and uses the learned prefix to condition a VLA policy. As a WAM policy, SpatialWAM emphasizes cross-view sharing: at each current or future time, one Gaussian scene is rendered across views for depth supervision, imposing a common geometric constraint. Our spatial and dynamics queries attend to video tokens through bidirectional attention, allowing geometric supervision to shape video representations for action generation. We have added discussions and citations of GaussianDream and GWM (Lu et al., 2025) in Section 2.2 and revised the characterization to distinguish temporal sharing from cross-view sharing.

  • Quantitative Comparisons: As requested, we have added GaussianDream, GWM, and X-WAM to the RoboCasa comparison in Table 1. We have also added GaussianDream and WSA₁ to the LIBERO comparison in Table 2. WAM4D does not report comparable RoboCasa or LIBERO results and is therefore discussed in the related work without a numerical entry.


W2 & Q2: Real-World Evaluation

Answer: We thank the reviewer for pointing out the missing details in our real-world experiments. Below, we clarify depth acquisition and processing, detail camera calibration and the workspace definition, present real-world depth visualizations, and discuss the factors contributing to the performance gains. We have added the missing details of our real-world experiments to Appendix D.

  • Depth acquisition and processing: The two external cameras and the wrist-mounted camera are RealSense D435i cameras. For each external camera, we use FoundationStereo (Wen et al., 2025) to estimate metric depth from stereo infrared images using known stereo baseline and intrinsics. We then align depth to RGB using known intrinsics and infrared-to-color extrinsics. The depth loss applies the workspace mask and excludes zero-depth pixels. Across all real-world training data, zero-depth rates are 0.77% overall, 2.78% in transparent-lid regions, and 2.43% in slender-straw regions. The processed depth maps (columns 2 and 5 of Figure 7 in Appendix D.3) capture the main scene structure and provide accurate geometric supervision.

  • Camera calibration and workspace definition: We calibrate the external cameras to the robot base using Tsai–Lenz eye-to-hand calibration from 30 poses of a checkerboard rigidly attached to the end effector. The right-handed workspace frame originates at the robot base, with xx forward, yy left, and zz upward. Its bounds are x∈[0,1.6]x\in[0,1.6] m, y∈[−0.8,0.8]y\in[-0.8,0.8] m, and z∈[0,1.0]z\in[0,1.0] m. To assess extrinsic accuracy, we reconstruct point clouds from 100 two-view training frame pairs using supervision depth and calibrated camera parameters. Mean two-view F1 scores are 72.52% at 5 mm and 85.21% at 20 mm, supporting the accuracy of our calibrated extrinsics.

  • Real-world depth visualization: As requested, we provide real-world supervision depth alongside depth rendered from SpatialWAM’s predicted Gaussian scenes in Figure 7 of Appendix D.3. These visualizations provide direct qualitative evidence that explicit 3D geometric supervision enables SpatialWAM to learn and reconstruct real-world scene geometry. Together with the real-world task results, they support the contribution of this mechanism to successful robotic manipulation.

  • Analysis of performance gains: To address the confounding between geometric learning and pre-training data, we additionally pre-train Cosmos Policy on the same data as SpatialWAM and post-train it on Task 3. It achieves 12% success versus 38% for SpatialWAM under matched pre-training data, supporting the contribution of explicit 3D geometric learning to our performance gains on this task. Regarding SpatialWAM-Sim, we agree that this comparison does not isolate noise effects. We have revised the manuscript to state that including real-world data in pre-training improves real-world task performance.

I thank the authors for the detailed response, which fully addresses my concerns. The revised related work now distinguishes temporal sharing from cross-view sharing, which clarifies the contribution relative to GaussianDream, and the expanded comparison tables support the central claim. The added real-world details on depth acquisition, calibration, and the workspace definition, together with the depth visualizations, make the setup reproducible. The additional control that pre-trains Cosmos Policy on the same data goes beyond what I asked for and helps separate the effect of geometric learning from that of the pre-training data. I will keep my positive score.

AuthorsOpenReview ↗

Thanks for your positive feedback!

Thank you very much for acknowledging our revisions and additional experiments and for your positive feedback! Your constructive comments and suggestions have helped us improve the quality of our paper. Thanks!

Reviewer_TuqGOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SpatialWAM is a world action model that learns spatially grounded video and spatial representations through explicit 3D geometric learning. Motivated by the gap between visual plausibility and geometric accuracy in video-based WAMs, the authors augment a Wan2.2 video expert with a randomly initialized spatial expert whose spatial queries and per-timestep dynamics queries extract features from current and future video tokens via bidirectional joint attention. A Spatio-Temporal Gaussian Modeling (STGM) module fuses these features to predict current and future 3D Gaussian scenes, each shared across views so that multi-view depth supervision constrains a single common geometry per timestep. SpatialWAM reaches 73.4% on RoboCasa with the fewest demonstrations, the best robustness on LIBERO-Plus, and large real-world gains on precision tasks; same-data ablations show improvements over video-only and 2D-depth pre-training, with analyses suggesting substantially improved post-training data efficiency.

Critical strengths

It converts "geometry helps WAMs" from field intuition into an isolated design claim. The conceptual advance is making geometric consistency a scene level constraint—one shared 3D Gaussian scene per timestep, supervised from all views—rather than per-view prediction as in X-WAM/WAM4D/WSA. Crucially, the same-data, matched-capacity 2D-depth baseline isolates the contribution of the shared scene itself, ruling out the most obvious alternative explanations.

The evaluation matches the paper's claims and supplies two evidence types the field lacks most. The largest gains appear exactly where geometry should matter: object/scene generalization, perturbation robustness, and real-world precision alignment—claims and evidence form a closed loop. In addition, the data-scaling study shows the pre-trained model with few demonstrations matches an unpretrained model with far more, and pre-training needs no action labels and can use any calibrated RGB-D video. These address the field's two bottlenecks: expensive demonstrations and hard-to-scale representation learning.

Critical weaknesses

The central claim—"accurate geometry is learned"—is never actually measured. The motivation is millimeter-level alignment, but the built geometry is a coarse 4 cm voxel grid, supervised only by depth from two external cameras, without any color. There are no quantitative geometry metrics (e.g., depth error, cross-view consistency), and no experiment showing the action module actually uses the spatial expert's features at inference. The improvement could therefore come from coarse scene-level priors or extra parameters, rather than the precise geometry the title advertises.

Comparisons with the closest concurrent work and with deployment cost are missing. The tables mostly cover VLAs and plain WAMs; geometry-aware WAMs (X-WAM, WAM4D, WSA) are only discussed in text and appear in no experiment, so readers cannot tell how much of the advantage comes from the shared-scene design versus a generic geometry pre-training recipe. Inference keeps all three experts and jointly denoises video and actions, yet latency, control frequency, and compute are never reported—conspicuous given concurrent work (e.g., Fast-WAM) whose selling point is removing test-time video generation.

Questions for authors

Your geometry is a coarse 4 cm grid supervised by two external depth views—far from the millimeter-level alignment in your motivation. Do the gains come from precise geometry or from rough scene-level structure? Can you report quantitative geometric accuracy (e.g., depth error, cross-view consistency), or a higher-resolution variant?

How was the real-world Cosmos Policy adapted (official weights or retrained? aligned data and interfaces?), and why is its score so much lower than published results? What provides depth for real-world post-training?

What are the inference latency and control frequency, and the extra cost of tri-expert inference compared to a video-only variant?

4 comments
AuthorsOpenReview ↗

Response to Reviewer TuqG

We sincerely thank the reviewer for the thoughtful review and constructive suggestions. We address the comments and questions below.


W1 & Q1: Geometric Accuracy

Answer: Thanks for suggesting geometric evaluation. We evaluate SpatialWAM and the 2D Depth variant from Table 3 (paper) on 93 held-out RoboCasa expert trajectories (22,065 frames), using models trained on identical pre-training data and post-training demonstrations. We use CVDCE (Cross-View Depth Consistency Error) to assess cross-view agreement by reprojecting predicted depths between views using camera parameters, then averaging differences against the target view’s predicted depths in both directions.

Table 1: Geometric Evaluation. We include the 2D Depth variant as a reference to assess the geometric benefits of explicit 3D scene modeling. MAE and AbsRel measure depth prediction accuracy. CVDCE measures cross-view depth consistency. Results are averaged over the current and eight future timestamps.

ModelMAE (mm) ↓AbsRel (%) ↓CVDCE (mm) ↓
2D Depth Variant43.175.3957.71
SpatialWAM34.454.4532.04

Table 1 shows that SpatialWAM lowers average MAE by 20.20% and CVDCE by 44.48% versus the 2D Depth variant. With weights unchanged, masking action tokens’ attention to spatial and dynamics tokens at inference drops RoboCasa success from 73.4% to 12.2%. Together, these results support that explicit 3D geometric learning yields more accurate, consistent spatial features used in action generation for successful manipulation.

Clarification of the 4 cm grid. In our implementation, each voxel predicts four Gaussians with learnable center offsets, scales, and rotations. The 4 cm voxel size specifies the spatial layout of the prediction grid, rather than the size of individual Gaussians. These continuous Gaussian attributes adapt to scene geometry, enabling the representation of details finer than the voxel spacing.


W2: Comparisons with Concurrent Methods

Answer: Thanks for the valuable suggestion. We added X-WAM on RoboCasa (Table 1) and WSA₁ on LIBERO (Table 2) to the revised paper. WAM4D is discussed in related work but lacks comparable results on either benchmark. Gains over the 2D Depth variant in Table 3 (paper), together with Table 1 (W1 & Q1), demonstrate our shared-scene design’s contribution to geometric accuracy and task performance.


Q2: Real-World Training Details

Answer: Thanks for requesting clarification. We post-train Cosmos Policy from Cosmos-Predict2-2B-Video2World using the same real-world demonstrations, three camera views, robot-state inputs, action representations, and control interfaces as SpatialWAM. Initialization and training configurations are now provided in Table 9 (Appendix D.2).

Cosmos Policy in Figure 3 is directly post-trained from a general video model without robot-data pre-training or geometric supervision, explaining its lower scores on our spatially demanding tasks. Pre-training Cosmos Policy (MD) on SpatialWAM’s data before Task 3 post-training raises success from 0% to 12%, versus SpatialWAM’s 38% (Table 9), supporting the benefit of explicit 3D geometric supervision under matched pre-training data.

Depth Supervision: We use FoundationStereo (Wen et al., 2025) to estimate metric depth from external D435i stereo IR images with known baseline and intrinsics, then align it to RGB using camera intrinsics and IR-to-color extrinsics. We have detailed this pipeline in Appendix D.1 and shown the processed depth in Figure 7 (Appendix D.3).


W2 & Q3: Inference Efficiency and Deployment

Answer: Thanks for asking about deployment. We benchmark Cosmos Policy, the dual-expert variant, and SpatialWAM on an RTX 4090 with three camera views, 32-step action chunks, torch.compile, and consistent measurement protocols, averaging latency over 50 runs. Fast-WAM serves as a reference with reported latency and estimated compute.

Table 2: Inference Efficiency on an RTX 4090. Compute and latency are reported per action chunk. The dual-expert variant retains the video and action experts. * Fast-WAM compute is estimated for three views. † Fast-WAM latency is taken from its official report.

MethodDenoising StepsCompute (TFLOPs) ↓Latency (ms) ↓Controller Frequency (Hz)
Fast-WAM104.42*190†—
Cosmos Policy547.0342620
Dual-Expert Variant523.8125320
SpatialWAM (Tri-Expert)526.6528720

Table 2 shows that SpatialWAM adds approximately 12% compute and 13% latency over the dual-expert variant, yet reduces both by 43% and 33%, respectively, versus Cosmos Policy, providing useful spatial features at limited cost. Following Cosmos Policy’s synchronous protocol, we execute all 32 actions at 20 Hz (1.6 s), then pause 287 ms for the next prediction. The 20 Hz refers to action execution.

AuthorsOpenReview ↗

Reminder for Discussion

Thank you again for your thoughtful and constructive comments. We truly appreciate the time and effort you have devoted to reviewing our manuscript. With the discussion period coming to an end, we would be glad to hear whether the added geometry and efficiency evaluations help address your concerns, or if any questions remain.

We thank the authors for the response. The new geometric evaluation, masking experiment, additional comparisons, and efficiency results largely address my concerns. In particular, the matched-data results provide convincing evidence that shared-scene modeling improves geometric quality and benefits action generation.

Overall, my main concerns are largely resolved, and I have raised my rating from weak reject to weak accept.

AuthorsOpenReview ↗

Thanks for your positive reassessment!

Thank you very much for acknowledging our additional experiments and for your positive reassessment! We are glad that the new evidence has largely addressed your concerns. Your constructive comments and suggestions have helped us strengthen the geometric evaluation and improve the quality of our paper. Thanks!

Reviewer_oMT9OpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SpatialWAM is a world action model that couples video-model spatio-temporal priors with explicit 3D geometric learning for robot manipulation. Built on a frozen-VAE Wan2.2-5B video backbone, it adds a spatial expert whose learnable spatial queries (an 8x8x5 workspace grid) and per-horizon dynamics queries attend to current and future video tokens via dual-expert joint attention. A Spatio-Temporal Gaussian Modeling (STGM) module fuses the two query types into current and future 3D Gaussian scenes — one shared scene per timestep — which are rendered to multi-view depth maps and supervised with a workspace-masked smooth-L1 loss (Eqs. 4-5). The design choice the paper emphasizes is that a single scene is rendered to all supervised views, so cross-view consistency is enforced by construction rather than by per-view depth regression.

Training is two-stage. Video-spatial pre-training (Eq. 7) jointly optimizes flow-matching video prediction and the depth loss on 252,649 trajectories / 851 hours from MimicGen, RoboCasa365, and DROID; because the geometric objective needs only calibrated multi-view RGB-D, this stage requires no action labels. Action post-training (Eq. 8) introduces a randomly initialized action expert in tri-expert joint attention and fully updates all three experts while retaining the video and depth terms. At inference all three experts jointly denoise video latents and a 32-step action chunk; STGM is dropped.

Validation: RoboCasa 73.4% with 50 demos/task (vs. Cosmos Policy 67.1% and methods using 300-3000 demos), LIBERO 97.8% (second to Cosmos Policy’s 98.5%), LIBERO-Plus 85.9% without adaptation (30,090 trials), and three real-world tasks averaging 72.0% vs. Cosmos Policy’s 22.0% (450 trials). The central controlled evidence is Table 3, where four pre-training variants share identical pre-training data: w/o pre-training 57.9, video-only 68.1, 2D depth 69.3, simulation-only 70.5, full 73.4. Supporting analyses cover hyperparameter sensitivity (H, d_a), post-training data scaling, and qualitative future-frame comparisons against the video-only variant.

Critical strengths
  1. The ablation holds pre-training data fixed, which is the comparison the central claim actually requires. The paper’s thesis is that explicit 3D scene modeling, not simply more video pre-training, produces the gains. Table 3 tests this directly: +5.3 over Video-Only Pre-Training and +4.1 over 2D Depth Pre-Training on the same corpus. Most geometry-aware VLA/WAM papers substantiate their geometric component only against published baselines trained on different data; isolating it this way is the right methodology and is what makes the contribution assessable at all. This design is the main reason I lean positive.

  2. The action-label-free pre-training claim is backed by a scaling result, not just asserted. Since the geometric objective needs only calibrated RGB-D, pre-training can in principle scale past action-annotated demonstrations. Figure 4 makes this concrete: with 100 generated demos/task the pre-trained model reaches 72.2%, comparable to 71.9% for the non-pre-trained model with 1,000 demos — i.e. pre-training substitutes for roughly an order of magnitude of post-training data. Combined with the LIBERO-Plus evaluation (85.9% across 7 perturbation categories with no adaptation), this is a meaningful consequence of the design rather than a benchmark number alone.

Critical weaknesses
  1. Every headline comparison is confounded; only the ablations are controlled, and the paper’s framing does not respect that distinction. Tables 1-2 are assembled from numbers reported in the original or benchmark papers (only pi-0.5 on RoboCasa is marked as reproduced), so SpatialWAM’s margins also reflect a 5B video backbone and a 252K-trajectory / 851-hour pre-training corpus that the baselines do not share. The real-world result is the sharpest instance: a 50.0-point gap over Cosmos Policy (72.0 vs. 22.0, with 0/50 on Straw Insertion) is reported with no statement of what pre-training or post-training budget Cosmos Policy received on the same 100 demos/task. A gap of that size across three tasks is more naturally read as a setup asymmetry than as an architectural effect, and the supplementary videos (single fixed external view, no trial indexing) cannot resolve it. The consequence is that the abstract’s claims ("exceeding Cosmos Policy by 50.0 percentage points") are not supported by a controlled comparison, while the claims that are controlled (Table 3) are the modest 4-5 point ones. A single pre-training-matched baseline would resolve this and would considerably strengthen the paper.

  2. No variance is reported anywhere, and the decisive gaps are the same size as the method’s own hyperparameter sensitivity. Simulation results are averaged over three seeds, but no standard deviation or interval appears in any of Tables 1-4 or 6-7; the real-world evaluation is a single run of 50 trials per task-method (binomial std alone is about 7 points at a 38% success rate). The differences the argument rests on are 2.9-5.3 points, while Table 4 shows comparable swings from hyperparameters that are not claimed to be contributions (d_a: 73.4 / 70.8 / 71.1; H: 73.4 / 69.2 / 63.9). Without seed-level spread, the 3D-Gaussian-vs-2D-depth conclusion is not clearly separable from training noise. The comparison is further weakened because the 2D Depth variant uses a separately parameterized depth expert rather than a capacity-matched STGM substitute, so architecture and objective are varied together. Additionally, no ablation isolates the one mechanism the paper repeatedly identifies as its distinguishing idea — the shared scene across views — from multi-view depth supervision in general.

Questions for authors

Requested clarifications:

  1. Pre-training/evaluation overlap. Does the pre-training corpus (RoboCasa365, MimicGen) contain the same tasks, scenes, or object sets as the RoboCasa-24 and LIBERO evaluation suites? If there is task-level overlap, how should the 50-demos/task comparison against methods using 300-3000 demos be interpreted?
  2. Baseline parity. Were any Table 1-2 baselines re-pre-trained on your corpus, or are all numbers as-published? Specifically for real-world Cosmos Policy: what pre-training checkpoint and what post-training schedule on the 100 demos/task did it receive? Was any per-task tuning done for either method?
  3. Variance. Can you report per-seed success rates or standard deviations for Tables 1, 3, and 4? For the real-world results, is 72.0% vs. 45.3% (SpatialWAM-Sim) stable across repeats?
  4. Capacity matching. What are the parameter counts and pre-training FLOPs for the 2D Depth Pre-Training depth expert vs. the spatial expert + STGM, and for the full model vs. each ablation variant?
  5. Isolating the shared scene. What is the result if multi-view depth is predicted per-view (independent per-view geometry, no shared Gaussian scene) with everything else held fixed? This is the ablation that would directly support the cross-view-consistency argument in Sections 2.2 and 3.2.
  6. Dynamics queries. What happens if the dynamics queries are removed and only the current scene (k = 0) is supervised? This would separate "3D scene modeling" from "future geometry prediction."
  7. Depth loss at post-training. Is retaining L_depth during post-training necessary (lambda_depth = 0 in Eq. 8)?
  8. Sim-only discrepancy. Simulation-Only Pre-Training costs 2.9 points on RoboCasa but 26.7 points in the real world. How much of this is DROID’s domain proximity to your real-world platform rather than representation quality? Does a DROID-only pre-training variant behave similarly?
  9. Calibration requirements. How sensitive is the geometric objective to extrinsic calibration error, and does it work with monocular-estimated depth? The "path to scaling beyond action-labeled demonstrations" claim depends on how strict the calibrated-RGB-D requirement is.
6 comments
AuthorsOpenReview ↗

Response to Reviewer oMT9 (1/2)

We sincerely thank the reviewer for the thorough review and constructive suggestions. We address the comments and questions below.


Q1: Pre-Training Data Overlap

Answer: Thanks for requesting clarification. Our pre-training corpus overlaps with RoboCasa-24 in tasks and scenes, but not objects, and has no overlap with LIBERO or LIBERO-Plus. The 50/300/3,000 demonstrations per task represent the action-labeled demonstration budgets for post-training, while our pre-training uses only calibrated RGB-D data without action annotations. These comparisons therefore reflect performance with varying amounts of action supervision. The improvements on LIBERO and LIBERO-Plus further support the effectiveness of our approach beyond the overlapping RoboCasa tasks.


Q2 & W1: Baseline Parity

Answer: Thanks for the valuable suggestion. Tables 1–2 use results reported in the original or benchmark papers, except for our reproduced π₀.₅ result on RoboCasa. These comparisons reflect differences in both methods and training resources, while the controlled ablations in Table 3 support our method’s contribution. For Figure 3, we initialize Cosmos Policy from Cosmos-Predict2-2B-Video2World and post-train it on 100 demonstrations per task using the schedule detailed in Table 9 (Appendix D.2). Neither method uses per-task tuning, and all tasks use the hyperparameters listed in Table 9.

Pre-Training-Matched Baseline: As requested, we additionally pre-train Cosmos Policy on the same data as SpatialWAM before post-training on Task 3. As shown in Table 9, its success rate increases from 0% to 12%, compared with SpatialWAM’s 38%, supporting the benefit of explicit 3D geometric supervision under matched pre-training data. Due to resource constraints, we have evaluated this baseline on Task 3 so far and will include its results on all three tasks in the revision.


Q3 & W2: Evaluation Variance

Answer: Thanks for pointing out this issue. We have added mean ± standard deviation for our results in Tables 1–4 to Table 8 (Appendix C.4), based on three evaluation runs with seeds 7, 116, and 3520. We acknowledge that the gains are similar in magnitude to the hyperparameter-induced changes in Table 4. To examine whether these gains are stable across evaluation runs, we measure variability with all other settings fixed. On RoboCasa, SpatialWAM achieves 73.39 ± 0.34% versus 2D Depth’s 69.33 ± 0.91%. The 4.06-point gain exceeds both evaluation standard deviations, supporting its stability across runs. Due to resource constraints, we conducted two additional real-world runs on Task 3 only, with 50 trials per method per run. Across three runs, SpatialWAM achieves 38%, 44%, and 38% (40.00 ± 3.46%), versus SpatialWAM-Sim’s 20%, 24%, and 18% (20.67 ± 3.06%). Its consistent 18–20-point advantage supports stability on Task 3.


Q4 & W2: Model Capacity

Answer: Thanks for the valuable suggestion. Table 1 shows that SpatialWAM and the 2D Depth variant have similar geometry-module sizes (659.55M vs. 637.68M, a 3.43% difference), with only a 0.39% difference in total parameters. These closely matched parameter counts support attributing the 4.06-point improvement to explicit 3D geometric learning rather than increased model capacity. Most additional forward compute comes from temporal fusion and Gaussian decoding (0.84 TFLOPs).

Table 1: Model Capacity and Compute. We compare the three variants from Table 3 (paper), measuring their pre-training models without the action expert. Compute is counted for a single forward pass with batch size 1 and three camera views at 224 × 224 resolution.

VariantGeometryGeometry Params (M)Total Params (B)Compute (TFLOPs)
Video-Only——5.004.65
2D DepthDepth expert + head637.685.645.15
SpatialWAMSpatial expert + STGM659.555.666.04
AuthorsOpenReview ↗

Response to Reviewer oMT9 (2/2)

Table 2: Additional Post-Training Ablations on RoboCasa. Success rates (%) are reported as mean ± standard deviation across three evaluation runs. All settings are identical to SpatialWAM except for the changes specified in each variant. Experimental settings are described in the corresponding responses.

QuestionVariantSuccess Rate (%)
—SpatialWAM73.39 ± 0.34
Q5Independent Per-View Gaussian Scenes69.75 ± 1.11
Q6w/o Dynamics Queries (Only Current Scene)69.08 ± 0.61
Q7w/o Depth Loss during Post-Training67.00 ± 0.46
Q9Perturbed Camera Extrinsics71.67 ± 0.50
Q9Monocular-Estimated Depth (MoGe-3)75.00 ± 0.93

Q5 & W2: Shared-Scene Design

Answer: Thanks for suggesting this ablation. We predict independent per-view Gaussian scenes with all other settings unchanged. Table 2 shows that RoboCasa success drops from 73.39% to 69.75%, demonstrating the benefit of shared scenes beyond multi-view depth supervision alone. Our geometric evaluation (Table 1, response to Reviewer R-TuqG) also shows lower cross-view depth consistency error than the 2D Depth variant (32.04 vs. 57.71 mm), supporting the cross-view-consistency argument in Sections 2.2 and 3.2.


Q6: Dynamics Queries

Answer: Thanks for this helpful suggestion. We remove the dynamics queries and supervise only the current scene (k=0k=0), keeping all other settings unchanged. Table 2 shows that success drops from 73.39% to 69.08%, demonstrating the contribution of future geometry prediction beyond current-scene 3D modeling.


Q7: Depth Supervision during Post-Training

Answer: Thanks for raising this question. We remove the depth loss during post-training by setting λdepth=0\lambda_{\mathrm{depth}}=0 in Eq. 8, keeping all other settings unchanged. Table 2 shows that success drops from 73.39% to 67.00%, demonstrating the necessity of retaining depth supervision during post-training to maintain full task performance.


Q8: Real-World Pre-Training Data

Answer: Thanks for the valuable suggestion. DROID shares a similar platform configuration with ours: a single arm, two external stereo cameras, and one wrist-mounted camera. We pre-train a DROID-only variant and post-train it on Task 3, obtaining 26% success, compared with 20% for SpatialWAM-Sim and 38% for SpatialWAM. This suggests that domain proximity may contribute to real-world transfer, while combining real and simulated pre-training data provides additional benefits beyond DROID alone.


Q9: Calibration and Depth Requirements

Answer: Thanks for raising these practical concerns. During post-training, we perturb camera extrinsics by randomly choosing translation or rotation with equal probability, then randomly selecting one of the x, y, or z axes. Translation offsets along the selected axis are sampled uniformly from [−10, 10] cm, and rotation offsets around the selected axis from [−10°, 10°]. All other settings remain unchanged. Table 2 shows that success decreases from 73.39% to 71.67%, a modest drop of 1.72 percentage points, indicating tolerance to calibration errors.

Monocular Depth: We replace the simulator ground-truth depth with monocular depth estimated by MoGe-3 [1] during post-training. Interestingly, success improves from 73.39% to 75.00% (Table 2). One possible explanation is that MoGe-3’s estimated depth introduces supervision noise that acts as regularization. Together, these two experiments show that geometric supervision remains effective with imperfect calibration and monocular-estimated depth, supporting the potential to scale beyond action-labeled demonstrations with less restrictive geometric data requirements.

[1] Lingyu Kong et al. MoGe-3: Fine-Detail Monocular Geometry Estimation with Self-Guided Sparse Volumetric Refinement. NeurIPS, 2026.

AuthorsOpenReview ↗

Reminder for Discussion

Thank you once more for your insightful and constructive comments and for the time you have dedicated to reviewing our paper. We hope our additional comparisons and ablations help address your concerns. As the discussion phase comes to an end, please let us know if you have any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their time, effort, and constructive feedback.


We appreciate the reviewers’ recognition of our geometric learning design and controlled experimental evidence:

  • “The cross-view rendering constraint on a shared 3D scene is a substantive mechanism, not a terminological distinction.” R-DkJ1
  • “The conceptual advance is making geometric consistency a scene level constraint—one shared 3D Gaussian scene per timestep, supervised from all views—rather than per-view prediction” R-TuqG
  • “The ablation holds pre-training data fixed, which is the comparison the central claim actually requires.” and “This design is the main reason I lean positive.” R-oMT9

All reviewers also recognized the value of pre-training without action annotations:

  • “the absence of an action-annotation requirement during pre-training extends the data boundary for representation learning from robot demonstrations to calibrated RGB-D videos.” R-DkJ1
  • “pre-training needs no action labels and can use any calibrated RGB-D video.” R-TuqG
  • “The action-label-free pre-training claim is backed by a scaling result, not just asserted.” R-oMT9

All three reviewers rated the presentation as “Clear and easy to follow.”


Updates and New Experiments:

  • Related Work and Positioning (R-DkJ1, R-TuqG). We clarify the distinction between temporal and cross-view sharing, expand the discussion of GaussianDream and GWM, and add available benchmark results for GaussianDream, GWM, X-WAM, and WSA₁.

  • Real-World Evaluation and Deployment (all reviewers). We detail depth processing, calibration, baseline training, and deployment, provide real-world depth visualizations, and clarify data overlap and comparison conditions. With matched pre-training data, SpatialWAM achieves 38% versus Cosmos Policy’s 12% on Task 3. On the same RTX 4090, deployment latency is 287 ms per action chunk, approximately 33% lower than Cosmos Policy’s.

  • Geometric Accuracy and Shared-Scene Design (R-TuqG, R-oMT9). SpatialWAM reduces depth MAE by 20.20% and cross-view consistency error by 44.48% relative to the 2D Depth variant. Blocking access to spatial and dynamics features reduces success from 73.4% to 12.2%. Independent per-view Gaussian scenes during post-training reduce success to 69.75%, supporting the shared-scene design.

  • Additional Analyses (R-oMT9). We report closely matched model capacities, evaluation standard deviations, and repeated Task 3 trials. Further ablations assess future geometry prediction, post-training depth supervision, and DROID-only pre-training. Extrinsic-perturbation and monocular-depth experiments demonstrate tolerance to imperfect geometric supervision during post-training.

We have marked revisions and additions to the manuscript in blue, primarily in Section 2.2, Tables 1–2, and Appendices C.4 and D. We thank all reviewers again for their thoughtful comments and constructive suggestions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.