acceptodds
Under review as a conference paper at ICLR 2027

WAM4D: Learning to Act through Geometry-Grounded World Prediction

24 views

Abstract

World-action models couple future visual prediction with robot action generation, yet visually plausible futures do not necessarily capture the spatial changes required for reliable manipulation. We introduce WAM4D, a geometry-grounded world-action model that connects future geometric prediction to robot control while retaining RGB-only sensing. Guided by a pretrained geometric decoder and future-depth supervision, spatial register tokens learn temporally structured features from RGB observation history within a shared video-action backbone. Asymmetric causal attention makes these geometry-supervised features directly available to action queries alongside RGB features, integrating geometric prediction and action generation into single-stage training without future-target leakage. At inference, history and register features are computed once per decision step and cached for flow-based action denoising, requiring neither future-video generation nor dense depth decoding. In controlled experiments on four RoboTwin 2.0 tasks, WAM4D achieves higher average success than an RGB-only WAM and a 4D-WAM reimplementation across their common evaluation conditions, with the largest gains under texture and dynamic lighting changes. Frozen-feature probes further show that its registers retain action-relevant information and are less sensitive to appearance perturbations than RGB-only WAM features. Complementary evaluations with an earlier register-block variant show competitive full-task RoboTwin performance, improved geometric prediction on RT-1 and Bridge-Data, and higher average sub-action success in real-world manipulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World-action models, 4D world modeling, Robot manipulation, Geometric representation learning.

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WUK8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WAM4D, a world action model that uses future depth supervision to train spatial register tokens within a shared video and action backbone. Asymmetric causal attention allows action queries to access these geometry features without accessing future visual targets. At deployment, history and register features are cached for action denoising, avoiding the generation of future videos and dense depth decoding. The paper evaluates control on RoboTwin and four tasks with real robots, and video and geometry prediction on RT-1 and Bridge-Data.

Critical strengths
  1. The method establishes an explicit connection between geometric supervision and action generation. Action queries directly access register features trained with depth supervision, while the causal visibility design prevents access to future visual targets. Caching these features and omitting dense depth decoding provides a practical deployment advantage.
  2. The experiments provide positive control evidence beyond prediction quality. Depth decoder ablations on a diagnostic subset of ten tasks improve policy success, and controlled perturbation experiments show benefits particularly under texture and dynamic lighting changes. Results across the full RoboTwin task suite also demonstrate competitive performance.
Critical weaknesses
  1. The correspondence between the reported results and the actual training configurations is insufficiently specified. The supplementary M7 implementation includes a consistency loss between register features from original and augmented images, as well as visual KV dropout. The paper only briefly mentions the consistency loss without defining its formulation, weight, or experimental scope.
  2. The evaluation on real robots omits the bottle placement stage. The task definition includes both grasping and placement, but Table 2 reports only Bottle S1. Its reported average therefore covers seven displayed stages rather than all eight defined stages.
Questions for authors
  1. Which supplementary configurations produced the main results, particularly Tables 4 and 5? Please specify whether register consistency and visual KV dropout were enable.
  2. Why is Bottle S2 absent from Table 2? Could you provide its success count for each method from the existing rollouts?
Reviewer_ePF5OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes WAM4D, a geometry-grounded World-Action Model that appends 240 learnable Spatial Register Tokens (3 coarse future frames x 80 tokens) to a shared Mixture-of-Transformers video-action backbone. During training, intermediate register readouts are supervised with future depth via a pretrained DA3 decoder; at inference, dense depth decoding and future video generation are skipped, and observed RGB plus spatial registers are prefilled and cached under an asymmetric causal mask so action denoising queries can read geometry-supervised register features.

Critical strengths

(1) Practical training-inference decoupling via Spatial Register Tokens: Supervising future 3D/4D geometry through compact spatial registers during training while caching their KV states once per action chunk at inference avoids test-time depth-sensor dependencies and cuts latency by 29.8% relative to video-generating LingBot-VA (592.34 ms vs. 843.57 ms, Table 1).

(2) Comprehensive evaluation across simulation, open datasets, and real-world tasks: The paper validates WAM4D across all 50 tasks in RoboTwin 2.0, a 10-task diagnostic ablation suite, 4-task environmental perturbations (Table 4), frozen-feature action probes (Table 5), RT-1/Bridge-Data depth evaluation (Table 12), and 4 real-world bimanual tasks on AstriBot S1 (Table 2), including a strong offline DA3 pseudo-depth experiment (77.2% vs. 71.7% without depth).

Critical weaknesses

(1) Missing conditioning in Spatial Register queries and static repeated initialization: Under the asymmetric causal mask in Section 3.3 (p. 5), spatial register queries attend only to history RGB and one another—excluding language instructions (text cross-attention) and robot state history even though both are already available during inference prefill—forcing the registers to predict future multi-task depth without knowing which object or goal the instruction targets. In addition, input registers are initialized by repeating 80 static vectors across future time slots (Eq. 6), and Appendix Table 9 (p. 21) shows high inter-episode cosine similarity (0.8829) and lower effective rank than RGB-only WAM features (18.80 vs. 28.88 out of 36).

(2) Confounded capacity-versus-geometry ablation and modest pretrained prior gains on Randomized scenes: In Tables 3 and 11, the No Depth baseline (71.7% Clean / 69.1% Randomized, 5.089B parameters) removes both the 240 spatial register tokens (+601M parameters) and L_depth relative to WAM4D (80.1% Clean / 75.4% Randomized, 5.690B parameters), without a Registers Only (lambda_depth = 0) or Masked Action->Register baseline to separate extra token/parameter capacity from geometric supervision; meanwhile, keeping the 240 registers increases single-chunk latency by 39.2% over action-only Fast-WAM (592.34 ms vs. 425.53 ms, Table 14). Furthermore, Table 6 shows that while Trainable pretrained DA3 substantially improves Clean success over Trainable random init (80.1% vs. 70.0%), the gain on Randomized scenes is only +0.6 pp (75.4% vs. 74.8%).

(3) Mixed results under broader domain randomization and minor table inconsistencies: On the full 50-task RoboTwin 2.0 Randomized split (Table 1), WAM4D (90.1%) trails Fast-WAM (91.8%) and LingBot-VA (91.50% in Table 13, reported as 91.6% in Table 1), its 4-task perturbation gains in Appendix Table 8 are primarily concentrated in Handover block, and its frozen-feature probe MSE on unseen Randomized episodes (0.0316, Table 5) remains higher than the State + instruction baseline (0.0271). Finally, Table 1's aggregate scores for LingBot-VA (91.6 / 92.3) and Motus (88.7 / 87.9) differ slightly from the 50-task averages in Table 13 (91.50 / 92.20 and 88.52 / 87.78), and real-robot Table 2 reports 7 sub-actions while omitting Bottle S2 defined in Figure 3 and Appendix Table 7.

Questions for authors
  1. Since language instructions and robot state history are already known during inference prefill, does allowing spatial register queries to attend to them improve multi-task future depth prediction and randomized control?

  2. Holding the 240 spatial registers (5.690B parameters) fixed, what is the 10-task Clean and Randomized success rate when (a) training with lambda_depth = 0 (Registers Only), and (b) masking out Action -> Register attention at inference, and why does Trainable pretrained DA3 improve over Trainable random init by only +0.6 pp on Randomized scenes (Table 6)?

  3. Could you reconcile the LingBot-VA and Motus averages between Table 1 (91.6 / 92.3 and 88.7 / 87.9) and Table 13 (91.50 / 92.20 and 88.52 / 87.78), and report the missing Bottle S2 real-robot sub-action success rate from Table 7 and Figure 3?

2 comments

Q1. Conditioning Spatial Registers on Instructions and Robot States

We tested explicit conditioning by allowing spatial registers to attend to robot states via self-attention and instructions via cross-attention, using the same 10-task Clean+Randomized training protocol.

ModelAbsRel ↓δ1 ↑CD1 ↓F-score ↑F-score-T ↑
WAM4D0.0490.9480.00990.7100.848
+ Instruction & State0.0480.9480.00970.7130.850

Conditioning slightly improves geometry (AbsRel: 0.049 → 0.048; F-score: 0.710 → 0.713), but policy success remains nearly unchanged:

ModelCleanRandomized
WAM4D80.1%75.4%
+ Instruction & State79.9%75.4%

The incremental benefit is thus limited in this setting. Action queries already access instructions and states directly, and observed RGB history may largely suffice for predicting future geometry.

Q2(a). Parameter-Controlled Depth-Supervision Ablation

Registers have lower effective rank than RGB features (18.80 vs. 28.88), potentially reflecting a more compact, geometry-focused representation that suppresses appearance variations less relevant to depth prediction. Although we repeat a learnable spatial grid across future time slots, spatiotemporal RoPE encodes target timestamps and spatial positions; attention to RGB history allows distinct, observation-dependent register representations. Feature probes and ablations confirm that the registers retain action-relevant information.

To separate geometry supervision from added capacity, we retained the same registers, architecture, and action-attention pathway while disabling the depth loss (λdepth=0\lambda_{\mathrm{depth}}=0). Ten-task results (%):

ModelCleanRandomized
RGB-only WAM (no registers/depth)71.769.1
WAM4D w/o Depth Loss72.169.3
WAM4D80.175.4

Registers alone yield only +0.4/+0.2 pp (Clean/Randomized); enabling future-depth supervision with the same registers adds +8.0/+6.1 pp. Thus register capacity alone cannot explain the gain, supporting geometry-grounded representation learning.

Q2(b). Masking Action-to-Register Attention

To test whether actions use register features, we masked action-to-register attention at inference without changing trained weights. Ten-task success (%):

ModelCleanRandomized
WAM4D80.175.4
w/o Action-to-Register Attention63.560.3
Difference−16.6 pp−15.1 pp

The substantial drops show that action queries rely on the spatial registers. Together with Q2(a), this supports that geometry-supervised representations are used for action generation.

Q2(c). Limited Randomized Gain from Pretrained DA3 (Table 6)

Compared with trainable random initialization, pretrained DA3 increases Clean success by +10.1 pp but Randomized success by only +0.6 pp. Randomly initialized depth supervision already raises Randomized success from 69.1% (no depth) to 74.8%, suggesting that most gains there stem from depth supervision, with a smaller additional benefit from pretrained weights.

DA3 was pretrained on features from its own visual encoder, whereas our decoder processes registers from a shared video–action backbone. The temporal upsampler, adapters, and decoder must therefore adapt to this feature-distribution mismatch. Pretrained geometry priors do not guarantee robustness to randomized appearances, and better geometric decoding need not yield proportional gains in closed-loop success.

Q3. Baseline Aggregation, Randomized Performance, and Real-Robot Reporting

We rechecked the baselines against their original papers. Motus: Per-task values taken from LingBot-VA's supplement differ slightly from the Motus paper; we corrected them, along with premature rounding of the aggregate. LingBot-VA: Its reported aggregate differs slightly from the mean of displayed per-task results; premature rounding also affected our main table. We will align both tables with official scores; our results and conclusions are unchanged.

MethodCleanRandomizedAverage
Motus88.6687.0287.84
LingBot-VA92.9391.5592.24
WAM4D (ours)94.2490.0892.16

We acknowledge that WAM4D trails LingBot-VA (91.55%) and Fast-WAM (91.78%) on full-suite Randomized success (90.08%) and will clarify this limitation.

For Bottle S2, the Table 2 value labeled “Bottle S1” measures the entire grasp-and-place operation, including placement, consistent with the LEGO-sorting subtask criterion. The separate S2 labels in Figure 3 and Appendix Table 7 reflect an outdated breakdown. We will reconcile these labels and specify the success criterion; no stage is missing from the reported result.

Reviewer_ePF5OpenReview ↗

Response to Authors

Thank you to the authors for the thorough experiments and clarifications.

  1. Addressed Concerns (Q1, Q2a, Q2b, Q3):
  • Capacity vs. Geometry (Q2a, Q2b): Holding the 240 registers fixed with lambda_depth = 0 (WAM4D w/o Depth Loss) yields only +0.4/+0.2 pp over the RGB-only WAM (vs. +8.0/+6.1 pp with depth supervision), and masking Action->Register attention at inference drops success by -16.6/-15.1 pp. This cleanly confirms that the gains come from depth-supervised register features rather than extra token capacity.
  • Conditioning & Reporting (Q1, Q3): The + Instruction & State ablation (79.9%/75.4% vs. 80.1%/75.4%) justifies the vision-only register design on RoboTwin (indicating the registers act mainly as a task-agnostic geometric extrapolator), and the Table 1/13 averages and Table 2/7 Bottle S1/S2 definitions are now reconciled. Please incorporate all new ablations and corrections into the revised manuscript.
  1. Remaining Limitations & Final Rating (Q2c, W2, W3): While the rebuttal resolves the methodological gaps, key empirical trade-offs remain: pretrained DA3 adds only +0.6 pp over random initialization on 10-task Randomized scenes (75.4% vs. 74.8%), and on the full 50-task Randomized split WAM4D (90.08%) still trails Fast-WAM (91.78%) and LingBot-VA (91.55%) with 39.2% higher latency than Fast-WAM. Balancing the strengthened experimental completeness against these Randomized and efficiency trade-offs, I maintain my positive rating (3: Weak accept).
Reviewer_DgpwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WAM4D, which uses future depth supervision to learn spatial register tokens for action prediction. Its main contribution is integrating these tokens into a shared video-action backbone and caching their features for action-only inference. Experiments show competitive control performance on RoboTwin 2.0 and improved geometric prediction on RT-1 and Bridge-Data.

Critical strengths
  1. World-action models can generate plausible futures while missing the geometry needed for reliable manipulation, and using future depth supervision to guide action prediction is intuitive.

  2. The paper compares depth supervision and decoder training choices and uses feature probes to show that spatial registers retain action-relevant information under the tested appearance changes.

Critical weaknesses
  1. The comparison with X-WAM is limited to video and geometry generation, leaving the advantage in manipulation performance unclear.
  2. The ablations do not compare future-depth supervision with current-frame depth supervision, so the specific benefit of predicting future geometry remains unclear.
Questions for authors
  1. A comparison with X-WAM on manipulation tasks under matched training and evaluation settings would help clarify the control benefits of the proposed method.
  2. An ablation using current-frame depth supervision with the same architecture and training budget would help determine whether predicting future geometry provides additional control gains.
1 comment

Thank you for recognizing the motivation for future geometry supervision and for suggesting two important controlled comparisons. We have added (1) a closed-loop manipulation comparison with X-WAM and (2) an ablation replacing future-depth prediction with current-depth reconstruction. The experiments follow the same training and evaluation protocol as our environmental-perturbation study.

Q1. Closed-loop manipulation comparison with X-WAM.

The new results are summarized below (success rate, %). We include the RGB-only WAM and our 4D-WAM reimplementation as additional references.

Evaluation settingRGB-only WAM4D-WAM (reimpl.)X-WAM (reimpl.)WAM4D (current depth)WAM4D (future depth)
Clean32.530.533.033.535.0
Table height21.523.022.021.023.0
Texture18.516.021.525.024.0
Clutter19.515.517.019.019.5
Dynamic lighting30.529.535.034.535.5
Average24.522.925.726.627.4

We trained an X-WAM reimplementation within our framework using the same data and training protocol. Under matched evaluation, WAM4D achieves a 27.4% average success rate, compared with 25.7% for X-WAM (+1.7 percentage points). We will add this result and clarify the evaluation protocol in the revision.

Q2. Current-depth versus future-depth supervision.

We replaced future-depth prediction with current-depth reconstruction by replacing the temporal upsampling layers with temporal averaging, while leaving the remainder of the architecture unchanged. This variant achieves 26.6% average success, compared with 27.4% for future-depth supervision (−0.8 percentage points). Future-depth supervision performs better in four of the five reported conditions, with the largest difference under changes in table height (23.0% versus 21.0%). However, current-depth supervision performs better under texture changes (25.0% versus 24.0%); thus, the benefit is not uniform.

We interpret this result as suggestive evidence that predicting future geometry can encourage representations of scene evolution that are relevant to action generation, beyond reconstructing the currently observed scene. We will revise the discussion accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.