- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes WAM4D, a world action model that uses future depth supervision to train spatial register tokens within a shared video and action backbone. Asymmetric causal attention allows action queries to access these geometry features without accessing future visual targets. At deployment, history and register features are cached for action denoising, avoiding the generation of future videos and dense depth decoding. The paper evaluates control on RoboTwin and four tasks with real robots, and video and geometry prediction on RT-1 and Bridge-Data.
Critical strengths
- The method establishes an explicit connection between geometric supervision and action generation. Action queries directly access register features trained with depth supervision, while the causal visibility design prevents access to future visual targets. Caching these features and omitting dense depth decoding provides a practical deployment advantage.
- The experiments provide positive control evidence beyond prediction quality. Depth decoder ablations on a diagnostic subset of ten tasks improve policy success, and controlled perturbation experiments show benefits particularly under texture and dynamic lighting changes. Results across the full RoboTwin task suite also demonstrate competitive performance.
Critical weaknesses
- The correspondence between the reported results and the actual training configurations is insufficiently specified. The supplementary M7 implementation includes a consistency loss between register features from original and augmented images, as well as visual KV dropout. The paper only briefly mentions the consistency loss without defining its formulation, weight, or experimental scope.
- The evaluation on real robots omits the bottle placement stage. The task definition includes both grasping and placement, but Table 2 reports only Bottle S1. Its reported average therefore covers seven displayed stages rather than all eight defined stages.
Questions for authors
- Which supplementary configurations produced the main results, particularly Tables 4 and 5? Please specify whether register consistency and visual KV dropout were enable.
- Why is Bottle S2 absent from Table 2? Could you provide its success count for each method from the existing rollouts?
Q1. Conditioning Spatial Registers on Instructions and Robot States
We tested explicit conditioning by allowing spatial registers to attend to robot states via self-attention and instructions via cross-attention, using the same 10-task Clean+Randomized training protocol.
Conditioning slightly improves geometry (AbsRel: 0.049 → 0.048; F-score: 0.710 → 0.713), but policy success remains nearly unchanged:
The incremental benefit is thus limited in this setting. Action queries already access instructions and states directly, and observed RGB history may largely suffice for predicting future geometry.
Q2(a). Parameter-Controlled Depth-Supervision Ablation
Registers have lower effective rank than RGB features (18.80 vs. 28.88), potentially reflecting a more compact, geometry-focused representation that suppresses appearance variations less relevant to depth prediction. Although we repeat a learnable spatial grid across future time slots, spatiotemporal RoPE encodes target timestamps and spatial positions; attention to RGB history allows distinct, observation-dependent register representations. Feature probes and ablations confirm that the registers retain action-relevant information.
To separate geometry supervision from added capacity, we retained the same registers, architecture, and action-attention pathway while disabling the depth loss (λdepth=0). Ten-task results (%):
Registers alone yield only +0.4/+0.2 pp (Clean/Randomized); enabling future-depth supervision with the same registers adds +8.0/+6.1 pp. Thus register capacity alone cannot explain the gain, supporting geometry-grounded representation learning.
Q2(b). Masking Action-to-Register Attention
To test whether actions use register features, we masked action-to-register attention at inference without changing trained weights. Ten-task success (%):
The substantial drops show that action queries rely on the spatial registers. Together with Q2(a), this supports that geometry-supervised representations are used for action generation.
Q2(c). Limited Randomized Gain from Pretrained DA3 (Table 6)
Compared with trainable random initialization, pretrained DA3 increases Clean success by +10.1 pp but Randomized success by only +0.6 pp. Randomly initialized depth supervision already raises Randomized success from 69.1% (no depth) to 74.8%, suggesting that most gains there stem from depth supervision, with a smaller additional benefit from pretrained weights.
DA3 was pretrained on features from its own visual encoder, whereas our decoder processes registers from a shared video–action backbone. The temporal upsampler, adapters, and decoder must therefore adapt to this feature-distribution mismatch. Pretrained geometry priors do not guarantee robustness to randomized appearances, and better geometric decoding need not yield proportional gains in closed-loop success.
Q3. Baseline Aggregation, Randomized Performance, and Real-Robot Reporting
We rechecked the baselines against their original papers. Motus: Per-task values taken from LingBot-VA's supplement differ slightly from the Motus paper; we corrected them, along with premature rounding of the aggregate. LingBot-VA: Its reported aggregate differs slightly from the mean of displayed per-task results; premature rounding also affected our main table. We will align both tables with official scores; our results and conclusions are unchanged.
We acknowledge that WAM4D trails LingBot-VA (91.55%) and Fast-WAM (91.78%) on full-suite Randomized success (90.08%) and will clarify this limitation.
For Bottle S2, the Table 2 value labeled “Bottle S1” measures the entire grasp-and-place operation, including placement, consistent with the LEGO-sorting subtask criterion. The separate S2 labels in Figure 3 and Appendix Table 7 reflect an outdated breakdown. We will reconcile these labels and specify the success criterion; no stage is missing from the reported result.