- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes 3DFlowWAM, a unified world action model that jointly generates future RGB video, a dense 3D Flow representation, and robot actions within a shared causal diffusion transformer. The central representation combines an image-aligned point map, which records the 3D position of each visible pixel, with world-centric scene flow, which records the 3D displacement of the same physical point over time. RGB, 3D Flow, and action tokens are integrated into temporally aligned prediction blocks and jointly denoised with modality-specific flow-matching losses. Training starts from a pretrained video model, continues with 3D Flow-aware mid-training on large-scale robot-video data, and then adapts to target embodiments and tasks. At inference, 3DFlowWAM uses RGB observations, the language instruction, and optional proprioceptive state to predict the next video, 3D Flow, and action chunk; it executes the action and updates its visual context in a closed loop without requiring an online depth sensor or 3D tracker. The paper validates the design on LIBERO, LIBERO-Plus, RoboTwin 2.0, and seven real-world tasks on ALOHA-style and Franka platforms.
Critical strengths
Point maps provide scene state while scene flow provides temporal 3D displacement; keeping both on the image grid preserves correspondence with RGB observations and avoids the ordering and alignment difficulties of unstructured point clouds. The shared causal diffusion transformer then allows action tokens to attend directly to both visual and geometric motion tokens, so the paper proposes an integrated world model rather than a pipeline of separately trained modules. The design also retains an understandable physical interpretation while remaining independent of a specific robot action coordinate system. Its deployment path is practical because the model predicts 3D Flow internally and needs only RGB observations and robot state at test time.
Critical weaknesses
-
The experiments do not fully isolate joint generation from auxiliary supervision and capacity. The comparison with and without a 3D Flow stream shows that the complete 3DFlowWAM system is effective, but it does not distinguish joint generation from a training-only 3D Flow reconstruction branch, additional model and token capacity, or an additional auxiliary objective. The representation study is informative but is limited to the Hang Mug task and changes both the represented quantity and its VAE/token budget. Additional matched controls would substantially strengthen attribution to the joint design itself.
-
A few reporting and evaluation details require clarification or correction. In Table 7, the displayed RoboTwin results for X-VLA under the Clean condition average to 72.88%, while the table reports a 50-task mean of 72.80%; either one cell or the reported mean appears to require correction. The manuscript also does not clearly state how sequence-level 3D normalization statistics are obtained at inference, how 3D Flow tokens are aligned with three RGB views when only the overhead view has flow supervision, or how invalid tracked pixels are excluded from the losses. These issues are correctable and do not undermine the overall empirical case, but resolving them would improve reproducibility and confidence.
Questions for authors
- Clarification of normalization and test-time information. How are the sequence-level center and scale for point maps and scene flow computed? Are the same statistics available at deployment without using future ground-truth geometry, and are they used only for decoding or also for action prediction?
- Clarification of multimodal token alignment and masks. Please specify how a single overhead-view 3D Flow stream is aligned with multiple RGB camera views at different spatial resolutions. Please also state whether invalid tracked pixels are explicitly masked from the Flow VAE loss and flow-matching objective.
- Clarification of code, datasets, and processed 3D Flow annotations. Do the authors plan to release the model code and any datasets or processed 3D Flow annotations used for mid-training and task adaptation? If so, please specify whether the release will include the Track4World and SAM 3 preprocessing pipeline, the confidence and foreground masks, and the normalization statistics needed to reproduce training and inference.
5 comments
Rebuttal for reviewer Efzg (2/4)
2. Scope of the representation study and the effects of VAE choice and token budget
We extended the original Hang Mug study to the five jointly trained RoboTwin tasks above and added Sparse 3D Flow (Appendix H, Table 10). All variants retain RGB and action prediction and use DreamZero initialization without additional mid-training.
Mean success over the five tasks:
| Prediction targets | Clean (%) | Randomized (%) |
|---|---|---|
| RGB + action | 45.2 | 36.8 |
| + 2D Flow | 54.2 | 47.6 |
| + Depth | 56.8 | 50.2 |
| + Sparse 3D Flow | 52.8 | 45.8 |
| + Point Maps | 57.4 | 52.8 |
| + Scene Flow | 52.4 | 47.4 |
| + Full 3D Flow | 65.0 | 59.6 |
Full 3D Flow achieves the highest success on all five tasks under both conditions, extending the evidence for complementary position and displacement supervision beyond Hang Mug.
We also compare representations on four ALOHA tasks using the same mid-trained initialization and matched post-training settings (Appendix H, Table 11):
| Supervision | Mean ALOHA success (%) |
|---|---|
| SAM3 + 2D Flow + Depth | 68.8 |
| SAM3 + Sparse 3D Flow | 68.8 |
| SAM3 + Object Flow | 76.3 |
| SAM3 + 3DFlowWAM | 78.8 |
Encoder choice.
We use modality-appropriate encoders: Wan VAE for Depth and 2D Flow, and the corresponding MotionCrafter VAE branches for Point Maps and Scene Flow. Full 3D Flow jointly encodes both fields. Sparse 3D Flow randomly samples 10 points per frame within the SAM 3 interaction region and supplies their displacements to the Scene Flow branch.
Token budget.
We examine token count separately by varying the Flow stream over 32, 64, 128, and 256 tokens per block, while keeping video and action tokens fixed at 512 and 48, respectively (Appendix K, Table 15; Appendix N, Table 16).
| Flow tokens per block | Mean Clean success (%) | Mean Randomized success (%) |
|---|---|---|
| 32 | 65.0 | 59.6 |
| 64 | 65.8 | 60.6 |
| 128 | 65.0 | 60.8 |
| 256 | 66.0 | 59.4 |
Mean success ranges from 65.0% to 66.0% under Clean evaluation and 59.4% to 60.8% under Randomized evaluation, without monotonic improvement as token count increases. These results support the effectiveness of a compact Flow representation across the tested budgets.
Rebuttal for reviewer Efzg (3/4)
3. Normalization and test-time information
We compute normalization statistics offline, following MotionCrafter. Reconstructed points are first transformed into a common coordinate frame referenced to the first camera pose. Let contain the valid point-map entries across the processed sequence. The center and scale are
\qquad s=\frac{1}{|\mathcal{D}|}\sum_{i\in\mathcal{D}}\|P_i-\mu\|_2.$$ Thus, the center is the mean 3D position, and the scale is the mean distance from that center. Point Maps and Scene Flow share this scale: $$\bar P=(P-\mu)/s, \qquad \bar F=F/s.$$ Displacements are scaled but not centered. We store these statistics with the targets and recover geometry for visualization using $P=s\bar P+\mu$ and $F=s\bar F$. The computation and inverse transformations are provided in **Appendix N, “Sequence-level center and scale.”** **Action inference requires neither these sequence statistics nor future ground-truth geometry.** The policy jointly predicts video, Flow, and action latents and directly outputs an action chunk, without decoding or denormalizing Flow. Track4World is therefore not required online. ### 4. Multi-view alignment and treatment of unreliable tracking **Alignment across camera views.** We construct Flow targets only for the head camera while retaining RGB inputs from all three cameras. Wrist cameras undergo substantial motion and frequent occlusion, making their Flow estimates less reliable and limiting the additional scene-motion information they provide. The three RGB views are spatially concatenated into a 2 × 2 grid, with one padded cell, before video encoding (Appendix N). Flow targets associate each head-camera pixel $(u,v)$ with its 3D position and displacement. This is correspondence at the target level, not one-to-one alignment between differently sized latent grids or between Flow and all three RGB views. Cross-view information is integrated through shared attention within each temporal block. **Masks and losses.** Tracking-confidence filtering and SAM 3 foreground masks are applied during offline target preprocessing. With a fixed camera, static-background Scene Flow should be near zero, but geometry and pose-estimation errors can introduce spurious displacement. We therefore set background Scene Flow to zero before encoding. MotionCrafter VAE remains frozen during policy training, so no additional VAE reconstruction loss is optimized. The latent flow-matching objective uses the encoded, filtered targets; it does **not** apply a separate per-pixel validity mask to the latent loss. Masks are used to prepare the supervision targets, rather than to exclude individual pixels directly from that loss.
Rebuttal for reviewer Efzg (1/4)
Thank you for the constructive feedback. We have added controls for Flow supervision, cross-modal interaction, and token budget, expanded the representation study, and clarified the implementation details below.
1. Flow supervision, additional capacity, and joint generation
Flow supervision versus additional capacity.
We added a capacity-matched control on five jointly trained RoboTwin tasks: Hang Mug, Handover Block, Stack Bowls Three, Place Can Basket, and Turn Switch. All variants use DreamZero initialization without additional mid-training (Appendix J, Table 14).
The control retains the full model's Flow tokens, input/output modules, and joint denoising, but sets the Flow loss weight to zero. Its performance remains close to the RGB–action baseline. Enabling Flow supervision increases mean Clean success from 46.0% to 65.0% and Randomized success from 37.8% to 59.6%, supporting the contribution of explicit 3D supervision beyond the added tokens and modules alone.
Joint denoising versus fixed reconstruction features.
We also evaluate a reconstruction-based branch that predicts Flow tokens once and caches its output as keys and values for the policy DiT. Both designs provide 3D information during inference, but reconstruction supplies fixed features while joint denoising updates Flow alongside video and action throughout sampling. Joint denoising increases mean Clean success from 60.6% to 65.0% and Randomized success from 56.2% to 59.6%.
Mean success over the five tasks (Appendix J, Table 14):
Together, these controls support explicit Flow supervision and iterative cross-modal updates over the tested fixed-conditioning design.
Flow supervision during real-world post-training.
On ALOHA, we start from the same Flow-aware mid-trained checkpoint and disable only the Flow loss during post-training, retaining the architecture, Flow tokens, and joint inference. Mean success is 65.0% without post-training Flow supervision and 78.8% with it (Appendix I, Table 12).
Final-stage task success from Table 12:
This comparison supports the benefit of continued Flow supervision during post-training, beyond the representation learned during mid-training. The stage-wise results also show improvements in later interactions: on Pens, success at both drawer bracing and closing increases from 70% to 90%.