acceptodds
Under review as a conference paper at ICLR 2027

3DFlowWAM: Bridging Video and Action through 3D Flow

15 views

Abstract

Robot actions are expressed in embodiment-specific coordinates, but their effects can be described through shared 3D geometry and motion. We introduce 3DFlowWAM, a world action model that uses 3D scene changes to connect video priors to robot control. Its central representation, 3D Flow, describes visible points through their 3D positions and displacements, preserving correspondence with video while remaining independent of robot action coordinates. By jointly predicting video, 3D Flow, and actions, the model learns to connect visual appearance with explicit spatial dynamics. We build on a pretrained video model through 3D Flow-aware mid-training, followed by task-specific post-training. We assess the contribution of 3D Flow through controlled comparisons on a common DreamZero backbone with matched video and action data. Experiments across simulation benchmarks and seven real-world tasks show that 3D Flow improves action learning both with and without mid-training, while mid-training further strengthens real-world adaptation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied AI, World Action Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EfzgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes 3DFlowWAM, a unified world action model that jointly generates future RGB video, a dense 3D Flow representation, and robot actions within a shared causal diffusion transformer. The central representation combines an image-aligned point map, which records the 3D position of each visible pixel, with world-centric scene flow, which records the 3D displacement of the same physical point over time. RGB, 3D Flow, and action tokens are integrated into temporally aligned prediction blocks and jointly denoised with modality-specific flow-matching losses. Training starts from a pretrained video model, continues with 3D Flow-aware mid-training on large-scale robot-video data, and then adapts to target embodiments and tasks. At inference, 3DFlowWAM uses RGB observations, the language instruction, and optional proprioceptive state to predict the next video, 3D Flow, and action chunk; it executes the action and updates its visual context in a closed loop without requiring an online depth sensor or 3D tracker. The paper validates the design on LIBERO, LIBERO-Plus, RoboTwin 2.0, and seven real-world tasks on ALOHA-style and Franka platforms.

Critical strengths

Point maps provide scene state while scene flow provides temporal 3D displacement; keeping both on the image grid preserves correspondence with RGB observations and avoids the ordering and alignment difficulties of unstructured point clouds. The shared causal diffusion transformer then allows action tokens to attend directly to both visual and geometric motion tokens, so the paper proposes an integrated world model rather than a pipeline of separately trained modules. The design also retains an understandable physical interpretation while remaining independent of a specific robot action coordinate system. Its deployment path is practical because the model predicts 3D Flow internally and needs only RGB observations and robot state at test time.

Critical weaknesses
  1. The experiments do not fully isolate joint generation from auxiliary supervision and capacity. The comparison with and without a 3D Flow stream shows that the complete 3DFlowWAM system is effective, but it does not distinguish joint generation from a training-only 3D Flow reconstruction branch, additional model and token capacity, or an additional auxiliary objective. The representation study is informative but is limited to the Hang Mug task and changes both the represented quantity and its VAE/token budget. Additional matched controls would substantially strengthen attribution to the joint design itself.

  2. A few reporting and evaluation details require clarification or correction. In Table 7, the displayed RoboTwin results for X-VLA under the Clean condition average to 72.88%, while the table reports a 50-task mean of 72.80%; either one cell or the reported mean appears to require correction. The manuscript also does not clearly state how sequence-level 3D normalization statistics are obtained at inference, how 3D Flow tokens are aligned with three RGB views when only the overhead view has flow supervision, or how invalid tracked pixels are excluded from the losses. These issues are correctable and do not undermine the overall empirical case, but resolving them would improve reproducibility and confidence.

Questions for authors
  1. Clarification of normalization and test-time information. How are the sequence-level center and scale for point maps and scene flow computed? Are the same statistics available at deployment without using future ground-truth geometry, and are they used only for decoding or also for action prediction?
  2. Clarification of multimodal token alignment and masks. Please specify how a single overhead-view 3D Flow stream is aligned with multiple RGB camera views at different spatial resolutions. Please also state whether invalid tracked pixels are explicitly masked from the Flow VAE loss and flow-matching objective.
  3. Clarification of code, datasets, and processed 3D Flow annotations. Do the authors plan to release the model code and any datasets or processed 3D Flow annotations used for mid-training and task adaptation? If so, please specify whether the release will include the Track4World and SAM 3 preprocessing pipeline, the confidence and foreground masks, and the normalization statistics needed to reproduce training and inference.
5 comments
AuthorsOpenReview ↗

Rebuttal for reviewer Efzg (1/4)

Thank you for the constructive feedback. We have added controls for Flow supervision, cross-modal interaction, and token budget, expanded the representation study, and clarified the implementation details below.

1. Flow supervision, additional capacity, and joint generation

Flow supervision versus additional capacity.
We added a capacity-matched control on five jointly trained RoboTwin tasks: Hang Mug, Handover Block, Stack Bowls Three, Place Can Basket, and Turn Switch. All variants use DreamZero initialization without additional mid-training (Appendix J, Table 14).

The control retains the full model's Flow tokens, input/output modules, and joint denoising, but sets the Flow loss weight to zero. Its performance remains close to the RGB–action baseline. Enabling Flow supervision increases mean Clean success from 46.0% to 65.0% and Randomized success from 37.8% to 59.6%, supporting the contribution of explicit 3D supervision beyond the added tokens and modules alone.

Joint denoising versus fixed reconstruction features.
We also evaluate a reconstruction-based branch that predicts Flow tokens once and caches its output as keys and values for the policy DiT. Both designs provide 3D information during inference, but reconstruction supplies fixed features while joint denoising updates Flow alongside video and action throughout sampling. Joint denoising increases mean Clean success from 60.6% to 65.0% and Randomized success from 56.2% to 59.6%.

Mean success over the five tasks (Appendix J, Table 14):

Model / controlClean (%)Randomized (%)
RGB + action45.236.8
3DFlowWAM w/o Flow loss46.037.8
Reconstruction branch with fixed KV60.656.2
3DFlowWAM with joint denoising65.059.6

Together, these controls support explicit Flow supervision and iterative cross-modal updates over the tested fixed-conditioning design.

Flow supervision during real-world post-training.
On ALOHA, we start from the same Flow-aware mid-trained checkpoint and disable only the Flow loss during post-training, retaining the architecture, Flow tokens, and joint inference. Mean success is 65.0% without post-training Flow supervision and 78.8% with it (Appendix I, Table 12).

Final-stage task success from Table 12:

Post-training settingChilies (%)Cabbage (%)Towel (%)Pens (%)Mean (%)
Without Flow loss7565507065.0
With Flow loss8060859078.8

This comparison supports the benefit of continued Flow supervision during post-training, beyond the representation learned during mid-training. The stage-wise results also show improvements in later interactions: on Pens, success at both drawer bracing and closing increases from 70% to 90%.

AuthorsOpenReview ↗

Rebuttal for reviewer Efzg (2/4)

2. Scope of the representation study and the effects of VAE choice and token budget

We extended the original Hang Mug study to the five jointly trained RoboTwin tasks above and added Sparse 3D Flow (Appendix H, Table 10). All variants retain RGB and action prediction and use DreamZero initialization without additional mid-training.

Mean success over the five tasks:

Prediction targetsClean (%)Randomized (%)
RGB + action45.236.8
+ 2D Flow54.247.6
+ Depth56.850.2
+ Sparse 3D Flow52.845.8
+ Point Maps57.452.8
+ Scene Flow52.447.4
+ Full 3D Flow65.059.6

Full 3D Flow achieves the highest success on all five tasks under both conditions, extending the evidence for complementary position and displacement supervision beyond Hang Mug.

We also compare representations on four ALOHA tasks using the same mid-trained initialization and matched post-training settings (Appendix H, Table 11):

SupervisionMean ALOHA success (%)
SAM3 + 2D Flow + Depth68.8
SAM3 + Sparse 3D Flow68.8
SAM3 + Object Flow76.3
SAM3 + 3DFlowWAM78.8

Encoder choice.
We use modality-appropriate encoders: Wan VAE for Depth and 2D Flow, and the corresponding MotionCrafter VAE branches for Point Maps and Scene Flow. Full 3D Flow jointly encodes both fields. Sparse 3D Flow randomly samples 10 points per frame within the SAM 3 interaction region and supplies their displacements to the Scene Flow branch.

Token budget.
We examine token count separately by varying the Flow stream over 32, 64, 128, and 256 tokens per block, while keeping video and action tokens fixed at 512 and 48, respectively (Appendix K, Table 15; Appendix N, Table 16).

Flow tokens per blockMean Clean success (%)Mean Randomized success (%)
3265.059.6
6465.860.6
12865.060.8
25666.059.4

Mean success ranges from 65.0% to 66.0% under Clean evaluation and 59.4% to 60.8% under Randomized evaluation, without monotonic improvement as token count increases. These results support the effectiveness of a compact Flow representation across the tested budgets.

AuthorsOpenReview ↗

Rebuttal for reviewer Efzg (3/4)

3. Normalization and test-time information

We compute normalization statistics offline, following MotionCrafter. Reconstructed points are first transformed into a common coordinate frame referenced to the first camera pose. Let D\mathcal{D} contain the valid point-map entries across the processed sequence. The center and scale are

\qquad s=\frac{1}{|\mathcal{D}|}\sum_{i\in\mathcal{D}}\|P_i-\mu\|_2.$$ Thus, the center is the mean 3D position, and the scale is the mean distance from that center. Point Maps and Scene Flow share this scale: $$\bar P=(P-\mu)/s, \qquad \bar F=F/s.$$ Displacements are scaled but not centered. We store these statistics with the targets and recover geometry for visualization using $P=s\bar P+\mu$ and $F=s\bar F$. The computation and inverse transformations are provided in **Appendix N, “Sequence-level center and scale.”** **Action inference requires neither these sequence statistics nor future ground-truth geometry.** The policy jointly predicts video, Flow, and action latents and directly outputs an action chunk, without decoding or denormalizing Flow. Track4World is therefore not required online. ### 4. Multi-view alignment and treatment of unreliable tracking **Alignment across camera views.** We construct Flow targets only for the head camera while retaining RGB inputs from all three cameras. Wrist cameras undergo substantial motion and frequent occlusion, making their Flow estimates less reliable and limiting the additional scene-motion information they provide. The three RGB views are spatially concatenated into a 2 × 2 grid, with one padded cell, before video encoding (Appendix N). Flow targets associate each head-camera pixel $(u,v)$ with its 3D position and displacement. This is correspondence at the target level, not one-to-one alignment between differently sized latent grids or between Flow and all three RGB views. Cross-view information is integrated through shared attention within each temporal block. **Masks and losses.** Tracking-confidence filtering and SAM 3 foreground masks are applied during offline target preprocessing. With a fixed camera, static-background Scene Flow should be near zero, but geometry and pose-estimation errors can introduce spurious displacement. We therefore set background Scene Flow to zero before encoding. MotionCrafter VAE remains frozen during policy training, so no additional VAE reconstruction loss is optimized. The latent flow-matching objective uses the encoded, filtered targets; it does **not** apply a separate per-pixel validity mask to the latent loss. Masks are used to prepare the supervision targets, rather than to exclude individual pixels directly from that loss.
Reviewer_fjdpOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents 3DFlowWAM, a WAM policy trained with supervision from 3D flow, video, and action. The training pipeline consists of three stages: (1) initialization from Wan, (2) mid-training on MolmoAct2, RoboCoin, and RoboMIND 2.0, and (3) downstream fine-tuning. Experimental results show only marginal improvements on LIBERO and RoboTwin 2.0, along with evaluations on seven real-world task steps.

Critical strengths

The evaluation seems thorough.

Critical weaknesses

I have seen many “WAM model” papers this year, each claiming superior performance on some simple tasks. However, their data recipes, architectures, and loss functions all differ, making it difficult to draw meaningful conclusions. Unfortunately, this paper follows the same trend. To make the work more scientifically meaningful, I would encourage the authors to focus on a controlled study of the roles of video and 3D flow prediction. Figure 6 seems promising in this regard, but the experimental setup is too vague for me to understand what is being evaluated.

In summary:

  • Engineering: The improvement in robotic capabilities appears limited.
  • Scientific contribution: The paper lacks clarity, with many entangled factors making it difficult to identify which components are actually meaningful.
Questions for authors

What is the takeaway of this paper? How can future work benefit from your discovery? and what supports this? (think about Figure 6, this is the only figure I find meaningful in this paper)

4 comments
AuthorsOpenReview ↗

Rebuttal for reviewer fjdp (1/3)

Thank you for the constructive feedback. We agree that differences in training recipes, including data sources, optimization steps, and training budgets, make it difficult to attribute gains in comparisons across WAMs. We therefore adopt the DreamZero architecture and training formulation and construct four controlled settings: DreamZero, 3DFlowWAM without mid-training, DreamZero*, and full 3DFlowWAM (Figure 5).

Corresponding comparisons use the same target demonstrations, adaptation steps, batch size, and optimization settings. The two mid-trained models also use the same approximately 1,300 hours of video/action data, differing in whether they include the 3D Flow stream and its supervision. Our main conclusions thus come from controlled comparisons within this framework, rather than cross-model rankings. Below, we clarify the experimental protocols and summarize four takeaways.

1. 3D Flow improves manipulation, and mid-training increases its benefit.

Without additional mid-training, adding 3D Flow improves ALOHA/Franka success from 30.0%/26.7% to 47.5%/35.6%. After mid-training on the same video/action data, DreamZero* reaches 57.5%/60.0%, while 3DFlowWAM reaches 78.8%/82.2% (Table 3). The gains over the corresponding video–action baseline therefore increase from 17.5%/8.9% to 21.3%/22.2%.

We also examine whether Flow supervision remains useful during post-training. Starting from the Flow-aware mid-trained checkpoint, disabling only the Flow loss during ALOHA post-training yields 65.0% success, compared with 78.8% when supervision is retained (Appendix I, Table 12). Both variants retain the same architecture, Flow tokens, and joint inference.

These comparisons support using 3D Flow supervision during both mid-training and target adaptation: mid-training strengthens its benefit, while continued supervision improves downstream performance.

2. 3D position and displacement provide complementary supervision.

Figure 6 evaluates how different prediction targets affect manipulation success, rather than geometric reconstruction accuracy. All variants retain RGB and action prediction and additionally predict Depth, 2D Flow, Point Maps, Scene Flow, or full 3D Flow. We use modality-appropriate encoders: Wan VAE for Depth and 2D Flow, and the corresponding MotionCrafter VAE branches for separate or joint encoding of Point Maps and Scene Flow.

The original experiment trains on Hang Mug alone. This task requires precise 6DoF alignment between the mug handle and the peg, making it useful for distinguishing spatial representations. Point Maps and Scene Flow achieve 52%/37% and 45%/21% Clean/Randomized success, respectively, while their combination reaches 60%/47%. Scene Flow alone also performs below 2D Flow under randomization (21% vs. 29%), suggesting that displacement without its associated 3D location provides insufficient spatial information.

We extend this study to five jointly trained RoboTwin tasks: Hang Mug, Handover Block, Stack Bowls Three, Place Can Basket, and Turn Switch (Appendix H, Table 10). These challenging tasks cover complementary demands: precise alignment, bimanual transfer, sequential stacking, placement into a receptacle, and contact-sensitive switch operation. Both the original single-task study and the new five-task study use DreamZero initialization without additional mid-training, allowing us to compare representations under a common adaptation setting. Their training scopes are reported separately in Appendix H.

The five-task mean success rates are:

Prediction targetsClean (%)Randomized (%)
RGB + action45.236.8
+ 2D Flow54.247.6
+ Depth56.850.2
+ Sparse 3D Flow52.845.8
+ Point Maps57.452.8
+ Scene Flow52.447.4
+ Full 3D Flow65.059.6

Full 3D Flow achieves the highest success on all five tasks under both conditions. The design takeaway is to model displacement together with the 3D position where it occurs: position specifies where the interaction happens, while displacement describes how the scene changes.

AuthorsOpenReview ↗

Rebuttal for reviewer fjdp (2/3)

3. The benefits may involve shared action-effect supervision and improved attention to interaction regions.

A. A shared prediction target for action effects.
3D Flow describes scene geometry and motion independently of a robot’s joint dimensions or control interface. It provides a shared supervision target without requiring action labels from different robots to be converted into the same action space. Because it captures spatial changes produced by an interaction, jointly predicting actions and 3D Flow encourages the model to learn control commands together with their expected effects.

We also examine generalization to changes in object placement and appearance (Figures 9–10). After adaptation on the same 40 IND demonstrations per task, 3DFlowWAM improves Pens/Towel OOD success from DreamZero*’s 50%/30% to 70%/40%, while IND success remains unchanged. These results support generalization under the tested distribution shifts.

B. Attention to interaction-relevant visual information.
Under the same observations and heat-map scales, the displayed action-to-video attention maps concentrate more strongly around gripper–object interactions when Flow is included (Figure 8). Appendix L (Figures 12–14) provides ten additional paired examples covering fruit transfer, fruit/vegetable sorting, and shoe manipulation. Within each pair, the layer, attention head, and heat-map scale are matched, with reference Scene Flow overlaid to identify moving regions.

These qualitative observations suggest that explicit position and displacement supervision may help action prediction use interaction-relevant visual information.

4. Scaling 3D Flow supervision requires attention to target quality.

We vary only SAM 3 foreground filtering during ALOHA adaptation, holding the mid-trained initialization, demonstrations, model, optimization settings, tracking outputs, and confidence filtering fixed. Filtering increases mean success from 52.5% to 78.8%, a gain of 26.3% (Figure 11, Appendix B.4).

With a fixed camera, static-background Scene Flow should be close to zero. Geometry and pose-estimation errors can nevertheless introduce spurious background displacement. We suppress this noise by setting background Scene Flow to zero before encoding, while retaining the full RGB observations.

The practical takeaway is that automatically generated 3D Flow targets require quality control as well as scale, particularly when tracking errors introduce motion unrelated to the interaction.

In summary, our controlled study provides guidance on when to use 3D Flow supervision, which spatial quantities to predict, how it may support action learning, and how to prepare reliable targets. We will emphasize these findings in the revision so that the contribution is clear beyond aggregate performance improvements.

AuthorsOpenReview ↗

Rebuttal for reviewer fjdp (3/3)

Our central motivation is that video prediction and action generation would benefit from an explicit intermediate representation of 3D interactions. Some VLA use language-based intermediate reasoning to guide action prediction. For WAMs which don't have such a modality, we investigate a complementary representation that explicitly describes where objects are and how they move in 3D, acting as a guidance for action generation.

To test this hypothesis, we conduct controlled comparisons within the DreamZero architecture and training paradigm (Fig. 5), matching data and training settings to examine the effects of Flow supervision and mid-training. Fig. 6 (Simulation tasks success rate) and our additional representation studies support Point Maps+Scene Flow as an effective combination of accessible supervision. We further scale joint training to 1,294 hours of robot data, improving ALOHA/Franka success rates from 57.5%/60.0% to 78.8%/82.2% against the matched video–action baseline.

Our method is deliberately simple: we adopt a straightforward extension of DreamZero: adding Flow tokens while keeping the original training paradigm, without additional architectural tricks. This simple design allows us to directly evaluate the role of 3D Flow and provide clear, controlled evidence for the four findings discussed above. More broadly, both VLA and WAM policies must translate 2D visual observations into actions in 3D space. We therefore view the study of explicit 3D Flow representations as an important step toward enabling future robotic systems to benefit from 3D grounding models.

Thank you again for your feedback. In our rebuttal, we clarified the paper’s main contribution and the practical takeaways from our controlled experiments. We will highlight these findings more prominently in the revised manuscript. We look forward to hearing your thoughts. If any concerns remain, please let us know, and we will do our best to address them during the remaining discussion period.

Reviewer_BPY8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The submission proposes 3DFlowWAM, a world action model that jointly generates future rgb video, a dense “3D Flow” signal, and robot actions. 3D flow pairs image-aligned point maps (where each visible point is in 3d) with world-frame scene flow (how it moves). Targets are built offline: from privileged simulator state on libero and robotwin, and from track4world tracking filtered by sam 3 foreground masks on real data. They are encoded with motioncrafter’s frozen 4d VAE and denoised together with video and action tokens in a causal diffusion transformer initialized from dream=Zero (WAN video model backbone), with ground-truth rgb history as context.

Training has two stages: 3d flow-aware mid-training on about 1.3k hours of robot data (about 220k trajectories, 13 embodiments), then task-specific post-training. The authors compare four setups on the same dreamzero backbone (with and without 3d flow, with and without mid-training). They report 88% on libero-plus and 91% / 90% on robotwin 2.0 (clean / randomized). On real robots they report 79% / 82% success on aloha / franka, against 58% / 60% for the video-action baseline, with about 2% added inference latency. Analyses cover the choice of 3d representation (hang mug), sam 3 filtering, attention maps, and a small out-of-distribution test.

Critical strengths
  1. Controlled comparisons on one backbone with matched data are the right design, and uncommon in this space.

DreamZero* and 3DFlowWAM share the same roughly 1,300 hours of mid-training and the same post-training data, and differ only in whether 3D Flow is predicted. Per the supplementary code, the added branch is a patch embedding and an output head, so the comparision is close to parameter-matched. On this footing the mid-trained real-robot gains are large and survive a significance check: about 58 to 79 on ALOHA (63/80 vs 46/80) and 60 to 82 on Franka (74/90 vs 54/90). The gains are largest on placement-heavy tasks (pens 60 to 90, towel 50 to 85, stacking 47 to 73), which fits the paper's argument.

  1. Cheap at deployment.

Flow is training-time supervision only, and test time needs RGB alone, with no tracker, depth or segmentation. Latency overhead is about 2–5% (table 4). The Hang Mug result is a sensible finding. Point maps (52 / 37, clean / randomized success %) and scene flow (45 / 21) each beat RGB only (31 / 18), and the combination is best (60 / 47). Scene flow alone falls below 2D flow under randomization (21 vs 29), suggesting motion helps only when anchored to 3D positon.

Critical weaknesses
  1. Novelty and positioning relative to prior work.

-- The 3D Flow representation appears to match MotionCrafter's point map plus scene flow [1], including its VAE and normalization. 3D flow is also an established intermediate representation in manipulation. ManiTrend [2] jointly predicts 3D flow, future frames and actions. General Flow [3], 3DFlowAction [4], 3D Flow Diffusion Policy [5], NovaFlow [6] and Im2Flow2Act [7] (in 2D) address closely related ideas but are not cited. The 3D tracking work the method builds on (TAPIP3D [8], DELTA, CoTracker3 [9], D4RT) receives little discussion.

-- In the reading of the reviewer, the contribution is integrating dense 3D flow into a video WAM and training it at scale, not a new representation as contribution 1 states.

-- The motivation in paragraphs 1 and 2 of the introduction is stated without evidence or citations. The "embodiment-independent" claim is not directly tested: flow is also predicted on the robot arm, the action heads are per-embodiment in the released code, and no cross-embodiment transfer is reported.

  1. Attribution of the gains to 3D geometry.

-- Without SAM 3 filtering, ALOHA success is 52.5, below DreamZero* (57.5). This leaves open whether the flow targets help through 3D geometry or through object-centric masking. A matched mask-only control would help distinguish the two. Sec. 3.1 and Sec. 5.4 also describe the filter differently (loss masking vs. zeroing background flow).

-- Flow in mid-training is not evaluated separately from flow in post-training. Without mid-training, gains are smaller and less consistent: the Franka difference (32/90 vs 24/90) is not statistically significant, and on LIBERO-Plus the model is below DreamZero on camera and noise perturbations. With mid-training, the overall LIBERO-Plus lead comes mainly from the Camera dimension (93.7 vs 82.0 next best), while the model trails the best reported results by 8 to 15 points on Robot, Language and Background.

-- On RoboTwin, DreamZero* (74.3) scores below Motus (77.6) and FlowWAM (82.4) without their pretraining. Most simulation baselines are taken from prior work, not rerun. The closest flow and 3D WAMs are not evaluated on the real robots, and X-WAM remains ahead on randomized RoboTwin.

-- Results come from single runs without error bars. The OOD evaluation uses 10 trials per task, and the test placement range overlaps the training range.


On exposition: None of the minor points below would change the assessment on their own, but together they make the paper hard to follow.

  • Key terms are introduced without definition or context. In the abstract, "connect video priors" and "pretrained video model" are ambiguous (generation model or encoder?), "3D Flow" is presented as new, and post-training first appears in passing at line 79.
  • Fig. 6 and Table 7 report different Hang Mug numbers. The starting checkpoint is Wan in line 79 but DreamZero in Sec. 4.2. Baselines are labeled "cited evaluations" although DreamZero* is trained by the authors. The "mid-trained" grouping in Table 3 does not distinguish models trained on the authors' corpus from those using other pretraining data, such as pi0.5.
  • The figures do not convey the training setup. Fig. 2 omits the mid-training and post-training stages that Sec. 4.2 relies on, and in Fig. 5 it is unclear which stage the token boxes refer to. Figs. 8 and 9 are hard to read without the appendix (for example, the red and blue regions in Fig. 9). The experiment figures also show flow mostly on the manipulator, not on the objects.
  • The paper compares against latent-action models (UniVLA, Motus) but does not discuss why an explicit 3D target should be preferred over such implicit alternatives, including video latents with inverse dynamics.
  • Missing: model size, compute, loss weights, flow resolution, history window, tracking cost, how SAM 3 prompts were chosen and how pi0.5 was fine-tuned.
  • The released code differs from the paper in places. It includes an option, not described in the paper, to condition on first-step flow; please clarify whether it is used in any reported evaluation and, if so, how first-step flow is obtained at test time. The code uses per-modality noise levels where Alg. 1 uses one, and its flow positional encoding is not aligned to the video grid, unlike what Sec. 3.2 describes. Actual flow and video token counts per block would make the Sec. 5.3 efficiency claim checkable.
  • Real-world tasks have 4 stages with binary scoring and no per-stage results. Fig. 7's lower flow loss after flow mid-training adds limited evidence, since that model was already trained on the same objective, and Fig. 8 rests on three examples.
  • Small errors: "Close-Loop Control" and other Fig. 1 labels, the Table 1 caption, "midtraining" vs. "mid-training" used interchangeably, and X-VLA's clean mean in Table 7 (recomputes to 72.88, reported as 72.80).

References

[1] MotionCrafter: Dense Geometry and Motion Reconstruction with a 4D VAE. CVPR 2026. arXiv:2602.08961.

[2] ManiTrend: Bridging Future Generation and Action Prediction with 3D Flow for Robotic Manipulation. arXiv:2502.10028, 2025.

[3] General Flow as Foundation Affordance for Scalable Robot Learning. CoRL 2024. arXiv:2401.11439.

[4] 3DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Model. arXiv:2506.06199, 2025.

[5] 3D Flow Diffusion Policy: Visuomotor Policy Learning via Generating Flow in 3D Space. arXiv:2509.18676, 2025.

[6] NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videos. arXiv:2510.08568, 2025.

[7] Flow as the Cross-Domain Manipulation Interface (Im2Flow2Act). CoRL 2024. arXiv:2407.15208.

[8] TAPIP3D: Tracking Any Point in Persistent 3D Geometry. NeurIPS 2025. arXiv:2504.14717.

[9] CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos. ICCV 2025. arXiv:2410.11831.

Questions for authors

Questions for the authors

  1. Fig. 6 compares representations on a single simulated task (Hang Mug), and its numbers differ from Table 7. ManiTrend [1] already predicts sparse 3D flow jointly with future frames and actions, and LaMP encodes motion as image-plane plus depth displacement. On at least one real platform, with the same backbone, how does dense 3D Flow compare with these two forms: sparse tracked 3D points and (u, v, depth) flow? A clear margin would show that the dense, world-frame form is itself a contribution, beyond integrating MotionCrafter latents into DreamZero.

  2. Without SAM 3 filtering, ALOHA success drops to 52.5, below DreamZero* at 57.5 (Fig. 11). This suggests the flow targets may be helping partly through object segmentation and through the arm's own motion. Could the authors report three variants on the four ALOHA tasks: (a) DreamZero* with the same SAM 3 masks as an auxiliary target and no 3D information; (b) 2D flow under the same filter; (c) 3D Flow with robot-arm pixels excluded from supervision. Variant (c) also bears on the embodiment-independence claim in line 59. Please also confirm whether the filter masks the loss (Sec. 3.1) or zeroes background flow (Sec. 5.4), and how the SAM 3 prompts are specified for each task.

  3. The four setups in Fig. 5 leave one combination untested: flow-aware mid-training followed by video-action post-training only. Results for this setting on ALOHA and Franka would show whether flow in mid-training helps by itself, or only when flow supervision is also present during adaptation.

  4. On RoboTwin, DreamZero* scores 74.3, below Motus (77.6) and FlowWAM (82.4) without their pretraining, and most simulation rows are taken from prior work. Could the authors provide:

    • DreamZero's post-training budget and hyperparameters;
    • one of the closest flow or 3D WAMs (X-WAM, FlowWAM or Motus) on at least one real platform, with matched demonstrations;
    • three seeds for Table 3 and for the per-dimension LIBERO-Plus results? Most of the LIBERO-Plus margin comes from the Camera dimension (93.7 vs. 82.0). A short explanation of why flow-aware mid-training helps under camera shifts in particular, and not under Robot, Language or Background shifts, would also be useful.

Clarifications

  • Is use_first_motion_as_condition enabled in any reported result? If so, how is first-step flow obtained at test time?
  • What are the flow and video token counts per block, and does the Table 4 timing include flow generation?
  • Which checkpoint initializes training (Wan, line 79, or DreamZero, Sec. 4.2)? What protocol produced the Fig. 6 numbers, and what history window is used at inference?
9 comments
AuthorsOpenReview ↗

Rebuttal for reviewer BPY8 (1/8)

We thank the reviewer for recognizing our controlled comparisons, real-world improvements, and low deployment overhead. We have reformulated our contribution and added experiments to clarify the roles of representation, supervision, and training stage. We address overlapping weaknesses and questions together below.

AuthorsOpenReview ↗

Rebuttal for reviewer BPY8 (2/8)

W1.1 & Q1: Relation to prior Flow methods and representation comparisons

A. Positioning and related work.

We have reformulated our positioning and expanded the related-work discussion. General Flow, Im2Flow2Act, and 3D Flow Diffusion Policy use Flow for policy learning and control; LaMP introduces motion priors into a VLA policy. 3DFlowAction and NovaFlow instead use predicted object motion to guide optimization-based control. Our focus is dense 3D supervision within a pretrained video-generation WAM, including its representation choices and benefits from large-scale mid-training.

The closest work, ManiTrend, jointly predicts sampled-point trajectories, future images, and actions using a GR-1-initialized causal Transformer. It also studies cross-embodiment pretraining. Our model starts from Wan and jointly denoises dense Point Map–Scene Flow latents with video and action tokens under a shared flow-matching objective. We therefore emphasize the representation and controlled training study within a video-diffusion WAM, rather than claiming the first use of Flow or joint prediction.

B. Additional representation comparisons.

We extended the original Hang Mug study to five jointly trained RoboTwin tasks: Hang Mug, Handover Block, Stack Bowls Three, Place Can Basket, and Turn Switch. These tasks cover precise alignment, bimanual transfer, stacking, placement, and contact-sensitive manipulation. All variants retain RGB and action prediction and use the same DreamZero-based initialization without additional mid-training (Appendix H, Table 10).

We use modality-appropriate encoders: Wan VAE for Depth and 2D Flow, and the corresponding MotionCrafter VAE branches for Point Maps and Scene Flow. Inspired by point-based Flow representations, our Sparse 3D Flow control randomly samples 10 points per frame within the SAM 3 interaction region and supplies only their displacements to the Scene Flow branch.

The following table summarizes the five-task means. Scores are success rates (%); Δ denotes the difference from the first-row baseline, not a relative percentage change.

RepresentationCleanRandomizedΔ CleanΔ Randomized
RGB + action45.236.8——
+ 2D Flow54.247.6+9.0+10.8
+ Depth56.850.2+11.6+13.4
+ Sparse 3D Flow52.845.8+7.6+9.0
+ Point Maps57.452.8+12.2+16.0
+ Scene Flow52.447.4+7.2+10.6
+ 3D Flow65.059.6+19.8+22.8

Full 3D Flow performs best on all five tasks under both evaluation conditions. Combining position and displacement therefore remains beneficial beyond the original Hang Mug experiment.

We also added representation comparisons on four ALOHA tasks, using the same mid-trained initialization and matched post-training settings. Both 2D Flow + Depth and Sparse 3D Flow achieve 68.8%, compared with 78.8% for full 3DFlowWAM (Table 11, detailed under W2.1). These controls compare sparse motion and combined image-plane motion/depth supervision within our backbone.

C. Tracking and scalability.

We have expanded the discussion of TAPIP3D, DELTA, CoTracker3, and D4RT. We will also connect these advances to scalability in the Introduction: automatic tracking allows existing videos to provide geometric motion supervision without manual 3D annotation. Track4World is suitable for our pipeline because its dense world-centric outputs provide the Point Maps and Scene Flow required by MotionCrafter VAE. All tracking is performed offline.

W1.2: Novelty of the representation

We agree that Point Maps, Scene Flow, and MotionCrafter’s VAE are established components. Our contribution is their integration into joint video–Flow–action generation and a controlled study of representation, training stage, and supervision quality. We have reformulated our positioning accordingly and will make this distinction explicit in Contribution 1.

W1.3: Motivation and embodiment independence

We will strengthen the Introduction with references supporting its motivation. By “embodiment-independent,” we mean that the definition of the supervision does not depend on joint dimensions or control interfaces. Different robots can share supervision about scene geometry and the expected spatial effects of their actions without converting their action labels into the same action space.

This does not require shared action heads: the control interfaces remain embodiment-specific. Our experiments test the utility of this shared supervision through matched adaptation on ALOHA and Franka, rather than claim zero-shot cross-embodiment policy transfer. The Object Flow control additionally demonstrates useful object-motion supervision without arm-motion targets during post-training (Table 11).

AuthorsOpenReview ↗

Rebuttal for reviewer BPY8 (3/8)

W2.1 & Q2: Geometry, masking, and filtering implementation

Foreground filtering suppresses background pseudo-motion in the training targets. With a fixed camera, static-background Scene Flow should be near zero, but geometry and pose-estimation errors can produce spurious displacement. Track4World provides DINO+SAM2-based foreground/background separation; we use SAM 3 for target preprocessing.

Keeping initialization, data, model, optimization, tracking outputs, and confidence filtering fixed, SAM 3 filtering raises ALOHA success from 52.5% to 78.8% (Figure 11; Appendix B.4).

The additional controls share the same mid-trained initialization and adaptation settings. They distinguish the effects of mask-only, motion, and geometric supervision during post-training (Appendix H, Table 11).

Post-training supervisionALOHA success (%)Δ vs. mask only
SAM3 mask only55.0—
SAM3 + 2D Flow63.8+8.8
SAM3 + 2D Flow + Depth68.8+13.8
SAM3 + Sparse 3D Flow68.8+13.8
SAM3 + Object Flow76.3+21.3
SAM3 + 3DFlowWAM78.8+23.8

Filtered 2D Flow improves success from 55.0% to 63.8%, showing that motion supervision contributes beyond mask-only targets. Adding depth further increases success to 68.8%. Object Flow reaches 76.3% while excluding robot-arm and gripper motion from the post-training targets. Thus, the improvements are not explained by segmentation targets or arm-motion supervision alone.

Object Flow is slightly below the full model (76.3% vs. 78.8%). One possible explanation is an encoding distribution shift: MotionCrafter’s VAE was trained on scene-motion fields, whereas Object Flow artificially zeros motion in moving robot-arm regions. This may affect encoding quality; we present it as a possible explanation rather than an established cause.

Implementation clarification. We set background Scene Flow to zero before VAE encoding; we do not apply a pixelwise mask directly to the latent flow-matching loss. MotionCrafter VAE remains frozen during policy training. The exact training prompts are the manuplated object and the two largest robot arms (Appendix O). Object Flow uses the same object prompt but excludes arm and gripper regions; category-specific visualization prompts are documented separately.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Thanks for all reviewers' valuable feedback

We thank all reviewers for their careful reading and constructive feedback. We appreciate their recognition of (1) our controlled comparisons with matched data, strong real-world gains, and low deployment overhead (BPY8); (2) the thorough evaluation (fjdp); and (3) our image-aligned, physically interpretable 3D representation and unified video–3D Flow–action modeling (EFzg).

In the revised PDF, we clarify the four controlled training setups and update the main text with five-task representation ablations and X-WAM real-world comparisons. The appendix adds real-world representation studies, controls without Flow supervision, reconstruction-based and token-budget comparisons, and stage-wise evaluation, together with additional visualizations and implementation details. We will incorporate these clarifications and additional results into the revision.

Dear Area Chair,

3DFlowWAM studies whether an explicit representation of where objects are and how they move in 3D can help a video world action model generate robot actions. We study this question by adding 3D Flow tokens to DreamZero and retaining its joint flow-matching training procedure.

Main contributions and findings

  1. Under strictly controlled comparisons, adding 3D Flow improves manipulation performance. We compare four DreamZero variants with and without Flow supervision, both with and without robot mid-training. Within each pair, we match the video and action data, downstream demonstrations, and training schedule. Flow improves performance in both settings. After mid-training, success rises from 57.5% to 78.8% on ALOHA and from 60.0% to 82.2% on Franka (Fig. 5, Table 3). In attention maps computed for identical observations on a common scale, action tokens attend more strongly to the interaction region with Flow supervision. This suggests a possible mechanism for the observed gains (Fig. 8).

  2. 3D position and motion provide complementary supervision. Jointly predicting Point Maps and Scene Flow outperforms predicting either alone on all five RoboTwin tasks under both Clean and Randomized conditions (Fig. 6, Appendix H).

  3. Scaling 3D Flow supervision through mid-training improves generalization. With 1,294 hours of robot mid-training, the advantage of Flow over its matched video–action baseline on LIBERO-Plus grows from 4.2% to 10.3%, with gains in six of seven perturbation categories. On the real-world Pens and Towel tasks, the two mid-trained models match in-distribution, while Flow improves success under changes in object placement and appearance (Table 8, Fig. 10).

  4. The quality of estimated Flow targets matters. With initialization, demonstrations, and tracker-confidence filtering held fixed, setting estimated motion in the static background to zero raises ALOHA success from 52.5% to 78.8%. This identifies a practical preprocessing step for training on fixed-camera robot data (Fig. 11).

Experiments and revisions added during rebuttal

  1. Clarified the training, inference, and figure configurations. We specified the training stages and token streams shown in Figs. 2 and 5. In Figure 5, we strictly control the variable. We also explained why the Hang Mug results differ between Fig. 6, which uses single-task training, and Table 7, which uses multi-task training. The revision clarifies checkpoint initialization, normalization, multi-view alignment, foreground filtering, and inference timing.

  2. Tested the contribution of Flow supervision during downstream adaptation. Starting from the same Flow-aware mid-trained checkpoint, we varied only the post-training Flow loss. Retaining this supervision raises ALOHA success from 65.0% to 78.8%. We also report success at each stage of the real-world tasks (Appendix I).

  3. Expanded the representation study to five RoboTwin tasks and four real-world ALOHA tasks. On ALOHA, dense 3D Flow reaches 78.8%, compared with 68.8% for both sparse 3D Flow and 2D Flow plus depth. We also added mask-only and object-motion controls to examine the contributions of foreground segmentation and robot-arm motion (Appendix H).

  4. Isolated Flow supervision. With identical tokens, parameters, and inference, enabling the Flow loss raises mean RoboTwin success from 46.0% to 65.0% under Clean conditions and from 37.8% to 59.6% under Randomized conditions (Appendix J).

  5. Tested joint denoising and the effect of the Flow token budget. Updating Flow jointly with video and action improves Clean/Randomized success from 60.6%/56.2% with fixed reconstruction features to 65.0%/59.6%. Separately, varying the budget from 32 to 256 Flow tokens yields no monotonic improvement (Appendices J–K).

We thank the reviewers for their constructive suggestions, which motivated these additional experiments. We summarize our key contributions and revisions above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.