acceptodds
Under review as a conference paper at ICLR 2027

FlowWAM: Optical Flow as a Unified Action Representation for World Action Models

30 views

Abstract

World Action Models (WAMs) leverage pretrained video generators for both world modeling and action prediction. However, directly leveraging such video generators for control raises a new challenge: how to represent actions in a suitable form that aligns with pretrained video generators while carrying enough motion cues for accurate control. Existing numerical actions fail to satisfy the former, and prior visual action representations overlook the temporal motion structure across frames. We address this issue with FlowWAM, a dual-stream diffusion framework that adopts optical flow as a unified, video-native action representation. Rather than attaching actions to the generator through a separate action space, FlowWAM natively internalizes them as a parallel video stream. Since flow videos share the same format as RGB videos and encode rich per-pixel displacement, FlowWAM jointly models both within a shared pretrained video generator, naturally implementing two modes of WAMs. In policy mode, FlowWAM generates flow for action prediction, while in world-model mode, it uses target flow sequences to guide future video generation. Moreover, since learning flow requires no action labels, FlowWAM can leverage large-scale action-unlabeled videos for pretraining. In policy mode, FlowWAM outperforms both VLA and WAM baselines on RoboTwin (93.7% average success), with action-unlabeled video pretraining notably boosting generalization to unseen randomized scenes, and surpasses dynamic-manipulation specialists on DOMINO (41.3% success). In world-model mode, its dense motion conditioning achieves the highest overall score on WorldArena, bringing an 18.4% relative improvement in trajectory accuracy. More results are available at https://anonymous-flowwam.github.io.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Model, Manipulation

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_mCvQOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors propose a world action model that uses optical flow as the unified action representation. They use common representation space for action and video prediction by encoding optical flow as RGB frames. Leveraging how optical flow can be extracted from videos without action labels, they pretrain the model on large-scale unlabeled videos. Their resulting model provides two operating modes, one as a world model and other as a robot action policy. Extensive evaluations are provided to establish strong performance of proposed model.

Critical strengths
  1. Simple change to WAMs that enables large-scale pretraining on unlabeled videos and dual operating modes using shared video generation backbone.
  2. Extensive evaluation across multiple benchmarks with comparison to strong baselines.
  3. Well documented real world experiments.
  4. Comprehensive ablation studies.
Critical weaknesses
  1. The unlabelled video pretraining actually utilizes supervision from additional structure preextracted from videos. The unlabelled pretraining claim could be more clearly stated in introduction.
  2. Inference latency is not clearly documented in main paper - consider moving that appendix table to main paper.
Questions for authors
  1. How important is the additional structure pre-extracted from videos for the unlabelled video pretraining?
  2. How would denoising step count of video generator affect performance? For distilled model, what would performance look for steps 1 and 2 compared to current 4?
  3. How important is RGB extraction for action policy mode? Could video generator predict only flow for that mode?
4 comments
AuthorsOpenReview ↗

Responses to Reviewer mCvQ (1/2)

We sincerely thank Reviewer mCvQ for the support and for highlighting our unified dual-mode formulation (Strength 1), extensive benchmark comparisons (Strength 2), real-world experiments (Strength 3), and comprehensive ablations (Strength 4). We agree with both presentation suggestions (W1 and W2) and have conducted the three ablation studies in Q1–Q3.

W1: Clarifying the Unlabeled Pretraining Description in the Introduction

In Stage 1, the model itself is trained purely on RGB and optical flow video streams without requiring teleoperated robot action labels. As you point out, this supervision relies on structure pre-extracted from the videos. The optical flow is estimated with RAFT, and our default EgoDex pretraining pipeline additionally uses 3D hand poses and embodiment retargeting to narrow the visual gap between human hands and robot arms. We have clarified this in the Introduction (Sec. 1) of the revised paper, which now states that the flow comes from an off-the-shelf estimator and that the videos are either raw or retargeted to robot arms. The retargeting pipeline is described in Sec. 3.3, and the two settings are compared in Appendix B.8.

W2: Documenting Inference Latency in the Main Paper

We agree and have moved Table 6 from Appendix A.7 into Sec. 4.5 of the revised paper, expanded with both latency and task success rate across configurations.

Q1: How Important is the Pre-Extracted Structure for Pretraining?

To quantify the contribution of embodiment retargeting during Stage-1 pretraining, we evaluated a variant pretrained purely on raw EgoDex human videos and raw RAFT optical flow (w/o retargeting, without any 3D hand tracking, inpainting, IK retargeting, or robot rendering):

Pretraining SettingIn-Dom CleanIn-Dom RandomIn-Dom Avg.OOD CleanOOD RandomOOD Avg.
w/o Pretraining87.60%87.00%87.30%86.60%38.06%62.33%
Raw EgoDex (w/o retargeting)91.84%91.42%91.63%87.82%45.96%66.89%
Retargeted EgoDex (Ours)93.74%93.68%93.71%89.24%53.04%71.14%
  • Raw human flow pretraining is effective even without retargeting: Pretraining purely on raw human videos and RAFT flow already improves in-domain success by +4.33 points and OOD Random success by +7.90 points. Adding embodiment retargeting brings a further +2.08 points in-domain (+7.08 points on OOD Random), which we attribute to narrowing the visual gap between human hands and robot arms. We have added this ablation to the appendix of the revised paper.
  • Importance of the pre-extracted flow itself: Table 4(a) of the main paper isolates this factor. Under the same backbone, retargeted EgoDex clips, and training schedule, pretraining on the retargeted RGB videos alone (iii) reaches 67.23% mean out-of-domain success, and adding the flow stream to pretraining (iv) raises it to 71.14%.
AuthorsOpenReview ↗

Responses to Reviewer mCvQ (2/2)

Q2: Effect of Reducing Video Denoising Step Count (from 4 Steps to 2 or 1 Steps)

We evaluated the distilled video generator across 4, 2, and 1 denoising steps across all 50 RoboTwin 2.0 tasks (measured on a single NVIDIA H100 GPU):

Video StepsTotal LatencyIn-Dom CleanIn-Dom RandomIn-Dom Avg.OOD Random
4 steps (Default)352 ms93.74%93.68%93.71%53.04%
2 steps265 ms91.88%91.16%91.52%49.86%
1 step223 ms89.94%88.38%89.16%47.72%
  • Latency vs. accuracy across step counts: Reducing from 4 steps (352 ms) to 2 steps (265 ms) cuts latency by 25% while retaining 91.52% in-domain success (2.19 points lower). Reducing further to 1 step (223 ms) lowers success to 89.16% (47.72% on OOD Random), so 2 to 4 steps offer a reasonable trade-off between speed and accuracy.

Q3: Importance of RGB Generation for Action Policy Mode (Could the Video Generator Predict Only Flow?)

We evaluated a Flow-Only variant on our backbone that denoises only future flow latents (without generating future RGB):

VariantIn-Dom CleanIn-Dom RandomIn-Dom Avg.OOD Random
Flow-Only (w/o future RGB)91.48%90.62%91.05%46.84%
FlowWAM (Joint RGB + Flow, Ours)93.74%93.68%93.71%53.04%
  1. Unified architecture for policy and world-model modes: Keeping both streams in the generator preserves architectural unification. If the generator only predicted flow, it could no longer synthesize future RGB videos as an action-conditioned world model (Sec. 4.3, WorldArena). Modeling RGB and flow jointly lets a single model serve both policy control and world modeling.
  2. Policy accuracy gain (+2.66 points In-Domain, +6.20 points OOD Random): While Flow-Only already achieves 91.05% in-domain success, our flow target represents robot-only motion and does not depict how objects react upon contact. Jointly generating future RGB raises in-domain success to 93.71% and OOD Random success to 53.04%, which we attribute to RGB co-generation enabling the shared DiT blocks to model object states and contact outcomes.

We hope these responses address your concerns, and we would be glad to clarify any further questions.

Thanks for the updates. Great work.

Thank you for your positive feedback and constructive suggestions throughout the review process. Your comments have been particularly helpful in further improving our paper.

Reviewer_KEbUOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3(+1)
Soundness
3
Read review
Paper summary

This paper proposes FlowWAM, which jointly predicts future RGB video and optical flow as two branches with a shared generation model, and then outputs robot control actions through an action expert. The model is first pretrained on EgoDex videos retargeted to robot embodiments, then jointly trained for video generation and action prediction using downstream robot demonstrations. In addition, the model can also generate future videos given optical flow, which is used for world model prediction.

Critical strengths
  1. The experiments are comprehensive and the results are competitive. The evaluation covers regular manipulation, dynamic manipulation, real robots, and world model prediction.
  2. The ablation studies are relatively thorough and provide validation of the role of optical flow. The RGB-only comparison shows that adding optical flow improves downstream performance; the optical flow replacement and temporal misalignment experiments further demonstrate that the policy indeed leverages the motion information in optical flow.
Critical weaknesses
  1. Unclear effect of flow prediction error. The existing perturbation experiments show that the model relies on optical flow and can tolerate a certain amount of noise, but authors do not directly evaluate the accuracy of the generated optical flow, nor do they use a comparison between GT optical flow and predicted optical flow to show how much of the current failure comes from optical flow prediction error.
  2. Unclear design choice for the "additional forward pass" of DiT. After the model generates the future latent, it runs DiT again. Why the future latent cannot be used directly, or why the hidden features from the last denoising step cannot be reused? There is a lack of corresponding ablation and latency comparison.
  3. Insufficient discussion of the choice of base model. Since optical flow is the core of the method, why not consider fine-tuning an existing optical flow prediction model such as FOFPred and then attaching an action predictor? Compared with this more direct route, the additional value of this paper's approach of jointly generating RGB and optical flow based on a general video model still lacks a convincing explanation and comparison.
Questions for authors
  1. How much benefit does the extra DiT forward pass bring? Please compare directly using the future latent, reusing the last denoising features, and the extra feature extraction scheme in this paper, and report success rate and end-to-end latency. How does the scheme that uses only optical flow and does not generate future RGB perform?
  2. Under fixed conditions, if the predicted optical flow is replaced with the ground-truth future optical flow, how much can the control success rate improve?
4 comments
AuthorsOpenReview ↗

Responses to Reviewer KEbU (1/2)

We sincerely thank Reviewer KEbU for recognizing our comprehensive evaluations across static, dynamic, real-world, and world-modeling benchmarks (Strength 1), and for noting our thorough flow ablations (Strength 2). Below, we address each of your concerns and questions.

W1 & Q2: Evaluation of Predicted Optical Flow Quality and Comparison with Ground-Truth Flow

1. Accuracy of the Generated Optical Flow (W1): To directly evaluate flow generation quality, we measured the pixel-wise end-point error (EPE) between FlowWAM's predicted flow and the demonstration flow across newly generated held-out demonstrations of all 50 RoboTwin 2.0 tasks. Averaged over all pixels, FlowWAM achieves an EPE of 0.58 px in-domain and 0.93 px out-of-domain. As a reference floor, encoding and decoding the demonstration flow through the frozen video VAE alone already incurs 0.47 px in-domain and 0.46 px out-of-domain.

2. Replacing Predicted Flow with Ground-Truth Flow Under Fixed Conditions (Q2): The recorded ground-truth flow is only valid along the demonstrated trajectory, since it no longer matches the robot's state once the executed actions deviate. Following your suggestion of fixed conditions, we therefore compare at the chunk level on held-out demonstrations of all 50 tasks, with identical observations and instructions and only the flow fed to the action expert replaced. This gives action error rather than a rollout success rate:

Flow Input to Action ExpertAction MSE, In-Dom. (mean / median) ↓\downarrowAction MSE, OOD (mean / median) ↓\downarrow
Predicted Flow (Ours)0.0116 / 0.001050.0799 / 0.0098
Ground-Truth Flow (Oracle)0.0071 / 0.000540.0470 / 0.0037

Replacing predicted flow with ground-truth flow lowers the mean action error by 39% in-domain and 41% out-of-domain, and the median error by 49% and 62%, achieving lower error on 50/50 tasks in-domain and 49/50 tasks out-of-domain. Flow prediction error therefore accounts for a substantial share of the chunk-level action error, while the remainder persists even given ground-truth flow. At the rollout level, the flow intervention study you noted (Appendix B.3, 10 tasks) points the same way: with the generated RGB unchanged, replacing the flow with that of another task or with zero motion lowers success from 94.2% to 1.2% and 0.5%, and shifting it by one latent frame lowers it to 36.4%. We have added both analyses to the appendix of the revised paper.

AuthorsOpenReview ↗

Responses to Reviewer KEbU (2/2)

W2 & Q1 (Part 1): Design Choice and Ablation for the "Additional Forward Pass" of DiT

1. Why we use the additional DiT forward pass:

  • Why not use future VAE latents directly? Generated VAE latents are a single-level representation in which the RGB and flow streams have not yet interacted with each other or with the language instruction, and they directly expose the residual error of few-step denoising. Passing them through the 30 DiT blocks yields multi-layer features in which RGB and flow interact via joint self-attention and attend to the instruction via cross-attention. Reading these layers gives the action expert a richer signal that is more stable against residual noise in the latents.
  • Why not reuse the hidden features from the last denoising step? During denoising, the last step's hidden states are computed on the pre-update noisy latent before the final denoising update completes (in our 4-step schedule, the input to the 4th step still contains substantial noise at t≈0.57t \approx 0.57). Running one forward pass at t=0t=0 ensures the action expert reads features from the fully denoised future latents.

2. Ablation Across the Three Schemes (4-Step Schedule, 50 RoboTwin Tasks):

Feature Extraction SchemeEnd-to-End LatencyIn-Dom CleanIn-Dom RandomIn-Dom Avg.OOD Random
(1) Directly using future VAE latents (expert retrained)293 ms87.96%86.54%87.25%45.36%
(2) Reusing last denoising step features293 ms92.62%92.74%92.68%50.88%
(3) Extra DiT pass at t=0t=0 (Ours)352 ms93.74%93.68%93.71%53.04%
  • Comparison across schemes: Bypassing DiT feature extraction (Scheme 1) lowers average success by 6.46 points in-domain and 7.68 points on OOD Random, while reusing the noisy pre-update features from the last denoising step (Scheme 2) lowers success by 1.03 points in-domain and 2.16 points on OOD Random. Scheme 2 is a viable lower-latency option (59 ms faster), and we keep the extra pass at t=0t=0 as the default because it yields the best accuracy under both protocols. In the revised paper, we have stated this extra pass in Sec. 3.2 and added the explanation and the full ablation to the appendix.

W3 & Q1 (Part 2): Discussion on FOFPred and Performance of Flow-Only Generation

1. Discussion on FOFPred and Choice of Base Model (W3): FOFPred is a valuable work showing that future optical flow prediction benefits both robot control and video generation, and we agree that predicting flow first and attaching an action head is a natural alternative route. Compared with FOFPred, building on a general video generation backbone (Wan2.2) to jointly model RGB and flow offers two advantages:

  • One video backbone for both modes vs. separate downstream modules: In FOFPred, each downstream use attaches a separate module (a diffusion policy network for control, or the existing flow-conditioned video model Go-with-the-Flow for video generation). In FlowWAM, RGB and flow are modeled jointly inside one video backbone, so the same backbone generates the flow plan in policy mode and serves as the action-conditioned world model (synthesizing future RGB videos conditioned on rendered robot flow, Sec. 4.3), without requiring an external video generator.
  • Native video priors and RGB–flow interaction: In FOFPred, the flow predictor is a dedicated model that combines a VLM with an image-editing diffusion transformer extended to full spatio-temporal attention. FlowWAM instead starts directly from a pretrained video generator, so the HSV flow stream inherits its spatiotemporal video priors and interacts with RGB scene dynamics across all shared DiT blocks. We have clarified the distinction from FOFPred in Related Work (Sec. 2) of the revised paper.

2. Controlled Comparison: Predicting Only Optical Flow vs. Joint RGB + Flow (W3 & Q1): While we do not fine-tune FOFPred itself, we evaluated a Flow-Only variant on our 5B backbone that predicts only future optical flow (without generating future RGB) and decodes actions from it. This addresses Q1 and gives a reference point for the "flow prediction + action predictor" route on the same backbone:

VariantIn-Dom CleanIn-Dom RandomIn-Dom Avg.OOD Random
Flow-Only (w/o future RGB)91.48%90.62%91.05%46.84%
FlowWAM (Joint RGB + Flow, Ours)93.74%93.68%93.71%53.04%

Jointly generating future RGB alongside flow preserves the unified world-model capability and further improves policy success by +2.66 points in-domain and +6.20 points on OOD Random, which we attribute to RGB co-generation enabling the shared DiT to model how manipulated objects react after contact.

We hope these responses address your concerns, and we would be glad to clarify any further questions.

Thank you to the authors for providing such detailed supplementary experiments and explanations. This indeed helped me better understand this work, and several designs also make more sense now. Optical flow is an important topic in WAMs, so although I think some of the designs still have room for improvement, I recommend an accept.

We sincerely thank the reviewer for their constructive feedback and thoughtful suggestions throughout the review process. Your comments have been valuable in helping us clarify our design choices and improve the overall quality of the paper. We appreciate the reviewer's time and support for our work.

Reviewer_rRLLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FlowWAM, a unified world action model that uses optical flow as a video-native action representation. By jointly modeling RGB and optical flow streams within a pretrained video diffusion model, FlowWAM supports both robotic action prediction and action-conditioned world modeling. The framework also enables pretraining on action-unlabeled videos. Extensive experiments on simulation benchmarks and real-world robotic tasks demonstrate promising performance.

Critical strengths

S1. The paper presents an intuitive and well-motivated approach to bridging the representation gap between robotic actions and pretrained video generation models.

S2. The unified framework supports both policy learning and world modeling, making the proposed approach versatile and practically relevant.

S3. The experimental evaluation is comprehensive, covering multiple benchmarks, real-world robotic tasks, and ablation studies. The reported improvements are encouraging.

Critical weaknesses

W1. Limited discussion of novelty relative to existing methods. The technical novelty could be better distinguished from existing optical-flow-based manipulation and world modeling methods, such as Motus and DynamicWAM. While the unified dual-stream design is interesting, it would be helpful to clarify which improvements specifically come from the proposed architecture rather than the use of optical flow itself.

W2. Limitations of robot-only optical flow. The method primarily relies on robot-only optical flow, which may not fully capture the motion of manipulated objects or complex interactions between the robot and its environment. Although the current results are promising, further discussion or experiments involving full-scene optical flow would help establish the generality of this representation.

W3. Scalability of action-unlabeled pretraining. The action-unlabeled pretraining pipeline involves several preprocessing steps, including hand tracking, motion retargeting, and robot rendering. While this enables learning from videos without action annotations, it remains unclear how easily the approach can scale to larger and more diverse video datasets, especially those with complex scenes or significant viewpoint variations.

W4. Computational efficiency. The proposed framework relies on a large pretrained video diffusion backbone and an additional action expert, potentially introducing considerable computational overhead. Although the paper reports optimized inference latency, a more detailed breakdown of training costs and the additional overhead introduced by the dual-stream architecture would help assess the practicality of the approach.

Questions for authors

Q1. Could the authors further clarify the key differences between FlowWAM and existing flow-based manipulation methods, particularly Motus and DynamicWAM?

Q2. Have the authors considered using full-scene optical flow instead of robot-only flow? How might this affect performance?

Q3. Could the authors provide more details on the computational overhead and scalability of the proposed framework?

3 comments
AuthorsOpenReview ↗

Responses to Reviewer rRLL (1/3)

We sincerely thank Reviewer rRLL for recognizing our intuitive motivation (S1), unified dual-mode framework (S2), and comprehensive evaluation across simulation and real-world benchmarks (S3). Below, we address each of your concerns and questions in detail.

W1 & Q1: Key Differences from Existing Flow-Based Methods and Isolating Architectural Gains from the Use of Optical Flow Itself

Both Motus and DynamicWAM are valuable flow-based works. Below, we compare their designs with FlowWAM and isolate our architectural gains from the use of optical flow itself.

1. How FlowWAM Differs from Motus and DynamicWAM

The central difference is that FlowWAM treats future optical flow as the action itself and keeps it as a full-resolution video stream inside the video generator. The same stream is generated in policy mode and provided in world-model mode.

  • Flow stays a spatial video stream. Motus compresses flow with a separate DC-AE-based autoencoder into a 14-dimensional latent action vector, and DynamicWAM encodes observed past flow as historical context alongside a separate kinematic branch. In FlowWAM, future flow is encoded as HSV video and passes through the same frozen VAE and shared DiT blocks as RGB, so it keeps a pixel-aligned map of how each part of the arm moves.
  • Flow is the predicted motion plan. DynamicWAM does not predict future flow, and Motus predicts latent actions before being fine-tuned to predict numerical robot actions. FlowWAM jointly denoises future RGB and flow, and the action expert reads this flow plan across all DiT layers to decode joint actions. The same RGB–flow format also allows pretraining on action-unlabeled videos without a separate action tokenizer.
  • The same stream conditions the world model. DynamicWAM does not provide an action-conditioned world-model mode, and Motus conditions video generation on action vectors outside the visual space of the video generator. FlowWAM renders a candidate action trajectory into flow and fixes the flow stream to it, so the same backbone becomes an action-conditioned world model.

Empirically, FlowWAM also outperforms Motus on RoboTwin 2.0 (93.71% vs. 87.84%, Table 1) and DynamicWAM on DOMINO (41.31% vs. 38.20%, Table 2). We have clarified the distinction from Motus and DynamicWAM in Related Work (Sec. 2) of the revised paper.

2. Gains from the Architecture vs. from Optical Flow Itself

Table 4(b)–(c) and Appendix B.6 separate the two factors:

  • Optical flow itself (raw (u,v)(u,v) flow vs. numerical actions): switching to raw flow improves policy success by 4.91 points and EWMScore by 2.54. Raw displacement tensors do not match the input format of the pretrained VAE and DiT, which limits this gain.
  • Video-native flow in the shared backbone (FlowWAM vs. raw (u,v)(u,v) flow): encoding the same flow as HSV video that shares the frozen VAE and DiT blocks with RGB improves policy success by 16.91 points and EWMScore by 8.51.
  • Joint RGB–flow generation (FlowWAM vs. Flow-Only, Appendix B.6): generating future RGB together with flow improves success by 2.66 points in-domain and 6.20 points on OOD Random, while also enabling world-model mode.

The first gain comes from the use of optical flow itself. The other two come from our design choices, video-native flow in the shared backbone and joint RGB–flow generation, and make up most of the improvement.

AuthorsOpenReview ↗

Responses to Reviewer rRLL (2/3)

W2 & Q2: Rationale of Robot-Only Optical Flow and Comparison with Full-Scene Optical Flow

Our choice of robot-only optical flow is driven by (1) maintaining a unified action representation across both policy and world-model modes, and (2) letting the coupled RGB stream capture object dynamics while the flow stream focuses on the embodiment motion plan. We also conducted a direct experiment comparing Full-Scene Optical Flow against Robot-Only Optical Flow across all 50 RoboTwin tasks.

1. Why Robot-Only Flow is Used:

  • Unified representation for world-model mode: In world-model mode (Sec. 3.1), the model is conditioned on I0I_0 and a candidate robot action trajectory to predict how the scene responds. Full-scene flow cannot serve as the input condition because future object motion is unknown at inference time. Using robot-only flow (deterministically rendered from candidate joint actions) keeps the same action representation across policy and world-model modes.
  • Object dynamics are already modeled by the parallel RGB stream: In our dual-stream DiT, the flow stream represents the embodiment motion plan, while the coupled RGB stream captures object motion and robot–environment interactions via bidirectional self-attention. This division also holds when objects move on their own, as on the DOMINO benchmark with 35 moving-target tasks, where FlowWAM uses the same robot-only flow and reaches 41.31% success (Table 2).

2. Empirical Comparison: Full-Scene Flow vs. Robot-Only Flow in Policy Mode: In policy mode, FlowWAM can also be trained directly on full-scene optical flow extracted via RAFT from raw demonstration videos (requiring no URDF rendering). We replace only the flow target and keep all other settings identical. We evaluated Full-Scene Flow across all 50 RoboTwin tasks:

Flow TargetIn-Dom CleanIn-Dom RandomIn-Dom Avg.OOD CleanOOD RandomOOD Avg.
Full-Scene Flow92.18%91.46%91.82%87.64%50.82%69.23%
Robot-Only Flow (Ours)93.74%93.68%93.71%89.24%53.04%71.14%
  • Modest gap with full-scene flow: Replacing robot-only flow with full-scene flow lowers average success by 1.89 points in-domain and 1.91 points out-of-domain, while remaining on par with the strongest in-domain baselines and surpassing all baselines under the out-of-domain protocol. A likely cause of this gap is that the wrist cameras move with the end effectors, so full-scene RAFT flow in the wrist views is dominated by background ego-motion. Our robot-only target instead uses a constant placeholder in the wrist regions (Appendix A.5), since each arm is nearly static relative to its own wrist camera, which leaves a cleaner embodiment motion signal for the action expert. In the revised paper, we have added this comparison to the appendix and clarified the robot-only flow definition in Sec. 3.3.
AuthorsOpenReview ↗

Responses to Reviewer rRLL (3/3)

W3 & Q3 (Part 1): Scalability of the Action-Unlabeled Pretraining Pipeline

The preprocessing steps you mention (hand tracking, motion retargeting, and robot rendering) belong to our default retargeted pipeline and are not required by the pretraining objective itself, which only needs RGB videos and their optical flow. To verify this, we pretrained a Stage-1 variant directly on raw EgoDex human videos (w/o retargeting). This variant skips 3D hand tracking, inpainting, IK retargeting, and robot rendering, and keeps only an off-the-shelf flow estimator (RAFT) applied to the raw videos:

Pretraining SettingIn-Dom CleanIn-Dom RandomIn-Dom Avg.OOD CleanOOD RandomOOD Avg.
w/o Pretraining87.60%87.00%87.30%86.60%38.06%62.33%
Raw EgoDex (w/o retargeting)91.84%91.42%91.63%87.82%45.96%66.89%
Retargeted EgoDex (Ours)93.74%93.68%93.71%89.24%53.04%71.14%
  • Pretraining helps even without retargeting: Pretraining directly on raw EgoDex videos and RAFT flow without retargeting improves in-domain success by +4.33 points and OOD Random success by +7.90 points. Adding embodiment retargeting brings a further +2.08 points in-domain and +7.08 points on OOD Random, which we attribute to narrowing the visual gap between human hands and robot arms.
  • Scalability: Because pretraining on EgoDex without retargeting already improves both RoboTwin protocols, retargeting is not a prerequisite for leveraging action-unlabeled videos, while offering an additional gain when 3D hand poses are available. Pretraining without retargeting needs neither 3D hand poses nor camera parameters and only requires an off-the-shelf flow estimator, so it adds no viewpoint-specific preprocessing. This setting already involves viewpoint variation, since EgoDex (314,839 training episodes) is recorded from a moving head-mounted camera.

W4 & Q3 (Part 2): Computational Efficiency, Training Costs, and Dual-Stream Overhead

Because FlowWAM reuses the frozen Wan2.2 VAE and shares all 30 blocks of the 5B DiT between RGB and optical flow, the flow stream adds only 1.19M parameters (0.02%) and little additional memory. Its main cost is DiT compute while the action expert is nearly unaffected. The table below compares FlowWAM with a single-stream RGB-only counterpart, measured under identical settings on NVIDIA H100 GPUs. Inference uses our default distilled setting (4 video steps / 10 action steps, Appendix A.7):

VariantTrain Step TimeTrain Peak MemInference Peak MemTotal LatencyRoboTwin Avg.
w/o Flow Stream (RGB-only)2.81 s67.3 GiB23.4 GiB298 ms88.62%
FlowWAM (Dual-Stream, Ours)4.77 s72.7 GiB23.7 GiB352 ms93.71%
  • Training overhead: The flow stream raises per-step training time from 2.81 s to 4.77 s, because each step runs two DiT passes (the denoising-loss pass and the feature-extraction pass for the action expert), both on twice as many tokens. Per-GPU peak memory grows by only 5.4 GiB (67.3 →\rightarrow 72.7 GiB), because memory is dominated by the weights, gradients, DDP gradient buffers, and AdamW states of the shared 5B backbone. For reference, Stage-1 pretraining runs for 34k steps on 64 A100 GPUs, and downstream training uses 32 H100 GPUs for about 40k steps, i.e., about 53 hours or roughly 1.7k H100 GPU-hours at the measured step time.
  • Inference overhead: In our default distilled setting, the flow stream adds 54 ms per action chunk (298 ms →\rightarrow 352 ms). Inference memory stays nearly the same (+0.3 GiB), because it is dominated by the resident model weights (22.8 GiB, including the 10.6 GiB T5 text encoder). In exchange, average task success improves by +5.09 points. In the revised paper, we have added the inference and training overhead of the flow stream to the appendix.

We hope these responses address your concerns, and we would be glad to clarify any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all reviewers for their thoughtful and constructive feedback. We appreciate that the reviewers found our use of optical flow as a video-native action representation well-motivated (rRLL, mCvQ), our unified framework for policy learning and world modeling versatile (rRLL, mCvQ), and our evaluation and ablations comprehensive (rRLL, KEbU, mCvQ).

Summary of revisions. Following the reviewers' suggestions, we have revised the manuscript. All changes are highlighted in blue in the revised PDF.

Main text

  • Sec. 1 (mCvQ): Clarified that action-free pretraining is supervised by flow from an off-the-shelf estimator, on raw or retargeted egocentric videos.
  • Sec. 2 (rRLL, KEbU): Sharpened the distinction from Motus and DynamicWAM, and added a discussion of FOFPred.
  • Sec. 3.2 (KEbU): Stated the extra DiT pass at timestep zero that provides features to the action expert.
  • Sec. 3.3 (rRLL): Defined the robot-only flow target and the reason for using it.
  • Sec. 4.5 (mCvQ, rRLL): Moved the latency table from Appendix A.7 (formerly Table 6) into the main text as Table 5, which now reports both latency and success rate for the undistilled model and for the distilled model run with 4, 2, and 1 video steps.

Appendix

  • A.6 (rRLL): Training and inference overhead of the flow stream.
  • A.7 (mCvQ): Effect of the video denoising step count.
  • B.4 (KEbU): Flow prediction accuracy and action error with ground-truth flow.
  • B.5 (KEbU): Comparison of feature extraction schemes for the action expert.
  • B.6 (KEbU, mCvQ): Generating flow without future RGB.
  • B.7 (rRLL): Full-scene versus robot-only flow target.
  • B.8 (rRLL, mCvQ): Pretraining without retargeting.
  • E.1 (rRLL, mCvQ): Clarified that only the retargeting step relies on 3D hand poses.

We address each reviewer's comments in detail in the individual responses below. Thank you again for your time and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.