- Rating
- 4
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Authors propose a world action model that uses optical flow as the unified action representation. They use common representation space for action and video prediction by encoding optical flow as RGB frames. Leveraging how optical flow can be extracted from videos without action labels, they pretrain the model on large-scale unlabeled videos. Their resulting model provides two operating modes, one as a world model and other as a robot action policy. Extensive evaluations are provided to establish strong performance of proposed model.
Critical strengths
- Simple change to WAMs that enables large-scale pretraining on unlabeled videos and dual operating modes using shared video generation backbone.
- Extensive evaluation across multiple benchmarks with comparison to strong baselines.
- Well documented real world experiments.
- Comprehensive ablation studies.
Critical weaknesses
- The unlabelled video pretraining actually utilizes supervision from additional structure preextracted from videos. The unlabelled pretraining claim could be more clearly stated in introduction.
- Inference latency is not clearly documented in main paper - consider moving that appendix table to main paper.
Questions for authors
- How important is the additional structure pre-extracted from videos for the unlabelled video pretraining?
- How would denoising step count of video generator affect performance? For distilled model, what would performance look for steps 1 and 2 compared to current 4?
- How important is RGB extraction for action policy mode? Could video generator predict only flow for that mode?
4 comments
Responses to Reviewer mCvQ (2/2)
Q2: Effect of Reducing Video Denoising Step Count (from 4 Steps to 2 or 1 Steps)
We evaluated the distilled video generator across 4, 2, and 1 denoising steps across all 50 RoboTwin 2.0 tasks (measured on a single NVIDIA H100 GPU):
| Video Steps | Total Latency | In-Dom Clean | In-Dom Random | In-Dom Avg. | OOD Random |
|---|---|---|---|---|---|
| 4 steps (Default) | 352 ms | 93.74% | 93.68% | 93.71% | 53.04% |
| 2 steps | 265 ms | 91.88% | 91.16% | 91.52% | 49.86% |
| 1 step | 223 ms | 89.94% | 88.38% | 89.16% | 47.72% |
- Latency vs. accuracy across step counts: Reducing from 4 steps (352 ms) to 2 steps (265 ms) cuts latency by 25% while retaining 91.52% in-domain success (2.19 points lower). Reducing further to 1 step (223 ms) lowers success to 89.16% (47.72% on OOD Random), so 2 to 4 steps offer a reasonable trade-off between speed and accuracy.
Q3: Importance of RGB Generation for Action Policy Mode (Could the Video Generator Predict Only Flow?)
We evaluated a Flow-Only variant on our backbone that denoises only future flow latents (without generating future RGB):
| Variant | In-Dom Clean | In-Dom Random | In-Dom Avg. | OOD Random |
|---|---|---|---|---|
| Flow-Only (w/o future RGB) | 91.48% | 90.62% | 91.05% | 46.84% |
| FlowWAM (Joint RGB + Flow, Ours) | 93.74% | 93.68% | 93.71% | 53.04% |
- Unified architecture for policy and world-model modes: Keeping both streams in the generator preserves architectural unification. If the generator only predicted flow, it could no longer synthesize future RGB videos as an action-conditioned world model (Sec. 4.3, WorldArena). Modeling RGB and flow jointly lets a single model serve both policy control and world modeling.
- Policy accuracy gain (+2.66 points In-Domain, +6.20 points OOD Random): While Flow-Only already achieves 91.05% in-domain success, our flow target represents robot-only motion and does not depict how objects react upon contact. Jointly generating future RGB raises in-domain success to 93.71% and OOD Random success to 53.04%, which we attribute to RGB co-generation enabling the shared DiT blocks to model object states and contact outcomes.
We hope these responses address your concerns, and we would be glad to clarify any further questions.
Thanks for the updates. Great work.
Thank you for your positive feedback and constructive suggestions throughout the review process. Your comments have been particularly helpful in further improving our paper.
Responses to Reviewer mCvQ (1/2)
We sincerely thank Reviewer mCvQ for the support and for highlighting our unified dual-mode formulation (Strength 1), extensive benchmark comparisons (Strength 2), real-world experiments (Strength 3), and comprehensive ablations (Strength 4). We agree with both presentation suggestions (W1 and W2) and have conducted the three ablation studies in Q1–Q3.
W1: Clarifying the Unlabeled Pretraining Description in the Introduction
In Stage 1, the model itself is trained purely on RGB and optical flow video streams without requiring teleoperated robot action labels. As you point out, this supervision relies on structure pre-extracted from the videos. The optical flow is estimated with RAFT, and our default EgoDex pretraining pipeline additionally uses 3D hand poses and embodiment retargeting to narrow the visual gap between human hands and robot arms. We have clarified this in the Introduction (Sec. 1) of the revised paper, which now states that the flow comes from an off-the-shelf estimator and that the videos are either raw or retargeted to robot arms. The retargeting pipeline is described in Sec. 3.3, and the two settings are compared in Appendix B.8.
W2: Documenting Inference Latency in the Main Paper
We agree and have moved Table 6 from Appendix A.7 into Sec. 4.5 of the revised paper, expanded with both latency and task success rate across configurations.
Q1: How Important is the Pre-Extracted Structure for Pretraining?
To quantify the contribution of embodiment retargeting during Stage-1 pretraining, we evaluated a variant pretrained purely on raw EgoDex human videos and raw RAFT optical flow (w/o retargeting, without any 3D hand tracking, inpainting, IK retargeting, or robot rendering):