- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
World4Act Policy jointly learns action generation and future RGB, point-map, and scene-flow prediction within an action-centered World–Action Model. It combines modality-specific VAEs, three-stage training, and the Robo4D_Dyn dataset, while using causal attention to remove future-world prediction at deployment. The results support the value of explicit 3D supervision for prediction and manipulation, with little additional action-inference latency.
Critical strengths
- The motivation is sound, and the representations are well chosen. Point maps and scene flow explicitly describe 3D positions and displacements while preserving image alignment, providing a natural interface for reusing video-generation priors.
- The training and deployment design is coherent. Specialized VAEs and progressive training accommodate different modalities, while causal attention makes future-world prediction optional at deployment. The reported action-inference latency is 0.37 s, close to GigaWorld-Policy’s 0.36 s.
- The ablations provide encouraging evidence. Table 3(a) examines geometry, motion, and joint supervision. The full model improves success by 9 percentage points over the internal RGB-only variant on both randomized RoboTwin and the real-world average, and by 3 points over Projective 4D.
Critical weaknesses
- Stronger action–dynamics coupling is not directly established. Table 1 evaluates Stage-II models, which are not action-conditioned, and therefore does not test whether different actions produce the correct predicted 3D outcomes. Although Stage III exposes the world branch to action representations, the paper provides neither controlled comparisons under different actions nor action-removal/shuffling ablations. The proposed coupling therefore remains insufficiently validated.
- The independent contribution of 3D scene flow remains unclear. The Projective-to-Direct comparison changes both geometry and motion targets, preventing attribution to scene flow alone. The RGB + Scene Flow ablation supports the value of motion supervision, but does not establish its advantage over 2D optical-flow supervision under otherwise matched conditions.
- The real-world evaluation needs clearer documentation. The four tasks in Table 2 lack a corresponding overview or clearly identified policy-execution sequences. Although the main text refers readers to the supplement, task-specific initial conditions and success criteria remain underspecified.
Questions for authors
- What are the spatial and temporal sampling settings for Stage I? Please provide the training resolution, sequence length, frame interval, and padding scheme for short sequences. Since scene-flow magnitude depends on the temporal interval, please also clarify how temporal sampling is handled across data sources.
- How accurately do the specialized VAEs reconstruct geometry and motion? Figure 6 is schematic, while Table 3(b) primarily reports downstream effects. Could the authors provide held-out reconstruction metrics against the unadapted RGB VAE, together with examples around motion boundaries and manipulated objects? Since the VAEs are subsequently frozen, their reconstruction fidelity warrants direct evaluation.
- Which data are used in Stage III, and how is training organized? Please specify the data sources, demonstrations per task, sampling proportions, and whether training is joint across tasks or task-specific. How are the clean and randomized RoboTwin data used?
3 comments
Response to Reviewer NuDS
Dear Reviewer,
I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.
Thank you for your time and effort in reviewing our paper.
Further discussion with Reviewer NuDS
Dear Reviewer NuDS,
Thank you for your thoughtful feedback on our paper. As the discussion period nears its end, we would like to see whether our previous responses have addressed your questions.
If any points would benefit from further clarification, please let us know. We would be glad to provide additional details.
Best regards,
Authors
Response to Reviewer NuDS
We thank the reviewer for the detailed comments. We address the concerns below.
W1: Stronger action–dynamics coupling is not established: no comparisons under different actions, no action-removal or shuffling ablations.
Response to W1: Following the reviewer's suggestion, we ran these controlled comparisons and ablations on RoboTwin 2.0. They establish the stronger action–dynamics coupling directly: the predicted 3D motion follows the conditioning actions, degrades when they are shuffled or removed, and is coupled more tightly than with projective targets (Table S6).
Table S6. Stage-III prediction under action interventions (EPE3D, dynamic regions).
W2: The advantage of 3D scene flow over 2D optical flow under matched conditions is not established.
Response to W2: With the geometry target fixed, 3D scene flow is ahead of 2D optical flow in all three settings (Table S7). Depth and optical flow are projections of the same 4RC labels, with VAEs adapted by the same Stage-I schedule. It lowers Stage-II EPE3D by 7.5% with point maps and 6.0% with depth, and raises success by 1–2 points in every setting.
Table S7. Geometry and motion targets varied independently.
W3: Real-world initial conditions and success criteria are underspecified.
Response to W3: We apologize for the missing task definitions. All methods share the protocol below, with 100 recorded initial configurations per task. Clean the Desk: 5 of 12 objects in a 60×40 cm area; all in the storage box within 90 s. Scan a QR Code: package within ±10 cm and ±30° yaw; code decoded within 45 s. Sweep up Trash: 6–8 paper scraps; all in the dustpan within 90 s. Stack Bowls: 3 bowls of different sizes within ±12 cm; nested and stable for 3 s within 60 s.
Q1: Stage-I sampling settings, and temporal sampling across sources?
Response to Q1: Scene flow is always expressed as displacement per 0.1 s. Each view is resized to 480×640; a clip has 13 point maps and 12 forward flow maps. Each source uses the frame stride closest to 0.1 s, and the flow is rescaled from the actual interval to 0.1 s. Sources below 10 Hz (RT-1, Bridge) give no flow targets. Clips shorter than 13 frames are discarded, so no temporal padding is used.
Q2: How accurately do the adapted VAEs reconstruct geometry and motion?
Response to Q2: Stage I is a key part of our method. Many prior works, including our baseline RynnWorld-4D, keep the pretrained VAE frozen. We find that fine-tuning it gives much better geometry and flow reconstruction: on 2,000 held-out clips it removes 69–75% of the error (Table S8). MotionCrafter (CVPR 2026) likewise shows that fine-tuning the geometry and motion VAEs is necessary. Fig. 6 will show real boundary and object examples.
Table S8. Held-out VAE reconstruction error (flow: EPE3D).
Q3: Which data are used in Stage III, and how are clean and randomized RoboTwin data used?
Response to Q3: Stage III uses target-platform demonstrations only, with 4D targets from 4RC. Simulation: one multi-task policy for all 50 RoboTwin 2.0 tasks, trained on 50 clean and 500 randomized demonstrations per task, with uniform task sampling; one checkpoint is evaluated under both conditions. Real world: one policy for the four tasks, 100 demonstrations each.